Typedef removal tool

Typedef removal tool

[Luis R. Rodriguez]

Anyone know if there is a typedef removal tool? When porting drivers
this needs to be done quite often.

FWIW I've started porting some driver and have come across some tools,
or written my own scripts, which have helped. Using spatch to port a
driver was such a tool I explored. I think I'll try to put together
some info on the driver project wiki for this. typdef removal tool
would still be neat.

  Luis

Re: Typedef removal tool

[Julia Lawall]

On Tue, 11 Aug 2009, Luis R. Rodriguez wrote:

> Anyone know if there is a typedef removal tool? When porting drivers
> this needs to be done quite often.
> 
> FWIW I've started porting some driver and have come across some tools,
> or written my own scripts, which have helped. Using spatch to port a
> driver was such a tool I explored. I think I'll try to put together
> some info on the driver project wiki for this. typdef removal tool
> would still be neat.

Nicolas Palix (in CC) did a bit of work on this recently using Coccinelle, 
but the problem is that simply removing the typedef is not enough, one has 
to come up with a name that is within the spirit of the names used in 
Linux, or perhaps recognize that the structure in question already exists 
in a more general form in the kernel an thus the declaration should be 
removed completely.

Someone else made a dedicated tool using a dictionary of the old and new 
names that at least partially addresses these problems.  Nicolas knows who 
that person is.

julia

Re: Typedef removal tool

[Luis R. Rodriguez]

On Tue, Aug 11, 2009 at 10:01 PM, Julia Lawall<julia@diku.dk> wrote:
> On Tue, 11 Aug 2009, Luis R. Rodriguez wrote:
>
>> Anyone know if there is a typedef removal tool? When porting drivers
>> this needs to be done quite often.
>>
>> FWIW I've started porting some driver and have come across some tools,
>> or written my own scripts, which have helped. Using spatch to port a
>> driver was such a tool I explored. I think I'll try to put together
>> some info on the driver project wiki for this. typdef removal tool
>> would still be neat.
>
> Nicolas Palix (in CC) did a bit of work on this recently using Coccinelle,
> but the problem is that simply removing the typedef is not enough, one has
> to come up with a name that is within the spirit of the names used in
> Linux, or perhaps recognize that the structure in question already exists
> in a more general form in the kernel an thus the declaration should be
> removed completely.
>
> Someone else made a dedicated tool using a dictionary of the old and new
> names that at least partially addresses these problems.  Nicolas knows who
> that person is.

Sounds like a very proper way to do this.

I wrote something not-so-proper and simple based on Joe's work that
just replaces a specified typedef name with one the user specifies.
Should do it for simple porting where you do not have to prove
correctness and avoid namespace collisions.

  Luis

Re: Typedef removal tool

[Nicolas Palix]

[-- Attachment #1: Type: text/plain, Size: 1991 bytes --]

On Wednesday 12 August 2009 09:45:35 Luis R. Rodriguez wrote:
> On Tue, Aug 11, 2009 at 10:01 PM, Julia Lawall<julia@diku.dk> wrote:
> > On Tue, 11 Aug 2009, Luis R. Rodriguez wrote:
> >
> >> Anyone know if there is a typedef removal tool? When porting drivers
> >> this needs to be done quite often.
> >>
> >> FWIW I've started porting some driver and have come across some tools,
> >> or written my own scripts, which have helped. Using spatch to port a
> >> driver was such a tool I explored. I think I'll try to put together
> >> some info on the driver project wiki for this. typdef removal tool
> >> would still be neat.
> >
> > Nicolas Palix (in CC) did a bit of work on this recently using Coccinelle,
> > but the problem is that simply removing the typedef is not enough, one has
> > to come up with a name that is within the spirit of the names used in
> > Linux, or perhaps recognize that the structure in question already exists
> > in a more general form in the kernel an thus the declaration should be
> > removed completely.
> >
> > Someone else made a dedicated tool using a dictionary of the old and new
> > names that at least partially addresses these problems.  Nicolas knows who
> > that person is.
> 
> Sounds like a very proper way to do this.
> 
> I wrote something not-so-proper and simple based on Joe's work that
> just replaces a specified typedef name with one the user specifies.
> Should do it for simple porting where you do not have to prove
> correctness and avoid namespace collisions.
> 

We have done this for the new Hyper-V drivers.
Joe Perches (CC) proposes a shell and perl solution (cvt_typedef_to_struct.bash)
with the typedefs file dictionary, attached to this email.

I have modified my typedef_str.cocci file to use his dictionary.
This cocci file generates a cocci file per typedef to remove.
The generated files could be applied with the modified version
of Joe's shell script or by hand, at your convenience.

>   Luis
> 


-- 
Nicolas Palix

[-- Attachment #2: cvt_typedef_to_struct.bash --]
[-- Type: application/x-shellscript, Size: 897 bytes --]

[-- Attachment #3: cvt_typedef_to_struct_with_cocci.bash --]
[-- Type: application/x-shellscript, Size: 595 bytes --]

[-- Attachment #4: typedefs --]
[-- Type: text/plain, Size: 4883 bytes --]

DEVICE_INFO:vmbus_device_info
GPA_RANGE:vmbus_gpa_range
HV_CONNECTION_INFO:hv_connection_info
HV_INPUT_POST_MESSAGE:hv_input_post_message
HV_INPUT_SIGNAL_EVENT:hv_input_signal_event
HV_MESSAGE:hv_message
HV_MESSAGE_HEADER:hv_message_header
HV_MESSAGE_PAGE:hv_message_page
HV_MONITOR_PAGE:hv_monitor_page
HV_MONITOR_PARAMETER:hv_monitor_parameter
HV_PORT_INFO:hv_port_info
HV_SYNIC_EVENT_FLAGS_PAGE:hv_synic_event_flags_page
HV_TIMER_MESSAGE_PAYLOAD:hv_timer_message_payload
MULTIPAGE_BUFFER:vmbus_multipage_buffer
NETVSC_DEVICE_INFO:netvsc_device_info
NETVSC_DRIVER_OBJECT:netvsc_driver_object
NVSP_1_MESSAGE_REVOKE_RECEIVE_BUFFER:nvsp_1_message_revoke_receive_buffer
NVSP_1_MESSAGE_REVOKE_SEND_BUFFER:nvsp_1_message_revoke_send_buffer
NVSP_1_MESSAGE_SEND_NDIS_VERSION:nvsp_1_message_send_ndis_version
NVSP_1_MESSAGE_SEND_RECEIVE_BUFFER:nvsp_1_message_send_receive_buffer
NVSP_1_MESSAGE_SEND_RECEIVE_BUFFER_COMPLETE:nvsp_1_message_send_receive_buffer_complete
NVSP_1_MESSAGE_SEND_RNDIS_PACKET:nvsp_1_message_send_rndis_packet
NVSP_1_MESSAGE_SEND_RNDIS_PACKET_COMPLETE:nvsp_1_message_send_rndis_packet_complete
NVSP_1_MESSAGE_SEND_SEND_BUFFER:nvsp_1_message_send_send_buffer
NVSP_1_MESSAGE_SEND_SEND_BUFFER_COMPLETE:nvsp_1_message_send_send_buffer_complete
NVSP_1_RECEIVE_BUFFER_SECTION:nvsp_1_receive_buffer_section
NVSP_MESSAGE:nvsp_message
NVSP_MESSAGE_HEADER:nvsp_message_header
NVSP_MESSAGE_INIT:nvsp_message_init
NVSP_MESSAGE_INIT_COMPLETE:nvsp_message_init_complete
PAGE_BUFFER:vmbus_page_buffer
PORT_INFO:vmbus_port_info
RCONDIS_CALL_MANAGER_PARAMETERS:rcondis_call_manager_parameters
RCONDIS_INDICATE_STATUS:rcondis_indicate_status
RCONDIS_MEDIA_PARAMETERS:rcondis_media_parameters
RCONDIS_MP_ACTIVATE_VC_COMPLETE:rcondis_mp_activate_vc_complete
RCONDIS_MP_ACTIVATE_VC_REQUEST:rcondis_mp_activate_vc_request
RCONDIS_MP_CREATE_VC:rcondis_mp_create_vc
RCONDIS_MP_CREATE_VC_COMPLETE:rcondis_mp_create_vc_complete
RCONDIS_MP_DEACTIVATE_VC_COMPLETE:rcondis_mp_deactivate_vc_complete
RCONDIS_MP_DEACTIVATE_VC_REQUEST:rcondis_mp_deactivate_vc_request
RCONDIS_MP_DELETE_VC:rcondis_mp_delete_vc
RCONDIS_MP_DELETE_VC_COMPLETE:rcondis_mp_delete_vc_complete
RCONDIS_MP_QUERY_REQUEST:rcondis_mp_query_request
RCONDIS_MP_SET_REQUEST:rcondis_mp_set_request
RCONDIS_SPECIFIC_PARAMETERS:rcondis_specific_parameters
RING_BUFFER_DEBUG_INFO:hv_ring_buffer_debug_info
RING_BUFFER_INFO:hv_ring_buffer_info
RNDIS_CO_ADDRESS_FAMILY:rndis_co_address_family
RNDIS_CONFIG_PARAMETER_INFO:rndis_config_parameter_info
RNDIS_DEVICE:rndis_device
RNDIS_DIAGNOSTIC_INFO:rndis_diagnostic_info
RNDIS_FILTER_DRIVER_OBJECT:rndis_filter_driver_object
RNDIS_FILTER_PACKET:rndis_filter_packet
RNDIS_FLOWSPEC:rndis_flowspec
RNDIS_HALT_REQUEST:rndis_halt_request
RNDIS_INDICATE_STATUS:rndis_indicate_status
RNDIS_INITIALIZE_COMPLETE:rndis_initialize_complete
RNDIS_INITIALIZE_REQUEST:rndis_initialize_request
RNDIS_KEEPALIVE_COMPLETE:rndis_keepalive_complete
RNDIS_KEEPALIVE_REQUEST:rndis_keepalive_request
RNDIS_OOBD:rndis_oobd
RNDIS_PACKET:rndis_packet
RNDIS_PER_PACKET_INFO:rndis_per_packet_info
RNDIS_QUERY_COMPLETE:rndis_query_complete
RNDIS_QUERY_REQUEST:rndis_query_request
RNDIS_REQUEST:rndis_request
RNDIS_RESET_COMPLETE:rndis_reset_complete
RNDIS_RESET_REQUEST:rndis_reset_request
RNDIS_SET_COMPLETE:rndis_set_complete
RNDIS_SET_REQUEST:rndis_set_request
STORVSC_DEVICE:storvsc_device
STORVSC_DEVICE_INFO:storvsc_device_info
STORVSC_DRIVER_OBJECT:storvsc_driver_object
STORVSC_REQUEST_EXTENSION:storvsc_request_extension
VMBUS_CHANNEL:vmbus_channel
VMBUS_CHANNEL_CLOSE_CHANNEL:vmbus_channel_close_channel
VMBUS_CHANNEL_DEBUG_INFO:vmbus_channel_debug_info
VMBUS_CHANNEL_GPADL_BODY:vmbus_channel_gpadl_body
VMBUS_CHANNEL_GPADL_CREATED:vmbus_channel_gpadl_created
VMBUS_CHANNEL_GPADL_HEADER:vmbus_channel_gpadl_header
VMBUS_CHANNEL_GPADL_TEARDOWN:vmbus_channel_gpadl_teardown
VMBUS_CHANNEL_GPADL_TORNDOWN:vmbus_channel_gpadl_torndown
VMBUS_CHANNEL_INITIATE_CONTACT:vmbus_channel_initiate_contact
VMBUS_CHANNEL_INTERFACE:vmbus_channel_interface
VMBUS_CHANNEL_MESSAGE_HEADER:vmbus_channel_message_header
VMBUS_CHANNEL_MESSAGE_TABLE_ENTRY:vmbus_channel_message_table_entry
VMBUS_CHANNEL_MSGINFO:vmbus_channel_msginfo
VMBUS_CHANNEL_OFFER_CHANNEL:vmbus_channel_offer_channel
VMBUS_CHANNEL_OPEN_CHANNEL:vmbus_channel_open_channel
VMBUS_CHANNEL_OPEN_RESULT:vmbus_channel_open_result
VMBUS_CHANNEL_QUERY_VMBUS_VERSION:vmbus_channel_query_vmbus_version
VMBUS_CHANNEL_RELID_RELEASED:vmbus_channel_relid_released
VMBUS_CHANNEL_RESCIND_OFFER:vmbus_channel_rescind_offer
VMBUS_CHANNEL_VERSION_RESPONSE:vmbus_channel_version_response
VMBUS_CHANNEL_VERSION_SUPPORTED:vmbus_channel_version_supported
VMBUS_CHANNEL_VIEW_RANGE_ADD:vmbus_channel_view_range_add
VMBUS_CHANNEL_VIEW_RANGE_REMOVE:vmbus_channel_view_range_remove
VMBUS_DRIVER_OBJECT:vmbus_driver_object
VSTOR_PACKET:vstor_packet
XFERPAGE_PACKET:netvsc_xferpage_packet

[-- Attachment #5: typedef_str.cocci --]
[-- Type: text/x-csrc, Size: 1346 bytes --]

//
// Generate cocci file with
//
//BASE=~/linux
//NEXT=~/linux/drivers/staging/hv
//spatch.opt -include_headers -sp_file typedef_str -dir $(NEXT) -patch $(BASE)
//
// Applying a generated cocci file
//
//.cocci.patch:
//	spatch.opt -inplace -include_headers -sp_file $< -dir $(NEXT) -patch $(BASE) > $@

@initialize:python@

f = open("typedefs", 'r')
d = dict()

for line in f:
    (key, sep, name) = line.partition(':')
    d[key] = name

f.close()

@r@
identifier st;
type typ;
@@

typedef struct st {
...
} typ ;

@script: python@
st << r.st;
typ << r.typ;
@@

name = "gen/%s.cocci" % typ
strname = d[typ]

print name
f = open(name, 'w')

f.write( "@rm_%s@\n"            % typ    )
f.write( "@@\n"                          )
f.write( "-typedef struct %s\n" % st	 )
f.write( "+struct %s\n"         % strname)
f.write( "{...}\n"		         )
f.write( "-%s\n"                % typ	 )
f.write( ";\n\n"	        	 )

f.write( "@fixtypedef_%s@\n"    % typ    )
f.write( "typedef %s;\n"        % typ	 )
f.write( "@@\n"                          )
f.write( "-%s\n"                % typ	 )
f.write( "+struct %s\n\n"       % strname)

f.write( "@fixstruct_%s@\n"     % st     )
f.write( "@@\n"                          )
f.write( "struct\n"                 	 )
f.write( "-%s\n"                % st	 )
f.write( "+%s\n"                % strname)

f.close()

Re: Typedef removal tool

[Luis R. Rodriguez]

On Wed, Aug 12, 2009 at 5:23 AM, Nicolas Palix<npalix@diku.dk> wrote:
> On Wednesday 12 August 2009 09:45:35 Luis R. Rodriguez wrote:
>> On Tue, Aug 11, 2009 at 10:01 PM, Julia Lawall<julia@diku.dk> wrote:
>> > On Tue, 11 Aug 2009, Luis R. Rodriguez wrote:
>> >
>> >> Anyone know if there is a typedef removal tool? When porting drivers
>> >> this needs to be done quite often.
>> >>
>> >> FWIW I've started porting some driver and have come across some tools,
>> >> or written my own scripts, which have helped. Using spatch to port a
>> >> driver was such a tool I explored. I think I'll try to put together
>> >> some info on the driver project wiki for this. typdef removal tool
>> >> would still be neat.
>> >
>> > Nicolas Palix (in CC) did a bit of work on this recently using Coccinelle,
>> > but the problem is that simply removing the typedef is not enough, one has
>> > to come up with a name that is within the spirit of the names used in
>> > Linux, or perhaps recognize that the structure in question already exists
>> > in a more general form in the kernel an thus the declaration should be
>> > removed completely.
>> >
>> > Someone else made a dedicated tool using a dictionary of the old and new
>> > names that at least partially addresses these problems.  Nicolas knows who
>> > that person is.
>>
>> Sounds like a very proper way to do this.
>>
>> I wrote something not-so-proper and simple based on Joe's work that
>> just replaces a specified typedef name with one the user specifies.
>> Should do it for simple porting where you do not have to prove
>> correctness and avoid namespace collisions.
>>
>
> We have done this for the new Hyper-V drivers.
> Joe Perches (CC) proposes a shell and perl solution (cvt_typedef_to_struct.bash)
> with the typedefs file dictionary, attached to this email.
>
> I have modified my typedef_str.cocci file to use his dictionary.
> This cocci file generates a cocci file per typedef to remove.
> The generated files could be applied with the modified version
> of Joe's shell script or by hand, at your convenience.

Great thanks Nicholas! I'll play with these.

BTW I was under the impression spatch wouldn't touch header files or
dive in to untangle macros. Is that incorrect? Would the cocci files
allow spatch to not only change typdef usages with simple struct uses
but also redefine a typedef declaration to a simple struct definition
through header files? For upstream work I don't expect as much macro
mess as you would with typical vendor drivers. For drivers which do
need to get ported I do expect an tangling web of macro mess one may
need to put up with at times.

  Luis

Re: Typedef removal tool

[Julia Lawall]

> BTW I was under the impression spatch wouldn't touch header files or
> dive in to untangle macros. Is that incorrect? Would the cocci files
> allow spatch to not only change typdef usages with simple struct uses
> but also redefine a typedef declaration to a simple struct definition
> through header files? For upstream work I don't expect as much macro
> mess as you would with typical vendor drivers. For drivers which do
> need to get ported I do expect an tangling web of macro mess one may
> need to put up with at times.

spatch looks at header files to a user-controllable degree: -no_includes 
means ignore all #includes, -local_includes means consider include 
files in the current directory, and -all_includes means consider all 
direct #includes.  There is no option to consider files that are included 
by included files.

If a header file is included, then spatch is willing to generate patches 
for transformations within the header file.  An inconvenience is that 
if a single header file is included multiple times, a patch will be 
generated for it each time as well.  In the worst case, these patches 
might not even be consistent.

There is also an increasing effort to take into account macro definitions, 
when they are available and useful.  I don't know the exact current status 
of that, however.  Perhaps Yoann (in CC) can explain.


In the general case, there it is not possible to write a rule to change a 
typedef to a simple structure declaration with the same name as the 
typedef.  The problem is that typedef declares something that is parsed as 
a type and the name of a struct is an identifier.  Spacth doesn't allow 
you to put a type in a position where an identifier is expected.  
Nicolas' code, however, gets around the problem by using python to 
generate new semantic patches.  In the new semantic patch, the typedef 
name is just a constant, and thus parsed as convenient for the position in 
which it appears (once as a type name and once as a struct name).


julia