recommended programming practices for writing (was Linux 2.6.29)

recommended programming practices for writing (was Linux 2.6.29)

[Adam Turk]

I have been reading the Linux 2.6.29 thread with interest.  I have written several (10 or so) C programs that write large amounts of data (between 1 and 2 GB file sizes are common).  A snippet of code looks like this:

if((fptr = fopen(outfilename,"w")) == NULL) {
  printf("File %s could not be created\n", outfilename);
}
else {
  fprintf(fptr,"%s\n",datablock);
  while(!writeouput(datablock,amount,tax)) {
    getnext(dtablock)
    fprintf(fptr,"%s\n",datablock);
  }
  fclose(fptr);
}

I learned C about 15 years ago and there was no mention of a fsync.  My C book doesn't mention fsync either.  Granted I have written only 25-30 applications in the last 15 years or so so I am not an expert C programmer. 

>From what Linus posted about git and checking the return from fclose I think I going to start doing that.  I also think I am going to start checking the return from fprintf and maybe write to a /tmp/file and then rename it.

So is there a C fsync that I should add before my fclose?
What is the proper way to write to files?

Is what I am thinking about doing something that would be good or is it just a lack of my understanding the problem?

Thanks,
Adam

_________________________________________________________________
Hotmail® is up to 70% faster. Now good news travels really fast.
http://windowslive.com/online/hotmail?ocid=TXT_TAGLM_WL_HM_70faster_032009

Re: recommended programming practices for writing (was Linux 2.6.29)

[Bernd Petrovitsch]

On Thu, 2009-03-26 at 15:03 -0400, Adam Turk wrote:
> I have been reading the Linux 2.6.29 thread with interest.  I have
> written several (10 or so) C programs that write large amounts of data
> (between 1 and 2 GB file sizes are common).  A snippet of code looks
> like this:
> 
> if((fptr = fopen(outfilename,"w")) == NULL) {
>   printf("File %s could not be created\n", outfilename);
> }
> else {
>   fprintf(fptr,"%s\n",datablock);
>   while(!writeouput(datablock,amount,tax)) {
>     getnext(dtablock)
>     fprintf(fptr,"%s\n",datablock);
>   }
>   fclose(fptr);
> }
> 
> I learned C about 15 years ago and there was no mention of a fsync.
> My C book doesn't mention fsync either.  Granted I have written only

Probably because fsync() is a system call (and not a standard C lib
function).

[...]
> From what Linus posted about git and checking the return from fclose I
> think I going to start doing that.  I also think I am going to start
> checking the return from fprintf and maybe write to a /tmp/file and
> then rename it.

For a really robust app it's probably not wrong. At least one gets an
error (e.g. "disk full") immediately and not only at fclose() time after
(trying to) write 2GB data.

> So is there a C fsync that I should add before my fclose?

"fflush(fptr);" flushes all of the buffers (managed by the C-lib) of
fptr and also delivers an error.
fsync() is to flush the in-kernel cached pages of that file.

> What is the proper way to write to files?

Basically just as you do above.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

RE: recommended programming practices for writing (was Linux 2.6.29)

[Adam Turk]

>> On Thu, 2009-03-26 at 15:03 -0400, Adam Turk wrote:
>> I have been reading the Linux 2.6.29 thread with interest. I have
>> written several (10 or so) C programs that write large amounts of data
>> (between 1 and 2 GB file sizes are common). A snippet of code looks
>> like this:
>>
>> if((fptr = fopen(outfilename,"w")) == NULL) {
>> printf("File %s could not be created\n", outfilename);
>> }
>> else {
>> fprintf(fptr,"%s\n",datablock);
>> while(!writeouput(datablock,amount,tax)) {
>> getnext(dtablock)
>> fprintf(fptr,"%s\n",datablock);
>> }
>> fclose(fptr);
>> }
>>
>> I learned C about 15 years ago and there was no mention of a fsync.
>> My C book doesn't mention fsync either. Granted I have written only
>
> Probably because fsync() is a system call (and not a standard C lib
> function).
>
> [...]
>> From what Linus posted about git and checking the return from fclose I
>> think I going to start doing that. I also think I am going to start
>> checking the return from fprintf and maybe write to a /tmp/file and
>> then rename it.
>
> For a really robust app it's probably not wrong. At least one gets an
> error (e.g. "disk full") immediately and not only at fclose() time after
> (trying to) write 2GB data.
>
>> So is there a C fsync that I should add before my fclose?
>
> "fflush(fptr);" flushes all of the buffers (managed by the C-lib) of
> fptr and also delivers an error.
> fsync() is to flush the in-kernel cached pages of that file.
>
>> What is the proper way to write to files?
>
> Basically just as you do above.
>
> Bernd
 
Thank you for the tips. 
 
Thanks,
Adam
_________________________________________________________________
Quick access to Windows Live and your favorite MSN content with Internet Explorer 8.
http://ie8.msn.com/microsoft/internet-explorer-8/en-us/ie8.aspx?ocid=B037MSN55C0701A

Re: recommended programming practices for writing (was Linux 2.6.29)

[Leon Woestenberg]

Hello Adam,

On Thu, Mar 26, 2009 at 8:03 PM, Adam Turk <bofh1234@hotmail.com> wrote:
>
> I have been reading the Linux 2.6.29 thread with interest.  I have written several (10 or so) C programs that write large amounts of data (between 1 and 2 GB file sizes are common).  A snippet of code looks like this:
>
> I learned C about 15 years ago and there was no mention of a fsync.  My C book doesn't mention fsync either.  Granted I have written only 25-30 applications in the last 15 years or so so I am not an expert C programmer.
>

Your question is really off-topic for the Linux Kernel mailing list,
but let me point you somewhere else:

There is (1) the C library and (2) the functions provided by your
operating system. Linux follows to some extend the POSIX functions for
(2).

A good read is the book "Linux Systems Programming" if you want to go
beyond what the C library offers, for starters.

Regards,
-- 
Leon

RE: recommended programming practices for writing (was Linux 2.6.29)

[Adam Turk]

>> On Thu, Mar 26, 2009 at 8:03 PM, Adam Turk  wrote:
>>
>> I have been reading the Linux 2.6.29 thread with interest.  I have written several (10 or so) C programs that write large amounts of data (between 1 and 2 GB file sizes are common).  A snippet of code looks like this:
>>
>> I learned C about 15 years ago and there was no mention of a fsync.  My C book doesn't mention fsync either.  Granted I have written only 25-30 applications in the last 15 years or so so I am not an expert C programmer.
>>
>
> Your question is really off-topic for the Linux Kernel mailing list,
> but let me point you somewhere else:
>
> There is (1) the C library and (2) the functions provided by your
> operating system. Linux follows to some extend the POSIX functions for
> (2).
>
> A good read is the book "Linux Systems Programming" if you want to go
> beyond what the C library offers, for starters.
>

I don't think it is really off topic as in the original topic thread there were comments about lazy programmers who didn't write their code properly.  Noone who made those comments gave examples of bad and good code so I decided to ask.  

Thank you for the recommendation.  I will check the local bookstores.

Thanks,
Adam

_________________________________________________________________
Windows Live™ SkyDrive: Get 25 GB of free online storage.
http://windowslive.com/online/skydrive?ocid=TXT_TAGLM_WL_skydrive_032009

Re: recommended programming practices for writing (was Linux 2.6.29)

[J.A. Magallón]

On Thu, 26 Mar 2009 15:03:28 -0400, Adam Turk <bofh1234@hotmail.com> wrote:

> 
> I have been reading the Linux 2.6.29 thread with interest.  I have written several (10 or so) C programs that write large amounts of data (between 1 and 2 GB file sizes are common).  A snippet of code looks like this:
> 
> if((fptr = fopen(outfilename,"w")) == NULL) {
>   printf("File %s could not be created\n", outfilename);
> }
> else {
>   fprintf(fptr,"%s\n",datablock);
>   while(!writeouput(datablock,amount,tax)) {
>     getnext(dtablock)
>     fprintf(fptr,"%s\n",datablock);
>   }
>   fclose(fptr);
> }
> 
> I learned C about 15 years ago and there was no mention of a fsync.  My C book doesn't mention fsync either.  Granted I have written only 25-30 applications in the last 15 years or so so I am not an expert C programmer. 
> 
> From what Linus posted about git and checking the return from fclose I think I going to start doing that.  I also think I am going to start checking the return from fprintf and maybe write to a /tmp/file and then rename it.
> 
> So is there a C fsync that I should add before my fclose?
> What is the proper way to write to files?
> 
> Is what I am thinking about doing something that would be good or is it just a lack of my understanding the problem?
> 

You _really_ should not worry bout that. You write a file the way C or Fortran
or ADA tells you to do so. I think no language standard worries about when
the file really is dumped to disk from the kernel side. Even fflush() just
guarantees that the _high level_ buffer in the C library is sent to the kernel
routines to write to the device. But none gives you a standard function to
get the pages really witten to disk, or wait even till the hardware queues
have really sent the data to the platter. You will never know that.

Assume this world is imperfect. Your computer can always crash. I think just
routines like link/unlink/rename could be near to metal. And with journaled
filesystems, even that.

>From what I have understood reading LKML these years, if you do something like

    fwrite(one megabyte)
    fsync()
    pause()
    fwrite(other megabyte)

and unplug the cord while in the pause, nobody guarantees you that the first
megabyte is on the disk. Live with it.

PD: I'm not saying this is good, I think it's just the way it is.

-- 
J.A. Magallon <jamagallon()ono!com>     \               Software is like sex:
                                         \         It's better when it's free
Mandriva Linux release 2009.1 (Cooker) for x86_64
Linux 2.6.28.2-desktop-1mnb (gcc 4.3.2 (GCC) #1 Wed Jan