Is STL or the Win32 SDK more flexiblefor low-level binary file I/O? 
Author Message
 Is STL or the Win32 SDK more flexiblefor low-level binary file I/O?

    I have a lot of messing around to do manipulating files byte-by-byte in
binary mode, inserting, cutting, detecting and removing EOF markers,
appending, then replacing the EOF, etc. etc. Is the STL more flexible for
this sort of stuff than the Win32 SDK functions?
    If so, any suggestions regarding STL functions to read up on would be
most helpful.

Thank-you in advance,
Steve C.



Sun, 02 Jan 2005 04:02:35 GMT  
 Is STL or the Win32 SDK more flexiblefor low-level binary file I/O?
It depends on what you understand flexibility...STL calls Win32 API
funcitons sooner or later. In my opinion, the most flexible way is to use
API directly...


Quote:
>     I have a lot of messing around to do manipulating files byte-by-byte
in
> binary mode, inserting, cutting, detecting and removing EOF markers,
> appending, then replacing the EOF, etc. etc. Is the STL more flexible for
> this sort of stuff than the Win32 SDK functions?
>     If so, any suggestions regarding STL functions to read up on would be
> most helpful.

> Thank-you in advance,
> Steve C.



Sun, 02 Jan 2005 07:45:39 GMT  
 Is STL or the Win32 SDK more flexiblefor low-level binary file I/O?
Using STL streams is about 10 times slower than the pure Win32 API.
Also consider using ANSI C FILE* functions as 'fopen', 'fclose', 'fread',
'fwrite'...:
they are platform independent, so you can port your code as easily as with
STL, and as my tests showed they are just as fast as pure API under MSVC6.

Regards,
Boris


Quote:
> It depends on what you understand flexibility...STL calls Win32 API
> funcitons sooner or later. In my opinion, the most flexible way is to use
> API directly...



> >     I have a lot of messing around to do manipulating files byte-by-byte
> in
> > binary mode, inserting, cutting, detecting and removing EOF markers,
> > appending, then replacing the EOF, etc. etc. Is the STL more flexible
for
> > this sort of stuff than the Win32 SDK functions?
> >     If so, any suggestions regarding STL functions to read up on would
be
> > most helpful.

> > Thank-you in advance,
> > Steve C.



Sun, 02 Jan 2005 15:15:52 GMT  
 Is STL or the Win32 SDK more flexiblefor low-level binary file I/O?
    Hello Kaan,
                        I was under the impression that much of the
STL is stand-alone and doesn't require the Win32 SDK. I
also thought that many STL functions were a lot faster and
written in assembly. Is this not the case?
    By flexibility, I meant the ability to jump around a lot in
the file, inserting, modifying and removing characters/bytes,
etc.

... Steve C.


Quote:
> It depends on what you understand flexibility...STL calls Win32 API
> funcitons sooner or later. In my opinion, the most flexible way is to use
> API directly...



Sun, 02 Jan 2005 15:38:19 GMT  
 Is STL or the Win32 SDK more flexiblefor low-level binary file I/O?
Thanks, Boris, for the extra info. I don't know where I got
the idea that STL is faster. ( Although speed is a consideration,
though, low-level flexibility is what I'm really after. Speed
would be a bonus. ) I'm now looking at the low-level functions
_open(), _close() etc. They should be both fast and flexible, I
hope.
( I'm currently using 'fopen()', 'fflush()' and 'fclose()', with
'fputc()', 'fgetc()', 'fputs()' and 'fgets()'. )

... Steve.


Quote:
> Using STL streams is about 10 times slower than the pure Win32 API.
> Also consider using ANSI C FILE* functions as 'fopen', 'fclose', 'fread',
> 'fwrite'...:
> they are platform independent, so you can port your code as easily as with
> STL, and as my tests showed they are just as fast as pure API under MSVC6.

> Regards,
> Boris



Sun, 02 Jan 2005 15:55:20 GMT  
 Is STL or the Win32 SDK more flexiblefor low-level binary file I/O?

Quote:

>     Hello Kaan,
>                         I was under the impression that much of the
> STL is stand-alone and doesn't require the Win32 SDK. I
> also thought that many STL functions were a lot faster and
> written in assembly. Is this not the case?

No.  All the implementations I've seen have been written in C++.

Quote:
>     By flexibility, I meant the ability to jump around a lot in
> the file, inserting, modifying and removing characters/bytes,
> etc.

You don't say how big the files you're working on are.  If you're doing a
lot of inserts into smallish files, you might be better loading them into
memory and operating on them there.   You might find that STL containers
like 'vector' or 'string' are useful for this.    I very much doubt that
you'll find that the STL is any use for directly handling binary files at
all - its 'streams' stuff is not really designed for that.

In terms of handwaving general performance, you should consider that the STL
file operations are written on top of the 'C' file operations which are in
turn written on top of the OS services.

You might find that something entirely Windows-specific like memory mapping
your file is a good option - all this depends on what you want to do, and
the amount of
data you want to do it to.

I'm afraid the answer to all questions along the lines of 'which gives the
best performance' is 'you need to profile your application'.

Will



Sun, 02 Jan 2005 17:11:42 GMT  
 Is STL or the Win32 SDK more flexiblefor low-level binary file I/O?
On Wed, 17 Jul 2002 10:15:52 +0300, "Boris Vidolov"

Quote:

>Using STL streams is about 10 times slower than the pure Win32 API.

What for? I suspect you are misusing STL streams. The overhead should
be less than 10%, assuming you are doing *exactly* the same thing.

Quote:
>Also consider using ANSI C FILE* functions as 'fopen', 'fclose', 'fread',
>'fwrite'...:
>they are platform independent, so you can port your code as easily as with
>STL, and as my tests showed they are just as fast as pure API under MSVC6.

What were your tests? There are some slight problems with MSVC6's
iostreams implementation, but I believe they are fixed in MSVC7.

Tom



Sun, 02 Jan 2005 18:11:49 GMT  
 Is STL or the Win32 SDK more flexiblefor low-level binary file I/O?
On Wed, 17 Jul 2002 10:11:42 +0100, "Will Dean"

Quote:

>You don't say how big the files you're working on are.  If you're doing a
>lot of inserts into smallish files, you might be better loading them into
>memory and operating on them there.   You might find that STL containers
>like 'vector' or 'string' are useful for this.    I very much doubt that
>you'll find that the STL is any use for directly handling binary files at
>all - its 'streams' stuff is not really designed for that.

No, but its 'streambufs' stuff is!

Quote:

>In terms of handwaving general performance, you should consider that the STL
>file operations are written on top of the 'C' file operations which are in
>turn written on top of the OS services.

Typically, the C++ implementation calls the API directly, without
indirection through the C standard file operations. This is true for
most implementations that I am aware of.

Quote:

>You might find that something entirely Windows-specific like memory mapping
>your file is a good option - all this depends on what you want to do, and
>the amount of
>data you want to do it to.

>I'm afraid the answer to all questions along the lines of 'which gives the
>best performance' is 'you need to profile your application'.

Quite. Although it is very easy to use the C++ streams stuff
suboptimally.

Tom



Sun, 02 Jan 2005 18:15:35 GMT  
 Is STL or the Win32 SDK more flexiblefor low-level binary file I/O?

Quote:

> What for? I suspect you are misusing STL streams. The overhead should
> be less than 10%, assuming you are doing *exactly* the same thing.

Funny that this should be discussed today, as I've just benchmarked the
following on VC7:

   FILE* outFile = fopen(pOutFile, "wt");
   for(int i = 0; i < 100000; i++)
   {
        fprintf(outFile, "%d %d %d\n", rand() % 100, rand() % 100, rand() %
360);
   }

against

   std::ofstream outStream(pOutFile);
   for(int i = 0; i < 100000; i++)
   {
    outStream << rand() % 100 << " " << rand() % 100 << " " << rand() % 360
<< std::endl;
   }

Which are pretty much the same thing.

The approximate times are as follows:

fprintf version - debug 570ms, release 400ms
stream version - debug 20500ms (!), release 1515ms

I'd be interested on comments as to what's going on here.

Personally, I detest the stream output syntax, so to find that it's between
3 and 35 times slower than the syntax I prefer is the final evidence I need
to convince me that it's not for me.   Maybe streams don't work well in the
presence of a sceptic....?

Will



Sun, 02 Jan 2005 18:28:06 GMT  
 Is STL or the Win32 SDK more flexiblefor low-level binary file I/O?

Quote:

> Typically, the C++ implementation calls the API directly, without
> indirection through the C standard file operations. This is true for
> most implementations that I am aware of.

The VC7 implementation calls fopen et al.  I know, because I've just stepped
into the library.

Quote:
> Quite. Although it is very easy to use the C++ streams stuff
> suboptimally.

And perhaps very hard to use it any other way....?

I'd really appreciate your views on the my other posting with code/timing
results in it.  Maybe I should use some STL component other than ofstream?

Will



Sun, 02 Jan 2005 19:03:14 GMT  
 Is STL or the Win32 SDK more flexiblefor low-level binary file I/O?
    Thank-you all for the response. I get the idea. STL is neither faster
nor more efficient than the base functions. I think I'll load the file into
memory, deal with it there in my own fashion, then write it back to disk.
(All using Win32 SDK functions).

... Steve C.



Mon, 03 Jan 2005 02:03:14 GMT  
 Is STL or the Win32 SDK more flexiblefor low-level binary file I/O?
std::endl flushes the file. Outputting "\n" does not. Try calling fflush
each time through the first loop and see performance plummet. You are
essentially disabling buffering.
--
With best wishes,
    Igor Tandetnik

"For every complex problem, there is a solution that is simple, neat,
and wrong." H.L. Mencken


Quote:

> > What for? I suspect you are misusing STL streams. The overhead
should
> > be less than 10%, assuming you are doing *exactly* the same thing.

> Funny that this should be discussed today, as I've just benchmarked
the
> following on VC7:

>    FILE* outFile = fopen(pOutFile, "wt");
>    for(int i = 0; i < 100000; i++)
>    {
>         fprintf(outFile, "%d %d %d\n", rand() % 100, rand() % 100,
rand() %
> 360);
>    }

> against

>    std::ofstream outStream(pOutFile);
>    for(int i = 0; i < 100000; i++)
>    {
>     outStream << rand() % 100 << " " << rand() % 100 << " " << rand()
% 360
> << std::endl;
>    }

> Which are pretty much the same thing.

> The approximate times are as follows:

> fprintf version - debug 570ms, release 400ms
> stream version - debug 20500ms (!), release 1515ms

> I'd be interested on comments as to what's going on here.

> Personally, I detest the stream output syntax, so to find that it's
between
> 3 and 35 times slower than the syntax I prefer is the final evidence I
need
> to convince me that it's not for me.   Maybe streams don't work well
in the
> presence of a sceptic....?

> Will



Mon, 03 Jan 2005 07:21:02 GMT  
 Is STL or the Win32 SDK more flexiblefor low-level binary file I/O?

Quote:

>Funny that this should be discussed today, as I've just benchmarked the
>following on VC7:

>   FILE* outFile = fopen(pOutFile, "wt");
>   for(int i = 0; i < 100000; i++)
>   {
>        fprintf(outFile, "%d %d %d\n", rand() % 100, rand() % 100, rand() %
>360);
>   }

>against

>   std::ofstream outStream(pOutFile);
>   for(int i = 0; i < 100000; i++)
>   {
>    outStream << rand() % 100 << " " << rand() % 100 << " " << rand() % 360
><< std::endl;
>   }

>Which are pretty much the same thing.

>The approximate times are as follows:

>fprintf version - debug 570ms, release 400ms
>stream version - debug 20500ms (!), release 1515ms

>I'd be interested on comments as to what's going on here.

>Personally, I detest the stream output syntax, so to find that it's between
>3 and 35 times slower than the syntax I prefer is the final evidence I need
>to convince me that it's not for me.   Maybe streams don't work well in the
>presence of a sceptic....?

No comment on your debug results, but your release results match my
VC6 experience with MT programs. To be fair, you should replace endl
with '\n', as I did.

I just traced through your code in an MT build which uses the CRT DLL,
and each C++ stream integer output operation does the following:

1. Creates and destroys several objects that have dtors, which
requires hidden exception code to be generated, in more than one
function.
2. Locks/unlocks a critical section for the stream buffer.
3. Locks/unlocks a critical section while copying and destroying a
locale object. (Done twice, once in operator<<, again in
num_put::_Iput.)
4. Locks/unlocks a critical section in use_facet. (Done twice, one in
operator<<, again in _Iput.)
5. Generates a printf format specifier and calls sprintf.
6. Creates a temporary std::string in num_put::_Iput.
7. Outputs the converted int to the stream buffer.

Those are just the highlights, but it's a lot of overhead to wrap
conversion of an integer. Single-threaded builds get rid of the
critical section activity, cutting the run time of your C++ stream
code. I find that the ratio of the run time of your C++ code to the
run time of your C code is about 1.76 for ST builds and 3.9 for MT
builds. The ratio of MT C to ST C is 1.13, while the ratio of MT C++
to ST C++ is 2.51.

Now, let's replace fprintf with the equivalent:

      char buf[40];
      sprintf(buf,"%d",rand() % 100);
      fputs(buf,outFile);
      fputc(' ',outFile);
      sprintf(buf,"%d",rand() % 100);
      fputs(buf,outFile);
      fputc(' ',outFile);
      sprintf(buf,"%d",rand() % 360);
      fputs(buf,outFile);
      fputc('\n',outFile);

The MT/ST ratio for this is 1.6, again due to critical section
locking. Comparing fputs to fprintf, the ST ratio is 1.15, and the MT
ratio is 1.63. Comparing C++ streams to fputs, the ST ratio is 1.53,
and the MT ratio is 2.4.

I conclude that breaking fprintf into individual operations as above
doesn't have a huge effect on ST run time (15%), though MT builds
suffer quite a bit (60%). This could be addressed (to some extent) if
VC exported the locking primitives and provided unlocked versions of
fputc and friends, because then I could lock the whole sequence of
operations and not incur that overhead with each call. Compared to the
C fprintf approach, the VC7 C++ stream approach is significantly
slower even in the ST case (76%), and greatly slower for the MT case
(290%).

--
Doug Harrison
Microsoft MVP - Visual C++
Eluent Software, LLC
http://www.eluent.com
Tools for VC++, VS.NET, and Windows



Mon, 03 Jan 2005 09:44:50 GMT  
 Is STL or the Win32 SDK more flexiblefor low-level binary file I/O?

{ snip lengthy analysis of my example }

Quote:
> I conclude that breaking fprintf into individual operations as above
> doesn't have a huge effect on ST run time (15%), though MT builds
> suffer quite a bit (60%). This could be addressed (to some extent) if
> VC exported the locking primitives and provided unlocked versions of
> fputc and friends, because then I could lock the whole sequence of
> operations and not incur that overhead with each call. Compared to the
> C fprintf approach, the VC7 C++ stream approach is significantly
> slower even in the ST case (76%), and greatly slower for the MT case
> (290%).

Thanks for this.  I'm pleased to see that you get similar results - and the
same conclusion that I came to looking through the library code that however
good the compiler and the programmer, the streams version was bound to be
slower.

There is such a strong trait in discussions of the STL for one party to go
'it's terribly slow' and the other to go 'that's only if you're using it
wrongly', that it's useful to have some real data on a simple example.

I get the impression that Igor will still think that it's my fault that the
streams version is 1.7 to 30 times as slow as the fprintf version even after
I've corrected the endl bug...

Will



Mon, 03 Jan 2005 15:33:32 GMT  
 Is STL or the Win32 SDK more flexiblefor low-level binary file I/O?

Quote:

> std::endl flushes the file. Outputting "\n" does not. Try calling
> fflush each time through the first loop and see performance plummet.
> You are essentially disabling buffering.

Thanks for this.  It does make a difference:

Debug (streams, with \n): 18000ms
Release: 700ms

So the release might be just about useful - it's slightly less than twice as
slow as the fprintf version.  Unfortunately, the debug build is still
impossibly slow - I don't think I would tolerate that in a real app.

(As a reference, adding fflush to the fprintf version slows it down to debug
1300, release 1000.)

There isn't much way that you can cut this other than that the streams stuff
is very much slower than fprintf for some kinds of simple formatted output.

Will



Mon, 03 Jan 2005 15:25:07 GMT  
 
 [ 25 post ]  Go to page: [1] [2]

 Relevant Pages 

1. NEED WAVE FILE DOCUMENTATION AND LOW LEVEL ROUTINES

2. Low-Level Database File Formats

3. Windows SDK or Win32 SDK?

4. Examples of persisting/streaming STL collections to OS files

5. Writing low-level network program using .NET

6. Capturing Lower Level Error Messages

7. Low Level Event Handling

8. Capturing Lower Level Error Messages

9. "Low level" programming

10. as low level as possible

11. low level i/o

12. MIDI LOW-LEVEL DOS

 

 
Powered by phpBB® Forum Software