*DECREASING* THE DEFAULT STACK SIZE 
Author Message
 *DECREASING* THE DEFAULT STACK SIZE

I've combed Help, the Microsoft site, dejanews, codeguru, and even google
for the answer to my question.

I'm currently making use of CreateFiber() to generate a potentially very large
number of threads which are cooperatively scheduled in a simulator I am
writing. The number of threads is sufficiently large that the system will run out
of memory if the default stack size of 1MB is used.

I followed the stacksize documentation to the STACKSIZE information section,
including how to change Project->Settings->Link->Output  Reserve and Commit
settings. The documentation says that the values are the size in bytes, so I changed
Reserve and Link to 8000 each. This should generate 8K ministacks for each of
these threads.

My application is a small application constructed using a DLL which I also wrote
where the threads are made.

I cleaned all projects and libraries, made certain that the Reserve and Commit
was set for both the application and the library project settings (wasn't sure
if the app needed it, the dll needed it, or both), and then rebuilt.

The application dies after generating 1,411 fibers. It dies in CreateFiber() with
an error code of 8 -- an OutOfMemory error. The simulation is of sufficient scale
that TENS OF THOUSANDS of fibers may eventually be required.

CreateFiber is given an argument of 8000 bytes in each case.

I'm a bit confused in that 1411 x 1 MB greatly exceeds the amount of total
swap space I have, so I'm not quite sure what exactly is going on. If it's
using the default stack size, it should die much earlier, but it appears to not
be using the 8K size I've specified either.

What am I doing wrong?

C//



Mon, 07 Jul 2003 02:21:35 GMT  
 *DECREASING* THE DEFAULT STACK SIZE
(Followups set to m.p.vc.l, m.p.vc not valid)


Quote:
> I'm currently making use of CreateFiber() to generate a potentially very
> large number of threads which are cooperatively scheduled in a simulator
> I am writing. The number of threads is sufficiently large that the system
> will run out of memory if the default stack size of 1MB is used.

 ...

Quote:
> The application dies after generating 1,411 fibers. It dies in
> CreateFiber() with an error code of 8 -- an OutOfMemory error. The
> simulation is of sufficient scale that TENS OF THOUSANDS of fibers may
> eventually be required.
> CreateFiber is given an argument of 8000 bytes in each case.
> I'm a bit confused in that 1411 x 1 MB greatly exceeds the amount of total
> swap space I have, so I'm not quite sure what exactly is going on. If it's
> using the default stack size, it should die much earlier, but it appears
> to not be using the 8K size I've specified either.
> What am I doing wrong?

As I understand it, Windows doesn't actually commit physical memory for
threads until the increased stack size requires it - it simply reserves
that large a space in the 4 GB address space each process has (under NT;
under Win 95/98 all apps share a single 4 GB space).

So - since 8000 bytes times 1411 threads is only about 11 MB, it'd seem
that either something other than the stack is using up a lot of memory,
or the stacks aren't being limited to only 8000 bytes per thread, always
presuming that CreateFiber is returning an accurate error code.

Do your threads allocate much in the way of dynamic memory on the heap,
vice automatic variables and call parameters which'd go on the local
stack?  Also, you might try allocating a good-sized chunk of memory
and freeing it before your calls to CreateFiber, just to verify that it
is in fact a memory problem and not an issue of running out of some
other kind of resources being reported as being out of memory.

--
-- Chase Vogelsberg, Senior Principal Analyst
-- Logicon Information Systems & Services
-- Wormwood and wine, and the bitter taste of ashes.



Mon, 07 Jul 2003 05:02:47 GMT  
 *DECREASING* THE DEFAULT STACK SIZE
Go to www.sysinternals.com and download the latest version of HandleEx.
This wonderful tool will show you all sorts of useful things.
In particular how many handles your process has and which types.

It may not be memory that is the problem. You may be running out of
something else.

Also use the Task Manager, assuming that you have NT4 or W2K, and watch the
memory trace in real time.
Try Perfmon as well it can provide useful information as well.

Nick

--


Quote:

> I've combed Help, the Microsoft site, dejanews, codeguru, and even google
> for the answer to my question.

> I'm currently making use of CreateFiber() to generate a potentially very
large
> number of threads which are cooperatively scheduled in a simulator I am
> writing. The number of threads is sufficiently large that the system will
run out
> of memory if the default stack size of 1MB is used.

> I followed the stacksize documentation to the STACKSIZE information
section,
> including how to change Project->Settings->Link->Output  Reserve and
Commit
> settings. The documentation says that the values are the size in bytes, so
I changed
> Reserve and Link to 8000 each. This should generate 8K ministacks for each
of
> these threads.

> My application is a small application constructed using a DLL which I also
wrote
> where the threads are made.

> I cleaned all projects and libraries, made certain that the Reserve and
Commit
> was set for both the application and the library project settings (wasn't
sure
> if the app needed it, the dll needed it, or both), and then rebuilt.

> The application dies after generating 1,411 fibers. It dies in
CreateFiber() with
> an error code of 8 -- an OutOfMemory error. The simulation is of
sufficient scale
> that TENS OF THOUSANDS of fibers may eventually be required.

> CreateFiber is given an argument of 8000 bytes in each case.

> I'm a bit confused in that 1411 x 1 MB greatly exceeds the amount of total
> swap space I have, so I'm not quite sure what exactly is going on. If it's
> using the default stack size, it should die much earlier, but it appears
to not
> be using the 8K size I've specified either.

> What am I doing wrong?

> C//



Mon, 07 Jul 2003 05:42:16 GMT  
 *DECREASING* THE DEFAULT STACK SIZE
You might want to double check whether your application stack is
actually getting set to 8000 bytes. Use dumpbin with the /headers option
to do this.

You can also use VirtualQueryEx() to walk your process memory space
to see how much memory is reserved and committed and in how many
blocks. Alternately, use VirtualQueryEx with a pointer to your stack,
in the main thread and in your fibers, to find out how big the stack is.
This will let you find out how much space is reserved and committed.

As somebody else said, the problem may be lack of address space more
than lack of memory. VirtualQueryEx will let you find out which it is.

Quote:

> I've combed Help, the Microsoft site, dejanews, codeguru, and even google
> for the answer to my question.

> I'm currently making use of CreateFiber() to generate a potentially very large
> number of threads which are cooperatively scheduled in a simulator I am
> writing. The number of threads is sufficiently large that the system will run out
> of memory if the default stack size of 1MB is used.

> I followed the stacksize documentation to the STACKSIZE information section,
> including how to change Project->Settings->Link->Output  Reserve and Commit
> settings. The documentation says that the values are the size in bytes, so I changed
> Reserve and Link to 8000 each. This should generate 8K ministacks for each of
> these threads.

> My application is a small application constructed using a DLL which I also wrote
> where the threads are made.

> I cleaned all projects and libraries, made certain that the Reserve and Commit
> was set for both the application and the library project settings (wasn't sure
> if the app needed it, the dll needed it, or both), and then rebuilt.

> The application dies after generating 1,411 fibers. It dies in CreateFiber() with
> an error code of 8 -- an OutOfMemory error. The simulation is of sufficient scale
> that TENS OF THOUSANDS of fibers may eventually be required.

> CreateFiber is given an argument of 8000 bytes in each case.

> I'm a bit confused in that 1411 x 1 MB greatly exceeds the amount of total
> swap space I have, so I'm not quite sure what exactly is going on. If it's
> using the default stack size, it should die much earlier, but it appears to not
> be using the 8K size I've specified either.

> What am I doing wrong?

> C//

--
.Bruce Dawson, Humongous Entertainment (we're hiring).
http://www.humongous.com/
Send job applications by e-mail, post technical questions
to the newsgroups please. Thanks.


Tue, 08 Jul 2003 02:02:56 GMT  
 *DECREASING* THE DEFAULT STACK SIZE


Fri, 19 Jun 1992 00:00:00 GMT  
 *DECREASING* THE DEFAULT STACK SIZE

I have been investigating this matter more fully and wish to offer an
update. While I have been partly successful in solving this problem,
I have uncovered some perplexing hidden constraints in the Create
Fiber()/CreateThread() interface and would like to share my experiences
with the group in any case.

Part of achieving some understanding of what was going on included
extensive reading about Windows memory management, specifically
including understanding the differences between Commit and Reserve
Memory.

As a reminder, my application was crashing after creating ~1411
Fibers, returning error code = 8 -- an OutOfMemoryError. Observation
of the application under Task Manager seemed to indicate that it
wasn't really out of memory, so I didn't understand what was going
on. I tried altering the Reserve/Commit settings under linker options,
but this didn't seem to change anything.

EDITBIN /HEADERS only showed that my application had been
correctly linked with the proper default Reserve/Commit settings.

The most informative thing that I did was to call GlobalMemoryStatus()
just after every CreateFiber() call and then print out the physical
and virtual memory availability. This showed that each CreateFiber()
call correctly consumed the amount of physical memory requested
(8K), but was swallowing down a full megabyte (1024K) of addressable
virtual space.

This is where things get even stranger. I noticed a strange relationship
between the amount of memory REQUESTED (read: NOT granted)
for physical memory consumption in the CreateFiber() call. For example,
if I request *1* byte of physical memory be allocated to the stack for
a fiber, 8K is committed (the link-specified minimum) but only 64K is
Reserved. I then played with this number quite aribitrarily and found that
at or near 1000 bytes of request, the Reserve size jumps in one single
increment to a full 1024K.

I'm a bit mystified by this.

While my solution is workable, bizarre behavior like this always leaves
me wishing I understood what was going on.

Furthermore, I'd truly like to be able to get minimum Reserve sizes
of smaller than 64K. In certain situations, I'd like to be able to drop
as low as 8K Reserve per stack. This isn't critical, but it would be
nice to be able to run higher density versions of my simulation on
a 32 bit machine.

I suppose I could always cave in and write my own register and
stack restoration code in inline assembly and achieve what it is
that the Fiber interface actually accomplishes, but I'd like to avoid
this if possible.

Anyone have any feedback?

Joe Kraska
Decision Support Technologies
Verizon Technology Organization
San Diego, CA



Sat, 12 Jul 2003 13:54:00 GMT  
 
 [ 6 post ] 

 Relevant Pages 

1. *DECREASING* THE DEFAULT STACK SIZE

2. /STACK increase the stack, means decrease the heap ?

3. /STACK increase the stack, means decrease the heap ?

4. How to increase the default stack size of MSVC++

5. MDB file size decrease

6. Changing project dependencies to decrease .exe size

7. MoveWindow Decreasing window size?

8. how to decrease DLL size

9. ATL to decrease Internet download size??

10. ATL to decrease Internet download size??

11. How to decrease the compound files size?

12. std::stack default constructor?

 

 
Powered by phpBB® Forum Software