Stack checking cdecl vs stdcall 
Author Message
 Stack checking cdecl vs stdcall

One of the nice new features of VC++ 6.0 is that if you mix up cdecl and
stdcall the debug version will detect this and notify you that your
stack pointer is messed up. This is a huge improvement over the
behaviour in previous versions where the debug version would typically
work (because of the stack frames) and the release version would crash
in an extremely hard to diagnose manner.

I depend on this feature so I was _extremely_ disappointed to find out
that they did not enable it when compile mfc42d.dll. This means that if
you make the common mistake of defining an ON_MESSAGE() handler like
this:

LRESULT OnDoMacro1();

then you will get NO WARNING! Your code will crash in extremely subtle
ways in release mode.

I just spent two hours tracking down this problem for a coworker. I'm
only annoyed because this problem was trivially avoidable.

--
.Bruce Dawson, Cavedog Entertainment.
Makers of Total Annihilation - http://www.*-*-*.com/



Sat, 18 May 2002 03:00:00 GMT  
 Stack checking cdecl vs stdcall


Quote:
>One of the nice new features of VC++ 6.0 is that if you mix up cdecl and
>stdcall the debug version will detect this and notify you that your
>stack pointer is messed up. This is a huge improvement over the
>behaviour in previous versions where the debug version would typically
>work (because of the stack frames) and the release version would crash
>in an extremely hard to diagnose manner.

>I depend on this feature so I was _extremely_ disappointed to find out
>that they did not enable it when compile mfc42d.dll. This means that if
>you make the common mistake of defining an ON_MESSAGE() handler like
>this:

>LRESULT OnDoMacro1();

>then you will get NO WARNING! Your code will crash in extremely subtle
>ways in release mode.

>I just spent two hours tracking down this problem for a coworker. I'm
>only annoyed because this problem was trivially avoidable.

I'm working on some debug-only code that will automatically check the
prototypes of functions in a message map at runtime.  Keep your eyes open
for a posting from me in the near future.

Regards,
Jon Sturgeon



Sun, 19 May 2002 03:00:00 GMT  
 Stack checking cdecl vs stdcall
Bruce,

Could you elaborate on the debug vs. release angle of this problem a
bit? Specifically, why doesn't this crash with the debug build?

Also, is this problem really related to mixing up cdecl and stdcall
prototypes? It seems like the problem that people always make is using
the wrong number of parameters, not mixing calling conventions.
Specifically, LRESULT OnDoMacro1() is cdecl, which is the correct
calling convention, isn't it?

Thanks,

Everett McKay


Quote:

> One of the nice new features of VC++ 6.0 is that if you mix up cdecl
and
> stdcall the debug version will detect this and notify you that your
> stack pointer is messed up. This is a huge improvement over the
> behaviour in previous versions where the debug version would typically
> work (because of the stack frames) and the release version would crash
> in an extremely hard to diagnose manner.

> I depend on this feature so I was _extremely_ disappointed to find out
> that they did not enable it when compile mfc42d.dll. This means that
if
> you make the common mistake of defining an ON_MESSAGE() handler like
> this:

> LRESULT OnDoMacro1();

> then you will get NO WARNING! Your code will crash in extremely subtle
> ways in release mode.

> I just spent two hours tracking down this problem for a coworker. I'm
> only annoyed because this problem was trivially avoidable.

> --
> .Bruce Dawson, Cavedog Entertainment.
> Makers of Total Annihilation - http://www.cavedog.com

Sent via Deja.com http://www.deja.com/
Before you buy.


Sat, 25 May 2002 03:00:00 GMT  
 Stack checking cdecl vs stdcall
The program works in debug mode because a stack frame is set up
using ebp and when this stack frame is taken down the correct
value of esp is restored - thus hiding the bug. In release mode, stack
frames are frequently not created, as an optimization.

This problem can happen when you mix cdecl and stdcall. It can also
happen if you call a stdcall function with the wrong number of
arguments because the callee will pop off the number of arguments
that it expects, even if an improper number was pushed.

Typically this problem requires that you be calling through a function
pointer, because otherwise it is very difficult to get a mismatch, due
to name mangling.

All non-var args member functions use the thiscall calling convention,
which is very similar to stdcall, but also implicitly passes the this
pointer
in ecx.

The problem happens in MFC because MFC uses function pointers and it
typecasts them, thus suppressing the errors that would otherwise catch
this problem. The problem isn't caught by the VC++ 6.0 stack value
checking because this is not enabled in mfc42d.dll.

Quote:

> Bruce,

> Could you elaborate on the debug vs. release angle of this problem a
> bit? Specifically, why doesn't this crash with the debug build?

> Also, is this problem really related to mixing up cdecl and stdcall
> prototypes? It seems like the problem that people always make is using
> the wrong number of parameters, not mixing calling conventions.
> Specifically, LRESULT OnDoMacro1() is cdecl, which is the correct
> calling convention, isn't it?

> Thanks,

> Everett McKay



> > One of the nice new features of VC++ 6.0 is that if you mix up cdecl
> and
> > stdcall the debug version will detect this and notify you that your
> > stack pointer is messed up. This is a huge improvement over the
> > behaviour in previous versions where the debug version would typically
> > work (because of the stack frames) and the release version would crash
> > in an extremely hard to diagnose manner.

> > I depend on this feature so I was _extremely_ disappointed to find out
> > that they did not enable it when compile mfc42d.dll. This means that
> if
> > you make the common mistake of defining an ON_MESSAGE() handler like
> > this:

> > LRESULT OnDoMacro1();

> > then you will get NO WARNING! Your code will crash in extremely subtle
> > ways in release mode.

> > I just spent two hours tracking down this problem for a coworker. I'm
> > only annoyed because this problem was trivially avoidable.

> > --
> > .Bruce Dawson, Cavedog Entertainment.
> > Makers of Total Annihilation - http://www.cavedog.com

> Sent via Deja.com http://www.deja.com/
> Before you buy.

--
.Bruce Dawson, Cavedog Entertainment.
Makers of Total Annihilation - http://www.cavedog.com


Sat, 25 May 2002 03:00:00 GMT  
 Stack checking cdecl vs stdcall
Note that the VC 6 compiler includes a feature to address just this sort of
problem.  Compile your debug build w/ /GZ and it will insert code to check
for incorrect calling conventions and a few other assorted errors.  When the
bad code executes, you will be notified.

-Jay


Quote:
> The program works in debug mode because a stack frame is set up
> using ebp and when this stack frame is taken down the correct
> value of esp is restored - thus hiding the bug. In release mode, stack
> frames are frequently not created, as an optimization.

> This problem can happen when you mix cdecl and stdcall. It can also
> happen if you call a stdcall function with the wrong number of
> arguments because the callee will pop off the number of arguments
> that it expects, even if an improper number was pushed.

> Typically this problem requires that you be calling through a function
> pointer, because otherwise it is very difficult to get a mismatch, due
> to name mangling.

> All non-var args member functions use the thiscall calling convention,
> which is very similar to stdcall, but also implicitly passes the this
> pointer
> in ecx.

> The problem happens in MFC because MFC uses function pointers and it
> typecasts them, thus suppressing the errors that would otherwise catch
> this problem. The problem isn't caught by the VC++ 6.0 stack value
> checking because this is not enabled in mfc42d.dll.


> > Bruce,

> > Could you elaborate on the debug vs. release angle of this problem a
> > bit? Specifically, why doesn't this crash with the debug build?

> > Also, is this problem really related to mixing up cdecl and stdcall
> > prototypes? It seems like the problem that people always make is using
> > the wrong number of parameters, not mixing calling conventions.
> > Specifically, LRESULT OnDoMacro1() is cdecl, which is the correct
> > calling convention, isn't it?

> > Thanks,

> > Everett McKay



> > > One of the nice new features of VC++ 6.0 is that if you mix up cdecl
> > and
> > > stdcall the debug version will detect this and notify you that your
> > > stack pointer is messed up. This is a huge improvement over the
> > > behaviour in previous versions where the debug version would typically
> > > work (because of the stack frames) and the release version would crash
> > > in an extremely hard to diagnose manner.

> > > I depend on this feature so I was _extremely_ disappointed to find out
> > > that they did not enable it when compile mfc42d.dll. This means that
> > if
> > > you make the common mistake of defining an ON_MESSAGE() handler like
> > > this:

> > > LRESULT OnDoMacro1();

> > > then you will get NO WARNING! Your code will crash in extremely subtle
> > > ways in release mode.

> > > I just spent two hours tracking down this problem for a coworker. I'm
> > > only annoyed because this problem was trivially avoidable.

> > > --
> > > .Bruce Dawson, Cavedog Entertainment.
> > > Makers of Total Annihilation - http://www.cavedog.com

> > Sent via Deja.com http://www.deja.com/
> > Before you buy.

> --
> .Bruce Dawson, Cavedog Entertainment.
> Makers of Total Annihilation - http://www.cavedog.com



Mon, 27 May 2002 03:00:00 GMT  
 Stack checking cdecl vs stdcall

Quote:
>Note that the VC 6 compiler includes a feature to address just this sort of
>problem.  Compile your debug build w/ /GZ and it will insert code to check
>for incorrect calling conventions and a few other assorted errors.  When the
>bad code executes, you will be notified.

Yes, but the point of the original post was that mfc42d.dll was not built
by Microsoft with this switch enabled, thus the check is not performed on
any functions called by MFC.  Such as message-map functions, which (IMHO)
are the number one source of such errors.

Regards,
Jon Sturgeon



Mon, 27 May 2002 03:00:00 GMT  
 
 [ 6 post ] 

 Relevant Pages 

1. cdecl vs stdcall convention...

2. Difference between cdecl, stdcall, and thiscall?

3. stdcall and cdecl Calling conventions?

4. *cdcel vs stdcall

5. TurboC 1.5 Stack check

6. TurboC 1.5 Stack check

7. Stack checking code generation bug in VC .net

8. MSC6.0 stack checking pragma

9. Stack checking in MSC 6.0

10. Newbie Question - Stack vs Heap

11. stack vs heap allocation.

12. how CSharp handles locals (heap vs stack)

 

 
Powered by phpBB® Forum Software