VB6 Newbie Arithmetic error 
Author Message
 VB6 Newbie Arithmetic error

Can someone tell me why:
All Variables declared as double,  0.018 - 0.017 returns 9.99999999999998E-03.


Mon, 30 Jan 2012 19:57:01 GMT  
 VB6 Newbie Arithmetic error

On Thu, 13 Aug 2009 04:57:01 -0700, Tony

Quote:

>Can someone tell me why:
>All Variables declared as double,  0.018 - 0.017 returns 9.99999999999998E-03.

Understanding IEEE Floating-Point Errors
http://support.microsoft.com/kb/42980

How To Work Around Floating-Point Errors
http://support.microsoft.com/kb/69333
--
Alfie [UK]
<http://www.delphia.co.uk/>
As they say at Microsoft - 'What do you want to reinstall today?'.



Mon, 30 Jan 2012 20:10:58 GMT  
 VB6 Newbie Arithmetic error


Quote:
> Can someone tell me why:
> All Variables declared as double,  0.018 - 0.017 returns
> 9.99999999999998E-03.

It's not an arithmetic error. It's just being displayed using scientific
notation.  For display, simply format it however you want.  For example:

    Dim dbl1 As Double
    Dim dbl2 As Double

    dbl1 = 0.018
    dbl2 = 0.017
    Debug.Print Format$(dbl1 - dbl2, "####0.#######")

--
Mike



Mon, 30 Jan 2012 20:15:07 GMT  
 VB6 Newbie Arithmetic error
Thanks Alfie, I looked everywhere for info like that - but could not find it!
Quote:

> On Thu, 13 Aug 2009 04:57:01 -0700, Tony

> >Can someone tell me why:
> >All Variables declared as double,  0.018 - 0.017 returns 9.99999999999998E-03.

> Understanding IEEE Floating-Point Errors
> http://support.microsoft.com/kb/42980

> How To Work Around Floating-Point Errors
> http://support.microsoft.com/kb/69333
> --
> Alfie [UK]
> <http://www.delphia.co.uk/>
> As they say at Microsoft - 'What do you want to reinstall today?'.



Tue, 31 Jan 2012 19:49:01 GMT  
 VB6 Newbie Arithmetic error
Thanks Mike, I'll try that.
Quote:



> > Can someone tell me why:
> > All Variables declared as double,  0.018 - 0.017 returns
> > 9.99999999999998E-03.

> It's not an arithmetic error. It's just being displayed using scientific
> notation.  For display, simply format it however you want.  For example:

>     Dim dbl1 As Double
>     Dim dbl2 As Double

>     dbl1 = 0.018
>     dbl2 = 0.017
>     Debug.Print Format$(dbl1 - dbl2, "####0.#######")

> --
> Mike



Tue, 31 Jan 2012 19:58:01 GMT  
 VB6 Newbie Arithmetic error


Quote:
> Thanks Alfie, I looked everywhere for info like that - but could not find
it!

"What Every Computer Scientist Should Know About Floating-Point Arithmetic"
http://docs.sun.com/source/806-3568/ncg_goldberg.html

All the ugly information

-ralph



Wed, 01 Feb 2012 00:35:09 GMT  
 
 [ 6 post ] 

 Relevant Pages 

1. Error handling / Arithmetic overflow error

2. errors in arithmetic

3. Problem with ODBC/VB6 read-only error (newbie)

4. Problem with ODBC/VB6 read-only error (newbie)

5. Very newbie problem with Object Error (Error 424)

6. arithmetic problem in a VB module

7. Simple arithmetic with DOCVARIABLES

8. Arithmetic operation resulted in an Overflow in FilePut

9. Performing arithmetic on unit.pixel properties

10. arithmetic overflow without overflow

11. Overriding Arithmetic Operators (help!)

12. BASIC source code for Arithmetic compression

 

 
Powered by phpBB® Forum Software