Problem with Compariing Floating Point Numbers 
Author Message
 Problem with Compariing Floating Point Numbers

Hi.

I have this weird scenarion.  I tried it out in the Immediate Pane

?9.8<9.11
False            -- it should be true.

I also tried ..

?cdbl("9.8")<cdbl("9.11")
False

What shoudl be done for VB to return a True?  I am aware of floating point
representations in Binary format is going to be somewhat different than
using a Base 10 format.. but am baffled as to what is to be done to override
this.

Please let me know ASAP.. Thanks in advance.

S.



Fri, 10 Oct 2003 15:01:36 GMT  
 Problem with Compariing Floating Point Numbers

Quote:
> I have this weird scenarion.  I tried it out in the Immediate Pane

> ?9.8<9.11
> False            -- it should be true.

9.8 is not less than 9.11.

9.8 = 9 80/100ths
9.11 = 9 11/100ths

Regards,

    -djm
--



Fri, 10 Oct 2003 15:04:58 GMT  
 Problem with Compariing Floating Point Numbers
LOL !
Your Windows Calculator must have a very weird bug.



Quote:


> > I have this weird scenarion.  I tried it out in the Immediate Pane

> > ?9.8<9.11
> > False            -- it should be true.

> 9.8 is not less than 9.11.

> 9.8 = 9 80/100ths
> 9.11 = 9 11/100ths

> Regards,

>     -djm
> --
> David Marteinson / Contract Programmer / 416-871-9175 /


- Show quoted text -



Fri, 10 Oct 2003 16:25:09 GMT  
 Problem with Compariing Floating Point Numbers
You don't think that perhaps 9.8 is in fact greater than 9.11?

Gareth


Quote:
> Hi.

> I have this weird scenarion.  I tried it out in the Immediate Pane

> ?9.8<9.11
> False            -- it should be true.

> I also tried ..

> ?cdbl("9.8")<cdbl("9.11")
> False

> What shoudl be done for VB to return a True?  I am aware of floating point
> representations in Binary format is going to be somewhat different than
> using a Base 10 format.. but am baffled as to what is to be done to
override
> this.

> Please let me know ASAP.. Thanks in advance.

> S.



Sat, 11 Oct 2003 00:27:02 GMT  
 
 [ 4 post ] 

 Relevant Pages 

1. Converting floating point number to IEEE 32-bit precision floating point format

2. Converting floating number to the IEEE 32/40-bit precision floating point format

3. TransferSpreadSheet method - floating point numbers

4. mask for floating-point number text box?

5. How to test in vba for floating point not a number values

6. IBasic - Errors when computing large Floating Point Numbers

7. VAX BASIC floating point number question

8. Code for converting decimal floating point numbers in VB6

9. 32 Bit Floating Point Numbers

10. Floating point numbers

11. Floating Point IEEE Numbers

12. TextBox that only accepts Currency / Floating Point Numbers

 

 
Powered by phpBB® Forum Software