BUG?? Weird Int rounding 
Author Message
 BUG?? Weird Int rounding

Hi,
I first noticed this in an Access query, but check this out:
Paste this code into a project and run it:

    Dim iResult As Integer

        iResult = Int((57 / 100) * 100)

        MsgBox "test 1 57/100*100 = " & iResult

The result you get the first time is 56, which is incorrect. Run it again
and
it calculates the result as 57.
Any ideas?
Cheers,
Kem



Sun, 12 May 2002 03:00:00 GMT  
 BUG?? Weird Int rounding
I get 57 every time....
Quote:

> Hi,
> I first noticed this in an Access query, but check this out:
> Paste this code into a project and run it:

>     Dim iResult As Integer

>         iResult = Int((57 / 100) * 100)

>         MsgBox "test 1 57/100*100 = " & iResult

> The result you get the first time is 56, which is incorrect. Run it again
> and
> it calculates the result as 57.
> Any ideas?
> Cheers,
> Kem



Sun, 12 May 2002 03:00:00 GMT  
 BUG?? Weird Int rounding
I get 56 on the first run, and then 57 each run thereafter. Running VB6 with
SP3. Strange!  ;-)


Quote:
> I get 57 every time....




Quote:
> > Hi,
> > I first noticed this in an Access query, but check this out:
> > Paste this code into a project and run it:

> >     Dim iResult As Integer

> >         iResult = Int((57 / 100) * 100)

> >         MsgBox "test 1 57/100*100 = " & iResult

> > The result you get the first time is 56, which is incorrect. Run it
again
> > and
> > it calculates the result as 57.
> > Any ideas?
> > Cheers,
> > Kem



Sun, 12 May 2002 03:00:00 GMT  
 BUG?? Weird Int rounding


Quote:
>Hi,
>I first noticed this in an Access query, but check this out:
>Paste this code into a project and run it:

>    Dim iResult As Integer

>        iResult = Int((57 / 100) * 100)

>        MsgBox "test 1 57/100*100 = " & iResult

>The result you get the first time is 56, which is incorrect. Run it again
>and
>it calculates the result as 57.
>Any ideas?
>Cheers,
>Kem

I am getting 57 every time. What version of VB and what SP are you
running.

GB



Mon, 13 May 2002 03:00:00 GMT  
 BUG?? Weird Int rounding
Kev

I get 57 all the time with VB6 and SP3.  However, we no doubt don't have the
same make and model of CPU.

When you do a calculation like 57/100 VB uses floating point arithmetic such
as it would if you were dealing explicitly with variables declared as singles
or doubles.  The representation of numbers in the floating point world is not
the same as in out decimal world, and thus a result that can be represented
simply for us as 0.57 may end up after a floating point division to be
translated back to our decimal system as 0.5700000000000000000002 o r
0.56999999999999999999.  There is no guarantee as to which way it will go or
as to whether it will be consistently higher or lower than the "exact"
value.  If it is the former, your Int() will give 57.  If it is the latter,
your Int() gives 56.

The m{*filter*}of the story is that if floating point is used implicitly or
explicitly,  you should always round before using Int(), e.g.

        iResult = Int(Round(57 / 100) * 100, 6)

As you have found out, if you don't do this you can produce some very
embarassing results.

John..........

Quote:

> Hi,
> I first noticed this in an Access query, but check this out:
> Paste this code into a project and run it:

>     Dim iResult As Integer

>         iResult = Int((57 / 100) * 100)

>         MsgBox "test 1 57/100*100 = " & iResult

> The result you get the first time is 56, which is incorrect. Run it again
> and
> it calculates the result as 57.
> Any ideas?
> Cheers,
> Kem



Tue, 14 May 2002 03:00:00 GMT  
 
 [ 5 post ] 

 Relevant Pages 

1. Int & Format rounding errors

2. Rounding Int Values

3. ???Bug in Int()-function of ACC95 and ACC97 ???

4. Bug when ising INT function

5. TERRIBLE BUG IN INT FUNCION???????????????????

6. BUG: VB4(16 bit) - Int and Fix functions don't work correctly

7. Bug in Int() and Numbers

8. Converting BitArray to Int and Int to BitArray

9. How to display int, short int range values

10. Int to Str or Str to Int ??

11. Converting ULONG (4 byte binary) int VB Long / SQL INT for DTS Import

12. Convert Int to Str & Str to Int

 

 
Powered by phpBB® Forum Software