My Number comes back different 
Author Message
 My Number comes back different

I am storing a percentage value in an Access database as a single. The
percentage is a variation percentage (ie -20% saves as 0.8 +20% saves as
1.2) The number is saved as (100 + [Entered %]) / 100 then when I read
the number from the database I use ([% from database] * 100) - 100. The
problem is when I use the above example of 20% it returns 20.000001.
The other numbers I have tested work OK.

Any one care to have a stab at what is causing this.

Cheers

Blair
--
Another Message from the Desk of
===============================================================

Business                                    Fax +61 3 5722 2587
Software........Design...Develop...Integrate



Fri, 03 Sep 1999 03:00:00 GMT  
 My Number comes back different

Quote:

> I am storing a percentage value in an Access database as a single. The
> percentage is a variation percentage (ie -20% saves as 0.8 +20% saves as
> 1.2) The number is saved as (100 + [Entered %]) / 100 then when I read
> the number from the database I use ([% from database] * 100) - 100. The
> problem is when I use the above example of 20% it returns 20.000001.
> The other numbers I have tested work OK.

> Any one care to have a stab at what is causing this.

> Cheers

> Blair

As you save it as SINGLE, I would not be surprised.
Jeff Hong YAN


Fri, 03 Sep 1999 03:00:00 GMT  
 My Number comes back different

Quote:

>I am storing a percentage value in an Access database as a single. The
>percentage is a variation percentage (ie -20% saves as 0.8 +20% saves as
>1.2) The number is saved as (100 + [Entered %]) / 100 then when I read
>the number from the database I use ([% from database] * 100) - 100. The
>problem is when I use the above example of 20% it returns 20.000001.
>The other numbers I have tested work OK.

>Any one care to have a stab at what is causing this.

>Cheers

>Blair
>--
>Another Message from the Desk of
>===============================================================

>Business                                Fax +61 3 5722 2587
>Software........Design...Develop...Integrate

Blair,

Single precision numbers are stored in 4 byte fields as
X*10^Y, where X is between -1 and 1, and Y is between about
-38 and about 38. The Y part takes up 2 bytes, so you only
have 2 bytes to represent the X part as a fraction and
rounding inevitably takes place.

If you are working with Integer values then comparisons can be
exact, but with floating-point numbers, like Single and
Double, you have to allow for these small errors as they are
unavoidable.  Changing from Single to Double will just reduce
the size of these errors.


                 Holland Numerics Ltd
                 Royston, Herts, UK
                 mobile: +44-(0)850-295556



Fri, 03 Sep 1999 03:00:00 GMT  
 My Number comes back different

Quote:

> I am storing a percentage value in an Access database as a single. The
> percentage is a variation percentage (ie -20% saves as 0.8 +20% saves as
> 1.2) The number is saved as (100 + [Entered %]) / 100 then when I read
> the number from the database I use ([% from database] * 100) - 100. The
> problem is when I use the above example of 20% it returns 20.000001.
> The other numbers I have tested work OK.
> Any one care to have a stab at what is causing this.

[snip]

Welcome to the wacky world of floating point math! Some decimal
fractions just can't be represented exactly by binary fractions,
and I think 0.2 and certain multiples of 0.2 are in that
category. Representing them with a Float or Double is like
trying to use a calculator to divide 2 by 3 or 22 by 7. (Remember
22/7 from school?)

You might be better off converting everything to Currency or
maybe using some of the various rounding tricks like (off the top
of my head):

' ChopFraction(20.9999, 2) should return 20.99
Public Function ChopFraction (ByVal X As Double, ByVal Y As Integer) As Double
        Dim Fac As Double: Fac = 10 ^ Y
        ChopFraction = Int(X * Fac) / Fac
End Function

--

WARNING: I cannot be held responsible for the above        They're   coming  to
because  my cats have  apparently  learned to type.        take me away, ha ha!



Sat, 04 Sep 1999 03:00:00 GMT  
 
 [ 4 post ] 

 Relevant Pages 

1. CurrentDB.Name comes back as DOS

2. MVPs REINSTATED - (MichKa - coming back?)

3. Coming back to collections

4. hProcess coming back from ShellExecuteEx is always 0

5. Data comes back that was Edited.

6. different numbers, which is lowest number

7. How to recognize tones (numbers) coming from voice modem

8. Bring back different visibility of properties!

9. ListView items with different back color

10. history.back(number)

11. Converting Date to Number -- and back

12. autonumber to start at a different number

 

 
Powered by phpBB® Forum Software