[010111]

i was adding a list of prices for a G5 proposal for a friend and i noticed a possible Calculator bug...

adding these numbers:

2411
299
99.95
197.90
119.95
695
1190

i get:

5012.799999999999999999

what!?

yeah... i can round up... but where did i subtract the 0.00000000000001 at?!

i did it on 2 machines. same deal.

weird.

[proton]

It's caused by binary mathematics. You can't accurately represent some numbers in binary that can be represented in decimal accurately.

For example in decimal numbers, 1/3 can not be accurately represented, and is represented as 0.33333 (repeater). Similarly, in binary numbers, things like 0.1 can not be accurately represented.

In short, you're seeing a limitation of using a computer, any computer.

- proton

[cg2k6]

While in Calculator just go to "view-->precision" and then change it to something less than the default 12 and you won't get this many digits. I don't know if that's what you wanted, but that will make it look nicer.

[010111]

Originally posted by proton:
It's caused by binary mathematics. You can't accurately represent some numbers in binary that can be represented in decimal accurately.

For example in decimal numbers, 1/3 can not be accurately represented, and is represented as 0.33333 (repeater). Similarly, in binary numbers, things like 0.1 can not be accurately represented.

In short, you're seeing a limitation of using a computer, any computer.

- proton

i understand that. but i was just *adding*... nothing too complex that would end up with funky fractional numbers. when i am adding numbers that never go over .00 spaces... you would think it woul dbe clever enough to limit the answer as such. mulitplying ... dividing... sure i could see coming up with 0.799999999999 as a possible answer. adding? i wouldn't think so.

[proton]

Here's my quick calculations:

2411 - can be represented accurately in floating point
299 - can be represented accurately in floating point
99.95 - can not be represented accurately in floating point
197.90 - can not be represented accurately in floating point
119.95 - can not be represented accurately in floating point
695 - can be represented accurately in floating point
1190 - can be represented accurately in floating point

In short, in a floating point calculator (which is what Apple has used for Calculator), 3 of your 7 numbers could not be accurately represented. While there are ways that you can work around these problems, it is probably more effort than is warranted in a simple Calculator. Maybe if you were writing a Graphing Calculator you'd want to work around it...

- proton

[010111]

Originally posted by proton:
Here's my quick calculations:

2411 - can be represented accurately in floating point
299 - can be represented accurately in floating point
99.95 - can not be represented accurately in floating point
197.90 - can not be represented accurately in floating point
119.95 - can not be represented accurately in floating point
695 - can be represented accurately in floating point
1190 - can be represented accurately in floating point

In short, in a floating point calculator (which is what Apple has used for Calculator), 3 of your 7 numbers could not be accurately represented. While there are ways that you can work around these problems, it is probably more effort than is warranted in a simple Calculator. Maybe if you were writing a Graphing Calculator you'd want to work around it...

- proton

but a graphing calculator is overkill for addition!

[chabig]

On my machine, Calculator (version 3.1) gives me 5012.8. Are you using Jaguar's calculator?

Chris

[010111]

Originally posted by chabig:
On my machine, Calculator (version 3.1) gives me 5012.8. Are you using Jaguar's calculator?

Chris

nope. the new panther calculator. i tried it on 2 different machines. both gave the .79999999999 thing.

[chabig]

OK. I reran the calculation on my machine and I got the same as you. I must have done something different the first time.

Anyway, under the View menu you can set the display precision. For some reason, Apple has set the default to 12, which is ridiculously high. For the type of math most people do, two to four digits to the right of the decimal point is sufficient. In your case, set it to 2 and you'll see the right answer: 5012.80.

Chris

[Cadaver]

Hint- to make any changes to Calculator (Panther's) you make, including decimal precision, window placement, etc., stick the next time you launch it, set things the way you like and hit cmd-w to close the window. Now, relaunch. Things will be as you left them. You can quit normally from this point on.

[KidRed]

Using PCalc which came with my pMac I got 5012.8. It's so much nicer then regular calculator.