Rounding numbers with a lot of decimals doesn't always work

I’m having trouble with a very simple script that works fine by itself but when I incorporate it into my ASOC app it screws up, sometimes. All I want to do is round some numbers with a lot of decimals. It behaves as expected until the last step, when I divide it by 10 (or 100, depending on what I multiplied it by) it throws a ton of decimals back into it. Why does 934 / 10 = 93.40000000000001?

roundIt(93.38094023943)

on roundIt(theNumber)
set theBigNumber to theNumber * 10
set theRound to round theBigNumber
set theRounded to theRound / 10
end roundIt

Results:

93.38094023943002
933.8094023943002
934
93.40000000000001

While this number works:

57.72347875864725
577.2347875864725
577
57.7

Hi,

that’s the way floating point numbers behave.

Here’s a good description: What Every Computer Scientist Should Know About Floating-Point Arithmetic

Ok, I read that head to tail and found it hilarious! :rolleyes:

I didn’t really read it, but I’m guessing it doesn’t really explain why it works when run from Applescript Editor but not from my app. I’m sure there’s an explanation, but it doesn’t really matter unless there’s a solution. So I guess I’m stuck coercing my numbers to text and lopping them off? The problem with that is sometimes it ends up something like 93.499999999999. I can lop off a whole lot of zeros but not all those nines.

I guess I can test what the second digit is after the decimal point and change it based on what comes after it. Pretty kludgy - it should fit right in with the rest of my app!

When you say you want to round the numbers, what are you using them for? If you’re actually after a rounded text display, you should use an NSNumberFormatter.

The number 0.1 for example doesn’t exists like the outcome of 1/3 in a decimal system doesn’t exists either, it’s infinite. What you see is called a rounding error, which means the “real” number cannot be stored in the floating point so it will store the most closest value. However floating points are not always the same size and therefore not equally precise.

When you try and copy 93.40000000000001 to SE you’ll see that the precision of this number is too large for SE and will use the digits it can store in the floating point (it doesn’t even round) which is 93.4000000000000 which will be presented as 93.4. So it’s quite obvious what happens here, your app is more precise than SE.

update:

This is a handler written by Nigel Garvey that can solve your problem:


quantiseIEEE(93.40001, 0.1)

on quantiseIEEE(n, q)
	if (n mod (q * 2)) ^ 2 > (q / 2) ^ 2 then return (n * 2 div q - n div q) * q
	n div q * q
end quantiseIEEE

I’m using a number formatter so my numbers display just how I want them to, but I need to export the data to a text file for use in a manufacturing application and that’s where the rounding errors show up.

I created a work-around by coercing to text, looking at the numbers in the string and changing them to what they should be and putting them back together, but it’s giving me occasional “The result of a numeric operation was too large” errors. One interesting thing I noticed is when I coerce a long number to text it actually rounds it - i.e.
15.25024589965682 becomes
15.250245899657 as text

I’ll try Nigel’s handler - looks like it will work.

Thanks for all the help!

So use a number formatter to produce the strings:

set theNums to paragraphs of "93.38094023943002
933.8094023943002
934
93.40000000000001"

set nf to current application's NSNumberFormatter's alloc()'s init()
nf's setMaximumFractionDigits:0
repeat with aNum in theNums
	log (nf's stringFromNumber:(aNum as real))
end repeat