Thursday, March 22, 2018

You are not logged in.

## #1 2016-07-27 10:51:48 am

stegor
Member
Registered: 2006-03-07
Posts: 45

### Rounding numbers with a lot of decimals doesn't always work

I'm having trouble with a very simple script that works fine by itself but when I incorporate it into my ASOC app it screws up, sometimes. All I want to do is round some numbers with a lot of decimals. It behaves as expected until the last step, when I divide it by 10 (or 100, depending on what I multiplied it by) it throws a ton of decimals back into it. Why does 934 / 10 = 93.40000000000001?

roundIt(93.38094023943)

on roundIt(theNumber)
set theBigNumber to theNumber * 10
set theRound to round theBigNumber
set theRounded to theRound / 10
end roundIt

Results:

93.38094023943002
933.8094023943002
934
93.40000000000001

While this number works:

57.72347875864725
577.2347875864725
577
57.7

Offline

## #2 2016-07-27 11:04:20 am

StefanK
Member
From:: St. Gallen, Switzerland
Registered: 2006-10-21
Posts: 11492
Website

### Re: Rounding numbers with a lot of decimals doesn't always work

Hi,

that's the way floating point numbers behave.

Here's a good description: What Every Computer Scientist Should Know About Floating-Point Arithmetic

regards

Stefan

Offline

## #3 2016-07-27 11:29:27 am

stegor
Member
Registered: 2006-03-07
Posts: 45

### Re: Rounding numbers with a lot of decimals doesn't always work

I didn't really read it, but I'm guessing it doesn't really explain why it works when run from Applescript Editor but not from my app. I'm sure there's an explanation, but it doesn't really matter unless there's a solution. So I guess I'm stuck coercing my numbers to text and lopping them off? The problem with that is sometimes it ends up something like 93.499999999999. I can lop off a whole lot of zeros but not all those nines.

I guess I can test what the second digit is after the decimal point and change it based on what comes after it. Pretty kludgy - it should fit right in with the rest of my app!

Offline

## #4 2016-07-27 06:17:12 pm

Shane Stanley
Member
From:: Australia
Registered: 2002-12-07
Posts: 5256

### Re: Rounding numbers with a lot of decimals doesn't always work

When you say you want to round the numbers, what are you using them for? If you're actually after a rounded text display, you should use an NSNumberFormatter.

www.macosxautomation.com/applescript/apps/
latenightsw.com

Offline

## #5 2016-07-28 02:20:08 am

DJ Bazzie Wazzie
Member
From:: the Netherlands
Registered: 2004-10-20
Posts: 2740
Website

### Re: Rounding numbers with a lot of decimals doesn't always work

stegor wrote:

I didn't really read it, but I'm guessing it doesn't really explain why it works when run from Applescript Editor but not from my app. I'm sure there's an explanation, but it doesn't really matter unless there's a solution.

The number 0.1 for example doesn't exists like the outcome of 1/3 in a decimal system doesn't exists either, it's infinite. What you see is called a rounding error, which means the "real" number cannot be stored in the floating point so it will store the most closest value. However floating points are not always the same size and therefore not equally precise.

When you try and copy 93.40000000000001 to SE you'll see that the precision of this number is too large for SE and will use the digits it can store in the floating point (it doesn't even round) which is 93.4000000000000 which will be presented as 93.4. So it's quite obvious what happens here, your app is more precise than SE.

update:

This is a handler written by Nigel Garvey that can solve your problem:

#### Applescript:

quantiseIEEE(93.40001, 0.1)

on quantiseIEEE(n, q)
if (n mod (q * 2)) ^ 2 > (q / 2) ^ 2 then return (n * 2 div q - n div q) * q
n div q * q
end quantiseIEEE

Last edited by DJ Bazzie Wazzie (2016-07-28 02:37:41 am)

Offline

## #6 2016-07-28 07:40:58 am

stegor
Member
Registered: 2006-03-07
Posts: 45

### Re: Rounding numbers with a lot of decimals doesn't always work

Shane Stanley wrote:

When you say you want to round the numbers, what are you using them for? If you're actually after a rounded text display, you should use an NSNumberFormatter.

I'm using a number formatter so my numbers display just how I want them to, but I need to export the data to a text file for use in a manufacturing application and that's where the rounding errors show up.

I created a work-around by coercing to text, looking at the numbers in the string and changing them to what they should be and putting them back together, but it's giving me occasional "The result of a numeric operation was too large" errors. One interesting thing I noticed is when I coerce a long number to text it actually rounds it - i.e.
15.25024589965682 becomes
15.250245899657 as text

I'll try Nigel's handler - looks like it will work.

Thanks for all the help!

Offline

## #7 2016-07-28 05:42:29 pm

Shane Stanley
Member
From:: Australia
Registered: 2002-12-07
Posts: 5256

### Re: Rounding numbers with a lot of decimals doesn't always work

stegor wrote:

I need to export the data to a text file

So use a number formatter to produce the strings:

#### Applescript:

set theNums to paragraphs of "93.38094023943002
933.8094023943002
934
93.40000000000001"

set nf to current application's NSNumberFormatter's alloc()'s init()
nf's setMaximumFractionDigits:0
repeat with aNum in theNums
log (nf's stringFromNumber:(aNum as real))
end repeat