Turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- JMP User Community
- :
- Discussions
- :
- Re: strangeness of floating point math

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

Highlighted
##

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

strangeness of floating point math

Jun 3, 2020 2:49 PM
(440 views)

I am using JMP15.

I recently realized that a script I've used for some time had a small error in it that stems from a floating point quirk which is illustrated below.

The first two lines calculate the same number -> 23.

When I use the ceiling function, the first version returns 24 and the second returns 23.

Should this be happening?

show(((1104/2016)*42));

show(((42*1104/2016)));

show(ceiling((1104/2016)*42));

show(ceiling((42*1104/2016)));

/*:

(1104 / 2016) * 42 = 23;

42 * 1104 / 2016 = 23;

Ceiling((1104 / 2016) * 42) = 24;

Ceiling(42 * 1104 / 2016) = 23;

3 REPLIES 3

Highlighted
##

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Re: strangeness of floating point math

You have already seen that this is a floating point quirk. Should it be happening, technically no, but there will always be strange "rounding issues" due to floating point calculations.

If you put the first calculation into a column (1104/2016)*42, change the width of the column to be 30 (arbitrarily large), format to Fixed Decimal with 20 decimal places you will see that the number is stored as: 23.00000000000000355271.

Dan Obermiller

Highlighted
##

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Re: strangeness of floating point math

2016: 2×2×2×2×2×3×3×7

1104: 2×2×2×2×3×23

42: 2×3×7

1104/2016 is (~~2×2×2×2×3×~~23) / (~~2×2×2×2×~~2×~~3×~~3×7)

23/(2×3×7) is a repeating fraction in base 10 AND base 2 (hex, base 16, shown with 4 bits per hex digit):

0.5476190476190476190476190... or 3FE186186186186... (lucky 12-bit repeat for visualization!)

Both of those are truncated, and imprecise. The actual value for the hex base 2 representation is rounded to 3FE1861861861862, which is slightly high (3FE is the exponent). Multiply by 42, after the divide, is also slightly high.

(42×1104)/2016 is ((~~2×3×7~~)×(~~2×2×2×2×3×~~23)) / (~~2×2×2×2×2×3×3×7~~)

Multiply by 42, ahead of the divide, gets nice reduction of the prime divisors.

Craige

Highlighted
##

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Re: strangeness of floating point math

I think I'm going to frame that answer

JMP Systems Engineer, Pharm and BioPharm Sciences