Friday, July 24, 2009

when precision bites

I love math... programming it is even more fun, then mixing integer math with decimal (floating point) math.... even more fun:
In [90]: 26*.25 + 18*.75
Out[90]: 20.0

In [91]: 26/4 + 18*3/4
Out[91]: 19

In [92]: 18*3/4
Out[92]: 13

In [93]: 18.0*3/4
Out[93]: 13.5

In [94]: 26.0/4 + 18.0*3/4
Out[94]: 20.0
pesky '.0'

No comments: