Support Board
Date/Time: Tue, 24 Jun 2025 00:46:08 +0000
Post From: Rounding error on spreadsheet?
| [2022-02-07 22:45:59] |
| Sawtooth - Posts: 4229 |
|
SC's documentation solution on this works in most situations, albeit not easily utilized. The problem is: J21 can also have floating point errors, especially with very small tick sizes like with ZB, ZN, 6J, etc. Here's how I deal with it: In H21, ROUND the ROUNDed tick size to its decimal length, then MROUND the prices to H21. H21: =ROUND(J21,LEN(TEXT(1/ROUND(1/J21,0)))-1) then: =MROUND(D3,$H$21)<MROUND(J$9,$H$21) Here is some useful info on the precision of operators: When comparing a FRACTIME to a TIMEVALUE, the >= operator ignores the = Read the comment at the bottom of post #4. |