fix ortho: add epsilon to floating point comparison

fix ortho: add epsilon to floating point comparison

Thanks to good bug description #1408, I've compared logs of
bad and good version. I found that sometimes there is
difference ~1E-15 between t->hi.y and t->lo.y because of
precision issues. It caused false positive conditional statement.

The workaround is to tolerate insignificant deviations.

fixes #1408, fixes #1990, fixes #1658