fix: unify and correct RectArea implementations
Two implementations of RectArea were provided that implemented the overflow
check based on whether there was a larger type than unsigned int available. It
is unnecessary to maintain multiple implementations of this as the overflow
check can be written in a width-independent way.
Moreover the `LLONG_MAX > UINT_MAX` alternative was incorrect. E.g. on x86-64
where this alternative is used, if `r->boundary[i + NUMDIMS] - r->boundary[i]`
is `UINT_MAX` and the accumulated value in `a_test` is `(long long)UINT_MAX`,
the multiplication (which is done on operands promoted to long long) overflows
causing undefined behavior. In practice, you would likely get a negative value,
that then erroneously passes the overflow check.
Related to #1898. Fixes #1906.
projects / graphviz / commit