Fix decode_font when size % 4 != 0 or data contains illegal bytes
When given a byte c, decode_chars expects that 0 <= c - 33 <= 63,
i. e. that only the six lowest bits of c - 33 are possibly set.
With this assumption, it shifts and adds together multiple c - 33 values.
When c > 96, c - 33 has high nonzero bits, which interferes with other
shifted terms. c < 33 is even worse: c - 33 is negative (if unsigned char
fits in int), and left-shifting negative numbers has undefined behavior.
Even before the shift, on common platforms with a two's complement
representation of negative integers (or if unsigned char does not fit in
int and is promoted to unsigned int), c - 33 has high nonzero bits, which
again interfere with other shifted terms.
To make matters worse, even perfectly valid encoded data is affected when
size % 4 != 0, as decode_font calls decode_chars with '\0', which leads
decode_chars to shift and add -33, causing undefined behavior and/or
incorrect output.
Take our cue from VSFilter and bit-mask c - 33 to keep only the six
relevant bits. To ensure that we get the same bits as VSFilter when
c < 33 and to avoid the undefined behavior of left-shifting negative
numbers, convert the number to unsigned before masking and shifting.
While we are at it, rewrite decode_chars entirely
to get rid of any GPL code from mkvtoolnix.
Related mkvtoolnix bug: https://github.com/mbunkus/mkvtoolnix/issues/1003
projects / libass / commit