]> granicus.if.org Git - libvpx/commit
Fix variance (signed integer) overflow
authorJames Zern <jzern@google.com>
Wed, 7 Nov 2012 00:58:11 +0000 (16:58 -0800)
committerJames Zern <jzern@google.com>
Wed, 7 Nov 2012 07:06:44 +0000 (23:06 -0800)
commit984734436d8f13fc44ca3f0f3d65e0d057d6fd20
treec73a2ac7047b86d5752ae234e7b6ff76a6af4912
parenta879b4e6d425e7ec26e6ed9f0ef3e84575a891d4
Fix variance (signed integer) overflow

In the variance calculations the difference is summed and later squared.
When the sum exceeds sqrt(2^31) the value is treated as a negative when
it is shifted which gives incorrect results.

To fix this we force the multiplication to be unsigned.

The alternative fix is to shift sum down by 4 before multiplying.
However that will reduce precision.

For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and
change).

This change is based on:
1698234 Missed some variance casts
fea3556 Fix variance overflow

Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
test/test.mk
test/variance_test.cc [new file with mode: 0644]
vp9/common/findnearmv.c
vp9/encoder/variance_c.c
vp9/encoder/x86/variance_mmx.c
vp9/encoder/x86/variance_sse2.c
vp9/encoder/x86/variance_ssse3.c