]> granicus.if.org Git - libvpx/commit
pickmode: Fix a pair of unsigned overflows.
authorAlex Converse <aconverse@google.com>
Mon, 9 May 2016 18:21:20 +0000 (11:21 -0700)
committerAlex Converse <aconverse@google.com>
Mon, 9 May 2016 18:21:20 +0000 (11:21 -0700)
commitd8a18186ba4482ff121aa526fbf6df31f9d90d35
tree381df65d582059823e0f10f4719d69091a36c3b3
parentbd2e7fa1c8b9923c741d338f33b63c1a92649af0
pickmode: Fix a pair of unsigned overflows.

block_variance: This operates on 8x8s and would be safe with a int32 *
int32 to uint32 multiply, but this is potentially unsafe for 12-bit
input. Unfortunately the code already segfaults on 12-bit input:
https://bugs.chromium.org/p/webm/issues/detail?id=1223

calculate_variance: This operates on up to a 32x32 of 8x8s and can
overflow even with 8-bit input (log2((256*32*32)**2) == 36).

BUG=https://bugs.chromium.org/p/webm/issues/detail?id=1220

Change-Id: I1ca4ff6092db9a7580da371ee9a21f403fdadc40
vp9/encoder/vp9_pickmode.c