]> granicus.if.org Git - libvpx/commit
Fix buffer overrun of postproc_state.limits
authorMatthias Räncker <theonetruecamper@gmx.de>
Thu, 6 Sep 2018 16:29:16 +0000 (18:29 +0200)
committerJohann Koenig <johannkoenig@google.com>
Wed, 12 Sep 2018 16:57:46 +0000 (16:57 +0000)
commita07707125f84cab52dc6b4d3f09ed911712198f2
tree3017e54d45a5b4b38adfc5499096bd8294358f06
parentec8be105189bae3004933a91cad3a40f3d1a1eb9
Fix buffer overrun of postproc_state.limits

Always allocate cpi->common.postproc_state.limits using unscaled width.

With ./configure --enable-pic --enable-decode-perf-tests
--enable-encode-perf-tests --enable-encode-perf-tests
--enable-vp9-highbitdepth --enable-better-hw-compatibility
--enable-internal-stats --enable-postproc --enable-vp9-postproc
--enable-error-concealment --enable-coefficient-range-checking
--enable-postproc-visualizer --enable-multi-res-encodin
--enable-vp9-temporal-denoising --enable-webm-io --enable-libyuv
segfaults tend to occur in VP9/DatarateOnePassCbrSvcSingleBR.* tests.

This is an analogue to issue
https://bugs.chromium.org/p/webm/issues/detail?id=1374
where a buffer allocated using a scaled width is reused after scaling
back to the original size. Unfortunately, in this case the unscaled
width doesn't appear to be known in the immediated context of the
allocation, so the the signature of vp9_post_proc_frame needs to be
changed to provide that information in order to provide a similar fix
as in #1374.

Signed-off-by: Matthias Räncker <theonetruecamper@gmx.de>
Change-Id: I6f943aafbb3484ee94c5b38d7fcdd9d53fce3e5f
vp9/common/vp9_postproc.c
vp9/common/vp9_postproc.h
vp9/decoder/vp9_decoder.c
vp9/encoder/vp9_encoder.c