From: paulwilkins <paulwilkins@google.com>
Date: Thu, 30 Jun 2016 12:31:39 +0000 (+0100)
Subject: Fix error in get_smooth_intra_threshold() for 10/12 bit.
X-Git-Tag: v1.6.1~465^2
X-Git-Url: https://granicus.if.org/sourcecode?a=commitdiff_plain;h=f9a3d08f1b4c1a41d87d9d74b640d68e9dacda8d;p=libvpx

Fix error in get_smooth_intra_threshold() for 10/12 bit.

This function seems to scale the threshold for testing an
SSE value in the wrong direction for 10 and 12 bit inputs.

Also for a true SSE the scalings should probably be << 4 and 8

Change-Id: Iba8047b3f70d04aa46d9688a824f3d49c1c58e90
---

diff --git a/vp9/encoder/vp9_firstpass.c b/vp9/encoder/vp9_firstpass.c
index 53a3ec7de..11b397cd6 100644
--- a/vp9/encoder/vp9_firstpass.c
+++ b/vp9/encoder/vp9_firstpass.c
@@ -532,10 +532,10 @@ static int get_smooth_intra_threshold(VP9_COMMON *cm) {
         ret_val = SMOOTH_INTRA_THRESH;
         break;
       case VPX_BITS_10:
-        ret_val = SMOOTH_INTRA_THRESH >> 2;
+        ret_val = SMOOTH_INTRA_THRESH << 4;
         break;
       case VPX_BITS_12:
-        ret_val = SMOOTH_INTRA_THRESH >> 4;
+        ret_val = SMOOTH_INTRA_THRESH << 8;
         break;
       default:
         assert(0 && "cm->bit_depth should be VPX_BITS_8, "