From: Ronald S. Bultje <rbultje@google.com>
Date: Fri, 3 May 2013 17:33:07 +0000 (-0700)
Subject: Fix overflow in RD error calculation code.
X-Git-Tag: v1.3.0~1106^2~61^2
X-Git-Url: https://granicus.if.org/sourcecode?a=commitdiff_plain;h=ee808e52bd0e371f0caad5ca7f29fbc4d5bf2aee;p=libvpx

Fix overflow in RD error calculation code.

Change-Id: I61ef1f198c876f9f79787ea7d7385a862cfbae19
---

diff --git a/vp9/encoder/vp9_rdopt.c b/vp9/encoder/vp9_rdopt.c
index f971d91c5..990be6527 100644
--- a/vp9/encoder/vp9_rdopt.c
+++ b/vp9/encoder/vp9_rdopt.c
@@ -603,7 +603,7 @@ static int block_error(int16_t *coeff, int16_t *dqcoeff,
 
   for (i = 0; i < block_size; i++) {
     int this_diff = coeff[i] - dqcoeff[i];
-    error += this_diff * this_diff;
+    error += (unsigned)this_diff * this_diff;
   }
   error >>= shift;