From: James Zern <jzern@google.com>
Date: Tue, 6 Nov 2012 01:14:32 +0000 (-0800)
Subject: rd_pick_inter_mode: prevent signed integer overflow
X-Git-Tag: v1.3.0~1217^2~118^2~6
X-Git-Url: https://granicus.if.org/sourcecode?a=commitdiff_plain;h=e47d9f1d072e99c8c952e8ffa278ff7aef2ef1b7;p=libvpx

rd_pick_inter_mode: prevent signed integer overflow

calculate the txfm_cache difference first as both values may be
INT64_MAX with the intent that they cancel each other out.

Change-Id: I214d072458e1b24f60289974e6302af1aff7b66c
---

diff --git a/vp9/encoder/rdopt.c b/vp9/encoder/rdopt.c
index ffff248ac..d0f905b1d 100644
--- a/vp9/encoder/rdopt.c
+++ b/vp9/encoder/rdopt.c
@@ -4090,7 +4090,9 @@ static void rd_pick_inter_mode(VP9_COMP *cpi, MACROBLOCK *x,
         for (i = 0; i < NB_TXFM_MODES; i++) {
           int64_t adj_rd;
           if (this_mode != B_PRED) {
-            adj_rd = this_rd + txfm_cache[i] - txfm_cache[cm->txfm_mode];
+            const int64_t txfm_mode_diff =
+                txfm_cache[i] - txfm_cache[cm->txfm_mode];
+            adj_rd = this_rd + txfm_mode_diff;
           } else {
             adj_rd = this_rd;
           }