From: Jingning Han <jingning@google.com>
Date: Sun, 5 Oct 2014 19:05:14 +0000 (-0700)
Subject: Fix an IOC issue in vp9_rd_pick_inter_mode_sb
X-Git-Tag: v1.4.0~674^2
X-Git-Url: https://granicus.if.org/sourcecode?a=commitdiff_plain;h=085b97aa5c17afbc07c8a342ba08a0bbc5b1b523;p=libvpx

Fix an IOC issue in vp9_rd_pick_inter_mode_sb

It is possible that the GOLDEN reference frame is not avaiable, in
which setting the predicted mv will be associated with a residual
value of INT_MAX. This commit checks this condition before
left shift and comparison with that of ALTREF frame, to avoid
overflow issue.

Change-Id: Ib98c3149dbdd016f2fe5beaafb13f67d469dd07c
---

diff --git a/vp9/encoder/vp9_rdopt.c b/vp9/encoder/vp9_rdopt.c
index fd67ab5bd..43300ae44 100644
--- a/vp9/encoder/vp9_rdopt.c
+++ b/vp9/encoder/vp9_rdopt.c
@@ -2675,7 +2675,7 @@ int64_t vp9_rd_pick_inter_mode_sb(VP9_COMP *cpi, MACROBLOCK *x,
   }
 
   if (cpi->sf.alt_ref_search_fp)
-    if (!cm->show_frame)
+    if (!cm->show_frame && x->pred_mv_sad[GOLDEN_FRAME] < INT_MAX)
       if (x->pred_mv_sad[ALTREF_FRAME] > (x->pred_mv_sad[GOLDEN_FRAME] << 1))
         mode_skip_mask[ALTREF_FRAME] |= INTER_ALL;