From: Yaowu Xu <yaowu@google.com>
Date: Wed, 9 Apr 2014 18:37:27 +0000 (-0700)
Subject: Fix the setting of mode_skip_mask
X-Git-Tag: v1.4.0~1857^2
X-Git-Url: https://granicus.if.org/sourcecode?a=commitdiff_plain;h=4f95a7fe3ef8e786dbb6c9c1dc3240da24538f92;p=libvpx

Fix the setting of mode_skip_mask

In mode selection loop, once mode_index pass mode_skip_start, all
modes with a different reference frame from current best mode are
masked out using mode_skip_mask.

However, the setting of mode_skip_mask may use an invalid mode if
there is no mode tested yet. This commit fixes the issue by making
sure a mode has been tested and selected. Otherwise, no mode will be
masked out because of their reference frame.

Change-Id: Ib0009e8a96836a65cf5347440fff8a2e1a67f29f
---

diff --git a/vp9/encoder/vp9_rdopt.c b/vp9/encoder/vp9_rdopt.c
index a3e513277..1066f6a17 100644
--- a/vp9/encoder/vp9_rdopt.c
+++ b/vp9/encoder/vp9_rdopt.c
@@ -3304,7 +3304,7 @@ int64_t vp9_rd_pick_inter_mode_sb(VP9_COMP *cpi, MACROBLOCK *x,
 
     // Look at the reference frame of the best mode so far and set the
     // skip mask to look at a subset of the remaining modes.
-    if (mode_index == mode_skip_start) {
+    if (mode_index == mode_skip_start && best_rd < INT64_MAX) {
       switch (vp9_mode_order[best_mode_index].ref_frame[0]) {
         case INTRA_FRAME:
           break;