From: Marco Paniconi <marpan@google.com>
Date: Tue, 31 Jul 2018 22:28:58 +0000 (-0700)
Subject: vp9: Clamp tx_size in model_rd_large
X-Git-Tag: v1.8.0~466^2
X-Git-Url: https://granicus.if.org/sourcecode?a=commitdiff_plain;h=cf7486676773e737d9538b0aaeaf59bf972b2a30;p=libvpx

vp9: Clamp tx_size in model_rd_large

For nonrd_pickmode: add clamp/check to make
sure tx_size is not set to lower than 8X8,
for the model_rd_large function (which is only
called for big block sizes).

No change in behavior.

Change-Id: I9c6093068e406ac16cfd6784ba75868906225378
---

diff --git a/vp9/encoder/vp9_pickmode.c b/vp9/encoder/vp9_pickmode.c
index 792a163fb..c81cf1258 100644
--- a/vp9/encoder/vp9_pickmode.c
+++ b/vp9/encoder/vp9_pickmode.c
@@ -416,6 +416,9 @@ static void model_rd_for_sb_y_large(VP9_COMP *cpi, BLOCK_SIZE bsize,
                      tx_mode_to_biggest_tx_size[cpi->common.tx_mode]);
   }
 
+  // The code below for setting skip flag assumes tranform size of at least 8x8,
+  // so force this lower limit on transform.
+  if (tx_size < TX_8X8) tx_size = TX_8X8;
   xd->mi[0]->tx_size = tx_size;
 
   if (cpi->oxcf.content == VP9E_CONTENT_SCREEN && x->zero_temp_sad_source &&