vp9: Clamp tx_size in model_rd_large

For nonrd_pickmode: add clamp/check to make
sure tx_size is not set to lower than 8X8,
for the model_rd_large function (which is only
called for big block sizes).

No change in behavior.

Change-Id: I9c6093068e406ac16cfd6784ba75868906225378

diff --git a/vp9/encoder/vp9_pickmode.c b/vp9/encoder/vp9_pickmode.c
index 792a163fbca8cf12465b2161bded97d4dab4087d..c81cf1258320dcb4b328b6fb39de542bb34de4a1 100644 (file)
--- a/vp9/encoder/vp9_pickmode.c
+++ b/vp9/encoder/vp9_pickmode.c
@@ -416,6 +416,9 @@ static void model_rd_for_sb_y_large(VP9_COMP *cpi, BLOCK_SIZE bsize,
                      tx_mode_to_biggest_tx_size[cpi->common.tx_mode]);
   }
 
+  // The code below for setting skip flag assumes tranform size of at least 8x8,
+  // so force this lower limit on transform.
+  if (tx_size < TX_8X8) tx_size = TX_8X8;
   xd->mi[0]->tx_size = tx_size;
 
   if (cpi->oxcf.content == VP9E_CONTENT_SCREEN && x->zero_temp_sad_source &&