Attempt to detect miscompilation due to bug in gcc 4.2

I don't know if this bug still affects latest x264, but it can't hurt to try to detect it.
Accordingly refuse to open the encoder if detected.
Apparently VLC (on Windows) has been distributed for some time with a completely
broken x264 due to the use of a completely broken compiler (gcc 4.2).  In
particular, the MV costs seem to be calculated incorrectly on win32 when linking
from an application compiled without -ffast-math to an application with
-ffast-math.
I am not entirely certain why this occurs, but the result is, unsurprisingly,
encoding quality that makes MPEG-2 look good, due to the motion search being
completely broken.

diff --git a/encoder/encoder.c b/encoder/encoder.c
index e17b43f8bf7d9be2af18b556fca23aeb285e32fc..6bf59222c050031b5ba4cc9c0b36d4bf8438afad 100644 (file)
--- a/encoder/encoder.c
+++ b/encoder/encoder.c
@@ -873,6 +873,11 @@ x264_t *x264_encoder_open( x264_param_t *param )
             goto fail;
     if( x264_analyse_init_costs( h, X264_LOOKAHEAD_QP ) )
         goto fail;
+    if( h->cost_mv[1][2013] != 24 )
+    {
+        x264_log( h, X264_LOG_ERROR, "MV cost test failed: x264 has been miscompiled!\n" );
+        goto fail;
+    }
 
     h->out.i_nal = 0;
     h->out.i_bitstream = X264_MAX( 1000000, h->param.i_width * h->param.i_height * 4