From a09ba29a55b9a43d346421210d94370065eeaf53 Mon Sep 17 00:00:00 2001 From: DRC Date: Wed, 7 Sep 2016 16:40:10 -0500 Subject: [PATCH] Fix unsigned int overflow in libjpeg memory mgr. When attempting to decode a malformed JPEG image (refer to https://bugzilla.mozilla.org/show_bug.cgi?id=1295044) with dimensions 61472 x 32800, the maximum_space variable within the realize_virt_arrays() function will exceed the maximum value of a 32-bit integer and will wrap around. The memory manager subsequently fails with an "Insufficient memory" error (case 4, in alloc_large()), so this commit simply causes that error to be triggered earlier, before UBSan has a chance to complain. Note that this issue did not ever represent an exploitable security threat, because the POSIX-based memory manager that we use doesn't ever do anything meaningful with the value of maximum_space. jpeg_mem_available() simply sets avail_mem = maximum_space, so the subsequent behavior of the memory manager is the same regardless of whether maximum_space is correct or not. This commit simply removes a UBSan warning in order to make it easier to detect actual security issues. --- ChangeLog.md | 7 +++++++ jmemmgr.c | 17 +++++++++++++---- 2 files changed, 20 insertions(+), 4 deletions(-) diff --git a/ChangeLog.md b/ChangeLog.md index 0cef8e6..aba5776 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -62,6 +62,13 @@ have 2x1 luminance and 1x1 chrominance sampling factors, and 4:4:0 JPEGs have 1x2 luminance and 1x1 chrominance sampling factors), but the JPEG specification and the libjpeg API both allow it. +7. Fixed an unsigned integer overflow in the libjpeg memory manager, detected +by the Clang undefined behavior sanitizer, that could be triggered by +attempting to decompress a specially-crafted malformed JPEG image. This issue +affected only 32-bit code and did not pose a security threat, but removing the +warning makes it easier to detect actual security issues, should they arise in +the future. + 1.5.0 ===== diff --git a/jmemmgr.c b/jmemmgr.c index 9174ad3..adccbdf 100644 --- a/jmemmgr.c +++ b/jmemmgr.c @@ -32,6 +32,7 @@ #include "jinclude.h" #include "jpeglib.h" #include "jmemsys.h" /* import the system-dependent declarations */ +#include #ifndef NO_GETENV #ifndef HAVE_STDLIB_H /* should declare getenv() */ @@ -650,18 +651,26 @@ realize_virt_arrays (j_common_ptr cinfo) maximum_space = 0; for (sptr = mem->virt_sarray_list; sptr != NULL; sptr = sptr->next) { if (sptr->mem_buffer == NULL) { /* if not realized yet */ + size_t new_space = (long) sptr->rows_in_array * + (long) sptr->samplesperrow * sizeof(JSAMPLE); + space_per_minheight += (long) sptr->maxaccess * (long) sptr->samplesperrow * sizeof(JSAMPLE); - maximum_space += (long) sptr->rows_in_array * - (long) sptr->samplesperrow * sizeof(JSAMPLE); + if (SIZE_MAX - maximum_space < new_space) + out_of_memory(cinfo, 10); + maximum_space += new_space; } } for (bptr = mem->virt_barray_list; bptr != NULL; bptr = bptr->next) { if (bptr->mem_buffer == NULL) { /* if not realized yet */ + size_t new_space = (long) bptr->rows_in_array * + (long) bptr->blocksperrow * sizeof(JBLOCK); + space_per_minheight += (long) bptr->maxaccess * (long) bptr->blocksperrow * sizeof(JBLOCK); - maximum_space += (long) bptr->rows_in_array * - (long) bptr->blocksperrow * sizeof(JBLOCK); + if (SIZE_MAX - maximum_space < new_space) + out_of_memory(cinfo, 11); + maximum_space += new_space; } } -- 2.40.0