DRC [Fri, 15 Dec 2017 22:45:25 +0000 (16:45 -0600)]
BUILDING.md: Adjust Android recipes for NDK r16+
NDK r16b moved some things around, so modify the Android build recipes
to take that into account while preserving compatibility with previous
NDK releases.
NOTE: the GCC 4.9 NDK toolchain is deprecated, so we will need to
develop new Android build recipes for libjpeg-turbo 1.6 that use the
Clang toolchain.
DRC [Tue, 5 Dec 2017 21:27:34 +0000 (15:27 -0600)]
Fix lib state when skipping to end of 1-scan image
If jpeg_skip_scanlines() is used to skip to the end of a single-scan
image, then we need to change the library state such that subsequent
calls to jpeg_consume_input() will return JPEG_REACHED_EOI rather than
JPEG_SUSPENDED. (NOTE: not necessary for multi-scan images, since the
scans are processed prior to any call to jpeg_skip_scanlines().)
Unless I miss my guess, using jpeg_skip_scanlines() in this manner
will prevent any markers at the end of the JPEG image from being
read, but I don't think there is any way around that without actually
reading the data, which would defeat the purpose of
jpeg_skip_scanlines().
DRC [Wed, 29 Nov 2017 20:23:31 +0000 (14:23 -0600)]
Travis: Work around xcode7.3 image bug
Refer to travis-ci/travis-ci#8552. This was supposed to be fixed on
November 15, then on November 28. Travis blew through both deadlines,
so I have no confidence that the issue will be fixed as promised in a
timely manner. Adding 'brew update' to .travis.yml slows the OS X
build, but there is no choice at the moment.
DRC [Thu, 16 Nov 2017 01:39:45 +0000 (19:39 -0600)]
Uniquify tjbenchtest log file names based on args
+ clean up log files when 'make testclean' is invoked
+ fix 'tjbenchtest -yuv -alloc'
+ fix tjexampletest so that it creates images under /tmp
+ clean up tjexampletest
DRC [Tue, 14 Nov 2017 21:30:06 +0000 (15:30 -0600)]
tjbench.exe: Fix decompression access violation
The program crashed when a JPEG image was passed on the command line,
because we were mixing our metaphors vis-a-vis malloc()/free() and
tjAlloc()/tjFree() (malloc()/free() uses the tjbench.exe heap,
whereas tjAlloc()/tjFree() uses the turbojpeg.dll heap.)
DRC [Tue, 14 Nov 2017 03:01:53 +0000 (21:01 -0600)]
Further partial image decompression fixes
- Referring to 073b0e88a192adebbb479ee2456beb089d8b5de7 and #185, the
reason why BMP and RLE didn't (and won't) work with partial image
decompression is that the output engines for both formats maintain a
whole-image buffer, which is used to reverse the order of scanlines.
However, it was straightforward to add -crop support for GIF and
Targa, which is useful for testing partial image decompression along
with color quantization.
- Such testing reproduced a bug reported by Mozilla (refer to PR #182)
whereby jpeg_skip_scanlines() would segfault if color quantization was
enabled. To fix this issue, read_and_discard_scanlines() now sets up
a dummy quantize function in the same manner that it sets up a dummy
color conversion function.
DRC [Thu, 9 Nov 2017 03:01:57 +0000 (21:01 -0600)]
djpeg -crop: Exit gracefully with non-PPM formats
... and document that only PPM/PGM output images are supported with the
-crop option for the moment.
I investigated the possibility of supporting -crop with -bmp, but even
after resetting the buffer dimensions, I still kept getting virtual
array access errors. It seems that doing this the "right way" would
require creating a re-initialization function for each image format's
destination manager. I'm disinclined to do that right now, given that
this feature was Google's baby (developed as a prerequisite for
including libjpeg-turbo in Android), and the -crop option in djpeg is
intended only as an example of how to use the partial image
decompression API. Real-world applications would need to handle this
in their own destination managers.
It would probably be possible to make this work with Targa by employing
a similar hack to the one we used with PPM, but Targa isn't popular
enough to bother.
The version of RPM on RHEL 5 and older platforms defines _libdir
as %{_exec_prefix}/%{_lib}, so defining _lib in the spec file redefined
_libdir. However, newer versions of RPM (probably >= 4.6, since that
was the version that introduced the ISA macros) define _libdir as either
%{_prefix}/lib or %{_prefix}/lib64. Thus, we need to explicitly
override _libdir in our spec file.
It is necessary for the C code to be aware of the machine's endianness,
which is why the TurboJPEG Java wrapper sets a different pixel format
for integer BufferedImages depending on ByteOrder.nativeOrder().
However, it isn't necessary to handle endianness in pure Java code such
as TJUnitTest (d'oh!) This was a product of porting the C version of
TJUnitTest too literally, and of insufficient testing (historically,
the big endian systems I had available for testing didn't have Java.)
DRC [Mon, 14 Aug 2017 15:54:27 +0000 (10:54 -0500)]
Java: Fix NullPointerException in YUVImage
planes == null is a valid argument to setBuf() if alloc == true, so we
need to make sure that planes is non-null before validating its length.
We also need to allocate one dimension of the planes array if it's null.
DRC [Wed, 28 Jun 2017 19:47:45 +0000 (14:47 -0500)]
TurboJPEG: Document xform issue w/ big marker data
If the source image for a transform operation has a lot of EXIF or ICC
data embedded in it, then it may cause the output image size to exceed
the worst-case size returned by tjBufSize() (because tjTransform()
transfers all markers to the output image.) This is only a problem if
TJFLAG_NOREALLOC is passed to the function. Since the TurboJPEG C API
doesn't require the destination image size to be set in this case, it
makes the documented assumption that the calling program has allocated
the destination buffer to exactly the size returned by tjBufSize().
Changing this assumption would change the API behavior and necessitate
a new function name (tjTransform2().) At the moment, it's easier to
just document this as a known issue, since it's easy to work around in
the C API.
The Java API is unfortunately a different story, since it must always
use TJFLAG_NOREALLOC (because, when using the TurboJPEG Java API, all
buffers are allocated on the Java heap, and thus they can't be
reallocated by the C code.) There is no easy way to work around this
without changing the C API as discussed above, because if the source
image contains large amounts of marker data, it's virtually impossible
to determine how big the output image will be.
DRC [Tue, 27 Jun 2017 19:22:39 +0000 (14:22 -0500)]
TJBench: Improve consistency of results
Given that libjpeg-turbo can often process hundreds of megapixels/second
on modern hardware, the default of one warmup iteration was essentially
meaningless. Furthermore, the -warmup option was a bit clunky, since
it required some foreknowledge of how fast the benchmarks were going to
execute.
This commit introduces a 1-second warmup interval for each benchmark by
default, and the -warmup option has been retasked to control the length
of that interval.
DRC [Tue, 27 Jun 2017 01:48:02 +0000 (20:48 -0500)]
TJBench: Fix errors when decomp. files w/ ICC data
Embedded ICC profiles can cause the size of a JPEG file to exceed the
size returned by tjBufSize() (which is really meant to be used for
compression anyhow, not for decompression), and this was causing a
segfault (C) or an ArrayIndexOutOfBoundsException (Java) when
decompressing such files with TJBench. This commit modifies the
benchmark such that, when tiled decompression is disabled, it re-uses
the source buffer as the primary JPEG buffer.
DRC [Mon, 8 May 2017 13:15:11 +0000 (08:15 -0500)]
Travis: Fix OS X build
The Travis xcode7.3 image now apparently includes GnuPG 1.4.x by
default, so use it instead of installing GnuPG 2. Using GnuPG 2.1.x,
the default version in Homebrew as of this writing, is problematic for
this reason:
https://wiki.archlinux.org/index.php/GnuPG#Unattended_passphrase
This code was submitted in the initial ARM NEON patches
(https://sourceforge.net/p/libjpeg-turbo/patches/7/) by Siarhei while he
was still a Nokia employee.
DRC [Sat, 18 Mar 2017 21:15:14 +0000 (16:15 -0500)]
Honor max_memory_to_use/JPEGMEM/-maxmemory
This re-introduces a feature of the obsolete system-specific libjpeg
memory managers-- namely the ability to limit the amount of main memory
used by the library during decompression or multi-pass compression.
This is mainly beneficial for two reasons:
- Works around a 2 GB limit in libFuzzer
- Allows security-sensitive applications to set a memory limit for the
JPEG decoder so as to work around the progressive JPEG exploit
(LJT-01-004) described here:
http://www.libjpeg-turbo.org/pmwiki/uploads/About/TwoIssueswiththeJPEGStandard.pdf
This commit also removes obsolete documentation regarding the MS-DOS
memory manager (which itself was removed long ago) and changes the
documentation of the -maxmemory switch and JPEGMEM environment variable
to reflect the fact that backing stores are never used in libjpeg-turbo.
DRC [Sat, 18 Mar 2017 18:24:50 +0000 (13:24 -0500)]
AppVeyor: Fix CI build
Something changed in the CI build environment, and our previous trick of
setting the Git URL to file://c:/projects/libjpeg-turbo no longer works.
Using cygpath to translate the Windows path to a MinGW-friendly format
is a better solution anyhow.
DRC [Sat, 18 Mar 2017 17:56:36 +0000 (12:56 -0500)]
TurboJPEG: Fix potential memory leaks
Referring to https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=746,
it seems that the values of local buffer pointers in TurboJPEG API
functions aren't always preserved if longjmp() returns control to a
point prior to the allocation of the local buffers. This is known to
be an issue with GCC 4.x and clang with -O1 and higher optimization
levels but not with GCC 5.x and later. It is unknown why GCC 5.x and
6.x do not suffer from the issue, but possibly the local buffer pointers
are not allocated on the stack when using those more recent compilers.
In any case, this commit modifies the TurboJPEG API library code such
that the jump buffer is always updated after any local buffer pointers
are allocated but before any subsequent libjpeg API functions are
called.
DRC [Fri, 20 Jan 2017 00:51:41 +0000 (18:51 -0600)]
Always tweak EXIF w/h tags w/ lossless transforms
... even if using libjpeg v6b emulation. Previously
adjust_exif_parameters() was only called with libjpeg v7/v8 emulation,
but due to a bug (which this commit also fixes), it only worked properly
with libjpeg v8 emulation.
DRC [Thu, 19 Jan 2017 23:50:59 +0000 (17:50 -0600)]
Fix error w/ lossless crop & libjpeg v7 emulation
The JPEG_LIB_VERSION #ifdef in jtransform_adjust_parameters() was
incorrect, which caused a "Bogus virtual array access" error when
attempting to use the lossless crop feature.
DRC [Sat, 10 Dec 2016 15:32:23 +0000 (09:32 -0600)]
LICENSE.md: Include text of BSD/zlib licenses
LICENSE.md is included in the binary distributions as well, so it
doesn't make much sense to refer to license headers in source files that
aren't necessarily going to be there.
DRC [Mon, 5 Dec 2016 20:02:59 +0000 (14:02 -0600)]
Fix build when CFLAGS contains -std=c89 (or -ansi)
This is a subtle point, but AC_C_INLINE defines "inline" to be either
"inline", "__inline__", or "__inline". The subsequent test for
"inline __attribute__((always_inline))" uses this definition. The
attribute is irrespective of the inline keyword, so whereas
"__inline__ __attribute__((always_inline))" works under C89,
"inline __attribute__((always_inline))" doesn't, and defining INLINE to
the latter causes the build to fail. The easiest way around this is
simply to define "inline" ahead of "INLINE" in jconfigint.h,
which causes the inline keyword detected by AC_C_INLINE to modify the
INLINE macro if necessary.
DRC [Mon, 5 Dec 2016 18:39:49 +0000 (12:39 -0600)]
Reorg AltiVec detection code
+ advertise that full AltiVec SIMD acceleration is now available on
OpenBSD.
The relevant compilers probably all support C99 or GNU's variation of
C90 that allows variables to be declared anywhere, but our policy is to
conform to the C90 standard, if for no other reason than that it
improves code readability.
Colin Cross [Fri, 2 Dec 2016 00:56:18 +0000 (16:56 -0800)]
Fix sign mismatch comparison warnings
Fixes:
rdppm.c:257:14: warning: comparison of integers of different signs: 'int' and 'unsigned int' [-Wsign-compare]
if (temp > maxval)
~~~~ ^ ~~~~~~
rdppm.c:284:14: warning: comparison of integers of different signs: 'int' and 'unsigned int' [-Wsign-compare]
if (temp > maxval)
~~~~ ^ ~~~~~~
rdppm.c:289:14: warning: comparison of integers of different signs: 'int' and 'unsigned int' [-Wsign-compare]
if (temp > maxval)
~~~~ ^ ~~~~~~
rdppm.c:294:14: warning: comparison of integers of different signs: 'int' and 'unsigned int' [-Wsign-compare]
if (temp > maxval)
DRC [Thu, 1 Dec 2016 07:58:34 +0000 (01:58 -0600)]
Travis: Use xcode7.3 image
The xcode7.2 image is verfallen, verlumpt, verblunget, verkackt
This also ensures that the build scripts are checked out from a
branch matching the libjpeg-turbo repository branch (not strictly
necessary when building from master, but it keeps the code in sync with
dev.)
DRC [Tue, 22 Nov 2016 04:58:18 +0000 (22:58 -0600)]
AppVeyor: Use built-in MSYS2 MinGW compilers
AppVeyor already has MinGW32 and MinGW64 flavors of GCC 5.3.0
installed under MSYS2, so there is no need to install our own builds of
MinGW. MinGW-builds is no longer an active project, and we were getting
occasional timeouts while wgetting those files from SourceForge.
Furthermore, GCC 5.3.0 should produce faster code than GCC 4.8.1.
DRC [Mon, 21 Nov 2016 02:59:55 +0000 (20:59 -0600)]
BUILDING.md: Clarifications and wordsmithing
Updated out-of-date information, wordsmithed and clarified many
sections, and generally cleaned up the build recipes (including a
complete overhaul of the iOS recipes.)
DRC [Fri, 18 Nov 2016 15:09:41 +0000 (09:09 -0600)]
BUILDING.md/README.md: Increment libjpeg SO age
Documentation buglet. This should have been changed in 6ed4d9d11085acd04dc7f2f899848693976dc010 to reflect the addition of
libjpeg API functions in libjpeg-turbo 1.5.
This commit also makes the "testclean" target clean up the 4:1:1 test
images. This was implemented in the autotools build system in 1f3635c4969f2319a01c9fe561958815b733227f but was left out of the CMake
build system due to an oversight.
DRC [Thu, 20 Oct 2016 22:55:55 +0000 (17:55 -0500)]
Travis: Deploy to S3 rather than SourceForge
This has the following advantages:
-- It doesn't require checking a private SSH key into the repository.
(With SourceForge, an SSH key is the "keys to the kingdom".)
-- If the S3 key is compromised, it is very easy to revoke it and
generate a new one.
-- The S3 bucket is isolated, so even if it becomes compromised, then
the damage that one could do is limited.
-- It's much easier to manage files through S3's web interface than
through SourceForge.
-- The files are served via HTTPS.
-- Travis fully supports S3 as a deployment target, so this simplifies
.travis.yml somewhat.
DRC [Thu, 20 Oct 2016 06:01:27 +0000 (01:01 -0500)]
Travis: GPG sign Linux binaries/source tarballs
Since we're still deploying our Linux/macOS CI artifacts to a web server
(specifically SourceForge Project Web Services) that doesn't support
HTTPS, it's a good idea to sign them. But since the private key has to
be checked into the repository, we use a different key for signing the
pre-releases (per project policy, the private signing keys for our
release binaries are never made available on any public server.)
DRC [Tue, 11 Oct 2016 16:58:20 +0000 (11:58 -0500)]
Win: Use YASM if it is in the PATH and NASM isn't
Previously, simd/CMakeLists.txt was hard-coded to use NASM, and it was
necessary to override the NASM variable in order to use YASM. This
commit changes the behavior such that NASM is still preferred, but YASM
will be used if it is in the PATH and NASM isn't available. This brings
the actual behavior in line with the behavior described in BUILDING.md.
DRC [Wed, 5 Oct 2016 19:36:46 +0000 (14:36 -0500)]
Travis: use correct repo/branch for off. builds
Pass the actual repository and branch that Travis is using into the
builtljt script, so the official builds it generates will come from
the same code base as the other tested builds.
DRC [Tue, 4 Oct 2016 18:41:48 +0000 (13:41 -0500)]
Fix 32-bit non-SIMD FP regression tests
- Introduce a new FLOATTEST value ("387") on Un*x systems that will
compare the floating point DCT/IDCT algorithms against the expected
results from the C algorithms when built using 32-bit code and
-mfpmath=387.
- Extend the Windows regression tests so that they work properly when
building libjpeg-turbo with 32-bit code and without SIMD, using either
Visual C++ (tested with 2008, 2010, 2015) or MinGW.
In the AArch64 ABI, as in many others, it's forbidden to read/store data
below the stack pointer. Some SIMD functions were doing just that
(stack pointer misuse) when trying to preserve callee-saved registers,
and this resulted in those registers being restored with incorrect
contents under certain circumstances.
This patch fixes that behavior, and callee-saved registers are now
stored above the stack pointer throughout the function call. The patch
also removes register saving in places where it is unnecessary for this
ABI, or it makes use of unused scratch regiters instead of callee-saved
registers.
Fixes #97. Closes #101.
Refer also to https://bugzilla.redhat.com/show_bug.cgi?id=1368569
The last iDevice to require ARMv6 was the iPhone 3G, which required iOS
4.2.1 or older. Our binaries have always required iOS 4.3 or newer,
so I'm not sure if the ARMv6 fork of our binaries was ever useful to
begin with. In any case, if it ever was useful, it no longer is. Fat
binaries can still be generated with ARMv6 support by invoking
{build_directory}/pkgscripts/makemacpkg manually.
Fix out-of-bounds write in partial decomp. feature
Reported by Clang UBSan (refer to
https://bugzilla.mozilla.org/show_bug.cgi?id=1301252 for test image.)
This appears to be a legitimate bug introduced by 3ab68cf563f6edc2608c085f5c8b2d5d5c61157e. Any component array, such
as first_MCU_col and last_MCU_col, should always be able to accommodate
MAX_COMPONENTS values. The aforementioned test image had 8 components,
which was not enough to make the out-of-bounds write bust out of the
jpeg_decomp_master struct (and fortunately the memory after last_MCU_col
is an integer used as a boolean, so stomping on it will do nothing other
than change the decoder state.) I crafted another special image that
has 10 components (the maximum allowable), but that was apparently not
enough to bust out of the allocated memory, either. Thus, it is
posited that the security threat posed by this bug is either extremely
minimal or non-existent.
NOTE: The jdhuff.c/jdphuff.c warnings should have already been silenced
by 8e9cef2e6f5156c4b055a04a8f979b7291fc6b7a, but apparently I need to
be REALLY clear that I'm trying to do pointer arithmetic rather than
dereference an array. Grrr...
Refer to:
https://bugzilla.mozilla.org/show_bug.cgi?id=1301250
https://bugzilla.mozilla.org/show_bug.cgi?id=1301256
When attempting to decode a malformed JPEG image (refer to
https://bugzilla.mozilla.org/show_bug.cgi?id=1295044) with dimensions
61472 x 32800, the maximum_space variable within the
realize_virt_arrays() function will exceed the maximum value of a 32-bit
integer and will wrap around. The memory manager subsequently fails
with an "Insufficient memory" error (case 4, in alloc_large()), so this
commit simply causes that error to be triggered earlier, before UBSan
has a chance to complain.
Note that this issue did not ever represent an exploitable security
threat, because the POSIX-based memory manager that we use doesn't ever
do anything meaningful with the value of maximum_space.
jpeg_mem_available() simply sets avail_mem = maximum_space, so the
subsequent behavior of the memory manager is the same regardless of
whether maximum_space is correct or not. This commit simply removes a
UBSan warning in order to make it easier to detect actual security
issues.
Normally, 4:2:2 JPEGs have horizontal x vertical luminance,chrominance
sampling factors of 2x1,1x1, and 4:4:0 JPEGs have horizontal x vertical
luminance,chrominance sampling factors of 1x2,1x1. However, it is
technically legal to create 4:2:2 JPEGs with sampling factors of
2x2,1x2 and 4:4:0 JPEGs with sampling factors of 2x2,2x1, since the
sums of the products of those sampling factors (2x2 + 1x2 + 1x2 and
2x2 + 2x1 + 2x1) are still <= 10. The libjpeg API correctly decodes
such images, so the TurboJPEG API should as well.
Currently, this only affects ARM, since it is the only platform that
accelerates YCbCr-to-RGB conversion but not merged upsampling. Even if
"plain" upsampling isn't accelerated, the combination of accelerated
color conversion + unaccelerated plain upsampling is still faster than
the unaccelerated merged upsampling algorithms.
In the AArch64 ABI, the high (unused) DWORD of a 32-bit argument's
register is undefined, so it was incorrect to use 64-bit
instructions to transfer a JDIMENSION argument in the 64-bit NEON SIMD
functions. The code worked thus far only because the existing compiler
optimizers weren't smart enough to do anything else with the register in
question, so the upper 32 bits happened to be all zeroes.
The latest builds of Clang/LLVM have a smarter optimizer, and under
certain circumstances, it will attempt to load-combine adjacent 32-bit
integers from one of the libjpeg structures into a single 64-bit integer
and pass that 64-bit integer as a 32-bit argument to one of the SIMD
functions (which is allowed by the ABI, since the upper 32 bits of the
32-bit argument's register are undefined.) This caused the
libjpeg-turbo regression tests to crash.
This patch tries to use the Wn registers whenever possible. Otherwise,
it uses a zero-extend instruction to avoid using the upper 32 bits of
the 64-bit registers, which are not guaranteed to be valid for 32-bit
arguments.
This eliminates "illegal instruction" errors when running libjpeg-turbo
under Linux on PowerPC chips that lack AltiVec support (e.g. the old
7XX/G3 models but also the newer e5500 series.)
The JSIMD_FORCE* environment variables previously meant "force the use
of this instruction set if it is available but others are available as
well", but that did nothing on ARM platforms, since there is only ever
one instruction set available. Since the ARM and MIPS CPU feature
detection code is less than bulletproof, and since there is only one
SIMD instruction set (currently) supported on those platforms, it makes
sense for the JSIMD_FORCE* environment variables on those platforms to
actually force the use of the SIMD instruction set, thus bypassing the
CPU feature detection code.
This addresses a concern raised in #88 whereby parsing /proc/cpuinfo
didn't work within a QEMU environment. This at least provides a
workaround, allowing users to force-enable or force-disable SIMD
instructions for ARM and MIPS builds of libjpeg-turbo.
projects / libjpeg-turbo / log