DRC [Tue, 24 May 2016 15:23:56 +0000 (10:23 -0500)]
Format copyright headers more consistently
The IJG convention is to format copyright notices as:
Copyright (C) YYYY, Owner.
We try to maintain this convention for any code that is part of the
libjpeg API library (with the exception of preserving the copyright
notices from Cendio's code verbatim, since those predate
libjpeg-turbo.)
Note that the phrase "All Rights Reserved" is no longer necessary, since
all Buenos Aires Convention signatories signed onto the Berne Convention
in 2000. However, our convention is to retain this phrase for any files
that have a self-contained copyright header but to leave it off of any
files that refer to another file for conditions of distribution and use.
For instance, all of the non-SIMD files in the libjpeg API library refer
to README.ijg, and the copyright message in that file contains "All
Rights Reserved", so it is unnecessary to add it to the individual
files.
The TurboJPEG code retains my preferred formatting convention for
copyright notices, which is based on that of VirtualGL (where the
TurboJPEG API originated.)
DRC [Wed, 11 May 2016 02:04:02 +0000 (21:04 -0500)]
Don't allow opaque source/dest mgrs to be swapped
Calling jpeg_stdio_dest() followed by jpeg_mem_dest(), or jpeg_mem_src()
followed by jpeg_stdio_src(), is dangerous, because the existing opaque
structure would not be big enough to accommodate the new source/dest
manager. This issue was non-obvious to libjpeg-turbo consumers, since
it was only documented in code comments. Furthermore, the issue could
also occur if the source/dest manager was allocated by the calling
program, but it was not allocated with enough space to accommodate the
opaque stdio or memory source/dest manager structs. The safest thing to
do is to throw an error if one of these functions is called when there
is already a source/dest manager assigned to the object and it was
allocated elsewhere.
DRC [Tue, 10 May 2016 01:00:46 +0000 (20:00 -0500)]
Build: Don't allow jpeg-7+ emul. w/o arith coding
The jpeg-7/jpeg-8 APIs/ABIs require arithmetic coding, and the jpeg-8
API/ABI requires the memory source/destination manager, so this commit
causes the build system to ignore --with-arith-enc/--without-arith-enc
and --with-arith-dec/--without-arith-dec (and the equivalent CMake
variables-- WITH_ARITH_ENC and WITH_ARITH_DEC) when v7/v8 API/ABI
emulation is enabled. Furthermore, the CMake build system now ignores
WITH_MEM_SRCDST whenever WITH_JPEG8 is specified (the autotools build
system already did that.)
mattsarett [Tue, 3 May 2016 14:33:43 +0000 (10:33 -0400)]
ARMv7 SIMD: Fix clang compatibility (Part 2)
GCC does support UAL syntax (strbeq) if the ".syntax unified" directive
is supplied. This directive is supported by all versions of GCC and
clang going back to 2003, so it should not create any backward
compatibility issues.
mattsarett [Mon, 2 May 2016 16:31:51 +0000 (12:31 -0400)]
ARMv7 SIMD: Fix clang compatibility
By design, clang only supports Unified Assembler Language (and not
pre-UAL syntax):
https://llvm.org/bugs/show_bug.cgi?id=23507
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0473c/BABJIHGJ.html
Thus, clang only supports the strbeq instruction and not streqb, but
unfortunately some versions of GCC only support streqb. Go, go
Gadget #ifdef...
DRC [Sun, 1 May 2016 16:42:15 +0000 (11:42 -0500)]
Fix CMake fallback BUILD var on non-U.S. machines
If wmic.exe wasn't available, then CMakeLists.txt would call
"cmd /C date /T" and parse the result in order to set the BUILD
variable. However, the parser assumed that the date was in MM/DD/YYYY
format, which is not generally the case unless the user's locale is U.S.
English with the default region/language settings for that locale.
This commit modifies CMakeLists.txt such that it uses the
string(TIMESTAMP) function available in CMake 2.8.11 and later to set
the BUILD variable, thus eliminating the need to use wmic.exe or any
other platform-specific hack.
This commit also modifies the build instructions to remove any reference
to CMake 2.6 (which hasn't been supported by our build system since
libjpeg-turbo 1.3.x.)
At one time, it was possible to use CMake to build under Cygwin, but
that hasn't worked since 1.4.1 (due to the Huffman codec changes that
now require SIZEOF_SIZE_T to be defined for non-WIN32 platforms) and may
have even been broken before that. Originally, we used the "date"
command under MSYS in order to obtain the default build number, but that
was rendered unnecessary by 5e3bb3e9 (v1.3 beta.) 9fe22dac (1.4 beta)
further modified CMakeLists.txt so that the "date" command was only used
on Cygwin, but for unexplained reasons, that commit also applied the
(now vestigial) code to all non-WIN32 platforms. This prevented
CMakeLists.txt from displaying an error if someone attempted to use the
CMake build system on Un*x platforms, and that may have been behind the
flurry of pull requests and issues-- including #21, #29, #37, #58, #73--
complaining that the CMake build system didn't work on Un*x platforms
(although it was not until #73 that this bug came to light.)
This commit removes all vestiges of Un*x support from the CMake build
system and makes it clear that CMake cannot be used to build
libjpeg-turbo on non-WIN32 platforms. It is our position that CMake
will not be supported on non-WIN32 platforms until/unless the autotools
build system is removed, and this will not happen without broad support
from the community (including major O/S vendors.) If you are in favor
of migrating the entire build system to CMake, then please make your
voice heard by commenting on #56.
Even though tjDecompressToYUV2() is mostly just a wrapper for
tjDecompressToYUVPlanes(), tjDecompressToYUV2() still calls
jpeg_read_header(), so it needs to properly set up the libjpeg error
handler prior to making this call. Otherwise, under very esoteric (and
arguably incorrect) use cases, a program could call tjDecompressToYUV2()
without first checking the JPEG header using tjDecompressHeader3(), and
if the header was corrupt, then the libjpeg API would invoke
my_error_exit(). my_error_exit() would in turn call longjmp() on the
previous value of myerr->setjmp_buffer, which was probably set in a
previous TurboJPEG function, such as tjInitDecompress(). Thus, when a
libjpeg error was triggered within the body of tjDecompressToYUV2(), the
PC would jump to the error handler of the previous TurboJPEG function,
and this usually caused stack corruption in the calling program (because
the signature and return type of the previous TurboJPEG function
probably wasn't the same as that of tjDecompressToYUV2().)
Increase severity of tjDecompressToYUV2() bug desc
Actually, what happened was that the longjmp() call within
my_error_exit() acted on the previous value of myerr->setjmp_buffer,
which was probably set in a previous TurboJPEG function, such as
tjInitDecompress(). Thus, when a libjpeg error was triggered within
the body of tjDecompressToYUV2(), the PC jumped to the error handler
of the previous TurboJPEG function, and this usually caused stack
corruption in the calling program (because the signature and return
type of the previous TurboJPEG function probably wasn't the same.)
Even though tjDecompressToYUV2() is mostly just a wrapper for
tjDecompressToYUVPlanes(), tjDecompressToYUV2() still calls
jpeg_read_header(), so it needs to properly set up the libjpeg error
handler prior to making this call. Otherwise, under very esoteric (and
arguably incorrect) use cases, a program can call tjDecompressToYUV2()
without first checking the JPEG header using tjDecompressHeader3(), and
if the header is corrupt, tjDecompressToYUV2() will abort without
triggering an error.
<sigh> GitHub doesn't render indented text the same as my local MarkDown
viewer (MacDown), so it's necessary to indent "... OR ..." by 3 spaces
so both will display it on the same indentation level as "Visual C++
2005 or later" and "MinGW".
Indent "... OR ..." to make it clear that the choice is between Visual
C++ and MinGW, not Visual C++ and MinGW + NASM. Move NASM to the top of
the list to make that even more clear. Make it clear that nasm.exe
should be in the PATH.
DRC [Thu, 31 Mar 2016 14:49:49 +0000 (09:49 -0500)]
cjpeg: Fix buf overrun caused by bad bin PPM input
This extends the fix in 6709e4a0cfa44d4f54ee8ad05753d4aa9260cb91 to
include binary PPM/PGM files, thus preventing a malformed binary
PPM/PGM input file from triggering an overrun of the rescale array and
potentially crashing cjpeg.
Note that this issue affected only cjpeg and not the underlying
libjpeg-turbo libraries, and thus it did not represent a security
threat.
DRC [Wed, 16 Mar 2016 12:10:35 +0000 (07:10 -0500)]
Add version/build info to global string table
This is a common practice in other infrastructure libraries, such as
OpenSSL and libpng, because it makes it easy to examine an application
binary and determine which version of the library the application was
linked against.
DRC [Sun, 6 Mar 2016 14:15:04 +0000 (08:15 -0600)]
Ensure that default Huffman tables are initialized
This prevents a malformed motion-JPEG frame (MJPEG frames lack Huffman
tables) from causing the "fast path" of the Huffman decoder to read
uninitialized memory. Essentially, this is doing the same thing for
MJPEG frames as 43d8cf4d4572fa50a37cccadbe71b9bee37de55d did for regular
images.
DRC [Wed, 2 Mar 2016 15:53:11 +0000 (09:53 -0600)]
Java: Fix parallel make with autotools
Running 'make -j{jobs}' on a build that was configured with Java
(--with-java) would previously cause an error:
make: *** No rule to make target `TJExample.class', needed by
`turbojpeg.jar'.
It seems that parallel make doesn't understand that the files in
$(JAVA_CLASSES) are all generated from the same invocation of javac, so
it tries to parallelize the building of those files (which of course
doesn't work.) This patch instead makes turbojpeg.jar depend on
classnoinst.stamp. This effectively creates a synchronization fence,
since that file is only created when all of the class files have been
built.
DRC [Mon, 29 Feb 2016 23:21:02 +0000 (17:21 -0600)]
Win/x64: Fix improper callee save of xmm8-xmm11
The x86-64 SIMD accelerations for Huffman encoding used incorrect
stack math to save xmm8-xmm11 on Windows. This caused TJBench to
always report 1 Mpixel/sec for the compression performance, and it
likely would have caused other application issues as well.
DRC [Mon, 29 Feb 2016 19:18:01 +0000 (13:18 -0600)]
Bump TurboJPEG C API revision to 1.5
The changes relative to 1.4.x are only cosmetic (using const pointers)
and should not affect API/ABI compatibility, but our practice is to
synchronize the API revision with the most recent release that provides
user-visible changes to the API.
DRC [Sat, 20 Feb 2016 00:32:10 +0000 (18:32 -0600)]
libjpeg API: Partial scanline decompression
This, in combination with the existing jpeg_skip_scanlines() function,
provides the ability to crop the image both horizontally and vertically
while decompressing (certain restrictions apply-- see libjpeg.txt.)
This also cleans up the documentation of the line skipping feature and
removes the "strip decompression" feature from djpeg, since the new
cropping feature is a superset of it.
DRC [Fri, 19 Feb 2016 19:16:56 +0000 (13:16 -0600)]
Build: Make the NASM autoconf variable persistent
Previously, if a custom value of this variable was specified when
running configure, then that value would be lost if configure was
automatically re-run (as a result of changes to configure.ac, for
instance.)
As a bonus, the NASM variable is now also listed when running
'configure --help', so it is obvious how to override the default
NASM command.
DRC [Fri, 19 Feb 2016 14:53:33 +0000 (08:53 -0600)]
Use consistent/modern code formatting for pointers
The convention used by libjpeg:
type * variable;
is not very common anymore, because it looks too much like
multiplication. Some (particularly C++ programmers) prefer to tuck the
pointer symbol against the type:
type* variable;
to emphasize that a pointer to a type is effectively a new type.
However, this can also be confusing, since defining multiple variables
on the same line would not work properly:
type* variable1, variable2; /* Only variable1 is actually a
pointer. */
This commit reformats the entirety of the libjpeg-turbo code base so
that it uses the same code formatting convention for pointers that the
TurboJPEG API code uses:
type *variable1, *variable2;
This seems to be the most common convention among C programmers, and
it is the convention used by other codec libraries, such as libpng and
libtiff.
DRC [Thu, 18 Feb 2016 22:02:28 +0000 (16:02 -0600)]
Reorder copyright messages in cjpeg/djpeg/jpegtran
Place the authors in the following order:
* libjpeg-turbo authors (2009-) in descending order of the date of their
most recent contribution to the project, then in ascending order of
the date of their first contribution to the project
* Upstream authors in descending order of the date of the first
inclusion of their code (this indicates that their code serves as the
foundation of this code.)
This also adds Siarhei to the author list, since he contributed ARM SIMD
code both as a Nokia employee and more recently as an independent
developer.
DRC [Tue, 9 Feb 2016 07:35:39 +0000 (01:35 -0600)]
Java: Avoid OOM error when running 'make test'
We need to garbage collect between iterations of the outside loop in
bufSizeTest() in order to avoid exhausting the heap when running with
Java 6 (which is still used on Linux to test the 32-bit version of
libjpeg-turbo in automated builds.)
gas-preprocessor.pl and/or the clang assembler apparently don't like
default values in macro arguments, and we need to use a separate const
section for each function (because of our use of adr, also necessitated
by the broken clang assembler.)
DRC [Mon, 8 Feb 2016 04:05:56 +0000 (22:05 -0600)]
ARM64: Avoid LD3/ST3 at run time, not compile time
... and only if ThunderX is detected. This can be easily expanded later
on to include other CPUs that are known to suffer from slow LD3/ST3, but
it doesn't make sense to disable LD3/ST3 for all non-Android Linux
platforms just because ThunderX is slow.
Note that this commit introduces a similar /proc/cpuinfo parser to that
of the ARM32 implementation. It is used to specifically check whether
the code is running on Cavium ThunderX and, if so, disable the ARM64
SIMD Huffman routines (which slow performance by an average of 8% on
that CPU.)
DRC [Sat, 6 Feb 2016 22:04:29 +0000 (16:04 -0600)]
Un*x: Enable testing cross-compiled builds
Don't include the all: target as a dependency of the tests when
cross-compiling, and ensure that the files generated by the tests are
removed, even if they were created read-only (or if the tests are being
run on a different type of system that doesn't correctly interpret the
file permissions.) This allows one to easily build the code on one
machine and run 'make test' on another.
DRC [Sat, 6 Feb 2016 20:09:20 +0000 (14:09 -0600)]
Win: Enable testing cross-compiled builds
When cross-compiling, CMakeLists.txt now generates the CTest script
using relative paths, so that CTest can more easily be executed on a
different machine from the build machine. Furthermore, Windows builds
are now tested using md5cmp, just like on Linux, rather than a CMake
script. This prevents issues with differing CMake locations between
the build and test machines.
This also removes some trailing spaces from the md5cmp code and improves
the readability of the test code in CMakeLists.txt.
DRC [Sat, 6 Feb 2016 18:18:44 +0000 (12:18 -0600)]
Fix MinGW build
jinclude.h can't be safely included multiple times, so instead of
including it in the shared (broken-out) headers, it should instead be
included by the source files that include one or more of those headers.
DRC [Fri, 5 Feb 2016 00:34:38 +0000 (18:34 -0600)]
Prevent overread when decoding malformed JPEG
The accelerated Huffman decoder was previously invoked if there were
> 128 bytes in the input buffer. However, it is possible to construct a
JPEG image with Huffman blocks > 430 bytes in length
(http://stackoverflow.com/questions/2734678/jpeg-calculating-max-size).
While such images are pathological and could never be created by a
JPEG compressor, it is conceivable that an attacker could use such an
artifially-constructed image to trigger an input buffer overrun in the
libjpeg-turbo decompressor and thus gain access to some of the data on
the calling program's heap.
This patch simply increases the minimum buffer size for the accelerated
Huffman decoder to 512 bytes, which should (hopefully) accommodate any
possible input.
This addresses a major issue (LJT-01-005) identified in a security audit
by Cure53.
DRC [Thu, 4 Feb 2016 16:59:21 +0000 (10:59 -0600)]
Guard against wrap-around in alloc functions
Because of the exposed nature of the libjpeg API, alloc_small() and
alloc_large() can potentially be called by external code. If an
application were to call either of those functions with
sizeofobject > SIZE_MAX - ALIGN_SIZE - 1, then the math in
round_up_pow2() would wrap around to zero, causing that function to
return a small value. That value would likely not exceed
MAX_ALLOC_CHUNK, so the subsequent size checks in alloc_small() and
alloc_large() would not catch the error.
A similar problem could occur in 32-bit builds if alloc_sarray() were
called with
samplesperrow > SIZE_MAX - (2 * ALIGN_SIZE / sizeof(JSAMPLE)) - 1
This patch simply ensures that the size argument to the alloc_*()
functions will never exceed MAX_ALLOC_CHUNK (1 billion). If it did,
then subsequent size checks would eventually catch that error, so we
are instead catching the error before round_up_pow2() is called.
This addresses a minor concern (LJT-01-001) expressed in a security
audit by Cure53.
DRC [Thu, 4 Feb 2016 15:20:41 +0000 (09:20 -0600)]
TurboJPEG: Avoid dangling pointers
This addresses a minor concern (LJT-01-002) expressed in a security
audit by Cure53. _tjInitCompress() and _tjInitDecompress() call
(respectively) jpeg_mem_dest_tj() and jpeg_mem_src_tj() with a pointer
to a dummy buffer, in order to set up the destination/source manager.
The dummy buffer should never be used, but it's still better to make it
static so that the pointer in the destination/source manager always
points to a valid region of memory.
DRC [Wed, 3 Feb 2016 20:02:13 +0000 (14:02 -0600)]
Adjust performance claims
Document the latest benchmarks on the Nexus 5X and change the "2-4x"
overall claim to "2-6x". The peak performance on x86 platforms was
already closer to 5x, and the addition of SIMD-accelerated Huffman
encoding gave it that extra push over the cliff.
DRC [Wed, 3 Feb 2016 05:17:06 +0000 (23:17 -0600)]
Use consistent formatting in ARM NEON SIMD code
There aren't really any best practices to follow here. I tried as best
as I could to adopt a standard that would ease any future maintenance
burdens. The basic tenets of that standard are:
* Assembly instructions always start on Column 5, and operands always
start on Column 21, except:
- The instruction and operand can be indented (usually by 2 spaces)
to indicate a separate instruction stream.
- If the instruction is within an enclosing .if block in a macro,
it should always be indented relative to the .if block.
* Comments are placed with an eye toward readability. There are always
at least 2 spaces between the end of a line of code and the associated
in-line comment. Where it made sense, I tried to line up the comments
in blocks, and some were shifted right to avoid overlap with
neighboring instruction lines. Not an exact science.
* Assembler directives and macros use 2-space indenting rules. .if
blocks are indented relative to the macro, and code within the .if
blocks is indented relative to the .if directive.
* No extraneous spaces between operands. Lining up the operands
vertically did not really improve readability-- personally, I think it
made it worse, since my eye would tend to lose its place in the
uniform columns of characters. Also, code with a lot of vertical
alignment is really hard to maintain, since changing one line could
necessitate changing a bunch of other lines to avoid spoiling the
alignment.
* No extraneous spaces in #defines or other directives. In general, the
only extraneous spaces (other than indenting spaces) are between:
- Instructions and operands
- Operands and in-line comments
This standard should be more or less in keeping with other formatting
standards used within the project.
Decompression speedup relative to libjpeg-turbo 1.4.2 (IFAST IDCT):
48-core ThunderX (RunAbove ARM Cloud), Linux, 64-bit: 51-98% (avg. 75%)
Minimal speedup (1-5%) observed on iPhone 5S (Cortex-A7)
NOTE: This commit avoids the st3 instruction for non-Android and
non-Apple builds, which may cause a performance regression against
libjpeg-turbo 1.4.x on ARM64 systems that are running plain Linux.
Since ThunderX is the only platform known to suffer from slow ld3 and
st3 instructions, it is probably better to check for the CPU type
at run time and disable ld3/st3 only if ThunderX is detected.
This commit also enables the use of ld3 on Android platforms, which
should be a safe bet, at least for now. This speeds up compression on
the afore-mentioned Nexus Cortex-A53 by 5.5-19% (avg. 12%) and on the
Nexus Cortex-A57 by 1.2-14% (avg. 6.3%), relative to the previous
commits.
DRC [Mon, 1 Feb 2016 17:28:55 +0000 (11:28 -0600)]
Update Android build instr. for ARMv8, PIE, etc.
* Include information on how to do a 64-bit ARMv8 build with the latest
NDK
* Suggest -fPIE and -pie as default CFLAGS (required for android-16 and
later.
* Remove -fstrict-aliasing flag (-Wall already includes it)
DRC [Mon, 1 Feb 2016 17:12:22 +0000 (11:12 -0600)]
Update Android build instr. for ARMv8, PIE, etc.
* Include information on how to do a 64-bit ARMv8 build with the latest
NDK
* Suggest -fPIE and -pie as default CFLAGS (required for android-16 and
later.
* Remove -fstrict-aliasing flag (-Wall already includes it)
DRC [Mon, 1 Feb 2016 17:03:39 +0000 (11:03 -0600)]
TJBench: Fix segfault on Android
For whatever reason, the "write" global variable in tjbench.c was
overriding the linkage with the write() system function. This may have
affected other platforms as well but was not known to.
DRC [Mon, 18 Jan 2016 22:40:07 +0000 (16:40 -0600)]
Provide pkg-config (.pc) scripts
This allows a project to use PKG_CHECK_MODULES() in its configure.ac
file to easily check for the presence of libjpeg-turbo and modify the
compiler/linker flags accordingly. Note that if a project relies solely
on pkg-config to check for libjpeg-turbo, then it will not be possible
to build that project using libjpeg or an earlier version of
libjpeg-turbo.
DRC [Sat, 16 Jan 2016 07:53:32 +0000 (01:53 -0600)]
Optimize ARM64 SIMD code for Cavium ThunderX
Per @ssvb:
ThunderX is an ARM64 chip that dedicates most of its transistor real
estate to providing 48 cores, so each core is not as fast as a result.
Each core is dual-issue & in-order for scalar instructions and has only
a single-issue half-width NEON unit, so the peak throughput is one
128-bit instruction per 2 cycles. So careful instruction scheduling is
important. Furthermore, ThunderX has an extremely slow implementation
of ld2 and ld3, so this commit implements the equivalent of those
instructions using ld1.
Compression speedup relative to libjpeg-turbo 1.4.2:
48-core ThunderX (RunAbove ARM Cloud), Linux, 64-bit: 58-85% (avg. 74%)
relative to jpeg-6b: 1.75-2.14x (avg. 1.95x)
Refer to #49 and #51 for discussion.
Closes #51.
This commit also wordsmiths the ChangeLog entry (the ARMv8 SIMD
implementation is "complete" only for compression-- it still lacks some
decompression algorithms, as does the ARMv7 implementation.)
DRC [Fri, 15 Jan 2016 15:29:11 +0000 (09:29 -0600)]
Complete the ARM64 NEON SIMD implementation
This adds 64-bit NEON coverage for all of the algorithms that are
covered by the 32-bit NEON implementation, except for h2v1 (4:2:2) fancy
upsampling (used when decompressing 4:2:2 JPEG images.) It also adds
64-bit NEON SIMD coverage for:
which are not covered in the 32-bit implementation.
Compression speedups relative to libjpeg-turbo 1.4.2:
Apple A7 (iPhone 5S), iOS, 64-bit: 113-150% (reported)
48-core ThunderX (RunAbove ARM Cloud), Linux, 64-bit: 2.1-33% (avg. 15%)
Refer to #44 and #49 for discussion
This commit also removes the unnecessary
if (simd_support & JSIMD_ARM_NEON)
statements from the jsimd* algorithm functions. Since the jsimd_can*()
functions check for the existence of NEON, the corresponding algorithm
functions will never be called if NEON isn't available.
DRC [Wed, 13 Jan 2016 09:13:20 +0000 (03:13 -0600)]
ARM32 NEON SIMD implementation of Huffman encoding
Full-color compression speedups relative to libjpeg-turbo 1.4.2:
800 MHz ARM Cortex-A9, iOS, 32-bit: 26-44% (avg. 32%)
Refer to #42 and #47 for discussion.
This commit also removes the unnecessary
if (simd_support & JSIMD_ARM_NEON)
statements from the jsimd* algorithm functions. Since the jsimd_can*()
functions check for the existence of NEON, the corresponding algorithm
functions will never be called if NEON isn't available. Removing those
if statements improved performance across the board by a couple of
percent.
projects / libjpeg-turbo / log