DRC [Fri, 9 Dec 2016 16:21:29 +0000 (10:21 -0600)]
Build: More GNUInstallDirs improvements
These improvements enable build systems to use GNUInstallDirs to define
custom directory variables.
- The set_dir() macro was renamed to GNUInstallDirs_set_install_dir(),
in keeping with the module's established macro naming convention.
- Rather than detecting whether the prefix has changed, the new
GNUInstallDirs_set_install_dir() macro instead examines whether the
default for the variable in question has changed. This allows for
more flexibility, since build systems may decide to change the
defaults based on factors other than the prefix. It also enables the
macro to work properly outside of the module.
- The module now performs directory variable substitution within the
body of GNUInstallDirs_get_absolute_install_dir().
- The JAVADIR variable is no longer included in GNUInstallDirs. That
directory is not part of the GNU spec, and it turns out that various
operating systems use different conventions for the location of Java
classes. Instead, the variable is now implemented in our build
system as a demonstration of the aforementioned GNUInstallDirs
enhancements.
DRC [Thu, 8 Dec 2016 20:43:59 +0000 (14:43 -0600)]
Build: Various improvements to install/pkg system
- GNUInstallDirs: any directory variable can now reference any other
directory variable by including its name in angle brackets (<>).
- Changed the documentation of the directory variables in BUILDING.md
accordingly. This commit also includes some formatting tweaks to
that section (using boldface for directory names, as is our
convention.)
- Changed the package scripts such that they use
CMAKE_INSTALL_DATAROOTDIR rather than CMAKE_INSTALL_DATADIR.
- We no longer override the install dir. defaults on Windows unless
performing an official build. It may be useful, for instance, to
use the GNU defaults when installing into an MSYS environment.
DRC [Thu, 8 Dec 2016 01:14:20 +0000 (19:14 -0600)]
Build: Minor tweaks to GNUInstallDirs defaults
It isn't actually necessary to specify `CMAKE_INSTALL_DEFAULT_MANDIR`
for our official build. Because `CMAKE_INSTALL_DEFAULT_DATAROOTDIR` is
blank for the official build, the default of "<DATAROOTDIR>/man" will
resolve to "man".
For the same reason, this commit changes the specification of
`CMAKE_INSTALL_DEFAULT_DOCDIR` and `CMAKE_INSTALL_DEFAULT_JAVADIR` in
the official build to be dependent on the data root directory (mainly to
make it obvious what we're doing.)
This commit also tweaks the example CMake command line in the directory
variable documentation so that it shows the correct location of the
CMake argument.
DRC [Thu, 8 Dec 2016 00:11:38 +0000 (18:11 -0600)]
Build: Fix Debug/RelWithDebInfo build with YASM
YASM requires a debug format to be specified with -g. Currently the
only combination that I can make work at all is DWARF-2/ELF (YASM
doesn't support Mach-O debugging at all, and its support for CV8/MSVC
and MinGW/DWARF-2 appears to be broken), so debugging is only enabled
automatically for ELF at the moment. For other formats, we don't
specify -g at all, which is how the old build system behaved.
DRC [Wed, 7 Dec 2016 16:54:54 +0000 (10:54 -0600)]
Build: Set install dirs in a more GNU-friendly way
This builds upon the existing GNUInstallDirs module in CMake but adds
the following features to that module:
- The ability to override the defaults for each install directory
through a new set of variables (`CMAKE_INSTALL_DEFAULT_*DIR`).
Before operating system vendors began shipping libjpeg-turbo, it was
meant to be a run-time drop-in replacement for the system's
distribution of libjpeg, so it has traditionally installed itself
under /opt/libjpeg-turbo on Un*x systems by default. On Windows, it
has traditionally installed itself under %SystemDrive%\libjpeg-turbo*,
which is not uncommon behavior for open source libraries (open source
SDKs tend to install outside of the Program Files directory so as to
avoid spaces in the directory name.) At least in the case of Un*x,
the install directory behavior is based somewhat on the Solaris
standard, which requires all non-O/S packages to install their files
under /opt/{package_name}. I adopted that standard for VirtualGL and
TurboVNC while working at Sun, because it allowed those packages to be
located under the same directory on all platforms. I adopted it for
libjpeg-turbo because it ensured that our files would never conflict
with the system's version of libjpeg. Even though many Un*x
distributions ship libjpeg-turbo these days, not all of them ship the
TurboJPEG API library or the Java classes or even the latest version
of the libjpeg API library, so there are still many cases in which it
is desirable to install a separate version of libjpeg-turbo than the
one installed by the system. Furthermore, installing the files under
/opt mimics the directory structure of our official binary packages,
and it makes it very easy to uninstall libjpeg-turbo.
For these reasons, our build system needs to be able to use
non-GNU-compliant defaults for each install directory if
`CMAKE_INSTALL_PREFIX` is set to the default value.
- For each directory variable, the module now detects changes to
`CMAKE_INSTALL_PREFIX` and changes the directory variable accordingly,
if the variable has not been changed by the user.
This makes it easy to switch between our "official" directory
structure and the GNU-compliant directory structure "on the fly"
simply by changing `CMAKE_INSTALL_PREFIX`. Also, this new mechanism
eliminated the need for the crufty mechanism that previously did the
same thing just for the library directory variable.
How it should work:
- If a dir variable is unset, then the module will set an internal
property indicating that the dir variable was initialized to its
default value.
- If the dir variable ever diverges from its default value, then the
internal property is cleared, and it cannot be set again without
unsetting the dir variable.
- If the install prefix changes, and if the internal property
indicates that the dir variable is still set to its default value,
and if the dir variable's value is not being manually changed at the
same time that the install prefix is being changed, then the dir
variable's value is automatically changed to the new default value
for that variable (as determined by the new install prefix.)
- The directory variables are now always cached, regardless of whether
they were set on the command line or not. This ensures that they can
easily be examined and modified after being set, regardless of how they
were set.
This was made possible by the introduction of the aforementioned
`CMAKE_INSTALL_DEFAULT_*DIR` variables.
- Improved directory variable documentation (based on descriptions at
https://www.gnu.org/prep/standards/html_node/Directory-Variables.html)
- The module now allows "<DATAROOTDIR>" to be used as a placeholder in
relative directory variables.
It is replaced "on the fly" with the actual path of
`CMAKE_INSTALL_DATAROOTDIR`.
This should more closely mimic the behavior of the old autotools build
system while retaining our customizations to it, and it should retain
the behavior of the old CMake build system.
DRC [Mon, 5 Dec 2016 22:52:54 +0000 (16:52 -0600)]
Build: Clean up inline keyword detection
Strict C89-conformant compilers don't support the "inline" keyword, but
most of them support "__inline__", and that keyword can be used with the
always_inline atribute as well. This commit also removes duplicate code
by using a foreach() loop to test the various keywords.
DRC [Sat, 3 Dec 2016 20:21:11 +0000 (14:21 -0600)]
Build: Fix issues when building as a Git submodule
- Replace CMAKE_SOURCE_DIR with CMAKE_CURRENT_SOURCE_DIR
- Replace CMAKE_BINARY_DIR with CMAKE_CURRENT_BINARY_DIR
- Don't use "libjpeg-turbo" in any of the package system filenames
(because CMAKE_PROJECT_NAME will not be the same if building LJT as
a submodule.)
DRC [Sat, 26 Nov 2016 00:50:11 +0000 (18:50 -0600)]
Build: Use wrapper script for gas-preprocessor.pl
The previous hack (adding ${CMAKE_ASM_COMPILER} to CMAKE_ASM_FLAGS)
didn't work in all cases, because more recent versions of CMake place
the includes ahead of the flags (which meant that the real assembler
wasn't the first argument to gas-preprocessor.pl.)
DRC [Wed, 23 Nov 2016 23:12:57 +0000 (17:12 -0600)]
Build: Fix RPATH handling
CMAKE_INSTALL_RPATH has to be set before the targets are defined (oops.)
This also explicitly turns on MACOSX_RPATH for the shared libraries
(which is the default with newer versions of CMake but not with 2.8.x.)
The old autotools/libtool build system hard-coded the install name
directory of the OS X shared libraries to libdir, which meant that any
executable that linked against those libraries would also be hard-coded
to look for the libjpeg-turbo libraries in that directory. @rpath makes
the OS X version of libjpeg-turbo behave like the Linux version, in the
sense that the executables under /opt/libjpeg-turbo/bin will
automatically pick up the libraries under /opt/libjpeg-turbo/lib* by
default, but other executables won't unless they are linked with -rpath.
DRC [Tue, 22 Nov 2016 04:58:18 +0000 (22:58 -0600)]
AppVeyor: Use built-in MSYS2 MinGW compilers
AppVeyor already has MinGW32 and MinGW64 flavors of GCC 5.3.0
installed under MSYS2, so there is no need to install our own builds of
MinGW. MinGW-builds is no longer an active project, and we were getting
occasional timeouts while wgetting those files from SourceForge.
Furthermore, GCC 5.3.0 should produce faster code than GCC 4.8.1.
DRC [Mon, 21 Nov 2016 02:59:55 +0000 (20:59 -0600)]
BUILDING.md: Clarifications and wordsmithing
Updated out-of-date information, wordsmithed and clarified many
sections, and generally cleaned up the build recipes (including a
complete overhaul of the iOS recipes.)
DRC [Fri, 18 Nov 2016 15:09:41 +0000 (09:09 -0600)]
BUILDING.md/README.md: Increment libjpeg SO age
Documentation buglet. This should have been changed in 6ed4d9d11085acd04dc7f2f899848693976dc010 to reflect the addition of
libjpeg API functions in libjpeg-turbo 1.5.
This commit also makes the "testclean" target clean up the 4:1:1 test
images. This was implemented in the autotools build system in 1f3635c4969f2319a01c9fe561958815b733227f but was left out of the CMake
build system due to an oversight.
DRC [Thu, 20 Oct 2016 22:55:55 +0000 (17:55 -0500)]
Travis: Deploy to S3 rather than SourceForge
This has the following advantages:
-- It doesn't require checking a private SSH key into the repository.
(With SourceForge, an SSH key is the "keys to the kingdom".)
-- If the S3 key is compromised, it is very easy to revoke it and
generate a new one.
-- The S3 bucket is isolated, so even if it becomes compromised, then
the damage that one could do is limited.
-- It's much easier to manage files through S3's web interface than
through SourceForge.
-- The files are served via HTTPS.
-- Travis fully supports S3 as a deployment target, so this simplifies
.travis.yml somewhat.
DRC [Thu, 20 Oct 2016 06:01:27 +0000 (01:01 -0500)]
Travis: GPG sign Linux binaries/source tarballs
Since we're still deploying our Linux/macOS CI artifacts to a web server
(specifically SourceForge Project Web Services) that doesn't support
HTTPS, it's a good idea to sign them. But since the private key has to
be checked into the repository, we use a different key for signing the
pre-releases (per project policy, the private signing keys for our
release binaries are never made available on any public server.)
DRC [Tue, 11 Oct 2016 16:58:20 +0000 (11:58 -0500)]
Win: Use YASM if it is in the PATH and NASM isn't
Previously, simd/CMakeLists.txt was hard-coded to use NASM, and it was
necessary to override the NASM variable in order to use YASM. This
commit changes the behavior such that NASM is still preferred, but YASM
will be used if it is in the PATH and NASM isn't available. This brings
the actual behavior in line with the behavior described in BUILDING.md.
DRC [Wed, 5 Oct 2016 19:42:35 +0000 (14:42 -0500)]
Travis CI: Fixes to support AVX2 code
-- Use trusty for SIMD builds. Ubuntu 12.04 is still using NASM 2.09.x,
which isn't new enough to support AVX2.
-- Add a special test for the SSE2 code path, since it is no longer the
default.
DRC [Wed, 5 Oct 2016 19:36:46 +0000 (14:36 -0500)]
Travis: use correct repo/branch for off. builds
Pass the actual repository and branch that Travis is using into the
builtljt script, so the official builds it generates will come from
the same code base as the other tested builds.
DRC [Tue, 4 Oct 2016 18:41:48 +0000 (13:41 -0500)]
Fix 32-bit non-SIMD FP regression tests
- Introduce a new FLOATTEST value ("387") on Un*x systems that will
compare the floating point DCT/IDCT algorithms against the expected
results from the C algorithms when built using 32-bit code and
-mfpmath=387.
- Extend the Windows regression tests so that they work properly when
building libjpeg-turbo with 32-bit code and without SIMD, using either
Visual C++ (tested with 2008, 2010, 2015) or MinGW.
In the AArch64 ABI, as in many others, it's forbidden to read/store data
below the stack pointer. Some SIMD functions were doing just that
(stack pointer misuse) when trying to preserve callee-saved registers,
and this resulted in those registers being restored with incorrect
contents under certain circumstances.
This patch fixes that behavior, and callee-saved registers are now
stored above the stack pointer throughout the function call. The patch
also removes register saving in places where it is unnecessary for this
ABI, or it makes use of unused scratch regiters instead of callee-saved
registers.
Fixes #97. Closes #101.
Refer also to https://bugzilla.redhat.com/show_bug.cgi?id=1368569
The last iDevice to require ARMv6 was the iPhone 3G, which required iOS
4.2.1 or older. Our binaries have always required iOS 4.3 or newer,
so I'm not sure if the ARMv6 fork of our binaries was ever useful to
begin with. In any case, if it ever was useful, it no longer is. Fat
binaries can still be generated with ARMv6 support by invoking
{build_directory}/pkgscripts/makemacpkg manually.
Fix out-of-bounds write in partial decomp. feature
Reported by Clang UBSan (refer to
https://bugzilla.mozilla.org/show_bug.cgi?id=1301252 for test image.)
This appears to be a legitimate bug introduced by 3ab68cf563f6edc2608c085f5c8b2d5d5c61157e. Any component array, such
as first_MCU_col and last_MCU_col, should always be able to accommodate
MAX_COMPONENTS values. The aforementioned test image had 8 components,
which was not enough to make the out-of-bounds write bust out of the
jpeg_decomp_master struct (and fortunately the memory after last_MCU_col
is an integer used as a boolean, so stomping on it will do nothing other
than change the decoder state.) I crafted another special image that
has 10 components (the maximum allowable), but that was apparently not
enough to bust out of the allocated memory, either. Thus, it is
posited that the security threat posed by this bug is either extremely
minimal or non-existent.
NOTE: The jdhuff.c/jdphuff.c warnings should have already been silenced
by 8e9cef2e6f5156c4b055a04a8f979b7291fc6b7a, but apparently I need to
be REALLY clear that I'm trying to do pointer arithmetic rather than
dereference an array. Grrr...
Refer to:
https://bugzilla.mozilla.org/show_bug.cgi?id=1301250
https://bugzilla.mozilla.org/show_bug.cgi?id=1301256
When attempting to decode a malformed JPEG image (refer to
https://bugzilla.mozilla.org/show_bug.cgi?id=1295044) with dimensions
61472 x 32800, the maximum_space variable within the
realize_virt_arrays() function will exceed the maximum value of a 32-bit
integer and will wrap around. The memory manager subsequently fails
with an "Insufficient memory" error (case 4, in alloc_large()), so this
commit simply causes that error to be triggered earlier, before UBSan
has a chance to complain.
Note that this issue did not ever represent an exploitable security
threat, because the POSIX-based memory manager that we use doesn't ever
do anything meaningful with the value of maximum_space.
jpeg_mem_available() simply sets avail_mem = maximum_space, so the
subsequent behavior of the memory manager is the same regardless of
whether maximum_space is correct or not. This commit simply removes a
UBSan warning in order to make it easier to detect actual security
issues.
Normally, 4:2:2 JPEGs have horizontal x vertical luminance,chrominance
sampling factors of 2x1,1x1, and 4:4:0 JPEGs have horizontal x vertical
luminance,chrominance sampling factors of 1x2,1x1. However, it is
technically legal to create 4:2:2 JPEGs with sampling factors of
2x2,1x2 and 4:4:0 JPEGs with sampling factors of 2x2,2x1, since the
sums of the products of those sampling factors (2x2 + 1x2 + 1x2 and
2x2 + 2x1 + 2x1) are still <= 10. The libjpeg API correctly decodes
such images, so the TurboJPEG API should as well.
Currently, this only affects ARM, since it is the only platform that
accelerates YCbCr-to-RGB conversion but not merged upsampling. Even if
"plain" upsampling isn't accelerated, the combination of accelerated
color conversion + unaccelerated plain upsampling is still faster than
the unaccelerated merged upsampling algorithms.
cpuid tells us whether the O/S uses extended state management via
XSAVE/XRSTOR, but we have to call xgetbv to verify that it is using
XSAVE/XRSTOR to manage the state of XMM/YMM registers.
In the AArch64 ABI, the high (unused) DWORD of a 32-bit argument's
register is undefined, so it was incorrect to use 64-bit
instructions to transfer a JDIMENSION argument in the 64-bit NEON SIMD
functions. The code worked thus far only because the existing compiler
optimizers weren't smart enough to do anything else with the register in
question, so the upper 32 bits happened to be all zeroes.
The latest builds of Clang/LLVM have a smarter optimizer, and under
certain circumstances, it will attempt to load-combine adjacent 32-bit
integers from one of the libjpeg structures into a single 64-bit integer
and pass that 64-bit integer as a 32-bit argument to one of the SIMD
functions (which is allowed by the ABI, since the upper 32 bits of the
32-bit argument's register are undefined.) This caused the
libjpeg-turbo regression tests to crash.
This patch tries to use the Wn registers whenever possible. Otherwise,
it uses a zero-extend instruction to avoid using the upper 32 bits of
the 64-bit registers, which are not guaranteed to be valid for 32-bit
arguments.
This fixes crashes that would occur when attempting to use
libjpeg-turbo's AVX2 extensions on older O/S's (such as Windows XP or
RHEL 5.) Even if the CPU supports AVX2, the O/S has to also support
saving/restoring YMM registers when switching contexts.
Refer to
https://software.intel.com/sites/default/files/m/d/4/1/d/8/11MC12_Avoiding_2BAVX-SSE_2BTransition_2BPenalties_2Brh_2Bfinal.pdf
for more information. This eliminates all AVX-SSE transitions detected
with the Intel SDE tool.
This eliminates "illegal instruction" errors when running libjpeg-turbo
under Linux on PowerPC chips that lack AltiVec support (e.g. the old
7XX/G3 models but also the newer e5500 series.)
The JSIMD_FORCE* environment variables previously meant "force the use
of this instruction set if it is available but others are available as
well", but that did nothing on ARM platforms, since there is only ever
one instruction set available. Since the ARM and MIPS CPU feature
detection code is less than bulletproof, and since there is only one
SIMD instruction set (currently) supported on those platforms, it makes
sense for the JSIMD_FORCE* environment variables on those platforms to
actually force the use of the SIMD instruction set, thus bypassing the
CPU feature detection code.
This addresses a concern raised in #88 whereby parsing /proc/cpuinfo
didn't work within a QEMU environment. This at least provides a
workaround, allowing users to force-enable or force-disable SIMD
instructions for ARM and MIPS builds of libjpeg-turbo.
DRC [Fri, 10 Jun 2016 21:03:49 +0000 (16:03 -0500)]
64-bit AVX2 SIMD: Restore instructive comments
This commit adds back instructive comments in the image-space
algorithms, similar to those in the SSE2 code. These comments make it
easier to follow the flow of data through the algorithms.
DRC [Tue, 24 May 2016 15:23:56 +0000 (10:23 -0500)]
Format copyright headers more consistently
The IJG convention is to format copyright notices as:
Copyright (C) YYYY, Owner.
We try to maintain this convention for any code that is part of the
libjpeg API library (with the exception of preserving the copyright
notices from Cendio's code verbatim, since those predate
libjpeg-turbo.)
Note that the phrase "All Rights Reserved" is no longer necessary, since
all Buenos Aires Convention signatories signed onto the Berne Convention
in 2000. However, our convention is to retain this phrase for any files
that have a self-contained copyright header but to leave it off of any
files that refer to another file for conditions of distribution and use.
For instance, all of the non-SIMD files in the libjpeg API library refer
to README.ijg, and the copyright message in that file contains "All
Rights Reserved", so it is unnecessary to add it to the individual
files.
The TurboJPEG code retains my preferred formatting convention for
copyright notices, which is based on that of VirtualGL (where the
TurboJPEG API originated.)
projects / libjpeg-turbo / log