Lasse Collin [Sun, 25 May 2014 18:45:56 +0000 (21:45 +0300)]
liblzma: Use lzma_alloc_zero() in LZ encoder initialization.
This avoids a memzero() call for a newly-allocated memory,
which can be expensive when encoding small streams with
an over-sized dictionary.
To avoid using lzma_alloc_zero() for memory that doesn't
need to be zeroed, lzma_mf.son is now allocated separately,
which requires handling it separately in normalize() too.
Thanks to Vincenzo Innocente for reporting the problem.
Lasse Collin [Fri, 25 Apr 2014 14:53:42 +0000 (17:53 +0300)]
Build: Fix the combination of --disable-xzdec --enable-lzmadec.
In this case "make install" could fail if the man page directory
didn't already exist at the destination. If it did exist, a
dangling symlink was created there. Now the link is omitted
instead. This isn't the best fix but it's better than the old
behavior.
Lasse Collin [Thu, 24 Apr 2014 15:06:24 +0000 (18:06 +0300)]
Update INSTALL.
Add a note about failing "make check". The source of
the problem should be fixed in libtool (if it really is
a libtool bug and not mine) but I'm unable to spend time
on that for now. Thanks to Nelson H. F. Beebe for reporting
the issue.
Add a note about a possible need to run "ldconfig" after
"make install".
Lasse Collin [Wed, 9 Apr 2014 14:26:10 +0000 (17:26 +0300)]
xz: Rename a variable to avoid a namespace collision on Solaris.
I don't know the details but I have an impression that there's
no problem in practice if using GCC since people have built xz
with GCC (without patching xz), but renaming the variable cannot
hurt either.
Lasse Collin [Sun, 12 Jan 2014 17:38:43 +0000 (19:38 +0200)]
Windows: Add config.h for building liblzma with MSVC 2013.
This is for building liblzma. Building xz tool too requires
a little more work. Maybe it will be supported, but for most
MSVC users it's enough to be able to build liblzma.
C99 support in MSVC 2013 is almost usable which is a big
improvement over earlier versions. It's "almost" because
there's a dumb bug that breaks mixed declarations after
an "if" statements unless the "if" statement uses braces:
Lasse Collin [Tue, 12 Nov 2013 14:29:48 +0000 (16:29 +0200)]
xz: Make --block-list and --block-size work together in single-threaded.
Previously, --block-list and --block-size only worked together
in threaded mode. Boundaries are specified by --block-list, but
--block-size specifies the maximum size for a Block. Now this
works in single-threaded mode too.
Lasse Collin [Tue, 22 Oct 2013 16:51:55 +0000 (19:51 +0300)]
xz: Take advantage of LZMA_FULL_BARRIER with --block-list.
Now if --block-list is used in threaded mode, the encoder
won't need to flush at each Block boundary specified via
--block-list. This improves performance a lot, making
threading helpful with --block-list.
The flush timer was reset after LZMA_FULL_FLUSH but since
LZMA_FULL_BARRIER doesn't flush, resetting the timer is
no longer done.
Lasse Collin [Wed, 2 Oct 2013 17:05:23 +0000 (20:05 +0300)]
liblzma: Support LZMA_FULL_FLUSH and _BARRIER in threaded encoder.
Now --block-list=SIZES works with in the threaded mode too,
although the performance is still bad due to the use of
LZMA_FULL_FLUSH instead of the new LZMA_FULL_BARRIER.
Lasse Collin [Tue, 17 Sep 2013 08:52:28 +0000 (11:52 +0300)]
Add native threading support on Windows.
Now liblzma only uses "mythread" functions and types
which are defined in mythread.h matching the desired
threading method.
Before Windows Vista, there is no direct equivalent to
pthread condition variables. Since this package doesn't
use pthread_cond_broadcast(), pre-Vista threading can
still be kept quite simple. The pre-Vista code doesn't
use anything that wasn't already available in Windows 95,
so the binaries should run even on Windows 95 if someone
happens to care.
Lasse Collin [Mon, 9 Sep 2013 17:37:03 +0000 (20:37 +0300)]
Build: Create liblzma.pc in a src/liblzma/Makefile.am.
Previously it was done in configure, but doing that goes
against the Autoconf manual. Autoconf requires that it is
possible to override e.g. prefix after running configure
and that doesn't work correctly if liblzma.pc is created
by configure.
A potential downside of this change is that now e.g.
libdir in liblzma.pc is a standalone string instead of
being defined via ${prefix}, so if one overrides prefix
when running pkg-config the libdir won't get the new value.
I don't know if this matters in practice.
Lasse Collin [Thu, 4 Jul 2013 11:18:46 +0000 (14:18 +0300)]
xz: Add preliminary support for --flush-timeout=TIMEOUT.
When --flush-timeout=TIMEOUT is used, xz will use
LZMA_SYNC_FLUSH if read() would block and at least
TIMEOUT milliseconds has elapsed since the previous flush.
This can be useful in realtime-like use cases where the
data is simultanously decompressed by another process
(possibly on a different computer). If new uncompressed
input data is produced slowly, without this option xz could
buffer the data for a long time until it would become
decompressible from the output.
If TIMEOUT is 0, the feature is disabled. This is the default.
This commit affects the compression side. Using xz for
the decompression side for the above purpose doesn't work
yet so well because there is quite a bit of input and
output buffering when decompressing.
The --long-help or man page were not updated yet.
The details of this feature may change.
Lasse Collin [Thu, 4 Jul 2013 10:25:11 +0000 (13:25 +0300)]
xz: Fix the test when to read more input.
Testing for end of file was no longer correct after full flushing
became possible with --block-size=SIZE and --block-list=SIZES.
There was no bug in practice though because xz just made a few
unneeded zero-byte reads.
Lasse Collin [Sun, 30 Jun 2013 15:02:27 +0000 (18:02 +0300)]
Man pages: Use similar syntax for synopsis as in xz.
The man pages of lzmainfo, xzmore, and xzdec had similar
constructs as the man page of xz had before the commit eb6ca9854b8eb9fbf72497c1cf608d6b19d2d494. Eric S. Raymond
didn't mention these man pages in his bug report, but
it's nice to be consistent.
Lasse Collin [Sat, 29 Jun 2013 12:59:13 +0000 (15:59 +0300)]
xz: Use non-blocking I/O for the output file.
Now both reading and writing should be without
race conditions with signals.
They might still be signal handling issues left.
Signals are blocked during many operations to avoid
EINTR but it may cause problems e.g. if writing to
stderr blocks when trying to display an error message.
Lasse Collin [Fri, 28 Jun 2013 20:48:05 +0000 (23:48 +0300)]
xz: Use the self-pipe trick to avoid a race condition with signals.
It is possible that a signal to set user_abort arrives right
before a blocking system call is made. In this case the call
may block until another signal arrives, while the wanted
behavior is to make xz clean up and exit as soon as possible.
After this commit, the race condition is avoided with the
input side which already uses non-blocking I/O. The output
side still uses blocking I/O and thus has the race condition.
Lasse Collin [Fri, 28 Jun 2013 15:09:47 +0000 (18:09 +0300)]
xz: Fix error detection of fcntl(fd, F_SETFL, flags) calls.
POSIX says that fcntl(fd, F_SETFL, flags) returns -1 on
error and "other than -1" on success. This is how it is
documented e.g. on OpenBSD too. On Linux, success with
F_SETFL is always 0 (at least accorinding to fcntl(2)
from man-pages 3.51).
Lasse Collin [Fri, 28 Jun 2013 14:36:47 +0000 (17:36 +0300)]
xz: Fix use of wrong variable in a fcntl() call.
Due to a wrong variable name, when writing a sparse file
to standard output, *all* file status flags were cleared
(to the extent the operating system allowed it) instead of
only clearing the O_APPEND flag. In practice this worked
fine in the common situations on GNU/Linux, but I didn't
check how it behaved elsewhere.
The original flags were still restored correctly. I still
changed the code to use a separate boolean variable to
indicate when the flags should be restored instead of
relying on a special value in stdout_flags.
Lasse Collin [Fri, 28 Jun 2013 11:55:37 +0000 (14:55 +0300)]
xz: Fix assertion related to posix_fadvise().
Input file can be a FIFO or something else that doesn't
support posix_fadvise() so don't check the return value
even with an assertion. Nothing bad happens if the call
to posix_fadvise() fails.
Lasse Collin [Wed, 26 Jun 2013 10:30:57 +0000 (13:30 +0300)]
xz: Check the value of lzma_stream_flags.version in --list.
It is a no-op for now, but if an old xz version is used
together with a newer liblzma that supports something new,
then this check becomes important and will stop the old xz
from trying to parse files that it won't understand.
Lasse Collin [Wed, 26 Jun 2013 09:17:00 +0000 (12:17 +0300)]
Build: Require Automake 1.12 and use serial-tests option.
It should actually still work with Automake 1.10 if
the serial-tests option is removed. Automake 1.13 started
using parallel tests by default and the option to get
the old behavior isn't supported before 1.12.
At least for now, parallel tests don't improve anything
in XZ Utils but they hide the progress output from
test_compress.sh.
Lasse Collin [Sun, 23 Jun 2013 14:36:47 +0000 (17:36 +0300)]
xz: Validate Uncompressed Size from Block Header in list.c.
This affects only "xz -lvv". Normal decompression with xz
already detected if Block Header and Index had mismatched
Uncompressed Size fields. So this just makes "xz -lvv"
show such files as corrupt instead of showing the
Uncompressed Size from Index.
Lasse Collin [Fri, 21 Jun 2013 18:50:26 +0000 (21:50 +0300)]
xz: Fix interaction between preset and custom filter chains.
There was somewhat illogical behavior when --extreme was
specified and mixed with custom filter chains.
Before this commit, "xz -9 --lzma2 -e" was equivalent
to "xz --lzma2". After it is equivalent to "xz -6e"
(all earlier preset options get forgotten when a custom
filter chain is specified and the default preset is 6
to which -e is applied). I find this less illogical.
This also affects the meaning of "xz -9e --lzma2 -7".
Earlier it was equivalent to "xz -7e" (the -e specified
before a custom filter chain wasn't forgotten). Now it
is "xz -7". Note that "xz -7e" still is the same as "xz -e7".
Hopefully very few cared about this in the first place,
so pretty much no one should even notice this change.
Lasse Collin [Sat, 27 Apr 2013 19:07:46 +0000 (22:07 +0300)]
Build: Use -Wvla with GCC if supported.
Variable-length arrays are mandatory in C99 but optional in C11.
The code doesn't currently use any VLAs and it shouldn't in the
future either to stay compatible with C11 without requiring any
optional C11 features.
Lasse Collin [Mon, 15 Apr 2013 16:29:09 +0000 (19:29 +0300)]
xzdec: Improve the --help message.
The options are now ordered in the same order as in xz's help
message.
Descriptions were added to the options that are ignored.
I left them in parenthesis even if it looks a bit weird
because I find it easier to spot the ignored vs. non-ignored
options from the list that way.
Lasse Collin [Sat, 23 Mar 2013 20:25:15 +0000 (22:25 +0200)]
liblzma: Be less picky in lzma_alone_decoder().
To avoid false positives when detecting .lzma files,
rare values in dictionary size and uncompressed size fields
were rejected. They will still be rejected if .lzma files
are decoded with lzma_auto_decoder(), but when using
lzma_alone_decoder() directly, such files will now be accepted.
Hopefully this is an OK compromise.
This doesn't affect xz because xz still has its own file
format detection code. This does affect lzmadec though.
So after this commit lzmadec will accept files that xz or
xz-emulating-lzma doesn't.
NOTE: lzma_alone_decoder() still won't decode all .lzma files
because liblzma's LZMA decoder doesn't support lc + lp > 4.
Lasse Collin [Sat, 23 Mar 2013 19:51:38 +0000 (21:51 +0200)]
liblzma: Fix another deadlock in the threaded encoder.
This race condition could cause a deadlock if lzma_end() was
called before finishing the encoding. This can happen with
xz with debugging enabled (non-debugging version doesn't
call lzma_end() before exiting).
Lasse Collin [Tue, 5 Mar 2013 17:14:50 +0000 (19:14 +0200)]
Avoid unneeded use of awk in xzless.
Use "read" instead of "awk" in xzless to get the version
number of "less". The need for awk was introduced in
the commit db5c1817fabf7cbb9e4087b1576eb26f0747338e.
Lasse Collin [Fri, 14 Dec 2012 18:13:32 +0000 (20:13 +0200)]
Make the progress indicator smooth in threaded mode.
This adds lzma_get_progress() to liblzma and takes advantage
of it in xz.
lzma_get_progress() collects progress information from
the thread-specific structures so that fairly accurate
progress information is available to applications. Adding
a new function seemed to be a better way than making the
information directly available in lzma_stream (like total_in
and total_out are) because collecting the information requires
locking mutexes. It's waste of time to do it more often than
the up to date information is actually needed by an application.
Jonathan Nieder [Mon, 19 Nov 2012 08:10:10 +0000 (00:10 -0800)]
xzless: Make "less -V" parsing more robust
In v4.999.9beta~30 (xzless: Support compressed standard input,
2009-08-09), xzless learned to parse ‘less -V’ output to figure out
whether less is new enough to handle $LESSOPEN settings starting
with “|-”. That worked well for a while, but the version string from
‘less’ versions 448 (June, 2012) is misparsed, producing a warning:
More precisely, modern ‘less’ lists the regexp implementation along
with its version number, and xzless passes the entire version number
with attached parenthetical phrase as a number to "test $a -gt $b",
producing the above confusing message.
$ less-444 -V | head -1
less 444
$ less -V | head -1
less 456 (no regular expressions)
So relax the pattern matched --- instead of expecting "less <number>",
look for a line of the form "less <number>[ (extra parenthetical)]".
While at it, improve the behavior when no matching line is found ---
instead of producing a cryptic message, we can fall back on a LESSPIPE
setting that is supported by all versions of ‘less’.
The implementation uses "awk" for simplicity. Hopefully that’s
portable enough.
Reported-by: Jörg-Volker Peetz <jvpeetz@web.de> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>