1 <section> <date> 13. Aug 2003 </date>
2 <h2> 64on32 largefile information </h2>
5 <h3> largefile problems </h3>
8 Through actual problems with handling files larger than 2 Gigabyte,
9 I had to face the fact that there are serious problems with the
10 largefile implementation around shared-libraries. This is bound
11 to the effect that 64on32 platforms allow a preprocessor #define
12 _FILE_OFFSET_BITS=64 which will shift the integretal type of
13 off_t from a 32bit to a 64bit entity.
17 That will in fact lead to problems if an application is compiled
18 with a different off_t size than the shared-library it will be
19 linked to. Among the problems are different sizes of the callframe
20 for those functions that take an argument of off_t type. Note how
21 the "seek" call uses an off_t in the middle of its arguments which
22 may be 32bit or 64bit depending on a preprocessor define. And
23 you know that zziplib wraps up "seek"-like calls as well.
27 The observations were largely made independent of the zziplib
28 however, as a zip-file does uses 32bit offsets in its header
29 fields anyway, and therefore a single zip-archive (or any of
30 its wrapped files) can not be larger than 2 Gigabyte anyway.
31 I would to refer you for deeper information to my website
32 about this problem-space at
35 <a href="http://ac-archive.sf.net/largefile" remap="url">
36 ac-archive.sf.net/largefile </a>
40 <h3> zziplib related </h3>
43 Still however the problems hold for zziplib usage when
44 the functions are linked dynamically as a shared-library.
45 Here we can face the fact that an application (or a higher
46 level library) uses a different off_t size than the
47 underlying zziplib shared-library.
51 If you read the <code>zzip/zzip.h</code> header file then
52 you will surely see a number of off_t usages around, here
53 they are wrapped in the form of zzip_off_t to get away with
54 platforms not predefining off_t in the first place. Those
55 functions are (at the time of writing):
57 <li> <code>zzip_telldir</code> (return type) </li>
58 <li> <code>zzip_seekdir</code> (second param) </li>
59 <li> <code>zzip_tell</code> (return type) </li>
60 <li> <code>zzip_seek</code> (second param of three)</li>
65 What might be not as obvious however: you will find
66 also the off_t type being used in the plugin-handlers
67 callback functions. That is based on the fact that
68 the plugin-structures is filled by default with the
69 posix-functions from the C library. A 64on32 platform
70 however offers quite usually a mixedmode C library
71 exporting two symbols for tell/seek calls to match
72 either 32bit off_t or 64bit off_t
74 <li> <code>zzip_plugin_io->seeks</code> (return type and second param) </li>
75 <li> <code>zzip_plugin_io->filesize</code> (return type) </li>
80 The problem here: the application might not make use of
81 zzip_seek/zzip_tell explicitly, but may be the internal
82 implementation of a zzip call uses io->seeks or io->filesize.
83 When an application uses plugin-io with these callbacks
84 overridden then surely problems will arise.
88 <h3> zziplib mixedmode option </h3>
91 I have extended the zziplib implementation to allow itself to
92 live fine on 64on32 systems. The 64on32 system are like linux
93 and solaris where the default off_t is 32bit and only by the
94 preprocessor hint they shift into 64bit. The C library on
95 these systems is a mixedmode one offering a pair for each of
96 the problematic functions - lseek <em>and</em> lseek64 for
101 The zziplib header file detects when it is present on a
102 64on32 system (through hints in configured zzip/conf.h)
103 and that _FILE_OFFSET_BITS has been set to 64bit. In that
104 case it does automatically issue #defines that shift the
105 symbol-name from zzip_seek into zzip_seek64. Likewise,
106 <em>all</em> the *_ext_io functions are renamed into
111 The zziplib library itself will also pick up the renamings
112 when it is compiled with 64bit off_t - in effect an application
113 with a 64bit-off_t dependency can only link with a zziplib
114 compiled in 64bit-off_t mode. If the application does not
115 use any call symbol with an off_t dependency then it does
116 not matter and the link will succeed. That's simply because
117 function calls without an off_t dependency will not be
118 renamed and they are the same for a 32bit-off_t zziplib or
119 a 64bit-off_t zziplib.
123 As an extra, the zziplib exports a few of its common calls
124 like being a mixedmode library when you compile it both in
125 64bit mode and as a shared library. In that case, the
126 resulting shared library will export symbol pairs for the
127 calls with an off_t dependency, i.e. both zzip_seek and
128 zzip_seek64 are present.
132 Note that for reasons of being a lightweight library, the
133 zziplib library does not export mixmode call pairs for
134 the *_ext_io family of functions. The current generation
135 of zziplib does call io->seeks unconditionally of any
136 case'ing flag and so far there are no problems with the
141 <h3> Implementation details </h3>
144 In the header file zzip/zzip.h you will find the define
145 ZZIP_LARGEFILE_RENAME which triggers the renaming process.
146 See zzip/conf.h on the conditions where it is being triggered.
150 For the implementation of the mixedmode symbol pairs, see
151 zzip/dir.c for an example for the zzip_seekdir/zzip_seekdir64
152 pair - here we use libtools -DPIC to detect the situation of
153 being compiled as shared-library, we use the preprocessor
154 #def ZZIP_LARGEFILE_RENAME to know we are on a 64on32
155 system compiled in 64bit-off_t, and we check the transitional
156 largefile API to be present by looking for EOVERFLOW errno.
160 When all the three are present then we simply #undef the
161 renaming preprocessor macro and define a function symbol
162 (without the renaming) and call the renamed symbol
163 already compiled a few lines before. We use the pre-off_t
164 type "long" for the 32bit entity of these calls. While
165 we mostly let the compiler do the shrink/expand of these
166 integer types, we do also sometimes check for overflows
171 <h3> rpm extras and pkg-config </h3>
174 The provided .spec file shows how to compile both variants
175 of the zziplib shared library and to install them in
176 parallel in the system. Also we provide doubled sets
177 of .pc files for pkg-config installation. That should make
178 it a lot easier for applications to link to the correct
183 Here are all the variants that you can find after installing
184 the vanilla rpm files from zziplib.sf.net:
186 $ pkg-config --list-all | sort | grep zzip
187 zziplib32 zziplib32 - ZZipLib - libZ-based ZIP-access Library
188 zziplib64 zziplib64 - ZZipLib - libZ-based ZIP-access Library
189 zziplib zziplib - ZZipLib - libZ-based ZIP-access Library
190 zzip-sdl-config zzip-sdl-config - SDL Config (for ZZipLib)
191 zzip-sdl-rwops zzip-sdl-rwops - SDL_rwops for ZZipLib
192 zzipwrap zzipwrap - Callback Wrappers for ZZipLib
193 zzip-zlib-config zzip-zlib-config - ZLib Config (for ZZipLib)