1 <!-- $Header: /cvsroot/pgsql/doc/src/sgml/regress.sgml,v 1.21 2001/09/21 18:37:05 petere Exp $ -->
4 <title id="regress-title">Regression Tests</title>
6 <sect1 id="regress-intro">
7 <title>Introduction</title>
10 The regression tests are a comprehensive set of tests for the SQL
11 implementation in <productname>PostgreSQL</productname>. They test
12 standard SQL operations as well as the extended capabilities of
13 <productname>PostgreSQL</productname>. The test suite was
14 originally developed by Jolly Chen and Andrew Yu, and was
15 extensively revised and repackaged by Marc Fournier and Thomas
16 Lockhart. From <productname>PostgreSQL</productname> 6.1 onward
17 the regression tests are current for every official release.
22 <sect1 id="regress-run">
23 <title>Running the Tests</title>
26 The regression test can be run against an already installed and
27 running server, or using a temporary installation within the build
28 tree. Furthermore, there is a <quote>parallel</quote> and a
29 <quote>sequential</quote> mode for running the tests. The
30 sequential method runs each test script in turn, whereas the
31 parallel method starts up multiple server processes to run groups
32 of tests in parallel. Parallel testing gives confidence that
33 interprocess communication and locking are working correctly. For
34 historical reasons, the sequential test is usually run against an
35 existing installation and the parallel method against a temporary
36 installation, but there are no technical reasons for this.
40 To run the regression tests after building but before installation,
43 <prompt>$ </prompt><userinput>gmake check</userinput>
45 in the top-level directory. (Or you can change to
46 <filename>src/test/regress</filename> and run the command there.)
47 This will first build several auxiliary files, such as
48 platform-dependent <quote>expected</quote> files and some sample
49 user-defined trigger functions, and then run the test driver
50 script. At the end you should see something like
53 ======================
55 ======================
58 or otherwise a note about what tests failed. See <xref
59 linkend="regress-evaluation"> below for more.
64 Because this test method runs a temporary server, it will not work
65 when you are the root user (the server will not start as root).
66 If you already did the build as root, you do not have to start all
67 over. Instead, make the regression test directory writable by
68 some other user, log in as that user, and restart the tests.
71 <prompt>root# </prompt><userinput>chmod -R a+w src/test/regress</userinput>
72 <prompt>root# </prompt><userinput>chmod -R a+w contrib/spi</userinput>
73 <prompt>root# </prompt><userinput>su - joeuser</userinput>
74 <prompt>joeuser$ </prompt><userinput>cd <build top-level directory></userinput>
75 <prompt>joeuser$ </prompt><userinput>gmake check</userinput>
77 (The only possible <quote>security risk</quote> here is that other
78 users might be able to alter the regression test results behind
79 your back. Use common sense when managing user permissions.)
82 Alternatively, run the tests after installation.
88 On some systems, the default Bourne-compatible shell
89 (<filename>/bin/sh</filename>) gets confused when it has to manage
90 too many child processes in parallel. This may cause the parallel
91 test run to lock up or fail. In such cases, specify a different
92 Bourne-compatible shell on the command line, for example:
95 <prompt>$ </prompt><userinput>gmake SHELL=/bin/ksh check</userinput>
102 To run the tests after installation<![%standalone-ignore;[ (see <xref linkend="installation">)]]>,
103 initialize a data area and start the
104 server, <![%standalone-ignore;[as explained in <xref linkend="runtime">, ]]> then type
106 <prompt>$ </prompt><userinput>gmake installcheck</userinput>
108 The tests will expect to contact the server at the local host and the
109 default port number, unless directed otherwise by <envar>PGHOST</envar> and <envar>PGPORT</envar>
110 environment variables.
114 <sect1 id="regress-evaluation">
115 <title>Test Evaluation</title>
118 Some properly installed and fully functional
119 <productname>PostgreSQL</productname> installations can
120 <quote>fail</quote> some of these regression tests due to
121 platform-specific artifacts such as varying floating point representation
122 and time zone support. The tests are currently evaluated using a simple
123 <application>diff</application> comparison against the outputs
124 generated on a reference system, so the results are sensitive to
125 small system differences. When a test is reported as
126 <quote>failed</quote>, always examine the differences between
127 expected and actual results; you may well find that the
128 differences are not significant. Nonetheless, we still strive to
129 maintain accurate reference files across all supported platforms,
130 so it can be expected that all tests pass.
134 The actual outputs of the regression tests are in files in the
135 <filename>src/test/regress/results</filename> directory. The test
136 script uses <application>diff</application> to compare each output
137 file against the reference outputs stored in the
138 <filename>src/test/regress/expected</filename> directory. Any
139 differences are saved for your inspection in
140 <filename>src/test/regress/regression.diffs</filename>. (Or you
141 can run <application>diff</application> yourself, if you prefer.)
145 <title>Error message differences</title>
148 Some of the regression tests involve intentional invalid input
149 values. Error messages can come from either the
150 <productname>PostgreSQL</productname> code or from the host
151 platform system routines. In the latter case, the messages may
152 vary between platforms, but should reflect similar
153 information. These differences in messages will result in a
154 <quote>failed</quote> regression test that can be validated by
160 <title>Locale differences</title>
163 The tests expect to run in plain <quote>C</quote> locale. This
164 should not cause any problems when you run the tests against a
165 temporary installation, since the regression test driver takes care
166 to start the server in C locale. However, if you run the tests
167 against an already-installed server that is using non-C locale settings,
168 you may see differences caused by varying rules for string sort order,
169 formatting of numeric and monetary values, and so forth.
173 In some locales the resulting differences are small and easily checked by
174 inspection. However, in a locale that changes the rules for formatting
175 of numeric values (typically by swapping the usage of commas and
176 decimal points), entry of some data values will fail, resulting in
177 extensive differences later in the tests where the missing data values
178 are supposed to be used.
183 <title>Date and time differences</title>
186 Some of the queries in the <filename>timestamp</filename> test will
187 fail if you run the test on the day of a daylight-savings time
188 changeover, or the day before or after one. These queries assume
189 that the intervals between midnight yesterday, midnight today and
190 midnight tomorrow are exactly twenty-four hours -- which is wrong
191 if daylight-savings time went into or out of effect meanwhile.
195 Most of the date and time results are dependent on the time zone
196 environment. The reference files are generated for time zone
197 <literal>PST8PDT</literal> (Berkeley, California) and there will be apparent
198 failures if the tests are not run with that time zone setting.
199 The regression test driver sets environment variable
200 <envar>PGTZ</envar> to <literal>PST8PDT</literal>, which normally
201 ensures proper results. However, your system must provide library
202 support for the <literal>PST8PDT</literal> time zone, or the time zone-dependent
203 tests will fail. To verify that your machine does have this
204 support, type the following:
206 <prompt>$ </prompt><userinput>env TZ=PST8PDT date</userinput>
208 The command above should have returned the current system time in
209 the <literal>PST8PDT</literal> time zone. If the <literal>PST8PDT</literal> database is not available,
210 then your system may have returned the time in GMT. If the
211 <literal>PST8PDT</literal> time zone is not available, you can set the time zone
214 PGTZ='PST8PDT7,M04.01.0,M10.05.03'; export PGTZ
219 There appear to be some systems that do not accept the
220 recommended syntax for explicitly setting the local time zone
221 rules; you may need to use a different <envar>PGTZ</envar>
222 setting on such machines.
226 Some systems using older time zone libraries fail to apply
227 daylight-savings corrections to dates before 1970, causing
228 pre-1970 <acronym>PDT</acronym> times to be displayed in <acronym>PST</acronym> instead. This will
229 result in localized differences in the test results.
234 <title>Floating point differences</title>
237 Some of the tests involve computing 64-bit (<type>double
238 precision</type>) numbers from table columns. Differences in
239 results involving mathematical functions of <type>double
240 precision</type> columns have been observed. The float8 and
241 geometry tests are particularly prone to small differences across
242 platforms, or even with different compiler optimization options.
243 Human eyeball comparison is needed to determine the real
244 significance of these differences which are usually 10 places to
245 the right of the decimal point.
249 Some systems signal errors from <function>pow()</function> and
250 <function>exp()</function> differently from the mechanism
251 expected by the current <productname>PostgreSQL</productname>
257 <title>Polygon differences</title>
260 Several of the tests involve operations on geographic data about
261 the Oakland/Berkeley, California street map. The map data is expressed as
262 polygons whose vertices are represented as pairs of <type>double
263 precision</type> numbers (decimal latitude and
264 longitude). Initially, some tables are created and loaded with
265 geographic data, then some views are created that join two
266 tables using the polygon intersection operator
267 (<literal>##</literal>), then a select is done on the view.
271 When comparing the results from different platforms, differences
272 occur in the 2nd or 3rd place to the right of the decimal
273 point. The SQL statements where these problems occur are the
276 SELECT * from street;
283 <title>Row ordering differences</title>
286 You might see differences in which the same rows are output in a
287 different order than what appears in the expected file. In most cases
288 this is not, strictly speaking, a bug. Most of the regression test
289 scripts are not so pedantic as to use an ORDER BY for every single
290 SELECT, and so their result row orderings are not well-defined
291 according to the letter of the SQL specification. In practice, since we are
292 looking at the same queries being executed on the same data by the same
293 software, we usually get the same result ordering on all platforms, and
294 so the lack of ORDER BY isn't a problem. Some queries do exhibit
295 cross-platform ordering differences, however. (Ordering differences
296 can also be triggered by non-C locale settings.)
300 Therefore, if you see an ordering difference, it's not something to
301 worry about, unless the query does have an ORDER BY that your result
302 is violating. But please report it anyway, so that we can add an
303 ORDER BY to that particular query and thereby eliminate the bogus
304 <quote>failure</quote> in future releases.
308 You might wonder why we don't order all the regress test queries explicitly to
309 get rid of this issue once and for all. The reason is that that would
310 make the regression tests less useful, not more, since they'd tend
311 to exercise query plan types that produce ordered results to the
312 exclusion of those that don't.
317 <title>The <quote>random</quote> test</title>
320 There is at least one case in the <quote>random</quote> test
321 script that is intended to produce random results. This causes
322 random to fail the regression test once in a while (perhaps once
323 in every five to ten trials). Typing
325 diff results/random.out expected/random.out
327 should produce only one or a few lines of differences. You need
328 not worry unless the random test always fails in repeated
329 attempts. (On the other hand, if the random test is
330 <emphasis>never</emphasis> reported to fail even in many trials
331 of the regression tests, you probably <emphasis>should</emphasis>
337 <![%standalone-ignore;[
338 <!-- We might want to move the following section into the developer's guide. -->
339 <sect1 id="regress-platform">
340 <title>Platform-specific comparison files</title>
343 Since some of the tests inherently produce platform-specific
344 results, we have provided a way to supply platform-specific result
345 comparison files. Frequently, the same variation applies to
346 multiple platforms; rather than supplying a separate comparison
347 file for every platform, there is a mapping file that defines
348 which comparison file to use. So, to eliminate bogus test
349 <quote>failures</quote> for a particular platform, you must choose
350 or make a variant result file, and then add a line to the mapping
351 file, which is <filename>resultmap</filename>.
355 Each line in the mapping file is of the form
357 testname/platformpattern=comparisonfilename
359 The test name is just the name of the particular regression test
360 module. The platform pattern is a pattern in the style of
361 <citerefentry><refentrytitle>expr</><manvolnum>1</></citerefentry> (that is, a regular expression with an implicit
362 <literal>^</literal> anchor
363 at the start). It is matched against the platform name as printed
364 by <filename>config.guess</filename> followed by
365 <literal>:gcc</literal> or <literal>:cc</literal>, depending on
366 whether you use the GNU compiler or the system's native compiler
367 (on systems where there is a difference). The comparison file
368 name is the name of the substitute result comparison file.
372 For example: some systems using older time zone libraries fail to apply
373 daylight-savings corrections to dates before 1970, causing
374 pre-1970 <acronym>PDT</acronym> times to be displayed in <acronym>PST</acronym> instead. This causes a
375 few differences in the <filename>horology</> regression test.
376 Therefore, we provide a variant comparison file,
377 <filename>horology-no-DST-before-1970.out</filename>, which includes
378 the results to be expected on these systems. To silence the bogus
379 <quote>failure</quote> message on <systemitem>HPPA</systemitem> platforms, <filename>resultmap</filename>
382 horology/hppa=horology-no-DST-before-1970
384 which will trigger on any machine for which the output of <command>config.guess</command>
385 begins with <quote><literal>hppa</literal></quote>. Other lines
386 in <filename>resultmap</> select the variant comparison file for other
387 platforms where it's appropriate.
395 <!-- Keep this comment at the end of the file
400 sgml-minimize-attributes:nil
401 sgml-always-quote-attributes:t
404 sgml-parent-document:nil
405 sgml-default-dtd-file:"./reference.ced"
406 sgml-exposed-tags:nil
407 sgml-local-catalogs:("/usr/lib/sgml/catalog")
408 sgml-local-ecat-files:nil