Tom Lane [Thu, 22 Sep 2011 15:29:18 +0000 (11:29 -0400)]
Make EXPLAIN ANALYZE report the numbers of rows rejected by filter steps.
This provides information about the numbers of tuples that were visited
but not returned by table scans, as well as the numbers of join tuples
that were considered and discarded within a join plan node.
There is still some discussion going on about the best way to report counts
for outer-join situations, but I think most of what's in the patch would
not change if we revise that, so I'm going to go ahead and commit it as-is.
Documentation changes to follow (they weren't in the submitted patch
either).
Marko Tiikkaja, reviewed by Marc Cousin, somewhat revised by Tom
Robert Haas [Wed, 21 Sep 2011 14:32:30 +0000 (10:32 -0400)]
Fix another bit of unlogged-table-induced breakage.
Per bug #6205, reported by Abel Abraham Camarillo Ojeda. This isn't a
particularly elegant fix, but I'm trying to minimize the chances of
causing yet another round of breakage.
Tom Lane [Tue, 20 Sep 2011 17:23:40 +0000 (13:23 -0400)]
Improve reporting of newlocale() failures in CREATE COLLATION.
The standardized errno code for "no such locale" failures is ENOENT, which
we were just reporting at face value, viz "No such file or directory".
Per gripe from Thom Brown, this might confuse users, so add an errdetail
message to clarify what it means. Also, report newlocale() failures as
ERRCODE_INVALID_PARAMETER_VALUE rather than using
errcode_for_file_access(), since newlocale()'s errno values aren't
necessarily tied directly to file access failures.
Tom Lane [Mon, 19 Sep 2011 03:46:04 +0000 (23:46 -0400)]
Fix another Assert issue exposed by CLOBBER_CACHE_ALWAYS.
plpgsql's exec_stmt_execsql was Assert'ing that a CachedPlanSource was
is_valid immediately after exec_prepare_plan. The risk factor in this case
is that after building the prepared statement, exec_prepare_plan calls
exec_simple_check_plan, which might try to generate a generic plan --- and
with CLOBBER_CACHE_ALWAYS or other unusual causes of invalidation, that
could result in an invalidation. However, that path could only be taken
for a SELECT query, for which we need not set mod_stmt. So in this case
I think it's best to just remove the Assert; it's okay to look at a
slightly-stale querytree for what we need here. Per buildfarm testing.
Tom Lane [Sat, 17 Sep 2011 05:47:33 +0000 (01:47 -0400)]
Fix Assert failure in new plancache code.
The regression tests were failing with CLOBBER_CACHE_ALWAYS enabled,
as reported by buildfarm member jaguar. There was an Assert in
BuildCachedPlan that asserted that the CachedPlanSource hadn't been
invalidated since we called RevalidateCachedQuery, which in theory can't
happen because we are holding locks on all the relevant database objects.
However, CLOBBER_CACHE_ALWAYS generates a false positive by making an
invalidation happen anyway; and on reflection, that could also occur as a
result of a badly-timed sinval reset due to queue overflow. We could just
remove the Assert and forge ahead with the not-really-stale querytree, but
it seems safer to do another RevalidateCachedQuery call just to make real
sure everything's OK.
Tom Lane [Fri, 16 Sep 2011 22:25:27 +0000 (18:25 -0400)]
Remove debug logging for pgstat wait timeout.
This reverts commit 79b2ee20c8a041a85dd230c4e787bef22edae57b, which proved
to not be very informative; it looks like the "pgstat wait timeout"
warnings in the buildfarm are just a symptom of running on heavily loaded
machines, and there isn't any weird mechanism causing them to appear.
To try to reduce the frequency of buildfarm failures from this effect,
increase PGSTAT_MAX_WAIT_TIME from 5 seconds to 10.
Also, arrange to not send a fresh inquiry message every single time through
the loop, as that seems more likely to cause problems (by swamping the
collector) than fix them. We'll now send an inquiry the first time through
the delay loop, and every 640 msec thereafter.
Tom Lane [Fri, 16 Sep 2011 18:47:20 +0000 (14:47 -0400)]
Avoid unnecessary page-level SSI lock check in heap_insert().
As observed by Heikki, we need not conflict on heap page locks during an
insert; heap page locks are only aggregated tuple locks, they don't imply
locking "gaps" as index page locks do. So we can avoid some unnecessary
conflicts, and also do the SSI check while not holding exclusive lock on
the target buffer.
Kevin Grittner, reviewed by Jeff Davis. Back-patch to 9.1.
Tom Lane [Fri, 16 Sep 2011 16:31:23 +0000 (12:31 -0400)]
Ensure generic plan gets used for a plpgsql expression with no parameters.
Now that a NULL ParamListInfo pointer causes significantly different
behavior in plancache.c, be sure to pass it that way when the expression
is known not to reference any plpgsql variables. Saves a few setup
cycles anyway.
Tom Lane [Fri, 16 Sep 2011 08:27:49 +0000 (04:27 -0400)]
gistendscan() forgot to free so->giststate.
This oversight led to a massive memory leak --- upwards of 10KB per tuple
--- during creation-time verification of an exclusion constraint based on a
GIST index. In most other scenarios it'd just be a leak of 10KB that would
be recovered at end of query, so not too significant; though perhaps the
leak would be noticeable in a situation where a GIST index was being used
in a nestloop inner indexscan. In any case, it's a real leak of long
standing, so patch all supported branches. Per report from Harald Fuchs.
Tom Lane [Fri, 16 Sep 2011 04:42:53 +0000 (00:42 -0400)]
Redesign the plancache mechanism for more flexibility and efficiency.
Rewrite plancache.c so that a "cached plan" (which is rather a misnomer
at this point) can support generation of custom, parameter-value-dependent
plans, and can make an intelligent choice between using custom plans and
the traditional generic-plan approach. The specific choice algorithm
implemented here can probably be improved in future, but this commit is
all about getting the mechanism in place, not the policy.
In addition, restructure the API to greatly reduce the amount of extraneous
data copying needed. The main compromise needed to make that possible was
to split the initial creation of a CachedPlanSource into two steps. It's
worth noting in particular that SPI_saveplan is now deprecated in favor of
SPI_keepplan, which accomplishes the same end result with zero data
copying, and no need to then spend even more cycles throwing away the
original SPIPlan. The risk of long-term memory leaks while manipulating
SPIPlans has also been greatly reduced. Most of this improvement is based
on use of the recently-added MemoryContextSetParent primitive.
Teach the makefile used to build stand-alone libpq on Windows that libpq
needs win32setlocale.c now. The cygwin and MSVC build scripts were changed
earlier, but this was neglected. This should fix bug report #6203 by Steve.
In the final emptying phase of the new GiST buffering build, set the
queuedForEmptying flag correctly on buffer when adding it to the queue.
Also, don't add buffer to the queue if it's there already. These were
harmless oversights; failing to set the flag just means that a buffer might
get added to the queue twice if more tuples are added to it (although that
can't actually happen at this point because all the upper buffers have
already been emptied), and having the same buffer twice in the emptying
queue is harmless. But better be tidy.
Tom Lane [Sun, 11 Sep 2011 20:29:42 +0000 (16:29 -0400)]
Invent a new memory context primitive, MemoryContextSetParent.
This function will be useful for altering the lifespan of a context after
creation (for example, by creating it under a transient context and later
reparenting it to belong to a long-lived context). It costs almost no new
code, since we can refactor what was there. Per my proposal of yesterday.
Peter Eisentraut [Sun, 11 Sep 2011 18:54:32 +0000 (21:54 +0300)]
Remove many -Wcast-qual warnings
This addresses only those cases that are easy to fix by adding or
moving a const qualifier or removing an unnecessary cast. There are
many more complicated cases remaining.
Peter Eisentraut [Sat, 10 Sep 2011 20:12:46 +0000 (23:12 +0300)]
Add missing format attributes
Add __attribute__ decorations for printf format checking to the places that
were missing them. Fix the resulting warnings. Add
-Wmissing-format-attribute to the standard set of warnings for GCC, so these
don't happen again.
The warning fixes here are relatively harmless. The one serious problem
discovered by this was already committed earlier in cf15fb5cabfbc71e07be23cfbc813daee6c5014f.
Tom Lane [Fri, 9 Sep 2011 21:59:11 +0000 (17:59 -0400)]
Simplify handling of the timezone GUC by making initdb choose the default.
We were doing some amazingly complicated things in order to avoid running
the very expensive identify_system_timezone() procedure during GUC
initialization. But there is an obvious fix for that, which is to do it
once during initdb and have initdb install the system-specific default into
postgresql.conf, as it already does for most other GUC variables that need
system-environment-dependent defaults. This means that the timezone (and
log_timezone) settings no longer have any magic behavior in the server.
Per discussion.
Tom Lane [Fri, 9 Sep 2011 17:23:41 +0000 (13:23 -0400)]
Move Timestamp/Interval typedefs and basic macros into datatype/timestamp.h.
As per my recent proposal, this refactors things so that these typedefs and
macros are available in a header that can be included in frontend-ish code.
I also changed various headers that were undesirably including
utils/timestamp.h to include datatype/timestamp.h instead. Unsurprisingly,
this showed that half the system was getting utils/timestamp.h by way of
xlog.h.
No actual code changes here, just header refactoring.
When building a GiST index that doesn't fit in cache, buffers are attached
to some internal nodes in the index. This speeds up the build by avoiding
random I/O that would otherwise be needed to traverse all the way down the
tree to the find right leaf page for tuple.
Tom Lane [Wed, 7 Sep 2011 21:06:10 +0000 (17:06 -0400)]
Fix corner case bug in numeric to_char().
Trailing-zero stripping applied by the FM specifier could strip zeroes
to the left of the decimal point, for a format with no digit positions
after the decimal point (such as "FM999.").
Reported and diagnosed by Marti Raudsepp, though I didn't use his patch.
Tom Lane [Wed, 7 Sep 2011 17:01:15 +0000 (13:01 -0400)]
Fix get_name_for_var_field() to deal with RECORD Params.
With 9.1's use of Params to pass down values from NestLoop join nodes
to their inner plans, it is possible for a Param to have type RECORD, in
which case the set of fields comprising the value isn't determinable by
inspection of the Param alone. However, just as with a Var of type RECORD,
we can find out what we need to know if we can locate the expression that
the Param represents. We already knew how to do this in get_parameter(),
but I'd overlooked the need to be able to cope in get_name_for_var_field(),
which led to EXPLAIN failing with "record type has not been registered".
To fix, refactor the search code in get_parameter() so it can be used by
both functions.
Bruce Momjian [Wed, 7 Sep 2011 13:47:51 +0000 (09:47 -0400)]
Fix to_date() and to_timestamp() to handle year masks of length < 4 so
they wrap toward year 2020, rather than the inconsistent behavior we had
before.
Simon Riggs [Wed, 7 Sep 2011 11:11:26 +0000 (12:11 +0100)]
Partially revoke attempt to improve performance with many savepoints.
Maintain difference between subtransaction release and commit introduced
by earlier patch.
Simon Riggs [Wed, 7 Sep 2011 08:09:47 +0000 (09:09 +0100)]
Emit cascaded standby message on shutdown only when appropriate.
Adds additional test for active walsenders and closes a race
condition for when we failover when a new walsender was connecting.
Reported and fixed bu Fujii Masao. Review by Heikki Linnakangas
Tom Lane [Tue, 6 Sep 2011 18:50:28 +0000 (14:50 -0400)]
Avoid possibly accessing off the end of memory in SJIS2004 conversion.
The code in shift_jis_20042euc_jis_2004() would fetch two bytes even when
only one remained in the string. Since conversion functions aren't
supposed to assume null-terminated input, this poses a small risk of
fetching past the end of memory and incurring SIGSEGV. No such crash has
been identified in the field, but we've certainly seen the equivalent
happen in other code paths, so patch this one all the way back.
Tom Lane [Tue, 6 Sep 2011 18:35:19 +0000 (14:35 -0400)]
Avoid possibly accessing off the end of memory in examine_attribute().
Since the last couple of columns of pg_type are often NULL,
sizeof(FormData_pg_type) can be an overestimate of the actual size of the
tuple data part. Therefore memcpy'ing that much out of the catalog cache,
as analyze.c was doing, poses a small risk of copying past the end of
memory and incurring SIGSEGV. No such crash has been identified in the
field, but we've certainly seen the equivalent happen in other code paths,
so patch this one all the way back.
Per valgrind testing by Noah Misch, though this is not his proposed patch.
I chose to use SearchSysCacheCopy1 rather than inventing special-purpose
infrastructure for copying only the minimal part of a pg_type tuple.
Tom Lane [Tue, 6 Sep 2011 16:36:40 +0000 (12:36 -0400)]
Add an "incompatibility" entry to 9.1 release notes about CREATE EXTENSION.
We've now seen more than one gripe from somebody who didn't get the memo
about how to install contrib modules in 9.1. Try to make it a little more
prominent that you aren't supposed to call the scripts directly anymore.
Tom Lane [Tue, 6 Sep 2011 16:14:51 +0000 (12:14 -0400)]
Update type-conversion documentation for long-ago changes.
This example wasn't updated when we changed the behavior of bpcharlen()
in 8.0, nor when we changed the number of parameters taken by the bpchar()
cast function in 7.3. Per report from lsliang.
Tom Lane [Mon, 5 Sep 2011 00:07:34 +0000 (20:07 -0400)]
Guard against using plperl's Makefile without specifying --with-perl.
The $(PERL) macro will be set by configure if it finds perl at all,
but $(perl_privlibexp) isn't configured unless you said --with-perl.
This results in confusing error messages if someone cd's into
src/pl/plperl and tries to build there despite the configure omission,
as reported by Tomas Vondra in bug #6198. Add simple checks to
provide a more useful report, while not disabling other use of the
makefile such as "make clean".
Back-patch to 9.0, which is as far as the patch applies easily.
Tom Lane [Sun, 4 Sep 2011 19:13:46 +0000 (15:13 -0400)]
Dig down into sub-selects to look for column statistics.
If a sub-select's output column is a simple Var, recursively look for
statistics applying to that Var, and use them if available. The need for
this was foreseen ages ago, but we didn't have enough infrastructure to do
it with reasonable speed until just now.
We punt and stick with default estimates if the subquery uses set
operations, GROUP BY, or DISTINCT, since those operations would change the
underlying column statistics (particularly, the relative frequencies of
different values) beyond recognition. This means that the types of
sub-selects for which this improvement applies are fairly limited, since
most subqueries satisfying those restrictions would have gotten flattened
into the parent query anyway. But it does help for some cases, such as
subqueries with ORDER BY or LIMIT.
Tom Lane [Sun, 4 Sep 2011 18:43:52 +0000 (14:43 -0400)]
Can't print PlannerGlobal's subroots list in outfuncs.
Since the subroots will surely link back to the same glob struct, this
necessarily leads to infinite recursion. Doh. Found while trying to
debug some other code.
Tom Lane [Sun, 4 Sep 2011 05:13:16 +0000 (01:13 -0400)]
Clean up the #include mess a little.
walsender.h should depend on xlog.h, not vice versa. (Actually, the
inclusion was circular until a couple hours ago, which was even sillier;
but Bruce broke it in the expedient rather than logically correct
direction.) Because of that poor decision, plus blind application of
pgrminclude, we had a situation where half the system was depending on
xlog.h to include such unrelated stuff as array.h and guc.h. Clean up
the header inclusion, and manually revert a lot of what pgrminclude had
done so things build again.
This episode reinforces my feeling that pgrminclude should not be run
without adult supervision. Inclusion changes in header files in particular
need to be reviewed with great care. More generally, it'd be good if we
had a clearer notion of module layering to dictate which headers can sanely
include which others ... but that's a big task for another day.
Tom Lane [Sun, 4 Sep 2011 02:14:45 +0000 (22:14 -0400)]
Remove unnecessary and circular #include.
storage/proc.h should not include replication/syncrep.h, especially not
when the latter includes storage/proc.h; but in any case this was a pretty
poor thing from a modular layering standpoint.
Tom Lane [Sat, 3 Sep 2011 20:17:34 +0000 (16:17 -0400)]
Fix typo in pg_srand48 (srand48 in older branches).
">" should be ">>". This typo results in failure to use all of the bits
of the provided seed.
This might rise to the level of a security bug if we were relying on
srand48 for any security-critical purposes, but we are not --- in fact,
it's not used at all unless the platform lacks srandom(), which is
improbable. Even on such a platform the exposure seems minimal.
Tom Lane [Sat, 3 Sep 2011 19:35:12 +0000 (15:35 -0400)]
Rearrange planner to save the whole PlannerInfo (subroot) for a subquery.
Formerly, set_subquery_pathlist and other creators of plans for subqueries
saved only the rangetable and rowMarks lists from the lower-level
PlannerInfo. But there's no reason not to remember the whole PlannerInfo,
and indeed this turns out to simplify matters in a number of places.
The immediate reason for doing this was so that the subroot will still be
accessible when we're trying to extract column statistics out of an
already-planned subquery. But now that I've done it, it seems like a good
code-beautification effort in its own right.
I also chose to get rid of the transient subrtable and subrowmark fields in
SubqueryScan nodes, in favor of having setrefs.c look up the subquery's
RelOptInfo. That required changing all the APIs in setrefs.c to pass
PlannerInfo not PlannerGlobal, which was a large but quite mechanical
transformation.
One side-effect not foreseen at the beginning is that this finally broke
inheritance_planner's assumption that replanning the same subquery RTE N
times would necessarily give interchangeable results each time. That
assumption was always pretty risky, but now we really have to make a
separate RTE for each instance so that there's a place to carry the
separate subroots.
Tom Lane [Fri, 2 Sep 2011 18:29:31 +0000 (14:29 -0400)]
Teach ANALYZE to clear pg_class.relhassubclass when appropriate.
In the past, relhassubclass always remained true if a relation had ever had
child relations, even if the last subclass was long gone. While this had
only marginal performance implications in most cases, it was annoying, and
I'm now considering some planner changes that would raise the cost of a
false positive. It was previously impractical to fix this because of race
condition concerns. However, given the recent change that made tablecmds.c
take ShareExclusiveLock on relations that are gaining a child (commit fbcf4b92aa64d4577bcf25925b055316b978744a), we can now allow ANALYZE to
clear the flag when it's no longer relevant. There is no additional
locking cost to do so, since ANALYZE takes ShareExclusiveLock anyway.
libpq compiles various pgport files like ecpg does, and needs similar Makefile
changes for the win32 setlocale() wrapper I put into ecpg, to make it compile
on MinGW.
Robert Haas [Thu, 1 Sep 2011 13:21:10 +0000 (09:21 -0400)]
Minor improvements to mbregress.sh script.
1. Use new dropdb --if-exists option, to avoid alarming the user if
the database being dropped doesn't already exist.
2. Bail out if createdb fails.
3. exit 1 if the checks fail.
4. Make it executable.
Robert Haas [Thu, 1 Sep 2011 12:28:26 +0000 (08:28 -0400)]
Fix "is db labeled test?" in chkselinuxenv script.
Don't test whether the number of labels is numerically equal to zero;
count(*) isn't going return zero anyway, and the current coding blows
up if it returns an empty string or an error.
Fix MinGW build, broken by my previous patch to add a setlocale() wrapper
on Windows. ecpglib doesn't link with libpgport, but picks and compiles
the .c files it needs individually. To cope with that, move the setlocale()
wrapper from chklocale.c to a separate setlocale.c file, and include that
in ecpglib.