Robert Haas [Thu, 7 Oct 2010 16:19:03 +0000 (12:19 -0400)]
Improve WAL reliability documentation, and add more cross-references to it.
In particular, we are now more explicit about the fact that you may need
wal_sync_method=fsync_writethrough for crash-safety on some platforms,
including MaxOS X. There's also now an explicit caution against assuming
that the default setting of wal_sync_method is either crash-safe or best
for performance.
Tom Lane [Wed, 6 Oct 2010 23:31:05 +0000 (19:31 -0400)]
Reduce the memory requirement for large ispell dictionaries.
This patch eliminates per-chunk palloc overhead for most small allocations
needed in the representation of an ispell dictionary. This saves close to
a factor of 2 on the current Czech ispell data. While it doesn't cover
every last small allocation in the ispell code, we are at the point of
diminishing returns, because about 95% of the allocations are covered
already.
Tom Lane [Wed, 6 Oct 2010 19:15:15 +0000 (15:15 -0400)]
Clean up temporary-memory management during ispell dictionary loading.
Add explicit initialization and cleanup functions to spell.c, and keep
all working state in the already-existing ISpellDict struct. This lets us
get rid of a static variable along with some extremely shaky assumptions
about usage of child memory contexts.
This commit is just code beautification and has no impact on functionality
or performance, but it opens the way to a less-grotty implementation of
Pavel's memory-saving hack, which will follow shortly.
Simon Riggs [Tue, 5 Oct 2010 23:19:05 +0000 (00:19 +0100)]
Correct docs for behaviour of ALTER DATABASE .. RENAME during Hot Standby.
Actual behaviour did not match documented behaviour and we have agreed
that it should be the docs that change.
Tom Lane [Sun, 3 Oct 2010 00:02:27 +0000 (20:02 -0400)]
Behave correctly if INSERT ... VALUES is decorated with additional clauses.
In versions 8.2 and up, the grammar allows attaching ORDER BY, LIMIT,
FOR UPDATE, or WITH to VALUES, and hence to INSERT ... VALUES. But the
special-case code for VALUES in transformInsertStmt() wasn't expecting any
of those, and just ignored them, leading to unexpected results. Rather
than complicate the special-case path, just ensure that the presence of any
of those clauses makes us treat the query as if it had a general SELECT.
Per report from Hitoshi Harada.
Tom Lane [Sat, 2 Oct 2010 22:21:41 +0000 (18:21 -0400)]
Throw an appropriate error if ALTER COLUMN TYPE finds a dependent trigger.
Actually making this case work, if the column is used in the trigger's
WHEN condition, will take some new code that probably isn't appropriate
to back-patch. For now, just throw a FEATURE_NOT_SUPPORTED error rather
than allowing control to reach the "unexpected object" case. Per bug #5688
from Daniel Grace. Back-patch to 9.0 where the possibility of such a
dependency was introduced.
Tom Lane [Thu, 30 Sep 2010 21:18:51 +0000 (17:18 -0400)]
Use a separate interpreter for each calling SQL userid in plperl and pltcl.
There are numerous methods by which a Perl or Tcl function can subvert
the behavior of another such function executed later; for example, by
redefining standard functions or operators called by the target function.
If the target function is SECURITY DEFINER, or is called by such a
function, this means that any ordinary SQL user with Perl or Tcl language
usage rights can do essentially anything with the privileges of the target
function's owner.
To close this security hole, create a separate Perl or Tcl interpreter for
each SQL userid under which plperl or pltcl functions are executed within
a session. However, all plperlu or pltclu functions run within a session
still share a single interpreter, since they all execute at the trust
level of a database superuser anyway.
Note: this change results in a functionality loss when libperl has been
built without the "multiplicity" option: it's no longer possible to call
plperl functions under different userids in one session, since such a
libperl can't support multiple interpreters in one process. However, such
a libperl already failed to support concurrent use of plperl and plperlu,
so it's likely that few people use such versions with Postgres.
Bruce Momjian [Tue, 28 Sep 2010 19:25:12 +0000 (19:25 +0000)]
Properly close files after read file failure to prevent potential
resource leak. Of course, any such failure aborts pg_upgrade, but might
as well be clean about it.
Tom Lane [Tue, 28 Sep 2010 16:08:56 +0000 (12:08 -0400)]
Fix PlaceHolderVar mechanism's interaction with outer joins.
The point of a PlaceHolderVar is to allow a non-strict expression to be
evaluated below an outer join, after which its value bubbles up like a Var
and can be forced to NULL when the outer join's semantics require that.
However, there was a serious design oversight in that, namely that we
didn't ensure that there was actually a correct place in the plan tree
to evaluate the placeholder :-(. It may be necessary to delay evaluation
of an outer join to ensure that a placeholder that should be evaluated
below the join can be evaluated there. Per recent bug report from Kirill
Simonov.
Back-patch to 8.4 where the PlaceHolderVar mechanism was introduced.
Robert Haas [Tue, 28 Sep 2010 00:55:27 +0000 (20:55 -0400)]
Add a SECURITY LABEL command.
This is intended as infrastructure to support integration with label-based
mandatory access control systems such as SE-Linux. Further changes (mostly
hooks) will be needed, but this is a big chunk of it.
Tom Lane [Mon, 27 Sep 2010 00:22:17 +0000 (20:22 -0400)]
Improve git_changelog as per discussion with Robert Haas.
1. Resurrect the behavior where old commits on master will have Branch:
labels for branches sprouted after the commit was made. I'm still
dubious about this mode, but if you want it, say --post-date or -p.
2. Annotate the Branch: labels with the release or branch in which the
commit was publicly released. For example, on a release branch you could
see
Branch: REL8_3_STABLE Release: REL8_3_2 [92c3a8004] 2008-03-29 00:15:37 +0000
showing that the fix was released in 8.3.2. Commits on master will
usually instead have notes like
Branch: master Release: REL8_4_BR [6fc9d4272] 2008-03-29 00:15:28 +0000
showing that this commit is ancestral to release branches 8.4 and later.
If no Release: marker appears, the commit hasn't yet made it into any
release.
3. Add support for release branches older than 7.4.
4. The implementation is improved by running git log on each branch only
back to where the branch sprouts from master. This saves a good deal
of time (about 50% of the runtime when generating the complete history).
We generate the post-date-mode tags via a direct understanding that
they should be applied to master commits made before the branch sprouted,
rather than backing into them via matching (which isn't any too
reliable when people used identical log messages for successive commits).
Tom Lane [Sun, 26 Sep 2010 05:51:20 +0000 (01:51 -0400)]
Still more tweaking of git_changelog.
1. Don't assume there's only one candidate match; check them all and use the
one with the closest timestamp. Avoids funny output when someone makes
several successive commits with the same log message, as certain people
have been known to do.
2. When the same commit (with the same SHA1) is reachable from multiple
branch tips, don't report it for all the branches; instead report it only
for the first such branch. Given our development practices, this case
arises only for commits that occurred before a given branch split off from
master. The original coding blamed old commits on *all* the branches,
which isn't terribly useful; the new coding blames such a commit only on
master.
Tom Lane [Sun, 26 Sep 2010 04:21:51 +0000 (00:21 -0400)]
Fix some more bugs in git_changelog.
1. Don't forget the last (oldest) commit on the oldest branch.
2. When considering which commit to print next, if two alternatives have
the same "distortion" score (which is actually the normal case, since
generally the "distortion" is 0), then choose the later timestamp to
print first. I don't know where Robert got the idea to ignore timestamps
and sort by branch age, but it wasn't a good idea: the resulting ordering
of commits was just plain bizarre anywhere that some branches had many
fewer commits than others, which is the typical situation for us.
Tom Lane [Sun, 26 Sep 2010 00:50:57 +0000 (20:50 -0400)]
Minor improvements to git_changelog.
Avoid depending on Date::Calc, which isn't in a basic Perl installation,
when we can equally well use Time::Local which is. Also fix the parsing
of timestamps to take heed of the timezone. (It looks like cvs2git emitted
all commit timestamps with zone GMT, so this refinement might've looked
unnecessary when looking at converted data; but it's needed now.)
Fix parsing of message bodies so that blank lines that may or may not get
emitted by "git log" aren't confused with real data. This avoids strange
formatting of the oldest commit on a branch.
Check child-process exit status, so that we actually notice if "git log"
fails, and so that we don't accumulate zombie children.
Tom Lane [Sat, 25 Sep 2010 23:03:50 +0000 (19:03 -0400)]
Fix another join removal bug: the check on PlaceHolderVars was wrong.
The previous coding would decide that join removal was unsafe upon finding
a PlaceHolderVar that needed to be evaluated at the inner rel and then used
above the join. However, this fails to cover the case of PlaceHolderVars
that refer to both the inner rel and some other rels. Per bug report from
Andrus.
Peter Eisentraut [Sat, 25 Sep 2010 06:57:09 +0000 (09:57 +0300)]
Fix man page markup for <cmdsynopsis> with multiple variants
Command synopses using <cmdsynopsis> with multiple variants previously used
<sbr> to break lines between variants. The new man page toolchain introduced
in 9.0 makes a mess out of that, and that markup was probably wrong all along,
because <sbr> is supposed to break lines within a synopsis, not between them.
So fix that by using multiple <cmdsynopsis> elements inside <refsynopsisdiv>.
Tom Lane [Thu, 23 Sep 2010 20:53:16 +0000 (16:53 -0400)]
Prevent show_session_authorization from crashing when session_authorization
hasn't been set.
The only known case where this can happen is when show_session_authorization
is invoked in an autovacuum process, which is possible if an index function
calls it, as for example in bug #5669 from Andrew Geery. We could perhaps
try to return a sensible value, such as the name of the cluster-owning
superuser; but that seems like much more trouble than the case is worth,
and in any case it could create new possible failure modes. Simply
returning an empty string seems like the most appropriate fix.
Back-patch to all supported versions, even those before autovacuum, just
in case there's another way to provoke this crash.
Tom Lane [Thu, 23 Sep 2010 19:34:56 +0000 (15:34 -0400)]
Avoid sharing subpath list structure when flattening nested AppendRels.
In some situations the original coding led to corrupting the child AppendRel's
subpaths list, effectively adding other members of the parent's list to it.
This was usually masked because we never made any further use of the child's
list, but given the right combination of circumstances, we could do so. The
visible symptom would be a relation getting scanned twice, as in bug #5673
from David Schmitt.
Backpatch to 8.2, which is as far back as the risky coding appears. The
example submitted by David only fails in 8.4 and later, but I'm not convinced
that there aren't any even-more-obscure cases where 8.2 and 8.3 would fail.
Tom Lane [Thu, 23 Sep 2010 19:16:49 +0000 (15:16 -0400)]
Make _outPathInfo print the relid set of the path's parent rel.
We can't actually print the parent RelOptInfo in toto, because that would
lead to infinite recursion. But it's safe enough to reach into the parent
and print its identifying relids, and that makes it a whole lot easier
to figure out what a Path represents. Should have done this years ago.
Initialize tableoid field correctly when dumping foreign data wrappers and
servers. AFAICT it's harmless at the moment because nothing can depend on
either, but as soon as we introduce an object type with such dependencies,
tableoid needs to be set or pg_dump will fail to interpret the dependencies
correctly. In theory, I guess the uninitialized garbage in tableoid could
cause the object to be mistaken for some other object with same OID as well.
Tom Lane [Thu, 23 Sep 2010 03:48:07 +0000 (23:48 -0400)]
Re-allow input of Julian dates prior to 0001-01-01 AD.
This was unintentionally broken in 8.4 while tightening up checking of
ordinary non-Julian date inputs to forbid references to "year zero".
Per bug #5672 from Benjamin Gigot.
Tom Lane [Thu, 23 Sep 2010 06:32:03 +0000 (02:32 -0400)]
More fixes for libpq's .gitignore file.
The previous patches failed to cover a lot of symlinks that are only
added in platform-specific cases. Make the lists match what's in the
Makefile for each branch.
Tom Lane [Tue, 21 Sep 2010 21:18:57 +0000 (17:18 -0400)]
Some more cleanup of CVS keyword noise.
Poking around for remaining occurrences of CVS keyword strings, I came
across one that apparently reflects the use of a $Revision: ...$ string
in the original input data. Dunno why anybody would be using that in
an MTA's Received: lines, but there it is. Put it back to the way that
it was originally, according to inspection of the CVS repo.
Tom Lane [Sat, 18 Sep 2010 20:10:15 +0000 (20:10 +0000)]
Make sure we wait for protocol-level EOF when ending binary COPY IN.
The previous coding just terminated the COPY immediately after seeing
the EOF marker (-1 where a row field count is expected). The expected
CopyDone or CopyFail message just got thrown away later, since we weren't
in COPY mode anymore. This behavior complicated matters for the JDBC
driver, and arguably was the wrong thing in any case since a CopyFail
message after the marker wouldn't be honored.
Note that there is a behavioral change here: extra data after the EOF
marker was silently ignored before, but now it will cause an error.
Hence not back-patching, although this is arguably a bug.
Tom Lane [Sat, 18 Sep 2010 18:37:01 +0000 (18:37 +0000)]
Give a suitable HINT when an INSERT's data source is a RowExpr containing
the same number of columns expected by the insert. This suggests that there
were extra parentheses that converted the intended column list into a row
expression.
Original patch by Marko Tiikkaja, rather heavily editorialized by me.
Magnus Hagander [Thu, 16 Sep 2010 20:37:13 +0000 (20:37 +0000)]
Treat exit code 128 (ERROR_WAIT_NO_CHILDREN) as non-fatal on Win32,
since it can happen when a process fails to start when the system
is under high load.
Per several bug reports and many peoples investigation.
Back-patch to 8.4, which is as far back as the "deadman-switch"
for shared memory access exists.
Tom Lane [Thu, 16 Sep 2010 02:54:01 +0000 (02:54 +0000)]
Fix two new-in-9.0 bugs in hstore.
There was an incorrect Assert in hstoreValidOldFormat(), which would cause
immediate core dumps when attempting to work with pre-9.0 hstore data,
but of course only in an assert-enabled build.
Also, ghstore_decompress() incorrectly applied DatumGetHStoreP() to a datum
that wasn't actually an hstore, but rather a ghstore (ie, a gist signature
bitstring). That used to be harmless, but could now result in misbehavior
if the hstore format conversion code happened to trigger. In reality,
since ghstore is not marked toastable (and doesn't need to be), this
function is useless anyway; we can lobotomize it down to returning the
passed-in pointer.
Both bugs found by Andrew Gierth, though this isn't exactly his proposed
patch.
Tom Lane [Wed, 15 Sep 2010 17:45:57 +0000 (17:45 +0000)]
Add a compatibility note about plpgsql's treatment of SELECT INTO rec.fld
when fld is of composite type. Per discussion of bug #5644 from Valentine
Gogichashvili.
Use a latch to make startup process wake up and replay immediately when
new WAL arrives via streaming replication. This reduces the latency, and
also allows us to use a longer polling interval, which is good for energy
efficiency.
We still need to poll to check for the appearance of a trigger file, but
the interval is now 5 seconds (instead of 100ms), like when waiting for
a new WAL segment to appear in WAL archive.
Simplify Windows implementation of latches. There's no need to keep a
dynamic pool of event handles, we can permanently assign one for each
shared latch. Thanks to that, we no longer need a separate shared memory
block for latches, and we don't need to know in advance how many shared
latches there is, so you no longer need to remember to update
NumSharedLatches when you introduce a new latch to the system.
Don't call OwnLatch while holding a spinlock. OwnLatch can elog() under
some "can't happen" scenarios, and spinlocks should only be held for
a few instructions anyway. As pointed out by Fujii Masao.
Tom Lane [Tue, 14 Sep 2010 23:15:29 +0000 (23:15 +0000)]
Fix join-removal logic for pseudoconstant and outerjoin-delayed quals.
In these cases a qual can get marked with the removable rel in its
required_relids, but this is just to schedule its evaluation correctly, not
because it really depends on the rel. We were assuming that, in effect,
we could throw away *all* quals so marked, which is nonsense. Tighten up
the logic to be a little more paranoid about which quals belong to the
outer join being considered for removal, and arrange for all quals that
don't belong to be updated so they will still get evaluated correctly.
Also fix another problem that happened to be exposed by this test case,
which was that make_join_rel() was failing to notice some cases where
a constant-false qual could be used to prove a join relation empty. If it's
a pushed-down constant false, then the relation is empty even if it's an
outer join, because the qual applies after the outer join expansion.
Per report from Nathan Grange. Back-patch into 9.0.
Don't warn about an in-progress online backup, when we're recovering from
an online backup instead of performing one. pg_ctl can detect that by
checking if recovery.conf exists.
Backup label file is renamed away early in recovery, so the window where
backup label exists during recovery is normally very small, but you can run
into it e.g if restore_command is set incorrectly and the startup process
never finds even the first WAL segment containing the checkpoint record to
start recovery from.
Remove prototype for non-existent function from walreceiver.h. Tidy up by
separating prototypes for functions in walreceiver.c and walreceiverfuncs.c
with comments.
Process options from the startup packed in walsender. Only few options
make sense for walsender, but for example application_name and client_encoding
do. We still don't apply per-role settings from pg_db_role_setting, because
that would require connecting to a database to read the table.
Joe Conway [Sat, 11 Sep 2010 18:38:58 +0000 (18:38 +0000)]
SERIALIZABLE transactions are actually implemented beneath the covers with
transaction snapshots, i.e. a snapshot registered at the beginning of
a transaction. Change variable naming and comments to reflect this reality
in preparation for a future, truly serializable mode, e.g.
Serializable Snapshot Isolation (SSI).
For the moment transaction snapshots are still used to implement
SERIALIZABLE, but hopefully not for too much longer. Patch by Kevin
Grittner and Dan Ports with review and some minor wording changes by me.
Introduce latches. A latch is a boolean variable, with the capability to
wait until it is set. Latches can be used to reliably wait until a signal
arrives, which is hard otherwise because signals don't interrupt select()
on some platforms, and even when they do, there's race conditions.
On Unix, latches use the so called self-pipe trick under the covers to
implement the sleep until the latch is set, without race conditions. On
Windows, Windows events are used.
Use the new latch abstraction to sleep in walsender, so that as soon as
a transaction finishes, walsender is woken up to immediately send the WAL
to the standby. This reduces the latency between master and standby, which
is good.
Preliminary work by Fujii Masao. The latch implementation is by me, with
helpful comments from many people.