Tom Lane [Sun, 21 May 2006 20:22:16 +0000 (20:22 +0000)]
Fix errors in fortuna PRNG reseeding logic that could cause a predictable
session key to be selected by pgp_sym_encrypt() in some cases. This only
affects non-OpenSSL-using builds. Marko Kreen
Tom Lane [Sun, 21 May 2006 20:19:23 +0000 (20:19 +0000)]
Modify libpq's string-escaping routines to be aware of encoding considerations
and standard_conforming_strings. The encoding changes are needed for proper
escaping in multibyte encodings, as per the SQL-injection vulnerabilities
noted in CVE-2006-2313 and CVE-2006-2314. Concurrent fixes are being applied
to the server to ensure that it rejects queries that may have been corrupted
by attempted SQL injection, but this merely guarantees that unpatched clients
will fail rather than allow injection. An actual fix requires changing the
client-side code. While at it we have also fixed these routines to understand
about standard_conforming_strings, so that the upcoming changeover to SQL-spec
string syntax can be somewhat transparent to client code.
Since the existing API of PQescapeString and PQescapeBytea provides no way to
inform them which settings are in use, these functions are now deprecated in
favor of new functions PQescapeStringConn and PQescapeByteaConn. The new
functions take the PGconn to which the string will be sent as an additional
parameter, and look inside the connection structure to determine what to do.
So as to provide some functionality for clients using the old functions,
libpq stores the latest encoding and standard_conforming_strings values
received from the backend in static variables, and the old functions consult
these variables. This will work reliably in clients using only one Postgres
connection at a time, or even multiple connections if they all use the same
encoding and string syntax settings; which should cover many practical
scenarios.
Clients that use homebrew escaping methods, such as PHP's addslashes()
function or even hardwired regexp substitution, will require extra effort
to fix :-(. It is strongly recommended that such code be replaced by use of
PQescapeStringConn/PQescapeByteaConn if at all feasible.
Tom Lane [Sun, 21 May 2006 20:10:42 +0000 (20:10 +0000)]
Add a new GUC parameter backslash_quote, which determines whether the SQL
parser will allow "\'" to be used to represent a literal quote mark. The
"\'" representation has been deprecated for some time in favor of the
SQL-standard representation "''" (two single quote marks), but it has been
used often enough that just disallowing it immediately won't do. Hence
backslash_quote allows the settings "on", "off", and "safe_encoding",
the last meaning to allow "\'" only if client_encoding is a valid server
encoding. That is now the default, and the reason is that in encodings
such as SJIS that allow 0x5c (ASCII backslash) to be the last byte of a
multibyte character, accepting "\'" allows SQL-injection attacks as per
CVE-2006-2314 (further details will be published after release). The
"on" setting is available for backward compatibility, but it must not be
used with clients that are exposed to untrusted input.
Thanks to Akio Ishida and Yasuo Ohgaki for identifying this security issue.
Tom Lane [Sun, 21 May 2006 20:05:21 +0000 (20:05 +0000)]
Change the backend to reject strings containing invalidly-encoded multibyte
characters in all cases. Formerly we mostly just threw warnings for invalid
input, and failed to detect it at all if no encoding conversion was required.
The tighter check is needed to defend against SQL-injection attacks as per
CVE-2006-2313 (further details will be published after release). Embedded
zero (null) bytes will be rejected as well. The checks are applied during
input to the backend (receipt from client or COPY IN), so it no longer seems
necessary to check in textin() and related routines; any string arriving at
those functions will already have been validated. Conversion failure
reporting (for characters with no equivalent in the destination encoding)
has been cleaned up and made consistent while at it.
Also, fix a few longstanding errors in little-used encoding conversion
routines: win1251_to_iso, win866_to_iso, euc_tw_to_big5, euc_tw_to_mic,
mic_to_euc_tw were all broken to varying extents.
Patches by Tatsuo Ishii and Tom Lane. Thanks to Akio Ishida and Yasuo Ohgaki
for identifying the security issues.
Alvaro Herrera [Fri, 19 May 2006 19:08:27 +0000 (19:08 +0000)]
Add last-vacuum/analyze-time columns to the stats collector, both manual and
issued by autovacuum. Add accessor functions to them, and use those in the
pg_stat_*_tables system views.
Catalog version bumped due to changes in the pgstat views and the pgstat file.
Patch from Larry Rosenman, minor improvements by me.
Tom Lane [Thu, 18 May 2006 19:56:46 +0000 (19:56 +0000)]
Fix choose_bitmap_and() so that partial index predicates are considered when
deciding whether a potential additional indexscan is redundant or not. As now
coded, any use of a partial index that was already used in a previous AND arm
will be rejected as redundant. This might be overly restrictive, but not
considering the point at all is definitely bad, as per example in bug #2441
from Arjen van der Meijden. In particular, a clauseless scan of a partial
index was *never* considered redundant by the previous coding, and that's
surely wrong. Being more flexible would also require some consideration
of how not to double-count the index predicate's selectivity.
Tom Lane [Thu, 18 May 2006 18:57:31 +0000 (18:57 +0000)]
When a bitmap indexscan is using a partial index, it is necessary to include
the partial index predicate in the scan's "recheck condition". Otherwise,
if the scan becomes lossy for lack of bitmap memory, we would fail to enforce
that returned rows satisfy the predicate. Noted while studying bug #2441
from Arjen van der Meijden.
Tom Lane [Thu, 18 May 2006 17:12:10 +0000 (17:12 +0000)]
Fix thinko in recent changes to handle ScalarArrayOpExpr as an indexable
condition: when there are multiple possible index paths involving
ScalarArrayOpExprs, they are logically to be ANDed together not ORed.
This thinko was a direct consequence of trying to put the processing
inside generate_bitmap_or_paths(), which I now see was a bit too cute.
So pull it out and make the callers do it separately (there are only two
that need it anyway). Partially responds to bug #2441 from Arjen van der Meijden.
There are some additional infelicities exposed by his example, but they
are also in 8.1.x, while this mistake is not.
Teodor Sigaev [Wed, 17 May 2006 16:34:59 +0000 (16:34 +0000)]
Reduce size of critial section during vacuum full, critical
sections now isn't nested. All user-defined functions now is
called outside critsections. Small improvements in WAL
protocol.
Tom Lane [Fri, 12 May 2006 22:44:36 +0000 (22:44 +0000)]
Fix the sense of the test on DH_check()'s return value. This was preventing
custom-generated DH parameters from actually being used by the server.
Found by Michael Fuhr.
Tom Lane [Fri, 12 May 2006 16:50:52 +0000 (16:50 +0000)]
Upon closer inspection, the sparc code in s_lock.c is dead code, and
always has been, because it's not got any .globl declaration! We've
been relying on the solaris_sparc.s code instead. Rip it out.
(Not back-patched, since this is just cosmetic cleanup.)
Tom Lane [Thu, 11 May 2006 19:15:36 +0000 (19:15 +0000)]
Code review for standard_conforming_strings patch. Fix it so it does not
throw warnings for 100%-SQL-standard constructs, clean up some minor
infelicities, try to un-break ecpg to the best of my ability. (It's not clear
how ecpg is going to find out the setting of standard_conforming_strings,
though.) I think pg_dump still needs work, too.
Tom Lane [Wed, 10 May 2006 23:18:39 +0000 (23:18 +0000)]
Clean up code associated with updating pg_class statistics columns
(relpages/reltuples). To do this, create formal support in heapam.c for
"overwrite" tuple updates (including xlog replay capability) and use that
instead of the ad-hoc overwrites we'd been using in VACUUM and CREATE INDEX.
Take the responsibility for updating stats during CREATE INDEX out of the
individual index AMs, and do it where it belongs, in catalog/index.c. Aside
from being more modular, this avoids having to update the same tuple twice in
some paths through CREATE INDEX. It's probably not measurably faster, but
for sure it's a lot cleaner than before.
Bruce Momjian [Mon, 8 May 2006 02:18:50 +0000 (02:18 +0000)]
Build server libpgport with all non-FRONTEND object files. This is to
fix a Win32 bug where pipe.c included a file that used FRONTEND, but it
wasn't on the server-build list.
Tom Lane [Mon, 8 May 2006 00:00:17 +0000 (00:00 +0000)]
Rewrite btree vacuuming to fold the former bulkdelete and cleanup operations
into a single mostly-physical-order scan of the index. This requires some
ticklish interlocking considerations, but should create no material
performance impact on normal index operations (at least given the
already-committed changes to make scans work a page at a time). VACUUM
itself should get significantly faster in any index that's degenerated to a
very nonlinear page order. Also, we save one pass over the index entirely,
except in the case where there were no deletions to do and so only one pass
happened anyway.
Original patch by Heikki Linnakangas, rework by Tom Lane.
Tom Lane [Sun, 7 May 2006 01:21:30 +0000 (01:21 +0000)]
Rewrite btree index scans to work a page at a time in all cases (both
btgettuple and btgetmulti). This eliminates the problem of "re-finding" the
exact stopping point, since the stopping point is effectively always a page
boundary, and index items are never moved across pre-existing page boundaries.
A small penalty is that the keys_are_unique optimization is effectively
disabled (and, therefore, is removed in this patch), causing us to apply
_bt_checkkeys() to at least one more tuple than necessary when looking up a
unique key. However, the advantages for non-unique cases seem great enough to
accept this tradeoff. Aside from simplifying and (sometimes) speeding up the
indexscan code, this will allow us to reimplement btbulkdelete as a largely
sequential scan instead of index-order traversal, thereby significantly
reducing the cost of VACUUM. Those changes will come in a separate patch.
Original patch by Heikki Linnakangas, rework by Tom Lane.
Tom Lane [Sat, 6 May 2006 15:51:07 +0000 (15:51 +0000)]
Further minor simplification of relcache startup: don't need a static
needNewCacheFile flag anymore, it can just be local in RelationCacheInitializePhase2.
Bruce Momjian [Fri, 5 May 2006 09:35:58 +0000 (09:35 +0000)]
Add/ cleanup:
< * %Disallow changing default expression of a SERIAL column?
> * %Disallow changing DEFAULT expression of a SERIAL column? 472a473,476
> * Add DEFAULT .. AS OWNER so permission checks are done as the table
> owner
>
> This would be useful for SERIAL nextval() calls and CHECK constraints.
Tom Lane [Thu, 4 May 2006 20:39:34 +0000 (20:39 +0000)]
Code review for contrib/pg_freespacemap. Add a storedpages column to
pg_freespacemap_relations --- while one could theoretically get that
number by counting rows in pg_freespacemap_pages, it's surely the hard
way to do it. Avoid expensive and inconvenient conversion to and from
text format. Minor code and docs cleanup.
Tom Lane [Thu, 4 May 2006 18:51:36 +0000 (18:51 +0000)]
Simplify relcache startup sequence. With the new design of InitPostgres
it's not necessary to have three separate calls anymore. This patch also
fixes things so we don't try to read pg_internal.init until after we've
obtained lock on the target database; which was fairly harmless, but it's
certainly cleaner this way.
Tom Lane [Thu, 4 May 2006 16:07:29 +0000 (16:07 +0000)]
Rethink the locking mechanisms used for CREATE/DROP/RENAME DATABASE.
The former approach used ExclusiveLock on pg_database, which being a
cluster-wide lock meant only one of these operations could proceed at
a time; worse, it also blocked all incoming connections in ReverifyMyDatabase.
Now that we have LockSharedObject(), we can use locks of different types
applied to databases considered as objects. This allows much more
flexible management of the interlocking: two CREATE DATABASEs need not
block each other, and need not block connections except to the template
database being used. Similarly DROP DATABASE doesn't block unrelated
operations. The locking used in flatfiles.c is also much narrower in
scope than before. Per recent proposal.
Tom Lane [Wed, 3 May 2006 22:45:26 +0000 (22:45 +0000)]
Create a syscache for pg_database-indexed-by-oid, and make use of it
in various places that were previously doing ad hoc pg_database searches.
This may speed up database-related privilege checks a little bit, but
the main motivation is to eliminate the performance reason for having
ReverifyMyDatabase do such a lot of stuff (viz, avoiding repeat scans
of pg_database during backend startup). The locking reason for having
that routine is about to go away, and it'd be good to have the option
to break it up.
Tom Lane [Wed, 3 May 2006 00:24:56 +0000 (00:24 +0000)]
Fix calculation of plan node extParams to account for the possibility that one
initPlan sets a parameter for another. This could not (I think) happen before
8.1, but it's possible now because the initPlans generated by MIN/MAX
optimization might themselves use initPlans. We attach those initPlans as
siblings of the MIN/MAX ones, not children, to avoid duplicate computation
when multiple MIN/MAX aggregates are present; so this leads to the case of an
initPlan needing the result of a sibling initPlan, which is not possible with
ordinary query nesting. Hadn't been noticed because in most contexts having
too much stuff listed in extParam is fairly harmless. Fixes "plan should not
reference subplan's variable" bug reported by Catalin Pitis.
Tom Lane [Tue, 2 May 2006 22:25:10 +0000 (22:25 +0000)]
Clean up API for ambulkdelete/amvacuumcleanup as per today's discussion.
This formulation requires every AM to provide amvacuumcleanup, unlike before,
but it's surely a whole lot cleaner. Also, add an 'amstorage' column to
pg_am so that we can get rid of hardwired knowledge in DefineOpClass().
Tom Lane [Tue, 2 May 2006 04:34:18 +0000 (04:34 +0000)]
Avoid assuming that statistics for a parent relation reflect the properties of
the union of its child relations as well. This might have been a good idea
when it was originally coded, but it's a fatally bad idea when inheritance is
being used for partitioning. It's better to have no stats at all than
completely misleading stats. Per report from Mark Liberman.
The bug arguably exists all the way back, but I've only patched HEAD and 8.1
because we weren't particularly trying to support partitioning before 8.1.
Eventually we ought to look at deriving union statistics instead of just
punting, but for now the drop kick looks good.
Tom Lane [Mon, 1 May 2006 23:22:43 +0000 (23:22 +0000)]
Provide a namespace.c function for lookup of an operator with exact
input datatypes given, and use this before trying OpernameGetCandidates.
This is faster than the old method when there's an exact match, and it
does not seem materially slower when there's not. And it definitely
makes some of the callers cleaner, because they didn't really want to
know about a list of candidates anyway. Per discussion with Atsushi Ogawa.
Tom Lane [Sun, 30 Apr 2006 21:15:33 +0000 (21:15 +0000)]
Code review for GRANT CONNECT patch. Spell the privilege as CONNECT not
CONNECTION, fix a number of places that were missed (eg pg_dump support),
avoid executing an extra search of pg_database during startup.
Tom Lane [Sun, 30 Apr 2006 18:30:40 +0000 (18:30 +0000)]
Improve the representation of FOR UPDATE/FOR SHARE so that we can
support both FOR UPDATE and FOR SHARE in one command, as well as both
NOWAIT and normal WAIT behavior. The more general code is actually
simpler and cleaner.
Tom Lane [Sat, 29 Apr 2006 20:13:07 +0000 (20:13 +0000)]
We only need to add thread.c on non-WIN32 platforms, since get_home_path
doesn't use pqGetpwuid on WIN32. Rather than try to figure out why it
won't build on WIN32, just remove it.