Tom Lane [Sun, 2 Apr 2017 23:19:16 +0000 (19:19 -0400)]
Remove reinvention of stringify macro.
We already have CppAsString2, there's no need for the MSVC support to
re-invent a macro to do that (and especially not to inject it in as
ugly a way as this).
Tom Lane [Sun, 2 Apr 2017 22:26:37 +0000 (18:26 -0400)]
Document psql's behavior of recalling the previously executed query.
Various psql slash commands that normally act on the current query buffer
will automatically recall and re-use the most recently executed SQL command
instead, if the current query buffer is empty. Although this behavior is
ancient (dating apparently to commit 77a472993), it was documented nowhere
in the psql reference page. For that matter, we'd never bothered to define
the concept of "current query buffer" explicitly. Fix that. Do some
wordsmithing on relevant command descriptions to improve clarity and
consistency.
Tom Lane [Sun, 2 Apr 2017 20:50:25 +0000 (16:50 -0400)]
Fix behavior of psql's \p to agree with \g, \w, etc.
In commit e984ef586 I (tgl) simplified the behavior of \p to just print
the current query buffer; but Daniel Vérité points out that this made it
inconsistent with the behavior of \g and \w. It should print the same
thing \g would execute. Fix that, and improve related comments.
Tom Lane [Sun, 2 Apr 2017 01:44:54 +0000 (21:44 -0400)]
Allow psql variable substitution to occur in backtick command strings.
Previously, text between backquotes in a psql metacommand's arguments
was always passed to the shell literally. That considerably hobbles
the usefulness of the feature for scripting, so we'd foreseen for a long
time that we'd someday want to allow substitution of psql variables into
the shell command. IMO the addition of \if metacommands has brought us to
that point, since \if can greatly benefit from some sort of client-side
expression evaluation capability, and psql itself is not going to grow any
such thing in time for v10. Hence, this patch. It allows :VARIABLE to be
replaced by the exact contents of the named variable, while :'VARIABLE'
is replaced by the variable's contents suitably quoted to become a single
shell-command argument. (The quoting rules for that are different from
those for SQL literals, so this is a bit of an abuse of the :'VARIABLE'
notation, but I doubt anyone will be confused.)
As with other situations in psql, no substitution occurs if the word
following a colon is not a known variable name. That limits the risk of
compatibility problems for existing psql scripts; but the risk isn't zero,
so this needs to be called out in the v10 release notes.
Kevin Grittner [Sat, 1 Apr 2017 20:21:05 +0000 (15:21 -0500)]
Fix two undocumented parameters to functions from ENR patch.
On ProcessUtility document the parameter, to match others.
On CreateCachedPlan drop the queryEnv parameter. It was not
referenced within the function, and had been added on the
assumption that with some unknown future usage of QueryEnvironment
it might be useful to do something there. We have avoided other
"just in case" implementation of unused paramters, so drop it here.
When the BRIN summary tuple for a page range becomes too "wide" for the
values actually stored in the table (because the tuples that were
present originally are no longer present due to updates or deletes), it
can be useful to remove the outdated summary tuple, so that a future
summarization can install a tighter summary.
This commit introduces a SQL-callable interface to do so.
Previously, only VACUUM would cause a page range to get initially
summarized by BRIN indexes, which for some use cases takes too much time
since the inserts occur. To avoid the delay, have brininsert request a
summarization run for the previous range as soon as the first tuple is
inserted into the first page of the next range. Autovacuum is in charge
of processing these requests, after doing all the regular vacuuming/
analyzing work on tables.
This doesn't impose any new tasks on autovacuum, because autovacuum was
already in charge of doing summarizations. The only actual effect is to
change the timing, i.e. that it occurs earlier. For this reason, we
don't go any great lengths to record these requests very robustly; if
they are lost because of a server crash or restart, they will happen at
a later time anyway.
Most of the new code here is in autovacuum, which can now be told about
"work items" to process. This can be used for other things such as GIN
pending list cleaning, perhaps visibility map bit setting, both of which
are currently invoked during vacuum, but do not really depend on vacuum
taking place.
The requests are at the page range level, a granularity for which we did
not have SQL-level access; we only had index-level summarization
requests via brin_summarize_new_values(). It seems reasonable to add
SQL-level access to range-level summarization too, so add a function
brin_summarize_range() to do that.
Authors: Álvaro Herrera, based on sketch from Simon Riggs. Reviewed-by: Thomas Munro.
Discussion: https://postgr.es/m/20170301045823.vneqdqkmsd4as4ds@alvherre.pgsql
Magnus Hagander [Sat, 1 Apr 2017 15:04:14 +0000 (17:04 +0200)]
Write "waiting for checkpoint" on regular progress row
When reporting progress, make the "waiting for checkpoint" test be
overwritten by the file-based progress once it's completed. This is more
consistent with how we report the rest of the progress.
Kevin Grittner [Sat, 1 Apr 2017 04:17:18 +0000 (23:17 -0500)]
Add infrastructure to support EphemeralNamedRelation references.
A QueryEnvironment concept is added, which allows new types of
objects to be passed into queries from parsing on through
execution. At this point, the only thing implemented is a
collection of EphemeralNamedRelation objects -- relations which
can be referenced by name in queries, but do not exist in the
catalogs. The only type of ENR implemented is NamedTuplestore, but
provision is made to add more types fairly easily.
An ENR can carry its own TupleDesc or reference a relation in the
catalogs by relid.
Although these features can be used without SPI, convenience
functions are added to SPI so that ENRs can easily be used by code
run through SPI.
The initial use of all this is going to be transition tables in
AFTER triggers, but that will be added to each PL as a separate
commit.
An incidental effect of this patch is to produce a more informative
error message if an attempt is made to modify the contents of a CTE
from a referencing DML statement. No tests previously covered that
possibility, so one is added.
Kevin Grittner and Thomas Munro
Reviewed by Heikki Linnakangas, David Fetter, and Thomas Munro
with valuable comments and suggestions from many others
Robert Haas [Sat, 1 Apr 2017 01:15:05 +0000 (21:15 -0400)]
Avoid GatherMerge crash when there are no workers.
It's unnecessary to return an actual slot when we have no tuple.
We can just return NULL, which avoids the risk of indexing into an
array that might not contain any elements.
Robert Haas [Sat, 1 Apr 2017 01:01:20 +0000 (21:01 -0400)]
Fix parallel query so it doesn't spoil row estimates above Gather.
Commit 45be99f8cd5d606086e0a458c9c72910ba8a613d removed GatherPath's
num_workers field, but this is entirely bogus. Normally, a path's
parallel_workers flag is supposed to indicate the number of workers
that it wants, and should be 0 for a non-partial path. In that
commit, I mistakenly thought that GatherPath could also use that field
to indicate the number of workers that it would try to start, but
that's disastrous, because then it can propagate up to higher nodes in
the plan tree, which will then get incorrect rowcounts because the
parallel_workers flag is involved in computing those values. Repair
by putting the separate field back.
Report by Tomas Vondra. Patch by me, reviewed by Amit Kapila.
Robert Haas [Sat, 1 Apr 2017 00:35:51 +0000 (20:35 -0400)]
Don't use bgw_main even to specify in-core bgworker entrypoints.
On EXEC_BACKEND builds, this can fail if ASLR is in use.
Backpatch to 9.5. On master, completely remove the bgw_main field
completely, since there is no situation in which it is safe for an
EXEC_BACKEND build. On 9.6 and 9.5, leave the field intact to avoid
breaking things for third-party code that doesn't care about working
under EXEC_BACKEND. Prior to 9.5, there are no in-core bgworker
entrypoints.
Tom Lane [Sat, 1 Apr 2017 00:24:12 +0000 (20:24 -0400)]
Fix unstable regression test result.
Commit e306df7f9 added a test case that depends on "the" being a
stop word, which it is not in non-English locales. Since the
point of the test is to check stopword behavior, fix by forcibly
selecting the 'english' configuration.
Tom Lane [Fri, 31 Mar 2017 22:11:25 +0000 (18:11 -0400)]
For foreign keys, check REFERENCES privilege only on the referenced table.
We were requiring that the user have REFERENCES permission on both the
referenced and referencing tables --- but this doesn't seem to have any
support in the SQL standard, which says only that you need REFERENCES
permission on the referenced table. And ALTER TABLE ADD FOREIGN KEY has
already checked that you own the referencing table, so the check could
only fail if a table owner has revoked his own REFERENCES permission.
Moreover, the symmetric interpretation of this permission is unintuitive
and confusing, as per complaint from Paul Jungwirth. So let's drop the
referencing-side check.
In passing, do a bit of wordsmithing on the GRANT reference page so that
all the privilege types are described in similar fashion.
Robert Haas [Fri, 31 Mar 2017 20:47:38 +0000 (16:47 -0400)]
Revert "Allow ON CONFLICT .. DO NOTHING on a partitioned table."
This reverts commit 8355a011a0124bdf7ccbada206a967d427039553, which
turns out to have been a misguided effort. We can't really support
this in a partitioning hierarchy after all for exactly the reasons
stated in the documentation removed by that commit. It's still
possible to use ON CONFLICT .. DO NOTHING (or for that matter ON
CONFLICT .. DO UPDATE) on individual partitions if desired, but
but to allow this on a partitioned table implies that we have some
way of evaluating uniqueness across the whole partitioning
hierarchy, which is false.
Shinoda Noriyoshi noticed that the old code was crashing (which we
could fix, though not in a nice way) and Amit Langote realized
that this was indicative of a fundamental problem with the commit
being reverted here.
Robert Haas [Fri, 31 Mar 2017 20:28:30 +0000 (16:28 -0400)]
Don't allocate storage for partitioned tables.
Also, don't allow setting reloptions on them, since that would have no
effect given the lack of storage. The patch does this by introducing
a new reloption kind for which there are currently no reloptions -- we
might have some in the future -- so it adjusts parseRelOptions to
handle that case correctly.
Bumped catversion. System catalogs that contained reloptions for
partitioned tables are no longer valid; plus, there are now fewer
physical files on disk, which is not technically a catalog change but
still a good reason to re-initdb.
Amit Langote, reviewed by Maksim Milyutin and Kyotaro Horiguchi and
revised a bit by me.
Simon Riggs [Thu, 30 Mar 2017 18:18:53 +0000 (14:18 -0400)]
Default monitoring roles
Three nologin roles with non-overlapping privs are created by default
* pg_read_all_settings - read all GUCs.
* pg_read_all_stats - pg_stat_*, pg_database_size(), pg_tablespace_size()
* pg_stat_scan_tables - may lock/scan tables
Top level role - pg_monitor includes all of the above by default, plus others
Author: Dave Page Reviewed-by: Stephen Frost, Robert Haas, Peter Eisentraut, Simon Riggs
Tom Lane [Thu, 30 Mar 2017 16:59:11 +0000 (12:59 -0400)]
Support \if ... \elif ... \else ... \endif in psql scripting.
This patch adds nestable conditional blocks to psql. The control
structure feature per se is complete, but the boolean expressions
understood by \if and \elif are pretty primitive; basically, after
variable substitution and backtick expansion, the result has to be
"true" or "false" or one of the other standard spellings of a boolean
value. But that's enough for many purposes, since you can always
do the heavy lifting on the server side; and we can extend it later.
Along the way, pay down some of the technical debt that had built up
around psql/command.c:
* Refactor exec_command() into a function per command, instead of
being a 1500-line monstrosity. This makes the file noticeably longer
because of repetitive function header/trailer overhead, but it seems
much more readable.
* Teach psql_get_variable() and psqlscanslash.l to suppress variable
substitution and backtick expansion on the basis of the conditional
stack state, thereby allowing removal of the OT_NO_EVAL kluge.
* Fix the no-doubt-once-expedient hack of sometimes silently substituting
mainloop.c's previous_buf for query_buf when calling HandleSlashCmds.
(It's a bit remarkable that commands like \r worked at all with that.)
Recall of a previous query is now done explicitly in the slash commands
where that should happen.
Corey Huinker, reviewed by Fabien Coelho, further hacking by me
Fujii Masao [Thu, 30 Mar 2017 16:31:15 +0000 (01:31 +0900)]
Simplify the example of VACUUM in documentation.
Previously a detailed activity report by VACUUM VERBOSE ANALYZE was
described as an example of VACUUM in docs. But it had been obsolete
for a long time. For example, commit feb4f44d296b88b7f0723f4a4f3945a371276e0b
updated the content of that activity report in 2003, but we had
forgotten to update the example.
So basically we need to update the example. But since no one cared
about the details of VACUUM output and complained about that mistake
for such long time, per discussion on hackers, we decided to get rid
of the detailed activity report from the example and simplify it.
Back-patch to all supported versions.
Reported by Masahiko Sawada, patch by me.
Discussion: https://postgr.es/m/CAD21AoAGA2pB3p-CWmTkxBsbkZS1bcDGBLcYVcvcDxspG_XAfA@mail.gmail.com
Andres Freund [Wed, 29 Mar 2017 20:16:49 +0000 (13:16 -0700)]
Remove support for version-0 calling conventions.
The V0 convention is failure prone because we've so far assumed that a
function is V0 if PG_FUNCTION_INFO_V1 is missing, leading to crashes
if a function was coded against the V1 interface. V0 doesn't allow
proper NULL, SRF and toast handling. V0 doesn't offer features that
V1 doesn't.
Thus remove V0 support and obsolete fmgr README contents relating to
it.
Author: Andres Freund, with contributions by Peter Eisentraut & Craig Ringer Reviewed-By: Peter Eisentraut, Craig Ringer
Discussion: https://postgr.es/m/20161208213441.k3mbno4twhg2qf7g@alap3.anarazel.de
Andres Freund [Wed, 29 Mar 2017 20:16:30 +0000 (13:16 -0700)]
Move contrib/seg to only use V1 calling conventions.
A later commit will remove V0 support.
Author: Andres Freund, with contributions by Craig Ringer Reviewed-By: Peter Eisentraut, Craig Ringer
Discussion: https://postgr.es/m/20161208213441.k3mbno4twhg2qf7g@alap3.anarazel.de
Peter Eisentraut [Wed, 29 Mar 2017 19:17:14 +0000 (15:17 -0400)]
pg_dump: Remove query truncation in error messages
Remove the behavior that a query mentioned in an error message would be
truncated to 128 characters. The queries that pg_dump runs are often
longer than that, and this behavior makes analyzing failures harder
unnecessarily.
Alvaro Herrera [Wed, 29 Mar 2017 15:18:48 +0000 (12:18 -0300)]
Simplify check of modified attributes in heap_update
The old coding was getting more complicated as new things were added,
and it would be barely tolerable with upcoming WARM updates and other
future features such as indirect indexes. The new coding incurs a small
performance cost in synthetic benchmark cases, and is barely measurable
in normal cases. A much larger benefit is expected from WARM, which
could actually bolt its needs on top of the existing coding, but it is
much uglier and bug-prone than doing it on this new code. Additional
optimization can be applied on top of this, if need be.
Reviewed-by: Pavan Deolasee, Amit Kapila, Mithun CY
Discussion: https://postgr.es/m/20161228232018.4hc66ndrzpz4g4wn@alvherre.pgsql
https://postgr.es/m/CABOikdMJfz69dBNRTOZcB6s5A0tf8OMCyQVYQyR-WFFdoEwKMQ@mail.gmail.com
Robert Haas [Wed, 29 Mar 2017 14:59:27 +0000 (10:59 -0400)]
Mark more functions parallel-restricted.
Commit 61c2e1a95f94bb904953a6281ce17a18ac38ee6d allowed parallel
query to be used in more places, revealing via buildfarm member
mandrill that several functions intended to be called from triggers
were incorrectly marked parallel-safe rather than parallel-restricted.
Report by Tom Lane. Patch by Rafia Sabih. Reviewed by me.
Robert Haas [Wed, 29 Mar 2017 13:44:29 +0000 (09:44 -0400)]
Plug race in dsa_attach.
With sufficiently bad luck, it was possible for a parallel worker to
attempt attach to a DSA area after all other backends have detached
from it, which is not legal. If the worker had waited a little longer
to get started, the DSM itself would have been destroyed, which is why
this wasn't noticed before.
Thomas Munro, per a report from Andreas Seltenreich
copyObject() is declared to return void *, which allows easily assigning
the result independent of the input, but it loses all type checking.
If the compiler supports typeof or something similar, cast the result to
the input type. This creates a greater amount of type safety. In some
cases, where the result is assigned to a generic type such as Node * or
Expr *, new casts are now necessary, but in general casts are now
unnecessary in the normal case and indicate that something unusual is
happening.
Reviewed-by: Mark Dilger <hornschnorter@gmail.com>
Peter Eisentraut [Wed, 29 Mar 2017 00:38:06 +0000 (20:38 -0400)]
Change 'diag' to 'note' in TAP tests
Reduce noise from TAP tests by changing 'diag' to 'note', so output only
goes to the test's log file not stdout, unless in verbose mode. This
also removes the junk on screen when running the TAP tests in parallel.
Alvaro Herrera [Fri, 24 Mar 2017 22:27:22 +0000 (19:27 -0300)]
Allow DSM segments to be created as pinned
dsm_create and dsm_attach assumed that a current resource owner was
always in place. Exploration with the API show that this is
inconvenient: sometimes one must create a dummy resowner, create/attach
the DSM, only to pin the mapping later, which is wasteful. Change
create/attach so that if there is no current resowner, the dsm is
effectively pinned right from the start.
Discussion: https://postgr.es/m/20170324232710.32acsfsvjqfgc6ud@alvherre.pgsql
Reviewed by Thomas Munro.
Tom Lane [Tue, 28 Mar 2017 22:05:03 +0000 (18:05 -0400)]
Make new expression eval code reject references to dropped columns.
Formerly, a Var referencing an already-dropped column was allowed and would
always produce a NULL value. However, that behavior was implemented in
slot_getattr which the new expression code doesn't use; thus there is now a
risk of returning theoretically-deleted data. We had regression test cases
that purported to exercise this, but they failed to expose any problem,
apparently because plpgsql filters the dropped column and produces an
output tuple that has a NULL there already.
Ideally the DROP or ALTER attempt in these test cases would get rejected
due to dependency checks; but until that happens, let's modify the behavior
so that we fail the query during executor start. This was already true for
the related case of a column having changed type underneath us, and there's
no obvious reason why we need to be laxer for dropped columns.
In passing, adjust the error messages in CheckVarSlotCompatibility to
include the composite type name. In the cases shown in the regression
tests this is always just "record", but it should be more useful in
actual stale-plan cases, where the slot tupdesc would be a table's
tupdesc directly.
Alvaro Herrera [Tue, 28 Mar 2017 15:52:55 +0000 (12:52 -0300)]
Remove direct uses of ItemPointer.{ip_blkid,ip_posid}
There are no functional changes here; this simply encapsulates knowledge
of the ItemPointerData struct so that a future patch can change things
without more breakage.
All direct users of ip_blkid and ip_posid are changed to use existing
macros ItemPointerGetBlockNumber and ItemPointerGetOffsetNumber
respectively. For callers where that's inappropriate (because they
Assert that the itempointer is is valid-looking), add
ItemPointerGetBlockNumberNoCheck and ItemPointerGetOffsetNumberNoCheck,
which lack the assertion but are otherwise identical.
Tom Lane [Tue, 28 Mar 2017 17:16:19 +0000 (13:16 -0400)]
Suppress implicit-conversion warnings seen with newer clang versions.
We were assigning values near 255 through "char *" pointers. On machines
where char is signed, that's not entirely kosher, and it's reasonable
for compilers to warn about it.
A better solution would be to change the pointer type to "unsigned char *",
but that would be vastly more invasive. For the moment, let's just apply
this simple backpatchable solution.
Peter Eisentraut [Tue, 28 Mar 2017 15:08:38 +0000 (11:08 -0400)]
dblink: Fix error reporting
The conname variable was not initialized in some code paths, resulting
in error reports referring to the "unnamed" connection rather than the
correct connection name.
Simon Riggs [Tue, 28 Mar 2017 14:05:21 +0000 (10:05 -0400)]
Cleanup slots during drop database
Automatically drop all logical replication slots associated with a
database when the database is dropped. Previously we threw an ERROR
if a slot existed. Now we throw ERROR only if a slot is active in
the database being dropped.
Peter Eisentraut [Tue, 28 Mar 2017 13:00:59 +0000 (09:00 -0400)]
Fix Perl code which had broken the Windows build
The previous change wanted to avoid modifying $_ in grep, but the code
just made the change in a local variable and then lost it. Rewrite the
code using a separate map and grep, which is clearer anyway.
Author: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
Tom Lane [Tue, 28 Mar 2017 00:14:36 +0000 (20:14 -0400)]
Show ignored constants as "$N" rather than "?" in pg_stat_statements.
The trouble with the original choice here is that "?" is a valid (and
indeed used) operator name, so that you could end up with ambiguous
statement texts like "SELECT ? ? ?". With this patch, you instead
see "SELECT $1 ? $2", which seems significantly more readable.
The numbers used for this purpose begin after the last actual $N parameter
in the particular query. The conflict with external parameters has its own
potential for confusion of course, but it was agreed to be an improvement
over the previous behavior.
Alvaro Herrera [Mon, 27 Mar 2017 17:52:19 +0000 (14:52 -0300)]
Fix uninitialized memory propagation mistakes
Valgrind complains that some uninitialized bytes are being passed around
by the extended statistics code since commit 7b504eb282ca2f, as reported
by Andres Freund. Silence it.
Tomas Vondra submitted a patch which he verified to fix the complaints
in his machine; however I messed with it a bit before pushing, so any
remaining problems are likely my (Álvaro's) fault.
Author: Tomas Vondra
Discussion: https://postgr.es/m/20170325211031.4xxoptigqxm2emn2@alap3.anarazel.de
Robert Haas [Mon, 27 Mar 2017 16:50:51 +0000 (12:50 -0400)]
Still more code review for single-page hash vacuuming.
Most seriously, fix use of incorrect block ID, per a report from
Jeff Janes that it causes a crash and a diagnosis from Amit Kapila.
Improve consistency between the hash and btree versions of this
code by adding back a PANIC that btree has, and by registering
data in the xlog record in the same way, per complaints from
Jeff Janes and Amit Kapila.
Tidy up some minor cosmetic points, per complaints from Amit
Kapila.
Patch by Ashutosh Sharma, reviewed by Amit Kapila, and tested by
Jeff Janes.
Teodor Sigaev [Mon, 27 Mar 2017 16:33:01 +0000 (19:33 +0300)]
Fsync directory after creating or unlinking file.
If file was created/deleted just before powerloss it's possible that
file system will miss that. To prevent it, call fsync() where creating/
unlinkg file is critical.
Author: Michael Paquier Reviewed-by: Ashutosh Bapat, Takayuki Tsunakawa, me
Alvaro Herrera [Mon, 27 Mar 2017 15:52:50 +0000 (12:52 -0300)]
Fix thinko in estimate_num_groups
The code for the reworked n-distinct estimation on commit 7b504eb282 was
written differently in a previous version of the patch, prior to commit;
on rewriting it, we missed updating an initializer. This caused the
code to (mistakenly) apply a fudge factor even in the case where a
single value is applied, leading to incorrect results.
This means that the 'relvarcount' variable name is now wrong. Add a
comment to try and make the situation clearer, and remove an incorrect
comment I added.
Problem noticed, and code patch, by Tomas Vondra. Additional commentary
by Álvaro.
Alvaro Herrera [Mon, 27 Mar 2017 15:40:42 +0000 (12:40 -0300)]
Rework the stats_ext test
As suggested by Tom Lane, avoid printing specific estimated cost values,
because they vary across architectures; instead, verify plan shapes (in
this case, HashAggregate vs. GroupAggregate), as we do in other planner
tests.
Peter Eisentraut [Mon, 27 Mar 2017 14:34:33 +0000 (10:34 -0400)]
Change default of log_directory to 'log'
The previous default 'pg_log' might have indicated by its "pg_" prefix
that it is an internal system directory. The new default is more in
line with the typical naming of directories with user-facing log files.
Together with the renaming of pg_clog and pg_xlog, this should clear up
that difference.
Alvaro Herrera [Mon, 27 Mar 2017 03:53:59 +0000 (00:53 -0300)]
Fix a couple of problems in pg_get_statisticsextdef
There was a thinko whereby we tested the wrong tuple after fetching it
from cache; avoid that by using generate_relation_name instead, which is
simpler. Also, the statistics name was not qualified, so add that. (It
could be argued that qualification should be conditional on the schema
not being on search path. We can add that later, but at least this form
is correct.)
Author: David Rowley, Álvaro Herrera
Discussion: https://postgr.es/m/CAKJS1f8RjLeVZJ2+93pdQGuZJeBF-ifsHaFMR-q-6-Z0qxA8cA@mail.gmail.com
Andrew Gierth [Mon, 27 Mar 2017 03:20:54 +0000 (04:20 +0100)]
Support hashed aggregation with grouping sets.
This extends the Aggregate node with two new features: HashAggregate
can now run multiple hashtables concurrently, and a new strategy
MixedAggregate populates hashtables while doing sorted grouping.
The planner will now attempt to save as many sorts as possible when
planning grouping sets queries, while not exceeding work_mem for the
estimated combined sizes of all hashtables used. No SQL-level changes
are required. There should be no user-visible impact other than the
new EXPLAIN output and possible changes to result ordering when ORDER
BY was not used (which affected a few regression tests). The
enable_hashagg option is respected.
Author: Andrew Gierth
Reviewers: Mark Dilger, Andres Freund
Discussion: https://postgr.es/m/87vatszyhj.fsf@news-spur.riddles.org.uk
Robert Haas [Mon, 27 Mar 2017 02:02:22 +0000 (22:02 -0400)]
Show more processes in pg_stat_activity.
Previously, auxiliary processes and background workers not connected
to a database (such as the logical replication launcher) weren't
shown. Include them, so that we can see the associated wait state
information. Add a new column to identify the processes type, so that
people can filter them out easily using SQL if they wish.
Before this patch was written, there was discussion about whether we
should expose this information in a separate view, so as to avoid
contaminating pg_stat_activity with things people might not want to
see. But putting everything in pg_stat_activity was a more popular
choice, so that's what the patch does.
Kuntal Ghosh, reviewed by Amit Langote and Michael Paquier. Some
revisions and bug fixes by me.
Tom Lane [Sun, 26 Mar 2017 23:14:47 +0000 (19:14 -0400)]
Improve performance of ExecEvalWholeRowVar.
In commit b8d7f053c, we needed to fix ExecEvalWholeRowVar to not change
the state of the slot it's copying. The initial quick hack at that
required two rounds of tuple construction, which is not very nice.
To fix, add another primitive to tuptoaster.c that does precisely what
we need. (I initially tried to do this by refactoring one of the
existing functions into two pieces; but it looked like that might hurt
performance for the existing case, and the amount of code that could
be shared is not very large, so I gave up on that.)
Tom Lane [Sun, 26 Mar 2017 22:14:03 +0000 (18:14 -0400)]
Use ExecPrepareExpr in place of ExecPrepareCheck where appropriate.
Change one more place where ExecInitCheck/ExecPrepareCheck's insistence
on getting implicit-AND-format quals wasn't really helpful, because the
caller had to do make_ands_implicit() for no reason that it cared about.
Using ExecPrepareExpr directly simplifies the code and saves cycles.
The only remaining use of these functions is to process
resultRelInfo->ri_PartitionCheck quals. However, implicit-AND format
does seem to be what we want for that, so leave it alone.
Tom Lane [Sun, 26 Mar 2017 21:35:35 +0000 (17:35 -0400)]
Fix unportable disregard of alignment requirements in RADIUS code.
The compiler is entitled to store a char[] local variable with no
particular alignment requirement. Our RADIUS code cavalierly took such
a local variable and cast its address to a struct type that does have
alignment requirements. On an alignment-picky machine this would lead
to bus errors. To fix, declare the local variable honestly, and then
cast its address to char * for use in the I/O calls.
Given the lack of field complaints, there must be very few if any
people affected; but nonetheless this is a clear portability issue,
so back-patch to all supported branches.
Noted while looking at a Coverity complaint in the same code.
Tom Lane [Sun, 26 Mar 2017 21:02:38 +0000 (17:02 -0400)]
Fix some minor resource leaks in PerformRadiusTransaction().
Failure to free serveraddrs pointed out by Coverity, failure to close
socket noted by code-reading. These bugs seem to be quite old, but
given the low probability of taking these error-exit paths and the
minimal consequences of the leaks (since the process would presumably
exit shortly anyway), it doesn't seem worth back-patching.
Tom Lane [Sun, 26 Mar 2017 19:57:02 +0000 (15:57 -0400)]
Improve implementation of EEOP_BOOLTEST_* opcodes.
Both Andres and I were happy with "*op->resvalue = *op->resvalue;",
but Coverity isn't; and it has a point, because some compilers might
not be smart enough to elide that. So remove it. In passing, also
avoid doing unnecessary assignments to *op->resnull when it's already
known to have the right value.
Peter Eisentraut [Sun, 26 Mar 2017 18:54:56 +0000 (14:54 -0400)]
doc: Clean up bibliography rendering for XSLT
In the DSSSL stylesheets, we had an extensive customization of the
bibliography rendering. Since the bibliography isn't that used much, it
doesn't seem worth doing an elaborate porting of that to XSLT. So this
just moves some things around, removes some unused things, and does some
minimal XSLT stylesheet customizations to make things look clean.
Andres Freund [Sat, 25 Mar 2017 22:32:01 +0000 (15:32 -0700)]
Remove unreachable code in expression evaluation.
The previous code still contained expression evaluation time support
for CaseExprs without a defresult. But transformCaseExpr() creates a
default expression if necessary.
Author: Andres Freund
Discussion: https://postgr.es/m/4834.1490480275@sss.pgh.pa.us
Andres Freund [Tue, 14 Mar 2017 22:45:36 +0000 (15:45 -0700)]
Faster expression evaluation and targetlist projection.
This replaces the old, recursive tree-walk based evaluation, with
non-recursive, opcode dispatch based, expression evaluation.
Projection is now implemented as part of expression evaluation.
This both leads to significant performance improvements, and makes
future just-in-time compilation of expressions easier.
The speed gains primarily come from:
- non-recursive implementation reduces stack usage / overhead
- simple sub-expressions are implemented with a single jump, without
function calls
- sharing some state between different sub-expressions
- reduced amount of indirect/hard to predict memory accesses by laying
out operation metadata sequentially; including the avoidance of
nearly all of the previously used linked lists
- more code has been moved to expression initialization, avoiding
constant re-checks at evaluation time
Future just-in-time compilation (JIT) has become easier, as
demonstrated by released patches intended to be merged in a later
release, for primarily two reasons: Firstly, due to a stricter split
between expression initialization and evaluation, less code has to be
handled by the JIT. Secondly, due to the non-recursive nature of the
generated "instructions", less performance-critical code-paths can
easily be shared between interpreted and compiled evaluation.
The new framework allows for significant future optimizations. E.g.:
- basic infrastructure for to later reduce the per executor-startup
overhead of expression evaluation, by caching state in prepared
statements. That'd be helpful in OLTPish scenarios where
initialization overhead is measurable.
- optimizing the generated "code". A number of proposals for potential
work has already been made.
- optimizing the interpreter. Similarly a number of proposals have
been made here too.
The move of logic into the expression initialization step leads to some
backward-incompatible changes:
- Function permission checks are now done during expression
initialization, whereas previously they were done during
execution. In edge cases this can lead to errors being raised that
previously wouldn't have been, e.g. a NULL array being coerced to a
different array type previously didn't perform checks.
- The set of domain constraints to be checked, is now evaluated once
during expression initialization, previously it was re-built
every time a domain check was evaluated. For normal queries this
doesn't change much, but e.g. for plpgsql functions, which caches
ExprStates, the old set could stick around longer. The behavior
around might still change.
Author: Andres Freund, with significant changes by Tom Lane,
changes by Heikki Linnakangas Reviewed-By: Tom Lane, Heikki Linnakangas
Discussion: https://postgr.es/m/20161206034955.bh33paeralxbtluv@alap3.anarazel.de
Tom Lane [Sat, 25 Mar 2017 21:25:28 +0000 (17:25 -0400)]
Remember to drop roles created by regression tests.
Commit e3920ac82 created "regress_subscription_user2" in subscription.sql,
but forgot to drop it, causing the regression tests to fail if run twice
without re-initdb'ing.
projects / postgresql / log