From: Peter Eisentraut Date: Fri, 31 Oct 2014 02:52:21 +0000 (-0400) Subject: doc: Wording and formatting improvements in new logical decoding docs X-Git-Tag: REL9_5_ALPHA1~1284 X-Git-Url: https://granicus.if.org/sourcecode?a=commitdiff_plain;h=50709a1d427831d48a5aa14655e9f57e3721a06b;p=postgresql doc: Wording and formatting improvements in new logical decoding docs --- diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index 7197e9dd0a..7559492b4d 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -17,16 +17,15 @@ The format in which those changes are streamed is determined by the output - plugin used. An example plugin is provided, and additional plugins can be + plugin used. An example plugin is provided in the PostgreSQL distribution. + Additional plugins can be written to extend the choice of available formats without modifying any core code. Every output plugin has access to each individual new row produced by INSERT and the new row version created by UPDATE. Availability of old row versions for UPDATE and DELETE depends on - the configured - REPLICA - IDENTITY. + the configured replica identity (see ). @@ -40,17 +39,21 @@ - Logical Decoding Example + Logical Decoding Examples + - The following example demonstrates the SQL interface. + The following example demonstrates controlling logical decoding using the + SQL interface. + Before you can use logical decoding, you must set to logical and - to at least 1. - Then, you should connect to the target database (in the example + to at least 1. Then, you + should connect to the target database (in the example below, postgres) as a superuser. + postgres=# -- Create a slot named 'regression_slot' using the output plugin 'test_decoding' postgres=# SELECT * FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding'); @@ -140,40 +143,47 @@ postgres=# SELECT pg_drop_replication_slot('regression_slot'); (1 row) + - The following example shows usage of the walsender interface using - the pg_recvlogical - shell command. It requires the replication configurations to be allowed - (see ) - and max_wal_senders to be set sufficiently high for - another connection. + The following example shows how logical decoding is controlled over the + streaming replication protocol, using the + program included in the PostgreSQL + distribution. This requires that client authentication is set up to allow + replication connections + (see ) and + that max_wal_senders is set sufficiently high to an + additional connection. -# pg_recvlogical -d postgres --slot test --create-slot -# pg_recvlogical -d postgres --slot test --start -f - -CTRL-Z -# psql -d postgres -c "INSERT INTO data(data) VALUES('4');" -# fg +$ pg_recvlogical -d postgres --slot test --create-slot +$ pg_recvlogical -d postgres --slot test --start -f - +ControlD +$ psql -d postgres -c "INSERT INTO data(data) VALUES('4');" +$ fg BEGIN 693 table public.data: INSERT: id[integer]:4 data[text]:'4' COMMIT 693 -CTRL-C -# pg_recvlogical -d postgres --slot test --drop-slot +ControlC +$ pg_recvlogical -d postgres --slot test --drop-slot + Logical Decoding Concepts Logical Decoding + Logical Decoding + Logical decoding is the process of extracting all persistent changes to a database's tables into a coherent, easy to understand format which can be interpreted without detailed knowledge of the database's internal state. + In PostgreSQL, logical decoding is implemented by decoding the contents of the write-ahead @@ -184,34 +194,40 @@ CTRL-C Replication Slots + replication slot logical replication + In the context of logical replication, a slot represents a stream of - changes which can be replayed to a client in the order they were made on + changes that can be replayed to a client in the order they were made on the origin server. Each slot streams a sequence of changes from a single database, sending each change exactly once (except when peeking forward in the stream). + PostgreSQL also has streaming replication slots (see ), but they are used somewhat differently there. + - Replication slots have an identifier which is unique across all databases + A replication slot has an identifier that is unique across all databases in a PostgreSQL cluster. Slots persist independently of the connection using them and are crash-safe. + Multiple independent slots may exist for a single database. Each slot has its own state, allowing different consumers to receive changes from different points in the database change stream. For most applications, a separate slot will be required for each consumer. + A logical replication slot knows nothing about the state of the receiver(s). It's even possible to have multiple different receivers using @@ -219,17 +235,19 @@ CTRL-C on from when the last receiver stopped consuming them. Only one receiver may consume changes from a slot at any given time. + Replication slots persist across crashes and know nothing about the state of their consumer(s). They will prevent removal of required resources even when there is no connection using them. This consumes storage because neither required WAL nor required rows from the system catalogs - can be removed by VACUUM as long as they are required by a replication - slot, so if a slot is no longer required it should be dropped. + can be removed by VACUUM as long as they are required by a replication + slot. So if a slot is no longer required it should be dropped. + Output Plugins @@ -237,63 +255,84 @@ CTRL-C representation into the format the consumer of a replication slot desires. + Exported Snapshots - When a new replication slot is created using the walsender interface a - snapshot is exported - (see ) which will show + When a new replication slot is created using the streaming replication interface, + a snapshot is exported + (see ), which will show exactly the state of the database after which all changes will be included in the change stream. This can be used to create a new replica by using SET TRANSACTION SNAPSHOT to read the state of the database at the moment the slot was created. This transaction can then be used to dump the - database's state at that point in time which afterwards can be updated + database's state at that point in time, which afterwards can be updated using the slot's contents without losing any changes. + Streaming Replication Protocol Interface + - The CREATE_REPLICATION_SLOT slot_name LOGICAL - options, DROP_REPLICATION_SLOT slot_name - and START_REPLICATION SLOT slot_name LOGICAL options - commands can be used to create, drop and stream changes from a replication - slot respectively. These commands are only available over a replication + The commands + + + CREATE_REPLICATION_SLOT slot_name LOGICAL options + + + + DROP_REPLICATION_SLOT slot_name + + + + START_REPLICATION SLOT slot_name LOGICAL options + + + are used to create, drop, and stream changes from a replication + slot, respectively. These commands are only available over a replication connection; they cannot be used via SQL. - See . + See for details on these commands. + - The pg_recvlogical command - (see ) can be used to control logical - decoding over a walsender connection. + The command can be used to control + logical decoding over a streaming replication connection. (It is uses + these commands internally.) + Logical Decoding <acronym>SQL</acronym> Interface + See for detailed documentation on the SQL-level API for interacting with logical decoding. + Synchronous replication (see ) is - only supported on replication slots used over the walsender interface. The + only supported on replication slots used over the streaming-replication interface. The function interface and additional, non-core interfaces do not support synchronous replication. + - System catalogs related to logical decoding + System Catalogs Related to Logical Decoding + The pg_replication_slots view and the pg_stat_replication view provide information about the current state of replication slots and - walsender connections respectively. These views apply to both physical and + streaming replication connections respectively. These views apply to both physical and logical replication. + Logical Decoding Output Plugins @@ -338,6 +377,7 @@ typedef void (*LogicalOutputPluginInit)(struct OutputPluginCallbacks *cb); Capabilities + To decode, format and output changes, output plugins can use most of the backend's normal infrastructure, including calling output functions. Read @@ -349,55 +389,60 @@ typedef void (*LogicalOutputPluginInit)(struct OutputPluginCallbacks *cb); ALTER TABLE user_catalog_table SET (user_catalog_table = true); CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true); - Any actions leading to xid assignment are prohibited. That, among others, - includes writing to tables, performing DDL changes and + Any actions leading to transaction ID assignment are prohibited. That, among others, + includes writing to tables, performing DDL changes, and calling txid_current(). Output Modes + Output plugin callbacks can pass data to the consumer in nearly arbitrary formats. For some use cases, like viewing the changes via SQL, returning - data in a datatype that can contain arbitrary data (i.e. bytea) is + data in a data type that can contain arbitrary data (e.g., bytea) is cumbersome. If the output plugin only outputs textual data in the - server's encoding it can declare that by + server's encoding, it can declare that by setting OutputPluginOptions.output_mode to OUTPUT_PLUGIN_TEXTUAL_OUTPUT instead of OUTPUT_PLUGIN_BINARY_OUTPUT in the startup - callback. In that case all the data has to be in the server's encoding - so a text can contain it. This is checked in assertion enabled + callback. In that case, all the data has to be in the server's encoding + so that a text datum can contain it. This is checked in assertion-enabled builds. Output Plugin Callbacks + An output plugin gets notified about changes that are happening via various callbacks it needs to provide. + Concurrent transactions are decoded in commit order, and only changes belonging to a specific transaction are decoded between the begin and commit callbacks. Transactions that were rolled back explicitly or implicitly never get - decoded. Successful SAVEPOINTs are + decoded. Successful savepoints are folded into the transaction containing them in the order they were executed within that transaction. + Only transactions that have already safely been flushed to disk will be - decoded. That can lead to a COMMIT not immediately being decoded in a + decoded. That can lead to a COMMIT not immediately being decoded in a directly following pg_logical_slot_get_changes() when synchronous_commit is set to off. + Startup Callback @@ -426,6 +471,7 @@ typedef struct OutputPluginOptions or OUTPUT_PLUGIN_BINARY_OUTPUT. See also . + The startup callback should validate the options present in ctx->output_plugin_options. If the output plugin @@ -433,8 +479,10 @@ typedef struct OutputPluginOptions use ctx->output_plugin_private to store it. + Shutdown Callback + The optional shutdown_cb callback is called whenever a formerly active replication slot is not used anymore and can @@ -446,9 +494,11 @@ typedef void (*LogicalDecodeShutdownCB) ( ); - + + Transaction Begin Callback + The required begin_cb callback is called whenever a start of a committed transaction has been decoded. Aborted transactions @@ -463,9 +513,11 @@ typedef void (*LogicalDecodeBeginCB) ( the transaction, like the time stamp at which it has been committed and its XID. - + + Transaction End Callback + The required commit_cb callback is called whenever a transaction commit has been @@ -480,13 +532,14 @@ typedef void (*LogicalDecodeCommitCB) ( + - Callback called for each individual change in a - transaction + Change Callback + The required change_cb callback is called for every individual row modification inside a transaction, may it be - an INSERT, UPDATE + an INSERT, UPDATE, or DELETE. Even if the original command modified several rows at once the callback will be called individually for each row. @@ -506,6 +559,7 @@ typedef void (*LogicalDecodeChangeCB) ( change describing the row modification are passed in. + Only changes in user defined tables that are not unlogged @@ -516,20 +570,23 @@ typedef void (*LogicalDecodeChangeCB) ( + - Functions for producing output from an output plugin + Functions for Producing Output + To actually produce output, output plugins can write data to the StringInfo output buffer in ctx->out when inside - the begin_cb, commit_cb + the begin_cb, commit_cb, or change_cb callbacks. Before writing to the output - buffer OutputPluginPrepareWrite(ctx, last_write) has + buffer, OutputPluginPrepareWrite(ctx, last_write) has to be called, and after finishing writing to the - buffer OutputPluginWrite(ctx, last_write) has to be + buffer, OutputPluginWrite(ctx, last_write) has to be called to perform the write. The last_write indicates whether a particular write was the callback's last write. + The following example shows how to output data to the consumer of an output plugin: @@ -541,8 +598,10 @@ OutputPluginWrite(ctx, true); + Logical Decoding Output Writers + It is possible to add more output methods for logical decoding. For details, see @@ -552,19 +611,22 @@ OutputPluginWrite(ctx, true); (see ). + - Synchronous replication support for Logical Decoding + Synchronous Replication Support for Logical Decoding + - Logical decoding may be used to to build + Logical decoding can be used to to build synchronous replication solutions with the same user interface as synchronous replication for streaming - replication. To do this, the walsender interface + replication. To do this, the streaming replication interface (see ) must be used to stream out data. Clients have to send Standby status update (F) (see ) messages, just like streaming replication clients do. + A synchronous replica receiving changes via logical decoding will work in diff --git a/doc/src/sgml/ref/pg_recvlogical.sgml b/doc/src/sgml/ref/pg_recvlogical.sgml index e8a60a9c26..a28dbc3f18 100644 --- a/doc/src/sgml/ref/pg_recvlogical.sgml +++ b/doc/src/sgml/ref/pg_recvlogical.sgml @@ -355,6 +355,14 @@ PostgreSQL documentation + + Examples + + + See for an example. + + + See Also