Fix datumSerialize infrastructure to not crash on non-varlena data.

Commit 1efc7e538 did a poor job of emulating existing logic for touching
Datums that might be expanded-object pointers.  It didn't check for typlen
being -1 first, which meant it could crash on fixed-length pass-by-ref
values, and probably on cstring values as well.  It also didn't use
DatumGetPointer before VARATT_IS_EXTERNAL_EXPANDED, which while currently
harmless is not according to documentation nor prevailing style.

I also think the lack of any explanation as to why datumSerialize makes
these particular nonobvious choices is pretty awful, so fix that.

Per report from Jarred Ward.  Back-patch to 9.6 where this code came in.

Discussion: https://postgr.es/m/6F61E6D2-2F5E-4794-9479-A429BE1CEA4B@simple.com

diff --git a/src/backend/utils/adt/datum.c b/src/backend/utils/adt/datum.c
index 071a7d4db168986a193966580da7216710259f79..81e3b52ec9bec8eda038884cbc9cf86779551df7 100644 (file)
--- a/src/backend/utils/adt/datum.c
+++ b/src/backend/utils/adt/datum.c
@@ -268,11 +268,11 @@ datumEstimateSpace(Datum value, bool isnull, bool typByVal, int typLen)
                /* no need to use add_size, can't overflow */
                if (typByVal)
                        sz += sizeof(Datum);
-               else if (VARATT_IS_EXTERNAL_EXPANDED(value))
+               else if (typLen == -1 &&
+                                VARATT_IS_EXTERNAL_EXPANDED(DatumGetPointer(value)))
                {
-                       ExpandedObjectHeader *eoh = DatumGetEOHP(value);
-
-                       sz += EOH_get_flat_size(eoh);
+                       /* Expanded objects need to be flattened, see comment below */
+                       sz += EOH_get_flat_size(DatumGetEOHP(value));
                }
                else
                        sz += datumGetSize(value, typByVal, typLen);
@@ -286,6 +286,13 @@ datumEstimateSpace(Datum value, bool isnull, bool typByVal, int typLen)
  *
  * Serialize a possibly-NULL datum into caller-provided storage.
  *
+ * Note: "expanded" objects are flattened so as to produce a self-contained
+ * representation, but other sorts of toast pointers are transferred as-is.
+ * This is because the intended use of this function is to pass the value
+ * to another process within the same database server.  The other process
+ * could not access an "expanded" object within this process's memory, but
+ * we assume it can dereference the same TOAST pointers this one can.
+ *
  * The format is as follows: first, we write a 4-byte header word, which
  * is either the length of a pass-by-reference datum, -1 for a
  * pass-by-value datum, or -2 for a NULL.  If the value is NULL, nothing
@@ -310,7 +317,8 @@ datumSerialize(Datum value, bool isnull, bool typByVal, int typLen,
                header = -2;
        else if (typByVal)
                header = -1;
-       else if (VARATT_IS_EXTERNAL_EXPANDED(value))
+       else if (typLen == -1 &&
+                        VARATT_IS_EXTERNAL_EXPANDED(DatumGetPointer(value)))
        {
                eoh = DatumGetEOHP(value);
                header = EOH_get_flat_size(eoh);