Add some more defenses against silly estimates to gincostestimate().

A report from Andy Colson showed that gincostestimate() was not being
nearly paranoid enough about whether to believe the statistics it finds in
the index metapage.  The problem is that the metapage stats (other than the
pending-pages count) are only updated by VACUUM, and in the worst case
could still reflect the index's original empty state even when it has grown
to many entries.  We attempted to deal with that by scaling up the stats to
match the current index size, but if nEntries is zero then scaling it up
still gives zero.  Moreover, the proportion of pages that are entry pages
vs. data pages vs. pending pages is unlikely to be estimated very well by
scaling if the index is now orders of magnitude larger than before.

We can improve matters by expanding the use of the rule-of-thumb estimates
I introduced in commit 7fb008c5ee59b040: if the index has grown by more
than a cutoff amount (here set at 4X growth) since VACUUM, then use the
rule-of-thumb numbers instead of scaling.  This might not be exactly right
but it seems much less likely to produce insane estimates.

I also improved both the scaling estimate and the rule-of-thumb estimate
to account for numPendingPages, since it's reasonable to expect that that
is accurate in any case, and certainly pages that are in the pending list
are not either entry or data pages.

As a somewhat separate issue, adjust the estimation equations that are
concerned with extra fetches for partial-match searches.  These equations
suppose that a fraction partialEntries / numEntries of the entry and data
pages will be visited as a consequence of a partial-match search.  Now,
it's physically impossible for that fraction to exceed one, but our
estimate of partialEntries is mostly bunk, and our estimate of numEntries
isn't exactly gospel either, so we could arrive at a silly value.  In the
example presented by Andy we were coming out with a value of 100, leading
to insane cost estimates.  Clamp the fraction to one to avoid that.

Like the previous patch, back-patch to all supported branches; this
problem can be demonstrated in one form or another in all of them.

diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index 15121bc4600adc318df858ac1d9a76cc672bfcb9..8d109aafe787b5e5eb161b91d0ed85de3ff51625 100644 (file)
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -7246,6 +7246,7 @@ gincostestimate(PG_FUNCTION_ARGS)
                                numEntries;
        GinQualCounts counts;
        bool            matchPossible;
+       double          partialScale;
        double          entryPagesFetched,
                                dataPagesFetched,
                                dataPagesFetchedBySel;
@@ -7274,42 +7275,59 @@ gincostestimate(PG_FUNCTION_ARGS)
                memset(&ginStats, 0, sizeof(ginStats));
        }
 
-       if (ginStats.nTotalPages > 0 && ginStats.nEntryPages > 0 && numPages > 0)
+       /*
+        * Assuming we got valid (nonzero) stats at all, nPendingPages can be
+        * trusted, but the other fields are data as of the last VACUUM.  We can
+        * scale them up to account for growth since then, but that method only
+        * goes so far; in the worst case, the stats might be for a completely
+        * empty index, and scaling them will produce pretty bogus numbers.
+        * Somewhat arbitrarily, set the cutoff for doing scaling at 4X growth; if
+        * it's grown more than that, fall back to estimating things only from the
+        * assumed-accurate index size.  But we'll trust nPendingPages in any case
+        * so long as it's not clearly insane, ie, more than the index size.
+        */
+       if (ginStats.nPendingPages < numPages)
+               numPendingPages = ginStats.nPendingPages;
+       else
+               numPendingPages = 0;
+
+       if (numPages > 0 && ginStats.nTotalPages <= numPages &&
+               ginStats.nTotalPages > numPages / 4 &&
+               ginStats.nEntryPages > 0 && ginStats.nEntries > 0)
        {
                /*
-                * We got valid stats.  nPendingPages can be trusted, but the other
-                * fields are data as of the last VACUUM.  Scale them by the ratio
-                * numPages / nTotalPages to account for growth since then.
+                * OK, the stats seem close enough to sane to be trusted.  But we
+                * still need to scale them by the ratio numPages / nTotalPages to
+                * account for growth since the last VACUUM.
                 */
                double          scale = numPages / ginStats.nTotalPages;
 
-               numEntryPages = ginStats.nEntryPages;
-               numDataPages = ginStats.nDataPages;
-               numPendingPages = ginStats.nPendingPages;
-               numEntries = ginStats.nEntries;
-
-               numEntryPages = ceil(numEntryPages * scale);
-               numDataPages = ceil(numDataPages * scale);
-               numEntries = ceil(numEntries * scale);
+               numEntryPages = ceil(ginStats.nEntryPages * scale);
+               numDataPages = ceil(ginStats.nDataPages * scale);
+               numEntries = ceil(ginStats.nEntries * scale);
                /* ensure we didn't round up too much */
-               numEntryPages = Min(numEntryPages, numPages);
-               numDataPages = Min(numDataPages, numPages - numEntryPages);
+               numEntryPages = Min(numEntryPages, numPages - numPendingPages);
+               numDataPages = Min(numDataPages,
+                                                  numPages - numPendingPages - numEntryPages);
        }
        else
        {
                /*
-                * It's a hypothetical index, or perhaps an index created pre-9.1 and
-                * never vacuumed since upgrading.  Invent some plausible internal
-                * statistics based on the index page count.  We estimate that 90% of
-                * the index is entry pages, and the rest is data pages.  Estimate 100
-                * entries per entry page; this is rather bogus since it'll depend on
-                * the size of the keys, but it's more robust than trying to predict
-                * the number of entries per heap tuple.
+                * We might get here because it's a hypothetical index, or an index
+                * created pre-9.1 and never vacuumed since upgrading (in which case
+                * its stats would read as zeroes), or just because it's grown too
+                * much since the last VACUUM for us to put our faith in scaling.
+                *
+                * Invent some plausible internal statistics based on the index page
+                * count (and clamp that to at least 10 pages, just in case).  We
+                * estimate that 90% of the index is entry pages, and the rest is data
+                * pages.  Estimate 100 entries per entry page; this is rather bogus
+                * since it'll depend on the size of the keys, but it's more robust
+                * than trying to predict the number of entries per heap tuple.
                 */
                numPages = Max(numPages, 10);
-               numEntryPages = floor(numPages * 0.90);
-               numDataPages = numPages - numEntryPages;
-               numPendingPages = 0;
+               numEntryPages = floor((numPages - numPendingPages) * 0.90);
+               numDataPages = numPages - numPendingPages - numEntryPages;
                numEntries = floor(numEntryPages * 100);
        }
 
@@ -7435,16 +7453,21 @@ gincostestimate(PG_FUNCTION_ARGS)
        /*
         * Add an estimate of entry pages read by partial match algorithm. It's a
         * scan over leaf pages in entry tree.  We haven't any useful stats here,
-        * so estimate it as proportion.
+        * so estimate it as proportion.  Because counts.partialEntries is really
+        * pretty bogus (see code above), it's possible that it is more than
+        * numEntries; clamp the proportion to ensure sanity.
         */
-       entryPagesFetched += ceil(numEntryPages * counts.partialEntries / numEntries);
+       partialScale = counts.partialEntries / numEntries;
+       partialScale = Min(partialScale, 1.0);
+
+       entryPagesFetched += ceil(numEntryPages * partialScale);
 
        /*
         * Partial match algorithm reads all data pages before doing actual scan,
-        * so it's a startup cost. Again, we haven't any useful stats here, so,
-        * estimate it as proportion
+        * so it's a startup cost.  Again, we haven't any useful stats here, so
+        * estimate it as proportion.
         */
-       dataPagesFetched = ceil(numDataPages * counts.partialEntries / numEntries);
+       dataPagesFetched = ceil(numDataPages * partialScale);
 
        /*
         * Calculate cache effects if more than one scan due to nestloops or array