Avoid integer overflow while sifting-up a heap in tuplesort.c.

If the number of tuples in the heap exceeds approximately INT_MAX/2,
this loop's calculation "2*i+1" could overflow, resulting in a crash.
Fix it by using unsigned int rather than int for the relevant local
variables; that shouldn't cost anything extra on any popular hardware.
Per bug #14722 from Sergey Koposov.

Original patch by Sergey Koposov, modified by me per a suggestion
from Heikki Linnakangas to use unsigned int not int64.

Back-patch to 9.4, where tuplesort.c grew the ability to sort as many
as INT_MAX tuples in-memory (commit 263865a48).

Discussion: https://postgr.es/m/20170629161637.1478.93109@wrigleys.postgresql.org

diff --git a/src/backend/utils/sort/tuplesort.c b/src/backend/utils/sort/tuplesort.c
index 64ef6e5032c46c1f242736bee06df5d0daa2f5df..4dd5407f741ee8db2b63cb1a13729ef6cd453073 100644 (file)
--- a/src/backend/utils/sort/tuplesort.c
+++ b/src/backend/utils/sort/tuplesort.c
@@ -3800,7 +3800,7 @@ tuplesort_heap_siftup(Tuplesortstate *state, bool checkIndex)
 {
        SortTuple  *memtuples = state->memtuples;
        SortTuple  *tuple;
-       int                     i,
+       unsigned int i,
                                n;
 
        Assert(!checkIndex || state->currentRun == RUN_FIRST);
@@ -3809,12 +3809,17 @@ tuplesort_heap_siftup(Tuplesortstate *state, bool checkIndex)
 
        CHECK_FOR_INTERRUPTS();
 
+       /*
+        * state->memtupcount is "int", but we use "unsigned int" for i, j, n.
+        * This prevents overflow in the "2 * i + 1" calculation, since at the top
+        * of the loop we must have i < n <= INT_MAX <= UINT_MAX/2.
+        */
        n = state->memtupcount;
        tuple = &memtuples[n];          /* tuple that must be reinserted */
        i = 0;                                          /* i is where the "hole" is */
        for (;;)
        {
-               int                     j = 2 * i + 1;
+               unsigned int j = 2 * i + 1;
 
                if (j >= n)
                        break;