Ensure 64bit arithmetic when calculating tapeSpace

In tuplesort.c:inittapes(), we calculate tapeSpace by first figuring
out how many 'tapes' we can use (maxTapes) and then multiplying the
result by the tape buffer overhead for each.  Unfortunately, when
we are on a system with an 8-byte long, we allow work_mem to be
larger than 2GB and that allows maxTapes to be large enough that the
32bit arithmetic can overflow when multiplied against the buffer
overhead.

When this overflow happens, we end up adding the overflow to the
amount of space available, causing the amount of memory allocated to
be larger than work_mem.

Note that to reach this point, you have to set work mem to at least
24GB and be sorting a set which is at least that size.  Given that a
user who can set work_mem to 24GB could also set it even higher, if
they were looking to run the system out of memory, this isn't
considered a security issue.

This overflow risk was found by the Coverity scanner.

Back-patch to all supported branches, as this issue has existed
since before 8.4.

diff --git a/src/backend/utils/sort/tuplesort.c b/src/backend/utils/sort/tuplesort.c
index d876369c6bb6145b6cbaf7b859f07cad3c3cc488..5d42480e604d57a6a7a31a863cf89875879ec6f9 100644 (file)
--- a/src/backend/utils/sort/tuplesort.c
+++ b/src/backend/utils/sort/tuplesort.c
@@ -1779,7 +1779,7 @@ inittapes(Tuplesortstate *state)
         * account for tuple space, so we don't care if LACKMEM becomes
         * inaccurate.)
         */
-       tapeSpace = maxTapes * TAPE_BUFFER_OVERHEAD;
+       tapeSpace = (long) maxTapes * TAPE_BUFFER_OVERHEAD;
        if (tapeSpace + GetMemoryChunkSpace(state->memtuples) < state->allowedMem)
                USEMEM(state, tapeSpace);