From: Alexander Korotkov <akorotkov@postgresql.org>
Date: Sun, 8 Sep 2019 18:07:30 +0000 (+0300)
Subject: Fix handling Inf and Nan values in GiST pairing heap comparator
X-Git-Url: https://granicus.if.org/sourcecode?a=commitdiff_plain;h=8f724002e2fe0e415f5a603462440d2f297f287c;p=postgresql

Fix handling Inf and Nan values in GiST pairing heap comparator

Previously plain float comparison was used in GiST pairing heap.  Such
comparison doesn't provide proper ordering for value sets containing Inf and Nan
values.  This commit fixes that by usage of float8_cmp_internal().  Note, there
is remaining problem with NULL distances, which are represented as Inf in
pairing heap.  It would be fixes in subsequent commit.

Backpatch to all supported versions.

Reported-by: Andrey Borodin
Discussion: https://postgr.es/m/CAPpHfdsNvNdA0DBS%2BwMpFrgwT6C3-q50sFVGLSiuWnV3FqOJuQ%40mail.gmail.com
Author: Alexander Korotkov
Reviewed-by: Heikki Linnakangas
Backpatch-through: 9.4
---

diff --git a/src/backend/access/gist/gistscan.c b/src/backend/access/gist/gistscan.c
index 058544e2ae..f5d8a01fe5 100644
--- a/src/backend/access/gist/gistscan.c
+++ b/src/backend/access/gist/gistscan.c
@@ -17,6 +17,7 @@
 #include "access/gist_private.h"
 #include "access/gistscan.h"
 #include "access/relscan.h"
+#include "utils/builtins.h"
 #include "utils/lsyscache.h"
 #include "utils/memutils.h"
 #include "utils/rel.h"
@@ -36,8 +37,10 @@ pairingheap_GISTSearchItem_cmp(const pairingheap_node *a, const pairingheap_node
 	/* Order according to distance comparison */
 	for (i = 0; i < scan->numberOfOrderBys; i++)
 	{
-		if (sa->distances[i] != sb->distances[i])
-			return (sa->distances[i] < sb->distances[i]) ? 1 : -1;
+		int			cmp = -float8_cmp_internal(sa->distances[i], sb->distances[i]);
+
+		if (cmp != 0)
+			return cmp;
 	}
 
 	/* Heap items go before inner pages, to ensure a depth-first search */