From: Matthew Fernandez <matthew.fernandez@gmail.com>
Date: Sun, 25 Dec 2022 20:25:49 +0000 (-0800)
Subject: twopi: fix crash with > 46341 nodes
X-Git-Url: https://granicus.if.org/sourcecode?a=commitdiff_plain;h=f3d95baeba5eac8724d8cac9e234e46cd2ba178f;p=graphviz

twopi: fix crash with > 46341 nodes

UBSan revealed the graph attached to #1999 was triggering an integer overflow in
this multiplication, later on causing a crash in `twopi`. Any number of nodes
≥⌈√INT_MAX⌉ exceeds INT_MAX during multiplication. This fix still does not
enable the graph to be processed in a reasonable amount of time, and it still
crashes later after several hours due to another integer overflow.

Gitlab: #1999
---

diff --git a/lib/twopigen/circle.c b/lib/twopigen/circle.c
index 6046a5ded..78faa5f19 100644
--- a/lib/twopigen/circle.c
+++ b/lib/twopigen/circle.c
@@ -8,6 +8,7 @@
  * Contributors: Details at https://graphviz.org
  *************************************************************************/
 
+#include    <assert.h>
 #include    <cgraph/alloc.h>
 #include    <twopigen/circle.h>
 #include    <ctype.h>
@@ -70,7 +71,8 @@ static bool isLeaf(Agraph_t * g, Agnode_t * n)
 static void initLayout(Agraph_t * g)
 {
     int nnodes = agnnodes(g);
-    uint64_t INF = (uint64_t)(nnodes * nnodes);
+    assert(nnodes >= 0);
+    uint64_t INF = (uint64_t)nnodes * (uint64_t)nnodes;
 
     for (Agnode_t *n = agfstnode(g); n; n = agnxtnode(g, n)) {
 	SCENTER(n) = INF;