twopi: fix crash with > 46341 nodes

UBSan revealed the graph attached to #1999 was triggering an integer overflow in
this multiplication, later on causing a crash in `twopi`. Any number of nodes
≥⌈√INT_MAX⌉ exceeds INT_MAX during multiplication. This fix still does not
enable the graph to be processed in a reasonable amount of time, and it still
crashes later after several hours due to another integer overflow.

Gitlab: #1999

diff --git a/lib/twopigen/circle.c b/lib/twopigen/circle.c
index 6046a5ded6e8d5985cd70729297179b1f89d55b7..78faa5f19229ebd36b61d0beffdf83b98bf36195 100644 (file)
--- a/lib/twopigen/circle.c
+++ b/lib/twopigen/circle.c
@@ -8,6 +8,7 @@
  * Contributors: Details at https://graphviz.org
  *************************************************************************/
 
+#include    <assert.h>
 #include    <cgraph/alloc.h>
 #include    <twopigen/circle.h>
 #include    <ctype.h>
@@ -70,7 +71,8 @@ static bool isLeaf(Agraph_t * g, Agnode_t * n)
 static void initLayout(Agraph_t * g)
 {
     int nnodes = agnnodes(g);
-    uint64_t INF = (uint64_t)(nnodes * nnodes);
+    assert(nnodes >= 0);
+    uint64_t INF = (uint64_t)nnodes * (uint64_t)nnodes;
 
     for (Agnode_t *n = agfstnode(g); n; n = agnxtnode(g, n)) {
        SCENTER(n) = INF;