pathplan: fix integer overflow with > 46341 nodes

After this change, a ASan+UBSan build of Graphviz can process the #1999 example
without crashing. Graphs with >46341 (⌈√INT_MAX⌉) nodes no longer cause an
integer overflow.

Gitlab: fixes #1999
Reported-by: Lockywolf

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5e7a67ad2738a7db074d77a0b73d37e72f6fecc0..30479deecbfd4db5582d83d1c78736e3f7a02824 100644 (file)
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -15,6 +15,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - The Autotools build system no longer errors when attempting libANN discovery
   during cross-compilation. This was a regression in Graphviz 7.0.6. #2335
+- Graphs with more than 46341 (⌈√INT_MAX⌉) nodes no longer crash `twopi`. #1999
 
 ## [7.0.6] – 2023-01-06
 

diff --git a/lib/pathplan/visibility.c b/lib/pathplan/visibility.c
index 94abb67dd4f40f584ae40ab206d0f0398dd2d4bf..52d21bf2ddf9476da3b185457bcd4a6150621167 100644 (file)
--- a/lib/pathplan/visibility.c
+++ b/lib/pathplan/visibility.c
@@ -8,7 +8,7 @@
  * Contributors: Details at https://graphviz.org
  *************************************************************************/
 
-
+#include <assert.h>
 #include <pathplan/vis.h>
 #include <stdbool.h>
 #include <stdlib.h>
@@ -33,8 +33,9 @@ static array2 allocArray(int V, int extra)
     array2 arr;
     COORD *p;
 
+    assert(V >= 0);
     arr = malloc((V + extra) * sizeof(COORD *));
-    p = calloc(V * V, sizeof(COORD));
+    p = calloc((size_t)V * (size_t)V, sizeof(COORD));
     for (i = 0; i < V; i++) {
        arr[i] = p;
        p += V;