delaunay_remove_holes: fix mismatch of calling convention in callback

The compiler said about this code:

  delaunay.c: In function ‘delaunay_remove_holes’:
  delaunay.c:45:9: warning: cast between incompatible function types from
    ‘gboolean (*)(GtsTriangle *)’ {aka ‘int (*)(struct _GtsTriangle *)’}
    to ‘gint (*)(void *, void *)’ {aka ‘int (*)(void *, void *)’}
    [-Wcast-function-type]
         (GtsFunc) triangle_is_hole, NULL);

This warning is not spurious. In particular, the mismatch in the number of
arguments passed to the callback means the callback function has a different
calling convention than expected by the code calling it. The result of this can
be stack corruption or incorrect interpretation of function arguments.

In practice, all major native calling conventions use registers for both
function types, so this is somewhat benign. However, this likely caused problems
on stack-based environments like WASM and JITs.

diff --git a/CHANGELOG.md b/CHANGELOG.md
index ad595527c2fe9b99569f6febe934ea3cf5200136..5a56cd8e2549b85c8e9d1ea70c205b743ad95ae8 100644 (file)
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -22,6 +22,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - remove Bashism from `gvmap.sh` #2151
 - Lefty artifacts are no longer installed when Lefty is disabled #2153
 - Smyrna artifacts are no longer installed when Smyrna is disabled
+- calling convention mismatch in delaunay.c’s GTS code
 
 ## [2.49.3] – 2021-10-22
 

diff --git a/lib/neatogen/delaunay.c b/lib/neatogen/delaunay.c
index 413cb0029e1aac39031269d12514daa28a468aa2..dc3953a51d0746dae9904b6394942ab6d0ba486f 100644 (file)
--- a/lib/neatogen/delaunay.c
+++ b/lib/neatogen/delaunay.c
@@ -23,26 +23,27 @@
 #ifdef HAVE_GTS
 #include <gts.h>
 
-static gboolean triangle_is_hole(GtsTriangle * t)
+static gint triangle_is_hole(void *triangle, void *ignored)
 {
+    GtsTriangle *t = triangle;
+    (void)ignored;
+
     GtsEdge *e1, *e2, *e3;
     GtsVertex *v1, *v2, *v3;
-    gboolean ret;
 
     gts_triangle_vertices_edges(t, NULL, &v1, &v2, &v3, &e1, &e2, &e3);
 
     if ((GTS_IS_CONSTRAINT(e1) && GTS_SEGMENT(e1)->v1 != v1) ||
        (GTS_IS_CONSTRAINT(e2) && GTS_SEGMENT(e2)->v1 != v2) ||
        (GTS_IS_CONSTRAINT(e3) && GTS_SEGMENT(e3)->v1 != v3))
-       ret = TRUE;
-    else ret = FALSE;
-    return ret;
+       return TRUE;
+
+    return FALSE;
 }
 
 static guint delaunay_remove_holes(GtsSurface * surface)
 {
-    return gts_surface_foreach_face_remove(surface,
-                                   (GtsFunc) triangle_is_hole, NULL);
+    return gts_surface_foreach_face_remove(surface, triangle_is_hole, NULL);
 }
 
 /* Derived classes for vertices and faces so we can assign integer ids