get_int_lsb_first: fix undefined behavior in little endian decoding

When this function was called with a byte length of 4 (something that happens
often in this file), the last iteration of this loop would shift an int left by
24. On a platform with 32-bit ints (most platforms Graphviz runs on) this shifts
_into_ the sign bit, something that is undefined behavior with respect to the C
standard.

The changes in this commit make the shift well defined and remove a
-Wsign-conversion warning.

diff --git a/CHANGELOG.md b/CHANGELOG.md
index b33b2c0412fe111bb0bb79b10f7353e7bbfa93ce..54b97a939a6a7e2d3b95230a3838edfa7b637e99 100644 (file)
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -26,6 +26,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Smyrna artifacts are no longer installed when Smyrna is disabled
 - calling convention mismatches in delaunay.c’s GTS code
 - impure assertion in `jacobi`
+- undefined behavior in libgvc’s reading of little endian numbers
 
 ## [2.49.3] – 2021-10-22
 

diff --git a/lib/gvc/gvusershape.c b/lib/gvc/gvusershape.c
index 6fa4eb898555496187d5edd5df4cb647a9163eb0..0a78c13411352c48170d32d92bc976853f707239 100644 (file)
--- a/lib/gvc/gvusershape.c
+++ b/lib/gvc/gvusershape.c
@@ -122,7 +122,7 @@ static boolean get_int_lsb_first(FILE *f, size_t sz, unsigned int *val) {
        ch = fgetc(f);
        if (feof(f))
            return FALSE;
-       *val |= (ch << 8*i);
+       *val |= (unsigned)ch << 8 * i;
     }
     return TRUE;
 }