vn_getf/vn_releasef should not accept negative file descriptors

C type coercion rules require that negative numbers be converted into
positive numbers via wraparound such that a negative -1 becomes a
positive 1. This causes vn_getf to return a file handle when it should
return NULL whenever a positive file descriptor existed with the same
value. We should check for a negative file descriptor and return NULL
instead.

This was caught by ClusterHQ's unit testing.

Reference:
http://stackoverflow.com/questions/50605/signed-to-unsigned-conversion-in-c-is-it-always-safe

Signed-off-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Andriy Gapon <avg@FreeBSD.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #450

diff --git a/module/spl/spl-vnode.c b/module/spl/spl-vnode.c
index 1e26b8e29a89e736f82e26a2a87e5606326bc889..4c62097dcc574b008d36abd1f9c079f793fd1b3b 100644 (file)
--- a/module/spl/spl-vnode.c
+++ b/module/spl/spl-vnode.c
@@ -648,6 +648,9 @@ vn_getf(int fd)
        vnode_t *vp;
        int rc = 0;
 
+       if (fd < 0)
+               return (NULL);
+
        /* Already open just take an extra reference */
        spin_lock(&vn_file_lock);
 
@@ -733,6 +736,9 @@ vn_releasef(int fd)
 {
        file_t *fp;
 
+       if (fd < 0)
+               return;
+
        spin_lock(&vn_file_lock);
        fp = file_find(fd);
        if (fp) {