Optimize vmem_alloc() retry path

For performance reasons the reworked kmem code maps vmem_alloc() to
kmalloc_node() for allocations less than spa_kmem_alloc_max.  This
allows for more concurrency in the system and less contention of
the virtual address space.  Generally, this is a good thing.

However, in the case when the kmalloc_node() fails it makes little
sense to retry it using kmalloc_node() again.  It will likely fail
in exactly the same way.  A smarter strategy is to abandon this
optimization and retry using spl_vmalloc() which is very likely
to succeed.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ned Bass <bass6@llnl.gov>
Closes #428

diff --git a/module/spl/spl-kmem.c b/module/spl/spl-kmem.c
index e97d5f25559de615f65d15da14fb075ffc45f356..92263f8f4ec4d8fce3514276a52ef01849a77b25 100644 (file)
--- a/module/spl/spl-kmem.c
+++ b/module/spl/spl-kmem.c
@@ -149,6 +149,7 @@ inline void *
 spl_kmem_alloc_impl(size_t size, int flags, int node)
 {
        gfp_t lflags = kmem_flags_convert(flags);
+       int use_vmem = 0;
        void *ptr;
 
        /*
@@ -182,7 +183,7 @@ spl_kmem_alloc_impl(size_t size, int flags, int node)
                 * impact performance so frequently manipulating the virtual
                 * address space is strongly discouraged.
                 */
-               if (unlikely(size > spl_kmem_alloc_max)) {
+               if ((size > spl_kmem_alloc_max) || use_vmem) {
                        if (flags & KM_VMEM) {
                                ptr = spl_vmalloc(size, lflags, PAGE_KERNEL);
                        } else {
@@ -195,6 +196,15 @@ spl_kmem_alloc_impl(size_t size, int flags, int node)
                if (likely(ptr) || (flags & KM_NOSLEEP))
                        return (ptr);
 
+               /*
+                * For vmem_alloc() and vmem_zalloc() callers retry immediately
+                * using spl_vmalloc() which is unlikely to fail.
+                */
+               if ((flags & KM_VMEM) && (use_vmem == 0))  {
+                       use_vmem = 1;
+                       continue;
+               }
+
                if (unlikely(__ratelimit(&kmem_alloc_ratelimit_state))) {
                        printk(KERN_WARNING
                            "Possible memory allocation deadlock: "