tests: Add missing checks of GC_malloc result (for out-of-memory)

tests: Add missing checks of GC_malloc result (for out-of-memory)

* tests/test.c (mktree): Insert CHECK_OUT_OF_MEMORY to report mktree()
out-of-memory failures.
* tests/trace_test.c (mktree): Test whether GC_MALLOC_ATOMIC returns
NULL (exit with an error code and the appropriate message printed in
this case).