From: Tyson Andre <tysonandre775@hotmail.com>
Date: Sat, 28 Sep 2019 13:54:43 +0000 (-0400)
Subject: Reduce memory used by token_get_all()
X-Git-Tag: php-7.4.0RC3~12
X-Git-Url: https://granicus.if.org/sourcecode?a=commitdiff_plain;h=7e0f7b7677e2fd76e83c2b731c3c0f3bd1a7279e;p=php

Reduce memory used by token_get_all()

Around a quarter of all strings in array tokens would have a string that's one
character long (e.g. ` `, `\`, `1`)

For parsing a large number of php files,
The memory increase dropped from 378374248 to 369535688 (2.5%)

Closes GH-4753.
---

diff --git a/ext/tokenizer/tokenizer.c b/ext/tokenizer/tokenizer.c
index 91ace6f701..3d343fec4d 100644
--- a/ext/tokenizer/tokenizer.c
+++ b/ext/tokenizer/tokenizer.c
@@ -110,7 +110,11 @@ static void add_token(zval *return_value, int token_type,
 		zval keyword;
 		array_init(&keyword);
 		add_next_index_long(&keyword, token_type);
-		add_next_index_stringl(&keyword, (char *) text, leng);
+		if (leng == 1) {
+			add_next_index_str(&keyword, ZSTR_CHAR(text[0]));
+		} else {
+			add_next_index_stringl(&keyword, (char *) text, leng);
+		}
 		add_next_index_long(&keyword, lineno);
 		add_next_index_zval(return_value, &keyword);
 	} else {