Nikita Popov [Tue, 3 Nov 2020 13:15:05 +0000 (14:15 +0100)]
Fix variance checks on resolved union types
This is a bit annoying: When preloading is used, types might be
resolved during inheritance checks, so we need to deal with CE
types rather than just NAME types everywhere.
Nikita Popov [Tue, 3 Nov 2020 10:50:14 +0000 (11:50 +0100)]
Don't ignore internal classes during preloading
When preloading, it's fine to make use of internal class information,
as we do not support Windows. It is also necessary to allow proper
variance checks against internal classes.
Alex Dowad [Sun, 18 Oct 2020 15:49:57 +0000 (17:49 +0200)]
Fix mbstring support for ARMSCII-8
- Identify filter was completely wrong.
- Respect `mb_substitute_character` rather than converting invalid bytes to
Unicode 0xFFFD (generic replacement character).
- Don't convert Unicode 0xFFFD to a valid ARMSCII-8 character.
- When converting ARMSCII-8 to ARMSCII-8, don't pass invalid bytes through
silently.
Alex Dowad [Sat, 29 Aug 2020 17:12:28 +0000 (19:12 +0200)]
Optimize (AND FIX) mb_check_encoding (cut execution time by 50%+)
Previously, `mb_check_encoding` did an awful lot of unneeded work. In order to
determine whether a string was valid or not, it would convert the whole string
into wchar (code points), which required dynamically allocating a (potentially
large) buffer. Then it would turn right around and convert that big 'ol buffer
of code points back to the original encoding again. Finally, it would check
whether any invalid bytes were detected during that long and onerous process.
The thing is, mbstring _already_ has machinery for detecting whether a string
is valid in a certain encoding or not, and it doesn't require copying any data
around or allocating buffers. Better yet, it can fail fast when an invalid byte
is found. Why not use it? It's sure a lot faster!
Further, the legacy code was also badly broken. Why? Because aside from
checking whether illegal characters were detected, it would also check whether
the conversion to and from wchars was lossless. But, some encodings have
more than one valid encoding for the same character. In such cases, it is
not possible to make the conversion to and from wchars lossless for every
valid character. So `mb_check_encoding` would actually reject good strings
in a lot of encodings!
Alex Dowad [Sun, 18 Oct 2020 13:30:03 +0000 (15:30 +0200)]
Fix mbstring support for CP1254 encoding
One funny thing: while the original author used Unicode 0xFFFD (generic
replacement character) for invalid bytes in CP1251 and CP1252, for CP1254
they used 0xFFFE, which is not a valid Unicode codepoint at all, but is a
reversed byte-order mark. Probably this was by mistake.
Anyways,
- Fixed identify filter, which was completely wrong.
- Don't convert Unicode 0xFFFE to a random (but valid) CP1254 byte.
- When converting CP1254 to CP1254, don't pass invalid bytes through silently.
Alex Dowad [Sun, 18 Oct 2020 13:12:11 +0000 (15:12 +0200)]
Fix mbstring support for CP1251 encoding
- Identify filter was as wrong as wrong can be.
- Invalid CP1251 byte 0x98 was converted to Unicode 0xFFFD (generic
replacement character), rather than respecting `mb_substitute_character`.
- Unicode 0xFFFD was converted to some random CP1251 byte.
- When converting CP1251 to CP1251, don't pass invalid bytes through silently.
Anatol Belski [Sun, 30 Aug 2020 12:14:04 +0000 (14:14 +0200)]
hash: Add MurmurHash3 with streaming support
The implementation is based on the upstream PMurHash. The following
variants are implemented
- murmur3a, 32-bit hash
- murmur3c, 128-bit hash for x86
- murmur3f, 128-bit hash for x64
The custom seed support is not targeted by this implementation. It will
need a major change to the API, so then custom arguments can be passed
through `hash_init`. For now, the starting hash is always zero.
Fixes bug #68109, closes #6059
Signed-off-by: Anatol Belski <ab@php.net> Co-Developed-by: Michael Wallner <mike@php.net> Signed-off-by: Michael Wallner <mike@php.net>
Alex Dowad [Sun, 18 Oct 2020 12:41:40 +0000 (14:41 +0200)]
Add test suite for CP1252 encoding
Also remove a bogus test (bug62545.phpt) which wrongly assumed that all invalid
characters in CP1251 and CP1252 should map to Unicode 0xFFFD (REPLACEMENT
CHARACTER).
mbstring has an interface to specify what invalid characters should be
replaced with; it's called `mb_substitute_character`. If a user wants to see
the Unicode 'replacement character', they can specify that using
`mb_substitute_character`. But if they specify something else, we should
follow that.
Alex Dowad [Sun, 18 Oct 2020 12:56:32 +0000 (14:56 +0200)]
Fix mbstring support for CP1252 encoding
It's a bit surprising how much was broken here.
- Identify filter was utterly and completely wrong.
- Instead of handling invalid CP1252 bytes as specified by
`mb_substitute_character`, it would convert them to Unicode 0xFFFD
(generic replacement character).
- When converting ISO-8859-1 to CP1252, invalid ISO-8859-1 bytes would
be passed through silently.
- Unicode codepoints from 0x80-0x9F were converted to CP1252 bytes 0x80-0x9F,
which is wrong.
- Unicode codepoint 0xFFFD was converted to CP1252 0x9F, which is very wrong.
Also clean up some unneeded code, and make the conversion table consistent with
others by using zero as a 'invalid' marker, rather than 0xFFFD.
Alex Dowad [Thu, 29 Oct 2020 21:10:04 +0000 (23:10 +0200)]
Convert numeric string array keys to integers correctly in JITted code
While fixing bugs in mbstring, one of my new test cases failed with a strange
error message stating: 'Warning: Undefined array key 1...', when clearly the
array key had been set properly.
GDB'd that sucker and found that JIT'd PHP code was calling directly into
`zend_hash_add_new` (which was not converting the numeric string key to an
integer properly). But where was that code coming from? I examined the disasm,
looked up symbols to figure out where call instructions were going, then grepped
the codebase for those function names. It soon became clear that the disasm I
was looking at was compiled from `zend_jit_fetch_dim_w_helper`.
Nikita Popov [Wed, 21 Oct 2020 13:01:47 +0000 (15:01 +0200)]
Add --repeat testing mode
This testing mode executes the test multiple times in the same
process (but in different requests). It is primarily intended to
catch tracing JIT bugs, but also catches state leaks across
requests.
Nikita Popov [Fri, 30 Oct 2020 10:11:16 +0000 (11:11 +0100)]
Fixed bug #80290
Dropping the dtor arg args[3] rather than using STR_COPY: Since
PHP 8, we no longer support separation in call_user_function(),
so we also don't need to worry about things like arguments being
replaced with references.
divinity76 [Tue, 27 Oct 2020 15:19:58 +0000 (16:19 +0100)]
Use constant size string in hash bench.php
I don't like the previous behaviour where the bytes to hash change
every time the code changes, that may make it difficult to compare
hash() performance changes over time.
Use a fixed number instead, and allow passing an override for a
different length.
Fix bug #72413: Segfault with get_result and PS cursors
We cannot simply switch to use_result here, because the fetch_row
methods in get_result mode and in use_result/store_result mode
are different: In one case it accepts a statement, in the other
a return value zval. Thus, doing a switch to use_result results
in a segfault when trying to fetch a row.
Actually supporting get_result with cursors would require adding
cursor support in mysqlnd_result, not just mysqlnd_ps. That would
be a significant amount of effort and, given the age of the issue,
does not appear to be particularly likely to happen soon.
As such, we simply generate an error when using get_result()
with cursors, which is much better than causing a segfault.
Instead, parameter binding needs to be used.
Nikita Popov [Thu, 29 Oct 2020 13:07:08 +0000 (14:07 +0100)]
Handle errors during PDO row fetch
The EOF flag also gets set on error, so we always end up ignoring
errors here.
However, we should only check errors for unbuffered results. For
buffered results, this function is guaranteed not to error, and
querying the errno may return an unrelated error.
Fix #44618: Fetching may rely on uninitialized data
Unless `SQLGetData()` returns `SQL_SUCCESS` or `SQL_SUCCESS_WITH_INFO`,
the `StrLen_or_IndPtr` output argument is not guaranteed to be properly
set. Thus we handle retrieval failure other than `SQL_ERROR` by
yielding `false` for those column values and raising a warning.