Fredrik Roubert [Mon, 5 Nov 2018 13:32:38 +0000 (14:32 +0100)]
ICU-20202 Merge #ifdef __cplusplus blocks.
For historical reasons (commit 3b12074b4092241a7e73ec8c7ea6be75548b1b82),
all C++ code (even #include statements) in cmemory.h is contained in an
#ifdef block at the end of the file. A recent bugfix inadvertently added
an additional #ifdef __cplusplus block at the beginning of the file to
add a new #include statement so that C++ #include statements now are
found in two places (commit 1bad36b91afa1fb33a0d45902a163a923d6e860a).
Markus Scherer [Fri, 9 Nov 2018 20:54:22 +0000 (12:54 -0800)]
ICU-20250 make UnicodeSet(intprop=value) faster
- fastpath for UnicodeSet.add(new last range)
- fewer UnicodeSet memory allocations:
initial internal list array, exponential array growth,
allocate strings list/set only when first one is added
- faster CodePointTrie.getRange(): fewer calls to filter function
- revert UnicodeSet(intprop=value) from trie ranges to range starts + lookup
- cache per-int-prop range starts: fewer lookups
Fredrik Roubert [Fri, 2 Nov 2018 16:50:34 +0000 (17:50 +0100)]
ICU-20169 Replace uprv_malloc() / uprv_free() with MemoryPool.
This resolves the immediate problem of brittle memory management
in the error handling code.
An obvious future improvement would be to replace the old C style
"plain struct with pointers" VariantListEntry, AttributeListEntry
and ExtensionListEntry with contemporary C++ style containers that
take care of ownership and memory management.
Fredrik Roubert [Mon, 29 Oct 2018 21:20:30 +0000 (22:20 +0100)]
ICU-20202 Replace CodePointMatcherWarehouse with MemoryPool.
The shared templated helper class MemoryPool can be used to replace the
local helper class CodePointMatcherWarehouse, reducing the amount of
specialized code needed.
Fredrik Roubert [Tue, 30 Oct 2018 22:35:03 +0000 (23:35 +0100)]
ICU-20158 Pass ByteSink from Locale::forLanguageTag() to uloc_forLanguageTag().
This eliminates the need for a scratch buffer in Locale::forLanguageTag()
and also the need for counting bytes required in uloc_forLanguageTag(),
something that ByteSink will now handle correctly.
Fredrik Roubert [Tue, 30 Oct 2018 21:12:07 +0000 (22:12 +0100)]
ICU-20202 Replace char* kwdBuf with MemoryPool<CharString>.
Instead of _appendLDMLExtensionAsKeywords() requiring to receive a pre-
allocated buffer of sufficient size to store all the temporary strings
it needs to store, have it use a MemoryPool<CharString> to allocate
storage space as needed.
Storing strings as individual CharString objects, instead of as NUL
delimited substrings in a contiguous memory area, also eliminates the
need for keeping track of string boundaries and NUL terminators.
Fredrik Roubert [Mon, 29 Oct 2018 21:50:27 +0000 (22:50 +0100)]
ICU-20202 Replace UVector with MemoryPool in uloc_keytype.
By moving the required call to uhash_close() into the destructor of
LocExtKeyData and using CharString instead of raw chunks of bytes
allocated with uprv_malloc(), it becomes easier to guarantee that
memory handling is correct, without leaks or overflows.
With the need for custom deleter functions removed, the code doesn't use
any of the additional functionality provided by UVector, it just needs a
simple way to keep track of allocated objects to delete them after it's
done using them, which MemoryPool does in a simpler and typesafe way.
Fredrik Roubert [Mon, 29 Oct 2018 21:29:12 +0000 (22:29 +0100)]
ICU-20202 Replace UVector with MemoryPool in CalendarDataSink.
CalendarDataSink doesn't use any of the additional functionality
provided by UVector, it just needs a simple way to keep track of
allocated objects to delete them after it's done using them.
Fredrik Roubert [Mon, 29 Oct 2018 21:12:00 +0000 (22:12 +0100)]
ICU-20202 Replace CharStringPool with MemoryPool.
The shared templated helper class MemoryPool is a drop-in replacement
for the local helper class CharStringPool, with a simpler implementation
and an interface that allows parameters to be passed to the constructor.
Fredrik Roubert [Wed, 24 Oct 2018 20:45:25 +0000 (22:45 +0200)]
ICU-20202 Add an internal helper class to own heap allocated objects.
This will be used first to replace existing locally defined memory
management helper classes such as CharStringPool (uloc_tag.cpp) and
CodePointMatcherWarehouse (numparse_affixes.h), then in new code.
Fredrik Roubert [Mon, 22 Oct 2018 20:28:20 +0000 (22:28 +0200)]
ICU-20132 Pass ByteSink from Locale::toLanguageTag() to uloc_toLanguageTag().
This eliminates the need for a scratch buffer in Locale::toLanguageTag()
and also the need for counting bytes required in uloc_toLanguageTag(),
something that ByteSink will now handle correctly and thereby
eliminating the bug where too few bytes required was returned.
Fredrik Roubert [Fri, 12 Oct 2018 12:33:03 +0000 (14:33 +0200)]
ICU-20080 Avoid strange compiler behaviour in ASSERT_EQUAL() macro.
Using temporary variables for the two values to be compared here makes
GCC compile the code just like we expect it to. (What it really is that
it otherwise does on some architechtures remains a mystery.)
This will make the tests pass as expected also on IA-32 with GCC.
Jeff Genovy [Fri, 5 Oct 2018 17:44:52 +0000 (10:44 -0700)]
ICU-20196 Fix MSVC/clang compiler warning C4251, export explicit template, required for building DLLs on Windows. Also fix a few other warnings as well.
ICU-9562 Update language tag mapping per the latest IANA registry
uloc_forLanguageTag has a few mapping tables to map grandfathered
language tags and deprecated language subtags to their preferred or
modern values.
Update them based on the latest version of the IANA
language subtag registry. [1]
Five grandfathered tags without a preferred value are still mapped to
what ICU has mapped them to for backward compatibility until the
wisdom of continuing to do so is reviewed.
In addition, map redundant language tags to their preferred values
regardless of whether they're followed by other subtags or not. (e.g.
zh-yue vs zh-yue-u-co-pinyin) .
Similary, ja-latn-hepburn-heploc is mapped to ja-latn-alaic97 (the
variant subtag 'hepburn-helploc' with the prefix 'ja-latn' has the
preferred value, 'alaic97') .
Update the mapping for deprecated language subtags (e.g. 'jw' to
'jv' and a bunch of 3-letter language codes).
Add a new table for deprecated region subtags to map them to their
modern values. (e.g. 'DD' to 'DE').
Add a new test case for deprecated language and region mapping and
a few more cases for updated grandfathered and redundant tag mapping.