Norbert Runge [Tue, 18 Dec 2018 23:07:38 +0000 (15:07 -0800)]
ICU-20217 Adds ICU fuzzer target originally from https://github.com/google/oss-fuzz/tree/master/projects/icu directly into ICU on GitHub.
Also, puts code under Unicode copyright notice. Code is from oss-fuzz project and was originally contributed by a Googler as well.
Shane Carr [Thu, 15 Nov 2018 07:38:54 +0000 (23:38 -0800)]
ICU-20138 Adding FormattedValue APIs in C, C++, and Java.
- Wires up FormattedNumber[Range] in applicable languages.
- Adds new header files and tests, with minor cleanup to old tests.
- Adds code to guarantee terminating NUL in FormattedNumber[Range].
- Cleanup of API docs for inherited methods in FormattedNumber[Range].
Shane Carr [Wed, 12 Dec 2018 01:14:58 +0000 (17:14 -0800)]
ICU-10923 Adding --filterDir option to genrb.
- Reads, parses, and applies the filter file syntax.
- Removes unused keys from the resource bundle.
- Adds sample filter txt file with test in intltest.
Shane Carr [Thu, 6 Dec 2018 08:33:10 +0000 (00:33 -0800)]
ICU-10923 Adding initial version of data file filtering.
- Reads filters.json or .hjson from ICU_DATA_FILTER_FILE environment variable
- Adds DepTarget for dependency semantics, and warns for missing deps.
- Fixes certain tests that crash with sliced locale data.
- Includes support for 4 filter types.
* Note: the diff will show changes to "runstatedir". This is
a feature which ICU does not use, and it is not in any released
autoconf version. Perhaps some distributions patched autoconf
locally.
Markus Scherer [Thu, 11 Oct 2018 23:04:05 +0000 (16:04 -0700)]
ICU-20203 update ICU to Unicode 12 beta
- data as of 2018-nov-26
- API constants for new blocks & scripts
- sync RBBIMonkeyTest.java test data with C++
Fredrik Roubert [Mon, 19 Nov 2018 14:12:05 +0000 (15:12 +0100)]
ICU-20276 Accept empty strings in Locale::setUnicodeKeywordValue().
The API documentation is perfectly clear about this, an empty string for
the value means that the keyword should be removed:
@param keywordValue value of the keyword to be set. If 0-length or
NULL, will result in the keyword being removed. No error is given if
that keyword does not exist.
Shane Carr [Fri, 16 Nov 2018 04:58:24 +0000 (20:58 -0800)]
ICU-20230 Implementing COMPACT field.
Adds some plumbing to allow MutablePatternModifier to set fields, and otherwise builds upon the infrastructure from the previous commit to add the MEASURE_UNIT field.
Shane Carr [Sat, 25 Aug 2018 06:39:10 +0000 (23:39 -0700)]
ICU-10923 Rewriting data build orchestration in Python.
- Creates new Python package in icu4c/data/buildtool
- Creates BUILRDULES.py in icu4c/data and icu4c/test/testdata, unified between Unix/Windows
- Removes most data build orchestration rules from makedata.mak, testdata.mak, data/Makefile.in, and test/testdata/Makefile.in
- Removes pool.res files and builds them on the fly instead
Fredrik Roubert [Mon, 5 Nov 2018 13:32:38 +0000 (14:32 +0100)]
ICU-20202 Merge #ifdef __cplusplus blocks.
For historical reasons (commit 3b12074b4092241a7e73ec8c7ea6be75548b1b82),
all C++ code (even #include statements) in cmemory.h is contained in an
#ifdef block at the end of the file. A recent bugfix inadvertently added
an additional #ifdef __cplusplus block at the beginning of the file to
add a new #include statement so that C++ #include statements now are
found in two places (commit 1bad36b91afa1fb33a0d45902a163a923d6e860a).
Markus Scherer [Fri, 9 Nov 2018 20:54:22 +0000 (12:54 -0800)]
ICU-20250 make UnicodeSet(intprop=value) faster
- fastpath for UnicodeSet.add(new last range)
- fewer UnicodeSet memory allocations:
initial internal list array, exponential array growth,
allocate strings list/set only when first one is added
- faster CodePointTrie.getRange(): fewer calls to filter function
- revert UnicodeSet(intprop=value) from trie ranges to range starts + lookup
- cache per-int-prop range starts: fewer lookups
Fredrik Roubert [Fri, 2 Nov 2018 16:50:34 +0000 (17:50 +0100)]
ICU-20169 Replace uprv_malloc() / uprv_free() with MemoryPool.
This resolves the immediate problem of brittle memory management
in the error handling code.
An obvious future improvement would be to replace the old C style
"plain struct with pointers" VariantListEntry, AttributeListEntry
and ExtensionListEntry with contemporary C++ style containers that
take care of ownership and memory management.
Fredrik Roubert [Mon, 29 Oct 2018 21:20:30 +0000 (22:20 +0100)]
ICU-20202 Replace CodePointMatcherWarehouse with MemoryPool.
The shared templated helper class MemoryPool can be used to replace the
local helper class CodePointMatcherWarehouse, reducing the amount of
specialized code needed.
Fredrik Roubert [Tue, 30 Oct 2018 22:35:03 +0000 (23:35 +0100)]
ICU-20158 Pass ByteSink from Locale::forLanguageTag() to uloc_forLanguageTag().
This eliminates the need for a scratch buffer in Locale::forLanguageTag()
and also the need for counting bytes required in uloc_forLanguageTag(),
something that ByteSink will now handle correctly.
Fredrik Roubert [Tue, 30 Oct 2018 21:12:07 +0000 (22:12 +0100)]
ICU-20202 Replace char* kwdBuf with MemoryPool<CharString>.
Instead of _appendLDMLExtensionAsKeywords() requiring to receive a pre-
allocated buffer of sufficient size to store all the temporary strings
it needs to store, have it use a MemoryPool<CharString> to allocate
storage space as needed.
Storing strings as individual CharString objects, instead of as NUL
delimited substrings in a contiguous memory area, also eliminates the
need for keeping track of string boundaries and NUL terminators.
Fredrik Roubert [Mon, 29 Oct 2018 21:50:27 +0000 (22:50 +0100)]
ICU-20202 Replace UVector with MemoryPool in uloc_keytype.
By moving the required call to uhash_close() into the destructor of
LocExtKeyData and using CharString instead of raw chunks of bytes
allocated with uprv_malloc(), it becomes easier to guarantee that
memory handling is correct, without leaks or overflows.
With the need for custom deleter functions removed, the code doesn't use
any of the additional functionality provided by UVector, it just needs a
simple way to keep track of allocated objects to delete them after it's
done using them, which MemoryPool does in a simpler and typesafe way.
Fredrik Roubert [Mon, 29 Oct 2018 21:29:12 +0000 (22:29 +0100)]
ICU-20202 Replace UVector with MemoryPool in CalendarDataSink.
CalendarDataSink doesn't use any of the additional functionality
provided by UVector, it just needs a simple way to keep track of
allocated objects to delete them after it's done using them.
Fredrik Roubert [Mon, 29 Oct 2018 21:12:00 +0000 (22:12 +0100)]
ICU-20202 Replace CharStringPool with MemoryPool.
The shared templated helper class MemoryPool is a drop-in replacement
for the local helper class CharStringPool, with a simpler implementation
and an interface that allows parameters to be passed to the constructor.
Fredrik Roubert [Wed, 24 Oct 2018 20:45:25 +0000 (22:45 +0200)]
ICU-20202 Add an internal helper class to own heap allocated objects.
This will be used first to replace existing locally defined memory
management helper classes such as CharStringPool (uloc_tag.cpp) and
CodePointMatcherWarehouse (numparse_affixes.h), then in new code.