Jeff Genovy [Thu, 30 Aug 2018 01:50:50 +0000 (18:50 -0700)]
ICU-13712 ICU4C does not report OOM if it fails to memory map the data file(s) (#30)
ICU does not report Out-Of-Memory (OOM) if it fails to memory map the data file(s) when calling the various platform API(s) to do so.
When you are using ICU with memory-mapped data file(s), and ICU fails to map the data file due to being out-of-memory, it does not bubble this failure up to the API that was called. You will instead get back the error U_MISSING_RESOURCE_ERROR, rather than U_MEMORY_ALLOCATION_ERROR, which might be a bit surprising to the caller of the API. This can lead to the application thinking that there are no resources for "en_US" or "en" (or even "root").
This change modifies ICU4C so that it will report back U_MEMORY_ALLOCATION_ERROR if OOM happens when attempting to load the data files.
Fredrik Roubert [Thu, 26 Jul 2018 18:15:45 +0000 (20:15 +0200)]
ICU-20006 Parameterize the TestSourceTargetSet2 test case.
By extracting this test case into a separate class and separating the
complex and time consuming building of the UnicodeSet disorderedMarks
data structure into an @BeforeClass shared setup method it becomes
possible to test the 18 different transliteration rules as a
parameterized test case.
This will lower the running time per test case and also aid in debugging
as it will make it immediately obvious which transliteration rules cause
test failures (and which don't).
Fredrik Roubert [Thu, 5 Jul 2018 19:48:13 +0000 (21:48 +0200)]
ICU-13483 Use a real time zone to make tests work in Android.
When running in Android, java.util.TimeZone.getDefault() will only
return a TimeZone object for a time zone that actually exists, so when
passing a bogus test TimeZone object to setDefault() and afterwards
calling getDefault() it won't return that same time zone.
Changing our tests to instead use a real existing (but otherwise unused)
time zone instead makes the tests work in Android as well.
Neither of these tests was testing the ability to set an arbitrarily
made up time zone as the default, so no test coverage is lost.
Markus Scherer [Tue, 28 Aug 2018 19:53:34 +0000 (12:53 -0700)]
ICU-13530 fix bugs, add tests, clarify docs (#83)
* ICU-13530 test & fix cptrie.getRange() when small null data block matches the start of a non-null fast data block
* ICU-13530 fix test bug
* ICU-13530 test & fix bug calculating worst-case data array capacity at start of compaction
* ICU-13530 docs: clarify buildImmutable() discards upper bits, trie then returns unsigned ints; range iteration slower than if ranges were stored directly
Fredrik Roubert [Thu, 16 Aug 2018 00:54:32 +0000 (02:54 +0200)]
ICU-20000 Workaround for BigDecimal.stripTrailingZeros() differences. (#57)
Different implementations of BigDecimal.stripTrailingZeros(), in
different versions of the JDK (and different versions of Android), have
differences in their handling of zero. To avoid this, ICU4J can return
BigDecimal.ZERO for any value that is equal to zero, instead of calling
BigDecimal.stripTrailingZeros() in this problematic case.
William Zhao [Thu, 23 Aug 2018 21:27:17 +0000 (14:27 -0700)]
ICU-13842 Use GetDynamicTimeZoneInformation instead of Windows registry access in wintz.cpp (#55)
Use GetDynamicTimeZoneInformation instead of accessing the Windows registries so that it does not rely on COM in wintz. So that UWP apps could also use this code.
* ICU-13530 change normalization main trie to UTrie3 with special values for lead surrogates; forbid non-inert surrogate code *points* because unable to store values different from code *units*; runtime code work around that for code point lookup and iteration; adjust UTS 46 for normalization no longer mapping unpaired surrogates to U+FFFD
X-SVN-Rev: 41122
* ICU-13530 simplenormperf bug fix and NFC base line
X-SVN-Rev: 41126
* ICU-13530 move normalization getRange skipping lead surrogates to API getRangeSkipLead()
X-SVN-Rev: 41182
* ICU-13530 switch CanonIterData and gennorm2 Norms to UTrie3
X-SVN-Rev: 41183
* ICU-13530 remove unused overwrite parameter from setRange()
X-SVN-Rev: 41184
* ICU-13530 getRange skip lead -> fixed surrogates
X-SVN-Rev: 41219
* ICU-13530 minor cleanup
X-SVN-Rev: 41221
* ICU-13530 UTS 46 code map unpaired surrogates to U+FFFD before normalization
X-SVN-Rev: 41224
* ICU-13530 minor internal-docs cleanup
X-SVN-Rev: 41225
* ICU-13530 rename UTrie3 to UCPTrie, and other name changes
X-SVN-Rev: 41226
* ICU-13530 add 8-bit data option; add type-any & valueBits-any for fromBinary(); macros consistently source type then data width
X-SVN-Rev: 41234
* ICU-13530 scrub the API docs for the proposal
X-SVN-Rev: 41319
* ICU-13530 tag internal definitions as such, or move them to an internal header
X-SVN-Rev: 41320
* ICU-13530 Java API skeleton
X-SVN-Rev: 41326
* ICU-13530 API feedback: ValueWidth, MutableCodePointTrie, base CodePointMap, ...
X-SVN-Rev: 41382
* ICU-13530 add UCPTrie valueWidth field and padding, and combine data pointers into a union
X-SVN-Rev: 41408
* ICU-13530 switch some macros to using dataAccess parameter: separate index vs. data lookups, no macro variant for each value width
X-SVN-Rev: 41409
* ICU-13530 StringIterator is no longer a java.util.Iterator (bad fit)
X-SVN-Rev: 41455
* ICU-13530 CodePointTrie.java code complete
X-SVN-Rev: 41518
* ICU-13530 finish Java port incl test; keep C++ parallel
* ICU-13530 adjust API for feedback: rename HandleValue to FilterValue, change getRange+getRangeFixedSurr(bool allSurr) to enum RangeOption+getRange(enum option); change remaining C macros to use dataAccess for 16/32/8-bit value widths; fix/clarify some API docs
* ICU-13530 add javadoc
* ICU-13530 document UCPTrie binary data format
* ICU-13530 update .nrm formatVersion 3->4, document change in surrogate handling with new trie
* ICU-13530 re-hardcode NFC data
* move trie swapper code into new file; add new files to Windows project files; turn off trie debugging
* ICU-13530 minor cleanup
* ICU-13530 test more range starts; fix a C test leak
* ICU-13530 regenerate Java data from scratch
* ICU-13530 review feedback changes: API docs typos, more @internal, C++11 field initializers, fix potential leak in MutableCodePointTrie::fromUCPTrie()
* ICU-13530 rename interface FilterValue to ValueFilter
Yoshito Umaoka [Tue, 14 Aug 2018 00:12:53 +0000 (20:12 -0400)]
ICU-13413 More clean-ups for code redundant/unnecessary for Java 7 and up
Simplified redundant code and removed unnecessary code path for Java 7 and later runtime in various places. Also cleaned up stale comments.
There is one test code change in com.ibm.icu.dev.test.localespi.NumberFormatTest - number keyword in test case was changed to Arab to arab. This test case was skipped with Java 6 runtime. It looks this code was note tested on Java 7 and later. @number=Arab does not work because it's case sensitive and must be all lower case letters.
sav-ix [Mon, 13 Aug 2018 16:18:20 +0000 (19:18 +0300)]
ICU-13138 fix ICU data binary naming for builds with Static layout (#44)
* ICU-13138 do not create symlink to Static ICU Data binary
For builds using mingw-w64 and MSVC with Static layout,
ICU installation folder contains '.dll' file, which is
a symbolic link to '.a' or '.lib' Data binary.
This have no sense anyway, since it can't be used as
binary with shared layout.
* ICU-13138 mingw: add lib prefix to Static Data binary name
For builds using mingw-w64 with Static layout, missing 'lib'
prefix in ICU Data binary name leads to Stubdata binary use
during ICU extras, samples and tests build. ICU Stubdata
binary also substitutes Data in installation directory.
The above leads to runtime errors of ICU tools, tests and
ICU-based software.
Thus need to add 'lib' prefix to ICU Data binary name during
Pkgdata run.
ICU-13413 Updated ULocale class to directly calls Java 7 Locale methods
Previously, ULocale had to work on Java 6, therefore, Java 7 specific Locale methods needed to be invoked by reflection. For the next release, we can directly call Java 7 Locale APIs, so rather complicated reflection implementation were replaced with simple API calls.
ICU-13413 Migrating some Utility class methods with Java 7 Objects class method
Previously, some developers accidentally introduced Java 7 Objects class utility methods. At that time, we once added Java 6 compatible implementation in com.ibm.icu.impl.Utility class. Now, we use Java 7 as the minimum supported Java version, so we can use Java 7 Objects class and methods.
There are some extra changes dropping generics type from constructor. For example from `ArrayList<String> output = new ArrayList<String>();` to `ArrayList<String> output = new ArrayList<>();`. The updated syntax is allowed since Java 7. We have eclipse project configured to normalize such expression, therefore, files touched by this commit were automatically updated. These changes are not directly related to Java 7 Objects method replacement.
Andy Heninger [Thu, 9 Aug 2018 18:28:55 +0000 (11:28 -0700)]
ICU-7270 Line Break rule LB8 updated for UAX-14 conformance. (#41)
Includes all line break tailorings.
Corresponding updates to monkey test rules.
State table builder, fix missed table optimization, uncovered by new rule.
Jeff Genovy [Wed, 8 Aug 2018 19:13:42 +0000 (12:13 -0700)]
ICU-13686 ICU4C: Prevent double file separators on data file paths. (#42)
ICU very often ends up with double separators in the file paths used to open files. This change prevents extra slashes on data file paths.
- The feedback from Markus was that the suffix shouldn't really start with a separator.
- It turns out that the suffix can be used for either a file extension (like .dat or .res) or for an actual item name as well (like ibm-5348_P100-1997.cnv).
- If the suffix is an extension then we don't want to append the separator at all. We only want to append it if the suffix is an item name.
- Also use a StringPiece to save repeated implicit calls to uprv_strlen.
Jeff Genovy [Wed, 8 Aug 2018 18:38:28 +0000 (11:38 -0700)]
ICU-13814 Fix define for excluding renaming from VS IntelliSense, so F12 works. (#22)
In the prior change for this ticket the logic was mixed up. For IntelliSense we actually want to turn *off* the renaming defines so that VS doesn't get confused when you press F12.
Fredrik Roubert [Mon, 6 Aug 2018 12:22:35 +0000 (14:22 +0200)]
ICU-20062 Set the Python -B flag to inhibit the writing of .pyc files.
This will prevent littering the source tree with spurious .pyc files.
The potential faster execution when re-running a script that has an
up-to-date .pyc file is negligible.
Fredrik Roubert [Tue, 10 Jul 2018 19:09:47 +0000 (21:09 +0200)]
ICU-20016 Skip testing pseudolocale region codes for display names.
At the moment, CLDR pseudolocale region codes don't have display names.
As long as that is so, ICU must not require them to exist in order to
not cause bogus test failures when building with pseudolocales.
Jeff Genovy [Mon, 6 Aug 2018 20:22:46 +0000 (13:22 -0700)]
ICU-20036 CurrencyPluralInfo class improve handling of OOM errors (#17)
ICU-20036 CurrencyPluralInfo class doesn't always check/handle OOM errors.
Changes include:
- Using LocalPointer instead of raw new/delete, in order to make the code cleaner.
- Using nullptr instead of NULL.
- Inspired by Andy's review feedback PluralRules changes, this change sets fPluralRules and fLocale to nullptr in the assignment operator in order to prevent possible double deletes in the failure case.
- More consistent about not checking for nullptr when calling delete.
- Using LocalUResourceBundlePointer in order to simply the code and not need manual deletes.
- Reduce memory usage by using the same LocalUResourceBundle with .getAlias() instead of allocating new ones.
Fredrik Roubert [Tue, 10 Jul 2018 21:00:43 +0000 (23:00 +0200)]
ICU-20018 Remove hardcoded expectations on available locales.
Both the number of installed locales and the name of the last locale
change with updates to and customizations of CLDR/ICU data so test
expectations on these values will cause bogus test failures.
Jeff Genovy [Fri, 3 Aug 2018 20:30:03 +0000 (13:30 -0700)]
ICU-20046 Improve OOM error checking in the RBNF class. (#24)
- There are a few locations in the RBNF class that don't check for out-of-memory (OOM) failures.
- Using LocalPointer to clean up the manual deletes.
- Change to use nullptr instead of NULL.
- A few minor typo fixes as well.
Jeff Genovy [Thu, 2 Aug 2018 23:23:07 +0000 (16:23 -0700)]
ICU-20041 Improve handling of OOM failures in NumberingSystem class. (#19)
ICU-20041 ICU4C NumberingSystem class doesn't handle out-of-memory (OOM) failures.
- Not all code paths in the NumberingSystem class check for OOM failures. This can lead to crashes in some cases as null pointers will be dereferenced without any checks.
- Change to use nullptr instead of NULL.
- Don't stomp on OOM errors when attempting to load resources. We should report back OOM to the caller.
- Use LocalPointer in order simplify the code and for automatic clean-up of memory.
- Use LocalUResourceBundlePointer as well to help simply things even more.
Jeff Genovy [Thu, 2 Aug 2018 06:33:03 +0000 (23:33 -0700)]
ICU-20042 Improve OOM handling in PluralRules class. (#20)
- PluralRules class doesn't handle out-of-memory (OOM) errors in some code paths.
- The clone and assignment operator (operator=) methods of construction don't take an error code parameter, meaning that if an OOM error occurs during the constructor, it will not reported back to the caller, and the caller has no way to know that the object is in a half-constructed state.
- Using an internal status variable for these above cases.
- Changes to the various PluralRules helper classes to check for OOM as well.
- Using nullptr instead NULL.
- Using LocalPointer instead of raw new/delete in some cases.
- Need to set mRules to nullptr, otherwise we can end up with double deletes in the failure case. (Thanks to Andy for the review).
- Using default member initializers for class members to avoid dangling pointers. Also allows for using default constructors too.
Jeff Genovy [Thu, 2 Aug 2018 05:44:39 +0000 (22:44 -0700)]
ICU-20034 ICU4C Locale assignment operator should set the locale to bogus if OOM occurs. (#14)
ICU-20034 ICU4C the Locale class's assignment operator should set the locale to "bogus" if an OOM error occurs when attempting to copy data over from the other locale.
Also need to check strdup, as that calls malloc and it can fail too.
Jeff Genovy [Fri, 20 Jul 2018 00:15:51 +0000 (17:15 -0700)]
ICU-20005 Update/clean-up the .gitattributes file after the automated conversion from the scripts. Also add some comments for others, and use the diff helpers for better diffs.
Jeff Genovy [Wed, 27 Jun 2018 19:25:52 +0000 (19:25 +0000)]
ICU-13778 DataTimePatternGenerator code refactor. Handle Out-of-Memory (OOM) errors, use LocalPointer to prevent memory leaks when OOM occurs, use an internal error code to better report errors during clone and copy construction, mark helper methods and parameters as const, use nullptr instead of NULL, minor spelling and formatting changes. (Note: All tests pass on Windows and Linux).
Fredrik Roubert [Thu, 21 Jun 2018 20:17:22 +0000 (20:17 +0000)]
ICU-13766 Make UnicodeMapTest#TestUnicodeMapGeneralCategory smaller.
There are 0x10FFFF code points and this test allocates memory for every
code point tested. Testing every single one of them is therefore memory
intensive and slow, without adding any real value to the test.
Instead an equivalent test coverage can be achieved by testing a much
smaller set of ranges of code points that cover all categories.
projects / icu / log