Ivan Maidanski [Sat, 20 Aug 2016 14:46:48 +0000 (17:46 +0300)]
Use C11 double-wide atomic intrinsics for Clang/x86 on Cygwin and OS X
Others (e.g., NDK clang/i686) still might need -latomic linker option.
* src/atomic_ops/sysdeps/gcc/x86.h
[AO_GCC_ATOMIC_TEST_AND_SET && __clang__]: Do not fallback to
non-intrinsic implementation if __APPLE_CC__ or __CYGWIN__.
Ivan Maidanski [Wed, 17 Aug 2016 16:51:30 +0000 (19:51 +0300)]
Fix (delete) comment for AO_and_full (x86)
* src/atomic_ops/sysdeps/gcc/x86.h [!AO_PREFER_GENERALIZED]
(AO_and_full): Remove incorrect comment (the code should work even for
i386).
* src/atomic_ops/sysdeps/sunc/x86.h [!AO_PREFER_GENERALIZED]
(AO_and_full): Likewise.
Hans Boehm [Mon, 15 Aug 2016 08:32:33 +0000 (11:32 +0300)]
Fix store-load ordering in AO_stack_pop_explicit_aux_acquire (PowerPC)
Issue #15.
The core issue is that AO_stack_pop_explicit_aux_acquire really needs
to ensure that the store to the blacklist via
AO_compare_and_swap_acquire becomes visible before the load to check
the list head. This effectively needs store-load ordering.
Currently the only ordering here is imposed by the _acquire on the
compare_and_swap. On PowerPC that turns into an lwsync, which is too
weak to enforce store to load ordering.
This patch should fix the issue. But this is suboptimal on x86, and
we may want to make the fence conditional on "not x86", where the CAS
already includes sufficient ordering. (With C++11 atomics, this would
also be tricky and probably involve making a bunch of accesses seq_cst.)
* src/atomic_ops_stack.c [AO_USE_ALMOST_LOCK_FREE]
(AO_stack_pop_explicit_aux_acquire): Call AO_compare_and_swap instead
of AO_compare_and_swap_acquire; call AO_nop_full just before
(first != AO_load(list)).
Ivan Maidanski [Mon, 25 Apr 2016 18:26:28 +0000 (21:26 +0300)]
Use GCC atomic intrinsics for x86 and x64 (gcc 4.8+ and clang 3.4+)
* src/atomic_ops/sysdeps/gcc/generic.h (AO_GCC_HAVE_double_SYNC_CAS):
Do not define if AO_SKIPATOMIC_double_compare_and_swap_ANY.
* src/atomic_ops/sysdeps/gcc/x86.h (AO_GCC_ATOMIC_TEST_AND_SET): Define
if gcc4.8+ (but not Intel compiler) or clang3.4+ (if not
AO_DISABLE_GCC_ATOMICS).
* src/atomic_ops/sysdeps/gcc/x86.h [AO_GCC_ATOMIC_TEST_AND_SET]: Do not
include all_aligned_atomic_load_store.h, test_and_set_t_is_char.h,
ordered_except_wr.h.
* src/atomic_ops/sysdeps/gcc/x86.h (AO_nop_full, AO_fetch_and_add_full,
AO_char_fetch_and_add_full, AO_short_fetch_and_add_full, AO_and_full,
AO_or_full, AO_xor_full, AO_test_and_set_full,
AO_compare_and_swap_full) [AO_GCC_ATOMIC_TEST_AND_SET]: Do not define.
* src/atomic_ops/sysdeps/gcc/x86.h: Include standard_ao_double_t.h if
AO_PREFER_BUILTIN_ATOMICS, or AO_GCC_ATOMIC_TEST_AND_SET and not
clang/x86.
* src/atomic_ops/sysdeps/gcc/x86.h (AO_double_load_acquire,
AO_double_store_release, AO_SKIPATOMIC_double_compare_and_swap_ANY):
Define if AO_GCC_ATOMIC_TEST_AND_SET and clang/x86 (unless
AO_PREFER_BUILTIN_ATOMICS).
* src/atomic_ops/sysdeps/gcc/x86.h
(AO_compare_double_and_swap_double_full) [x86]: Define also for
clang/x86 if AO_GCC_ATOMIC_TEST_AND_SET (unless
AO_PREFER_BUILTIN_ATOMICS).
* src/atomic_ops/sysdeps/gcc/x86.h
(AO_compare_double_and_swap_double_full) [x86_64]: Remove TODO item;
check __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16 to detect presence of
cmpxchg16b.
* src/atomic_ops/sysdeps/gcc/x86.h: Include generic.h if
AO_GCC_ATOMIC_TEST_AND_SET.
* src/atomic_ops/sysdeps/standard_ao_double_t.h (double_ptr_storage):
Use unsigned __int128 instead of __m128 for GCC x86_64 atomic
intrinsics.
Ivan Maidanski [Thu, 21 Apr 2016 07:59:18 +0000 (10:59 +0300)]
Do not force GCC atomic intrinsics usage if AO_PREFER_BUILTIN_ATOMICS
The macro should only prevent usage of assembly completely but only
if at least some of GCC atomic intrinsics are available.
* src/atomic_ops/sysdeps/gcc/arm.h (AO_PREFER_BUILTIN_ATOMICS): Remove
from the condition whether to include generic.h.
* src/atomic_ops/sysdeps/gcc/mips.h (AO_PREFER_BUILTIN_ATOMICS):
Likewise.
Ivan Maidanski [Mon, 18 Apr 2016 07:19:20 +0000 (10:19 +0300)]
Remove AO_T_IS_INT definition for GCC atomic intrinsics case
(Code refactoring)
In case of GCC/C11 atomic intrinsics are used, AO_int_* are defined in
generic-arithm.h and generic-small.h files.
* src/atomic_ops/sysdeps/gcc/arm.h (AO_T_IS_INT): Do not define if
AO_GCC_ATOMIC_TEST_AND_SET.
* src/atomic_ops/sysdeps/gcc/mips.h (AO_T_IS_INT): Likewise.
* src/atomic_ops/sysdeps/gcc/nios2.h (AO_T_IS_INT): Do not define.
Ivan Maidanski [Wed, 13 Apr 2016 23:19:58 +0000 (02:19 +0300)]
Avoid __atomic_and/or/xor_fetch if unsupported (clang/arm-v5te)
* src/atomic_ops/sysdeps/gcc/arm.h (AO_SKIPATOMIC_ANY_and_ANY,
AO_SKIPATOMIC_ANY_or_ANY, AO_SKIPATOMIC_ANY_xor_ANY): Define new
internal macro (before include "generic.h") if
AO_GCC_ATOMIC_TEST_AND_SET and __clang__, and not AO_ARM_HAVE_LDREX.
* src/atomic_ops/sysdeps/gcc/generic-arithm.h: Regenerate.
* src/atomic_ops/sysdeps/gcc/generic-arithm.template
(AO_XSIZE_and_XBAR): Do not define if AO_SKIPATOMIC_ANY_and_ANY.
* src/atomic_ops/sysdeps/gcc/generic-arithm.template
(AO_XSIZE_or_XBAR): Do not define if AO_SKIPATOMIC_ANY_or_ANY.
* src/atomic_ops/sysdeps/gcc/generic-arithm.template
(AO_XSIZE_xor_XBAR): Do not define if AO_SKIPATOMIC_ANY_xor_ANY.
Ivan Maidanski [Mon, 11 Apr 2016 19:20:07 +0000 (22:20 +0300)]
Use GCC atomic intrinsics for ARM (gcc 4.8+ and clang 3.5+)
* src/atomic_ops/sysdeps/gcc/arm.h (AO_GCC_ATOMIC_TEST_AND_SET): Define
if gcc 4.8+ or clang 3.5+ or AO_PREFER_BUILTIN_ATOMICS unless
AO_DISABLE_GCC_ATOMICS.
* src/atomic_ops/sysdeps/gcc/arm.h (AO_nop_write, AO_store,
AO_char_store, AO_short_store): Do not define (in assembly code) if
AO_PREFER_BUILTIN_ATOMICS.
* src/atomic_ops/sysdeps/gcc/arm.h (AO_SKIPATOMIC_store,
AO_SKIPATOMIC_store_release, AO_SKIPATOMIC_char_store,
AO_SKIPATOMIC_char_store_release, AO_SKIPATOMIC_short_store,
AO_SKIPATOMIC_short_store_release, AO_SKIPATOMIC_int_store,
AO_SKIPATOMIC_int_store_release): Define if AO_BROKEN_TASKSWITCH_CLREX.
* src/atomic_ops/sysdeps/gcc/arm.h (AO_nop_full, AO_test_and_set,
AO_fetch_and_add, AO_fetch_and_add1, AO_fetch_and_sub1, AO_and, AO_or,
AO_xor, AO_char_fetch_and_add, AO_short_fetch_and_add,
AO_compare_and_swap, AO_fetch_compare_and_swap, AO_double_load,
AO_double_store, AO_double_compare_and_swap, AO_test_and_set_full): Do
not define (in assembly code) if AO_GCC_ATOMIC_TEST_AND_SET.
* src/atomic_ops/sysdeps/gcc/arm.h: Do not include atomic_store.h,
all_atomic_only_load.h, char_atomic_store.h, short_atomic_store.h,
all_aligned_atomic_load_store.h if AO_GCC_ATOMIC_TEST_AND_SET.
* src/atomic_ops/sysdeps/gcc/arm.h: Include generic.h (at the end of
the file) if AO_GCC_ATOMIC_TEST_AND_SET.
* src/atomic_ops/sysdeps/gcc/generic-small.h: Regenerate.
* src/atomic_ops/sysdeps/gcc/generic-small.template (AO_XSIZE_store):
Do not define if AO_SKIPATOMIC_XSIZE_store.
* src/atomic_ops/sysdeps/gcc/generic-small.template
(AO_XSIZE_store_release): Do not define if
AO_SKIPATOMIC_XSIZE_store_release.
Ivan Maidanski [Mon, 4 Apr 2016 21:27:50 +0000 (00:27 +0300)]
Use GCC atomic intrinsics for MIPS (GCC 4.9+ and clang 3.5+)
* src/atomic_ops/sysdeps/gcc/mips.h: Include generic.h and do not
include all_aligned_atomic_load_store.h if AO_PREFER_BUILTIN_ATOMICS or
GCC 4.9+ or clang 3.5+ unless AO_DISABLE_GCC_ATOMICS.
* src/atomic_ops/sysdeps/gcc/mips.h (AO_MIPS_SET_ISA, AO_MIPS_LL_1,
AO_MIPS_SC, AO_MIPS_LL, AO_nop_full, AO_fetch_and_add, AO_test_and_set,
AO_compare_and_swap, AO_fetch_compare_and_swap): Do not define if
generic.h included.
Ivan Maidanski [Wed, 6 Apr 2016 08:40:45 +0000 (11:40 +0300)]
Avoid atomic_compare_exchange_n if no __GCC_HAVE_SYNC_COMPARE_AND_SWAP_n
* src/atomic_ops/sysdeps/gcc/aarch64.h (AO_GCC_FORCE_HAVE_CAS,
AO_GCC_HAVE_double_SYNC_CAS): Define macro before include generic.h
if __clang__ (workaround).
* src/atomic_ops/sysdeps/gcc/generic-small.h: Regenerate.
* src/atomic_ops/sysdeps/gcc/generic-small.template
(AO_XSIZE_fetch_compare_and_swap, AO_XSIZE_compare_and_swap): Do not
define unless AO_GCC_HAVE_XSIZE_SYNC_CAS.
* src/atomic_ops/sysdeps/gcc/generic.h (AO_GCC_HAVE_char_SYNC_CAS,
AO_GCC_HAVE_short_SYNC_CAS, AO_GCC_HAVE_int_SYNC_CAS,
AO_GCC_HAVE_SYNC_CAS): New internal macro (based on
__GCC_HAVE_SYNC_COMPARE_AND_SWAP_<n> or AO_GCC_FORCE_HAVE_CAS presence).
* src/atomic_ops/sysdeps/gcc/generic.h (AO_GCC_HAVE_double_SYNC_CAS):
New internal macro if AO_HAVE_DOUBLE_PTR_STORAGE.
* src/atomic_ops/sysdeps/gcc/generic.h (AO_double_compare_and_swap):
Check AO_GCC_HAVE_double_SYNC_CAS instead of AO_HAVE_DOUBLE_PTR_STORAGE.
Ivan Maidanski [Tue, 5 Apr 2016 21:10:56 +0000 (00:10 +0300)]
Eliminate redundant lwsync 2nd call in CAS_full on fail (gcc/PowerPC)
* src/atomic_ops/sysdeps/gcc/powerpc.h (AO_compare_and_swap_full,
AO_fetch_compare_and_swap_full): Do not call AO_lwsync 2nd time if
CAS is not successful.
Ivan Maidanski [Fri, 1 Apr 2016 17:34:30 +0000 (20:34 +0300)]
New macro AO_PREFER_BUILTIN_ATOMICS to rely on C11 atomics fully (AArch64)
If AO_PREFER_BUILTIN_ATOMICS is defined then inline assembly
implementation is not used.
* src/atomic_ops/sysdeps/gcc/aarch64.h (AO_SKIPATOMIC_double_load,
AO_SKIPATOMIC_double_load_acquire): Define if AO_PREFER_BUILTIN_ATOMICS.
* src/atomic_ops/sysdeps/gcc/aarch64.h (AO_nop_write, AO_double_load,
AO_double_load_acquire, AO_double_store, AO_double_store_release,
AO_double_compare_and_swap, AO_double_compare_and_swap_acquire,
AO_double_compare_and_swap_release, AO_double_compare_and_swap_full):
Do not define if AO_PREFER_BUILTIN_ATOMICS.
* src/atomic_ops/sysdeps/gcc/aarch64.h (AO_nop_write): Add comment;
remove TODO.
* src/atomic_ops/sysdeps/gcc/generic.h: Remove TODO about including
this header for other targets.
* src/atomic_ops/sysdeps/gcc/generic.h (AO_double_load): Do not define
if AO_SKIPATOMIC_double_load.
* src/atomic_ops/sysdeps/gcc/generic.h (AO_double_load_acquire): Do not
define if AO_SKIPATOMIC_double_load_acquire.
Marek Vasut [Tue, 15 Mar 2016 07:09:26 +0000 (10:09 +0300)]
Add initial nios2 architecture support
* src/Makefile.am (nobase_private_HEADERS): Add nios2.h.
* src/atomic_ops.h: Include nios2.h if __nios2__.
* src/atomic_ops/sysdeps/gcc/nios2.h: New file.
Ivan Maidanski [Mon, 8 Feb 2016 07:13:47 +0000 (10:13 +0300)]
Eliminate 'signed-to-unsigned value extension' compiler warning in AO_malloc
* src/atomic_ops_malloc.c (msbs): Change type from int to unsigned
char.
* src/atomic_ops_malloc.c (msb): Change return type from int to
unsigned.
* src/atomic_ops_malloc.c (msb, AO_malloc): Change type of v, result,
log_sz local variables from int to unsigned.
Frank Schaefer [Sat, 9 May 2015 05:26:43 +0000 (05:26 +0000)]
Support n32 ABI for mips64
Without the proposed change, the existing preprocessor checks for
mips64 support do not account for the n32 ABI (when building with
"gcc -mabi=n32", test_atomic and test_stack both fail, and
test_malloc appears to hang indefinitely while spinning at 100%).
* src/atomic_ops/sysdeps/gcc/mips.h (AO_MIPS_SET_ISA, AO_MIPS_LL_1,
AO_MIPS_SC): Define depending on _MIPS_SIM value.
James Cowgill [Thu, 8 Jan 2015 16:05:57 +0000 (16:05 +0000)]
Remove inclusion of acquire_release_volatile.h on mips
I'm not entirely sure how this ended up here. It might have been the case
that MIPS processors used to do this, but the ISA manuals don't say anything
about volatile loads / stores having aquire / release semnatics, so just
remove it to be safe.
Steve Capper [Wed, 29 Oct 2014 22:16:14 +0000 (01:16 +0300)]
Relax shareability domain for dmb st in AO_nop_write (ARM/AArch64)
* src/atomic_ops/sysdeps/gcc/aarch64.h (AO_nop_write): Replace "dmb st"
to "dmb ishst" (this way the barriers will occupy the inner shareable
domain).
* src/atomic_ops/sysdeps/gcc/arm.h (AO_nop_write): Likewise.
Fix missing output folder on making auto-generated headers (Automake)
Create the directories before sed needs them (otherwise build fails
when sed tries to redirect output to a non-existent directory in case
of build root folder is not identical to source root folder).
Ivan Maidanski [Fri, 13 Jun 2014 12:35:55 +0000 (16:35 +0400)]
Eliminate 'variable set but not used' Cppcheck warnings in test_stack
* tests/test_stack.c (run_one_test): Define "j" local variable only
if VERBOSE.
* tests/test_stack.c (main): Do not define "sum" local variable (and
don not compute the sum) if NO_TIMES.
Ivan Maidanski [Sun, 4 May 2014 12:25:40 +0000 (16:25 +0400)]
Restore contribution info in ChangeLog for authors not listed in git log
(deleted in commit 36a9c66)
* ChangeLog (7.0, 1.0): Add information about contribution of persons
mentioned in AUTHORS but not recorded in git log.
* src/atomic_ops/sysdeps/gcc/cris.h: Move contribution information
from comment to ChangeLog.
Ivan Maidanski [Sat, 17 Aug 2013 11:05:36 +0000 (15:05 +0400)]
Do not use LDREXD/STREXD for Clang3.2/arm (and earlier)
* src/atomic_ops/sysdeps/gcc/arm.h (AO_ARM_HAVE_LDREXD): Do not define
for pre-Clang3.3 (since it does not allocate register pairs for
LDREXD/STREXD instructions properly); add comment.
Ivan Maidanski [Sat, 17 Aug 2013 09:58:59 +0000 (13:58 +0400)]
Fix ARM char/short fetch_and_add and double-CAS operands width (GCC/Clang)
* src/atomic_ops/sysdeps/gcc/arm.h (AO_char_fetch_and_add,
AO_short_fetch_and_add): Use 32-bit int type for "tmp" and "result"
local variables instead of char/short type (resolve Clang3.3 warning
"size being stored is truncated, use a modifier to specify the size"
enabled by -Wasm-operand-widths compiler option); cast "incr"
argument to int to prevent Clang3.2 warning about value
truncation.
* src/atomic_ops/sysdeps/gcc/arm.h (AO_double_compare_and_swap): Swap
assembly code operands ("new_val.AO_whole" and "addr") to prevent
Clang3.3 warning about operand truncation.
* src/atomic_ops/sysdeps/gcc/arm.h (AO_double_load, AO_double_store,
AO_double_compare_and_swap): Specify that LDREXD and STREXD use 2
adjacent registers (thus preventing Clang3.3 from register allocation
failures leading to "registers may not be the same" or
"even register required" GAS errors).
Ivan Maidanski [Sat, 17 Aug 2013 09:04:23 +0000 (13:04 +0400)]
Tests: workaround GCC 4.4.3 warning reported for list_atomic.c 'val'
* tests/list_atomic.template (XSIZE_list_atomicXX): Use "static"
modifier for "val" local variable to initialize it (cannot use direct
assignment to zero because "val" type could be a SIMD vector type if
XCTYPE is AO_double_t) to workaround GCC v4.4.3
"'val' is used uninitialized in this function" warning reported for
AO_load.
Ivan Maidanski [Sat, 17 Aug 2013 08:13:40 +0000 (12:13 +0400)]
Support ARMv8 target (gcc/arm)
* src/atomic_ops/sysdeps/gcc/arm.h (__ARM_ARCH_8A__): Detect new macro
(treated same as __ARM_ARCH_7A__).
* src/atomic_ops/sysdeps/gcc/arm.h (AO_ARM_HAVE_SWP): Do not define
for ARMv8 (since SWP{B} obsoleted); add comment.
* src/atomic_ops/sysdeps/gcc/arm.h (AO_compare_and_swap): Add TODO
item (for deprecated IT block containing wide Thumb instruction).
Ivan Maidanski [Tue, 13 Aug 2013 20:40:04 +0000 (00:40 +0400)]
tests: Add char/short/int/AO_double_t and dd_acquire cases to list_atomic
* tests/Makefile.am (list_atomic.c): Iterate over XSIZE/XCTYPE (in
addition to XX); expand XX also as _dd_acquire_read.
* tests/list_atomic.template (XSIZE_list_atomicXX): Define 'val' local
variable only if used (needed for AO_double_t case).
Ivan Maidanski [Sun, 11 Aug 2013 10:49:05 +0000 (14:49 +0400)]
tests: Parameterize list_atomic template with XSIZE
* tests/Makefile.am (list_atomic.c): Substitute XSIZE/XCTYPE for AO_t.
* tests/list_atomic.template (list_atomicXX): Parameterize with XSIZE
and XCTYPE (except for AO_nop and AO_test_and_set).
Ivan Maidanski [Sun, 11 Aug 2013 07:33:12 +0000 (11:33 +0400)]
list_atomic: Add makefile rule to test list_atomic.template syntax
* tests/Makefile.am (BUILT_SOURCES, CLEANFILES): Add "list_atomic.o".
* tests/Makefile.am (list_atomic.o): New rule (to verify list_atomic.c
is valid C code).
* tests/list_atomic.template (list_atomicXX): Use "volatile" for "val"
local variable.
* tests/list_atomic.template (list_atomicXX): Define "oldval", "newval",
"ts" and "incr" local variables only if used; cast string literals to
void (to prevent compiler warnings).
* tests/list_atomic.template (list_atomicXX): Use AO_t type for "incr"
local variable instead of C long type.
* tests/list_atomic.template (list_atomicXX): Define "oldval",
"newval", "incr" as statitc local variables, and remove explicit
initialization to zero (to void problems with initialization of these
variables for AO_double_t case if the latter is a struct).