Azat Khuzhin [Thu, 11 Aug 2016 13:15:45 +0000 (16:15 +0300)]
tests: use waitpid(..., WNOWAIT) to fix failing of main/fork under solaris
According to solaris docs:
"One instance of a SIGCHLD signal is queued for each child process whose
status has changed. If waitpid() returns because the status of a child
process is available, and WNOWAIT was not specified in options, any pending
SIGCHLD signal associated with the process ID of that child process is
discarded. Any other pending SIGCHLD signals remain pending."
And interesting thing that it works if you add sleep(1) before waitpid(), and
also if you run with --verbose (some race or what).
But linux doesn't support WNOWAIT in waitpid() so add detection into
cmake/autotools.
Azat Khuzhin [Thu, 11 Aug 2016 10:43:14 +0000 (13:43 +0300)]
test: replace sleeping with syncing pair in main/fork
About this syncing pair:
- read endpoint, must be blocked, to make it a checkpoint or smth like this
- write endpoint, must be nonblocking, to avoid readcb hung
During testing sometimes bufferevent/bufferevent_connect_fail_eventcb fails,
and after some digging I found that this was the case when connect() doesn't
return ECONNREFUSED immediately, and instead next operation will return it (in
our case evbuffer_read()->readv(), needless to say that after this
bufferevent_writecb() called and it checks error with getsockopt() but of
course it doesn't return any error), so this patch checks "errno" after
"readv()" and installs "bufev::connection_refused" flag, to handle this from
writecb and only once.
Azat Khuzhin [Tue, 9 Aug 2016 12:49:01 +0000 (15:49 +0300)]
Merge branch 'appveyor-cmake-v6'
This patch set introduces matrix for appveyor and fixes bugs related to win32,
during debugging this matrix. This should greatly reduce number of work for
making a release ('check under windows' part only).
Right now tests on appveyor still failes in summary, but we should dig into
problems, not hide them.
Test matrix looks like this:
- autotools
- autotools --disable-openssl
- autotools --disable-thread-support
- autotools --disable-debug-mode
- autotools --disable-malloc-replacement
- cmake
- cmake -DEVENT__DISABLE_OPENSSL=ON
- cmake -DEVENT__DISABLE_THREAD_SUPPORT=ON
- cmake -DEVENT__DISABLE_DEBUG_MODE=ON
- cmake -DEVENT__DISABLE_MM_REPLACEMENT=ON
- cmake -DEVENT__ENABLE_VERBOSE_DEBUG=ON
- cmake -DCMAKE_C_FLAGS='-DUNICODE -D_UNICODE'
And takes 90min (was 7min, and this is the sad part).
* appveyor-cmake-v6:
test/ssl: use send()/recv()/EVUTIL_ERR_RW_RETRIABLE()/EVUTIL_SOCKET_ERROR() to fix win32
test/https_basic: increase timeout for complete write (fixes win32)
cmake: check for ZLIB_INCLUDE_DIR, since we can have only library without headers
autotools/win32: fix searching ssl library
appveyor/autotools: link with openssl by passing LDFLAGS/CFLAGS
appveyor: image already had openssl installed
cmake/win32: do not compile regress_thread on -DEVENT__DISABLE_THREAD_SUPPORT=ON
cmake/win32: do not compile evthread_win32 on -DEVENT__DISABLE_THREAD_SUPPORT=ON
appveyor: check -DUNICODE -D_UNICODE according to ReleaseChecklist (cmake only)
cmake: fix -DEVENT__ENABLE_VERBOSE_DEBUG (typo on -DUSE_DEBUG)
test: fix building with --disable-thread-support under win32
cmake: do not use stderr for notifications/version-info
autoconf: fix --disable-thread-support build under win32
appveyor: ignore failure of mingw-get
appveyor: drop shallow_clone, since we use tags for detecting version in cmake
appveyor: support cmake & autotools using build matrix (like travis-ci has)
Azat Khuzhin [Mon, 8 Aug 2016 14:25:11 +0000 (17:25 +0300)]
appveyor/autotools: link with openssl by passing LDFLAGS/CFLAGS
This never works, hence green tests in production. and now it works:
https://ci.appveyor.com/project/azat/libevent/build/2.1.5.100/job/eib4xqfb7d3ruo16
(look by /"evthread_win32.c")
Azat Khuzhin [Mon, 8 Aug 2016 12:34:57 +0000 (15:34 +0300)]
appveyor: image already had openssl installed
Fixes messages like "Version mismatch for openssl: compiled with 1000201f but
running with 1000208f" [1].
According to next link:
http://help.appveyor.com/discussions/questions/1132-openssl-installation-issues
Refs: appveyor/ci#576
[1]: https://ci.appveyor.com/project/azat/libevent/build/2.1.5.95/job/dcy9cfoj3rfo4gdh
Azat Khuzhin [Sun, 7 Aug 2016 20:07:44 +0000 (23:07 +0300)]
cmake: do not use stderr for notifications/version-info
For all except "STATUS" message() will write to stderr, and this also includes
case when you does not have <mode> at all. Plus for every message in stderr
powershell generates exception and make appveyor fails.
Azat Khuzhin [Sun, 7 Aug 2016 18:30:04 +0000 (21:30 +0300)]
appveyor: ignore failure of mingw-get
Since according to doc [1], appveyour will fail the build on every powershell's
exception:
"PowerShell script is considered successful if it finishes without exception."
Azat Khuzhin [Sun, 19 Jun 2016 11:15:41 +0000 (14:15 +0300)]
appveyor: support cmake & autotools using build matrix (like travis-ci has)
This will allow us to avoid possible compilation regressions, and also some
failures in different environments (for example because of different
sizeof(size_t)).
For example we can avoid next issues:
- #361
- #314
- #311
(And I'm pretty sure that this is not complete list)
And of course it reduce job for maintainers.
But it also have one downside, appveyor doesn't have such concurrency like
travis-ci, so it can take a while to finish the whole build, but we can resolve
this later, or reduce build matrix to only one for autotools and one for cmake.
Mark Mentovai [Fri, 8 Jul 2016 13:43:39 +0000 (09:43 -0400)]
Check for Mac OS X 10.4 kqueue bug properly
EV_ERROR is a bit in struct kevent::flags. Other bits may be set too.
Actually we have osx builds on travis-ci, but it uses osx 10.9.5, and we don't
have warnings there, since I guess that there is no OR'ing with previous flag
in case of error, while in 10.12 there is OR.
Eduardo Panisset [Fri, 17 Jun 2016 17:46:32 +0000 (10:46 -0700)]
be_filter: avoid data stuck under active watermarks
Suppose we have bufferevent filter attached to bufferevent socket.
Read high watermark for bufferevent filter is configured to 4096 bytes.
Socket receives 4343 bytes. Due to watermark, 4096 bytes are transferred
from socket input buffer to filter input buffer and 247 bytes are left
in bufferevent socket.
Suppose that no more data is received through socket.
At this point 247 bytes will sit forever in input buffer of bufferevent
socket.
The patch attached solves this issue registering read callback to
filter's input buffer if it reaches its read high water mark and data
was left in corresponding underlying's input buffer.
This read callback calls filter process input function as soon as filter
input buffer falls below its read high watermark and there still is data
left in underlying input buffer. Callback is deregistered as soon as
filter input buffer falls below its read high watermark.
David Paschich [Sun, 22 May 2016 04:05:11 +0000 (21:05 -0700)]
Fix bufferevent_pair to properly set BEV_EVENT_{READING,WRITING} on flush.
Here's some fun. From `bufferevent.h`:
```
#define BEV_EVENT_READING 0x01 /**< error encountered while reading */
#define BEV_EVENT_WRITING 0x02 /**< error encountered while writing */
```
And from `event.h`:
```
/** Wait for a socket or FD to become readable */
#define EV_READ 0x02
/** Wait for a socket or FD to become writeable */
#define EV_WRITE 0x04
```
Library users have to be very careful to get this right; it turns out, the
library itself got this wrong in the `bufferevent_pair` code. It appears that
in most of the code, only `BEV_EVENT_FINISHED` will indicate whether it's read
or write; on error or timeout, it appears that "both" is assumed and not set in
the callback. I read through all the other places where `BEV_EVENT_FINISHED` is
passed to an event callback; it appears that the pair code is the only spot
that got it wrong.
azat: add TT_FORK to avoid breaking clean env, and rebase commit message
(copied from #359) Fixes: #359
Azat Khuzhin [Tue, 7 Jun 2016 15:31:48 +0000 (18:31 +0300)]
buffer: evbuffer_add_buffer(): clean empty chains from destination buffer
@EMPanisset reported a problem (#358) with evbuffer_remove_buffer(), but
actually I think that the problem is in evbuffer_add_buffer() which introduces
this empty chain, all other callers (except evbuffer_prepend_buffer(), but it
doesn't have this problem though) should be safe.
And FWIW the only API that allows empty chains is evbuffer_add_reference(), and
we can add check there to avoid such issues, but for now I leaved this without
fixing, since I think that evbuffer_add_reference() with empty chains can be
used as a barrier (but this can be tricky).
Azat Khuzhin [Tue, 7 Jun 2016 11:56:32 +0000 (14:56 +0300)]
test/buffer: evbuffer_remove_buffer() with empty chains (evbuffer_add_buffer())
Using:
- evbuffer_add()
- evbuffer_add_buffer() -- the one that has problem
- evbuffer_add_reference() -- the only one that allows empty chains to be added
- evbuffer_remove_buffer()
Possible failures after this patch set (not always, IOW in some builds this
issues aren't real issues):
- some failures but mostly because of timing issues, must be fixed separately.
- https://travis-ci.org/azat/libevent/jobs/129430229 # on brew update
- https://travis-ci.org/azat/libevent/jobs/129430221 # some locking issues
* travis-ci-os-matrix-v2:
automake: define serial-tests only if automake have this option
test/automake: don't use paralell test harness (since automake 1.12)
travis-ci/osx: relink gcc/g++ instead of clang
travis-ci: enable multi-os mode (osx, linux)
travis-ci: increase matrix (--disable-foo)
travis-ci: adjust alignment
Azat Khuzhin [Wed, 11 May 2016 07:18:25 +0000 (10:18 +0300)]
test/automake: don't use paralell test harness (since automake 1.12)
Starting from automake 1.2 there is parallel test harness, that redirects all
output to some log, which serial-test doesn't do.
So in case of new runner we can get no output for 10 minutes, for example on my
desktop:
$ time make verify VERBOSE=1
PASS: test/test-script.sh
============================================================================
Testsuite summary for libevent 2.1.5-beta
============================================================================
# TOTAL: 1
# PASS: 1
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
real 25m31.735s
user 0m13.753s
sys 0m7.648s
And this means that this will fail on travis-ci, since it has timeout for 10
minutes. Sure we can use `travis wait 60` instead, but I think that it is
better to fix this by writing result to output, instead of hacking around, so
let's use serial-tests instead of parallel always.
And now it works on travis-ci under linux because it has automake 1.11 while
osx has at least 1.12.
Azat Khuzhin [Thu, 31 Mar 2016 17:45:47 +0000 (20:45 +0300)]
listener: unlock lev on error in listener_read_cb()
Without this patch:
$ regress --no-fork +listener/error_unlock
listener/error_unlock: [warn] Error from accept() call: Too many open files
[err] ../evthread.c:220: Assertion lock->count == 0 failed in ../evthread.c
Aborted (core dumped)
Marcus Sundberg [Sat, 26 Mar 2016 19:11:43 +0000 (20:11 +0100)]
evbuffer_add: Use last_with_datap if set, not last.
evbuffer_add() would always put data in the last chain, even if there
was available space in a previous chain, and in doing so it also
failed to update last_with_datap, causing subsequent calls to other
functions that do look at last_with_datap to add data in the middle
of the evbuffer instead of at the end.
Fixes the evbuffer_add() part of issue #335, and the evbuffer/add2 and
evbuffer/add3 tests, and also prevents wasting space available in the
chain pointed to by last_with_datap.
Marcus Sundberg [Sat, 26 Mar 2016 13:14:44 +0000 (14:14 +0100)]
test/regress: add tests for evbuffer_add() breakage on empty last chain
The evbuffer/add* tests currenly break on 2.0.21, 2.0.22 and 2.1 HEAD
due to issue #335. The evbuffer/reference2 test breaks on 2.0.21 and
2.0.22 due to commit b18c04dd not being applied.
../http.c:589:6: warning: logical not is only applied to the left hand side of this comparison [-Wlogical-not-parentheses]
if (!req->kind == EVHTTP_REQUEST || !REQ_VERSION_ATLEAST(req, 1, 1))
^ ~~
If the length of ```base_name``` is 0, then line 3125 reads 1 byte before the buffer. This will trigger a crash on ASAN-protected builds.
To reproduce:
Build libevent with ASAN:
```
$ CFLAGS='-fomit-frame-pointer -fsanitize=address' ./configure && make -j4
```
Put the attached ```resolv.conf``` and ```poc.c``` in the source directory and then do:
```
$ gcc -fsanitize=address -fomit-frame-pointer poc.c .libs/libevent.a
$ ./a.out
=================================================================
==22201== ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60060000efdf at pc 0x4429da bp 0x7ffe1ed47300 sp 0x7ffe1ed472f8
READ of size 1 at 0x60060000efdf thread T0
```
P.S. we can add a check earlier, but since this is very uncommon, I didn't add it.
Azat Khuzhin [Thu, 24 Mar 2016 09:49:47 +0000 (12:49 +0300)]
evdns: avoid double-free in evdns_base_free() for probing requests
http/cancel_by_host_no_ns:
OK ../test/regress_http.c:1384: assert(regress_dnsserver(data->base, &portnum, search_table))
OK ../test/regress_http.c:1387: assert(dns_base)
OK ../test/regress_http.c:1423: assert(evcon)
OK ../test/regress_http.c:1444: assert(evhttp_make_request(evcon, req, EVHTTP_REQ_GET, "/delay") != -1): 0 vs -1
OK ../test/regress_http.c:1455: assert(test_ok == 2): 2 vs 2
OK ../test/regress_http.c:1480: assert(evhttp_make_request(evcon, req, EVHTTP_REQ_GET, "/test") != -1): 0 vs -1[msg] Nameserver 127.0.0.1:55948 has failed: request timed out.
[msg] All nameservers have failed
OK ../test/regress_http.c:1274: assert(!req)
OK ../test/regress_http.c:1505: assert(evhttp_make_request(evcon, req, EVHTTP_REQ_GET, "/test") != -1): 0 vs -1
OK ../test/regress_http.c:1274: assert(!req)==19199== Invalid read of size 8
==19199== at 0x4CC285: evdns_cancel_request (evdns.c:2849)
==19199== by 0x4CEDB2: evdns_nameserver_free (evdns.c:4018)
==19199== by 0x4CEF5B: evdns_base_free_and_unlock (evdns.c:4052)
==19199== by 0x4CF13B: evdns_base_free (evdns.c:4088)
==19199== by 0x4617A3: http_cancel_test (regress_http.c:1518)
==19199== by 0x490A78: testcase_run_bare_ (tinytest.c:105)
==19199== by 0x490D5A: testcase_run_one (tinytest.c:252)
==19199== by 0x491699: tinytest_main (tinytest.c:434)
==19199== by 0x47E0E0: main (regress_main.c:461)
==19199== Address 0x61e56d0 is 0 bytes inside a block of size 48 free'd
==19199== at 0x4C2AE6B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==19199== by 0x4AAFFF: event_mm_free_ (event.c:3516)
==19199== by 0x4C5ADD: request_finished (evdns.c:693)
==19199== by 0x4CEE95: evdns_base_free_and_unlock (evdns.c:4040)
==19199== by 0x4CF13B: evdns_base_free (evdns.c:4088)
==19199== by 0x4617A3: http_cancel_test (regress_http.c:1518)
==19199== by 0x490A78: testcase_run_bare_ (tinytest.c:105)
==19199== by 0x490D5A: testcase_run_one (tinytest.c:252)
==19199== by 0x491699: tinytest_main (tinytest.c:434)
==19199== by 0x47E0E0: main (regress_main.c:461)
==19199== Block was alloc'd at
==19199== at 0x4C2BBD5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==19199== by 0x4AAEB2: event_mm_calloc_ (event.c:3459)
==19199== by 0x4CAAA2: nameserver_send_probe (evdns.c:2327)
==19199== by 0x4C50FF: nameserver_prod_callback (evdns.c:494)
==19199== by 0x4A564C: event_process_active_single_queue (event.c:1646)
==19199== by 0x4A5B95: event_process_active (event.c:1738)
==19199== by 0x4A6296: event_base_loop (event.c:1961)
==19199== by 0x4A5C1D: event_base_dispatch (event.c:1772)
==19199== by 0x46172C: http_cancel_test (regress_http.c:1507)
==19199== by 0x490A78: testcase_run_bare_ (tinytest.c:105)
==19199== by 0x490D5A: testcase_run_one (tinytest.c:252)
==19199== by 0x491699: tinytest_main (tinytest.c:434)
==19199==
Azat Khuzhin [Thu, 24 Mar 2016 11:05:28 +0000 (14:05 +0300)]
Merge branch 'bufev-cancellations-v5'
This patch set fixes bufferevent via http request cancellations (connect() and
dns-request), it survives tests and cancel.. with --no-fork, so this must be ok
(though I have one patch for dns layer pending).
But I'm not sure about cancel.. unit tests on win32, will fix disable them
later if they will differs (plus maybe we must make them skip-by-default?).
Fixes: #333
* bufev-cancellations-v5:
http: set fd to -1 unconditioally, to avoid leaking of DNS requests
test/http: cover NS timed out during request cancellations separatelly
http: avoid leaking of fd in evhttp_connection_free()
http: get fd from be layer during connection reset
be_sock: cancel in-progress dns requests
evdns: export cancel via callbacks in util (like async lib core/extra issues)
test/http: request cancellation with resolving/{conn,write}-timeouts in progress
Azat Khuzhin [Thu, 24 Mar 2016 10:38:05 +0000 (13:38 +0300)]
http: set fd to -1 unconditioally, to avoid leaking of DNS requests
Otherwise:
http/cancel_by_host_ns_timeout_inactive_server: [msg] Nameserver 127.0.0.1:37035 has failed: request timed out.
[msg] All nameservers have failed
OK
1 tests ok. (0 skipped)
==26211==
==26211== FILE DESCRIPTORS: 3 open at exit.
==26211== Open file descriptor 2: /dev/pts/47
==26211== <inherited from parent>
==26211==
==26211== Open file descriptor 1: /dev/pts/47
==26211== <inherited from parent>
==26211==
==26211== Open file descriptor 0: /dev/pts/47
==26211== <inherited from parent>
==26211==
==26211==
==26211== HEAP SUMMARY:
==26211== in use at exit: 1,112 bytes in 5 blocks
==26211== total heap usage: 149 allocs, 144 frees, 18,826 bytes allocated
==26211==
==26211== 40 bytes in 1 blocks are indirectly lost in loss record 1 of 5
==26211== at 0x4C2BBD5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==26211== by 0x4AAEB2: event_mm_calloc_ (event.c:3459)
==26211== by 0x498F5B: evbuffer_add_cb (buffer.c:3309)
==26211== by 0x4A0EF5: bufferevent_socket_new (bufferevent_sock.c:366)
==26211== by 0x4BFADF: evhttp_connection_base_bufferevent_new (http.c:2375)
==26211== by 0x4BFC8F: evhttp_connection_base_new (http.c:2427)
==26211== by 0x460DAA: http_cancel_test (regress_http.c:1417)
==26211== by 0x490A78: testcase_run_bare_ (tinytest.c:105)
==26211== by 0x490D5A: testcase_run_one (tinytest.c:252)
==26211== by 0x491699: tinytest_main (tinytest.c:434)
==26211== by 0x47E0E0: main (regress_main.c:461)
==26211==
==26211== 136 bytes in 1 blocks are indirectly lost in loss record 2 of 5
==26211== at 0x4C2BBD5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==26211== by 0x4AAEB2: event_mm_calloc_ (event.c:3459)
==26211== by 0x491FF0: evbuffer_new (buffer.c:365)
==26211== by 0x49A1BE: bufferevent_init_common_ (bufferevent.c:300)
==26211== by 0x4A0E44: bufferevent_socket_new (bufferevent_sock.c:353)
==26211== by 0x4BFADF: evhttp_connection_base_bufferevent_new (http.c:2375)
==26211== by 0x4BFC8F: evhttp_connection_base_new (http.c:2427)
==26211== by 0x460DAA: http_cancel_test (regress_http.c:1417)
==26211== by 0x490A78: testcase_run_bare_ (tinytest.c:105)
==26211== by 0x490D5A: testcase_run_one (tinytest.c:252)
==26211== by 0x491699: tinytest_main (tinytest.c:434)
==26211== by 0x47E0E0: main (regress_main.c:461)
==26211==
==26211== 136 bytes in 1 blocks are indirectly lost in loss record 3 of 5
==26211== at 0x4C2BBD5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==26211== by 0x4AAEB2: event_mm_calloc_ (event.c:3459)
==26211== by 0x491FF0: evbuffer_new (buffer.c:365)
==26211== by 0x49A1FB: bufferevent_init_common_ (bufferevent.c:305)
==26211== by 0x4A0E44: bufferevent_socket_new (bufferevent_sock.c:353)
==26211== by 0x4BFADF: evhttp_connection_base_bufferevent_new (http.c:2375)
==26211== by 0x4BFC8F: evhttp_connection_base_new (http.c:2427)
==26211== by 0x460DAA: http_cancel_test (regress_http.c:1417)
==26211== by 0x490A78: testcase_run_bare_ (tinytest.c:105)
==26211== by 0x490D5A: testcase_run_one (tinytest.c:252)
==26211== by 0x491699: tinytest_main (tinytest.c:434)
==26211== by 0x47E0E0: main (regress_main.c:461)
==26211==
==26211== 536 bytes in 1 blocks are indirectly lost in loss record 4 of 5
==26211== at 0x4C2BBD5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==26211== by 0x4AAEB2: event_mm_calloc_ (event.c:3459)
==26211== by 0x4A0E15: bufferevent_socket_new (bufferevent_sock.c:350)
==26211== by 0x4BFADF: evhttp_connection_base_bufferevent_new (http.c:2375)
==26211== by 0x4BFC8F: evhttp_connection_base_new (http.c:2427)
==26211== by 0x460DAA: http_cancel_test (regress_http.c:1417)
==26211== by 0x490A78: testcase_run_bare_ (tinytest.c:105)
==26211== by 0x490D5A: testcase_run_one (tinytest.c:252)
==26211== by 0x491699: tinytest_main (tinytest.c:434)
==26211== by 0x47E0E0: main (regress_main.c:461)
==26211==
==26211== 1,112 (264 direct, 848 indirect) bytes in 1 blocks are definitely lost in loss record 5 of 5
==26211== at 0x4C2BBD5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==26211== by 0x4AAEB2: event_mm_calloc_ (event.c:3459)
==26211== by 0x4D0564: evdns_getaddrinfo (evdns.c:4685)
==26211== by 0x4B13BA: evutil_getaddrinfo_async_ (evutil.c:1575)
==26211== by 0x4A139E: bufferevent_socket_connect_hostname (bufferevent_sock.c:524)
==26211== by 0x4C02DB: evhttp_connection_connect_ (http.c:2588)
==26211== by 0x4C04DD: evhttp_make_request (http.c:2643)
==26211== by 0x4615FF: http_cancel_test (regress_http.c:1504)
==26211== by 0x490A78: testcase_run_bare_ (tinytest.c:105)
==26211== by 0x490D5A: testcase_run_one (tinytest.c:252)
==26211== by 0x491699: tinytest_main (tinytest.c:434)
==26211== by 0x47E0E0: main (regress_main.c:461)
==26211==
==26211== LEAK SUMMARY:
==26211== definitely lost: 264 bytes in 1 blocks
==26211== indirectly lost: 848 bytes in 4 blocks
==26211== possibly lost: 0 bytes in 0 blocks
==26211== still reachable: 0 bytes in 0 blocks
==26211== suppressed: 0 bytes in 0 blocks
Azat Khuzhin [Tue, 22 Mar 2016 20:36:19 +0000 (23:36 +0300)]
http: avoid leaking of fd in evhttp_connection_free()
Since we do close fd there if we don't have BEV_OPT_CLOSE_ON_FREE, and
evcon->fd can be incorrect (non -1), so just get it from the underlying
bufferevent to fix this.
And after this patch the following tests report 0 instead of 2307 fd leaks:
$ valgrind --leak-check=full --show-reachable=yes --track-fds=yes --error-exitcode=1 regress --no-fork http/cancel..
==11299== FILE DESCRIPTORS: 3 open at exit.
Azat Khuzhin [Tue, 22 Mar 2016 16:29:50 +0000 (19:29 +0300)]
http: get fd from be layer during connection reset
Since it can be non -1, and we must close it, otherwise we will have problems.
And after this patch the following tests report fd 2307 instead of 2309 fd leaks:
$ valgrind --leak-check=full --show-reachable=yes --track-fds=yes --error-exitcode=1 regress --no-fork http/cancel..
==10853== FILE DESCRIPTORS: 2307 open at exit.
Azat Khuzhin [Mon, 21 Mar 2016 14:09:17 +0000 (17:09 +0300)]
be_sock: cancel in-progress dns requests
Before this patch we didn't have such functionality and this can cause some
issues when we are cancelling HTTP request while after this we will get a
response/timeout from the NS and in this case error_cb will be called and can
damage newly started HTTP request.
We could also have problems with connect, but we don't have them since we
closes the fd in HTTP layer.
This is not so good that this is done in be_sock, but it is internal, so we can
change without pain. Plus I don't like that callback-via-evutil, but since we
have event_extra we need do like that.
And after this patch the following tests doesn't report leaks:
$ valgrind --leak-check=full --show-reachable=yes --track-fds=yes --error-exitcode=1 regress --no-fork http/cancel..
...
==10469== FILE DESCRIPTORS: 2309 open at exit.
...
==10469== HEAP SUMMARY:
==10469== in use at exit: 0 bytes in 0 blocks
==10469== total heap usage: 33,846 allocs, 33,846 frees, 4,617,651 bytes allocated
==10469==
==10469== All heap blocks were freed -- no leaks are possible
v2: do under lock
v3: reset dns request
v4: ignore EVUTIL_EAI_CANCEL in regular callback
This patches not 100% for http layer, but more for be_sock, but it was simpler
to add them here, plus it also shows some bugs with fd leaking in http layer.
Right now we have next picture (we can also play with timeouts/attempts for
evdns to make tests fail, IOW to track the failures even without valgrind):
$ valgrind --leak-check=full --show-reachable=yes --track-fds=yes --error-exitcode=1 regress --no-fork http/cancel..
http/cancel: OK
http/cancel_by_host: OK
http/cancel_by_host_no_ns: [msg] Nameserver 127.0.0.1:42489 has failed: request timed out.
[msg] All nameservers have failed
OK
http/cancel_by_host_inactive_server: OK
http/cancel_inactive_server: OK
http/cancel_by_host_no_ns_inactive_server: [msg] Nameserver 127.0.0.1:51370 has failed: request timed out.
[msg] All nameservers have failed
OK
http/cancel_by_host_server_timeout: OK
http/cancel_server_timeout: OK
http/cancel_by_host_no_ns_server_timeout: [msg] Nameserver 127.0.0.1:45054 has failed: request timed out.
[msg] All nameservers have failed
OK
9 tests ok. (0 skipped)
==3202==
==3202== FILE DESCRIPTORS: 2309 open at exit.
...
==8403== HEAP SUMMARY:
==8403== in use at exit: 1,104 bytes in 5 blocks
==8403== total heap usage: 10,916 allocs, 10,911 frees, 1,458,818 bytes allocated
==8403==
==8403== 40 bytes in 1 blocks are indirectly lost in loss record 1 of 5
==8403== at 0x4C2BBD5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8403== by 0x4AAD2D: event_mm_calloc_ (event.c:3459)
==8403== by 0x498E48: evbuffer_add_cb (buffer.c:3309)
==8403== by 0x4A0DE2: bufferevent_socket_new (bufferevent_sock.c:366)
==8403== by 0x4BF8BA: evhttp_connection_base_bufferevent_new (http.c:2369)
==8403== by 0x4BFA6A: evhttp_connection_base_new (http.c:2421)
==8403== by 0x460CFC: http_cancel_test (regress_http.c:1413)
==8403== by 0x490965: testcase_run_bare_ (tinytest.c:105)
==8403== by 0x490C47: testcase_run_one (tinytest.c:252)
==8403== by 0x491586: tinytest_main (tinytest.c:434)
==8403== by 0x47DFCD: main (regress_main.c:461)
==8403==
==8403== 136 bytes in 1 blocks are indirectly lost in loss record 2 of 5
==8403== at 0x4C2BBD5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8403== by 0x4AAD2D: event_mm_calloc_ (event.c:3459)
==8403== by 0x491EDD: evbuffer_new (buffer.c:365)
==8403== by 0x49A0AB: bufferevent_init_common_ (bufferevent.c:300)
==8403== by 0x4A0D31: bufferevent_socket_new (bufferevent_sock.c:353)
==8403== by 0x4BF8BA: evhttp_connection_base_bufferevent_new (http.c:2369)
==8403== by 0x4BFA6A: evhttp_connection_base_new (http.c:2421)
==8403== by 0x460CFC: http_cancel_test (regress_http.c:1413)
==8403== by 0x490965: testcase_run_bare_ (tinytest.c:105)
==8403== by 0x490C47: testcase_run_one (tinytest.c:252)
==8403== by 0x491586: tinytest_main (tinytest.c:434)
==8403== by 0x47DFCD: main (regress_main.c:461)
==8403==
==8403== 136 bytes in 1 blocks are indirectly lost in loss record 3 of 5
==8403== at 0x4C2BBD5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8403== by 0x4AAD2D: event_mm_calloc_ (event.c:3459)
==8403== by 0x491EDD: evbuffer_new (buffer.c:365)
==8403== by 0x49A0E8: bufferevent_init_common_ (bufferevent.c:305)
==8403== by 0x4A0D31: bufferevent_socket_new (bufferevent_sock.c:353)
==8403== by 0x4BF8BA: evhttp_connection_base_bufferevent_new (http.c:2369)
==8403== by 0x4BFA6A: evhttp_connection_base_new (http.c:2421)
==8403== by 0x460CFC: http_cancel_test (regress_http.c:1413)
==8403== by 0x490965: testcase_run_bare_ (tinytest.c:105)
==8403== by 0x490C47: testcase_run_one (tinytest.c:252)
==8403== by 0x491586: tinytest_main (tinytest.c:434)
==8403== by 0x47DFCD: main (regress_main.c:461)
==8403==
==8403== 528 bytes in 1 blocks are indirectly lost in loss record 4 of 5
==8403== at 0x4C2BBD5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8403== by 0x4AAD2D: event_mm_calloc_ (event.c:3459)
==8403== by 0x4A0D02: bufferevent_socket_new (bufferevent_sock.c:350)
==8403== by 0x4BF8BA: evhttp_connection_base_bufferevent_new (http.c:2369)
==8403== by 0x4BFA6A: evhttp_connection_base_new (http.c:2421)
==8403== by 0x460CFC: http_cancel_test (regress_http.c:1413)
==8403== by 0x490965: testcase_run_bare_ (tinytest.c:105)
==8403== by 0x490C47: testcase_run_one (tinytest.c:252)
==8403== by 0x491586: tinytest_main (tinytest.c:434)
==8403== by 0x47DFCD: main (regress_main.c:461)
==8403==
==8403== 1,104 (264 direct, 840 indirect) bytes in 1 blocks are definitely lost in loss record 5 of 5
==8403== at 0x4C2BBD5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8403== by 0x4AAD2D: event_mm_calloc_ (event.c:3459)
==8403== by 0x4D0326: evdns_getaddrinfo (evdns.c:4682)
==8403== by 0x4B1213: evutil_getaddrinfo_async_ (evutil.c:1568)
==8403== by 0x4A1255: bufferevent_socket_connect_hostname (bufferevent_sock.c:517)
==8403== by 0x4C00B6: evhttp_connection_connect_ (http.c:2582)
==8403== by 0x4C02B8: evhttp_make_request (http.c:2637)
==8403== by 0x4614EC: http_cancel_test (regress_http.c:1496)
==8403== by 0x490965: testcase_run_bare_ (tinytest.c:105)
==8403== by 0x490C47: testcase_run_one (tinytest.c:252)
==8403== by 0x491586: tinytest_main (tinytest.c:434)
==8403== by 0x47DFCD: main (regress_main.c:461)
==8403==
==8403== LEAK SUMMARY:
==8403== definitely lost: 264 bytes in 1 blocks
==8403== indirectly lost: 840 bytes in 4 blocks
==8403== possibly lost: 0 bytes in 0 blocks
==8403== still reachable: 0 bytes in 0 blocks
==8403== suppressed: 0 bytes in 0 blocks
Azat Khuzhin [Sun, 13 Mar 2016 10:05:25 +0000 (13:05 +0300)]
test/http: exit from the loop in the errorcb to wait cancellation
This will make cancellation tests more graceful, that said that error_cb can
not be called sometimes if you will break the loop in cancel.
Plus drop that define for function generations, since function body changed,
and it is not generic anymore, plus that macro didn't used by anyone else.
Azat Khuzhin [Fri, 11 Mar 2016 16:58:05 +0000 (19:58 +0300)]
http: fix EVHTTP_CON_READ_ON_WRITE_ERROR when it doesn't supported by OS
For example win32 doesn't accept such things (maybe via overloaded IO, I'm not
sure), also I looked into curl and seems that the behaviour is the same (IOW
like with EVHTTP_CON_READ_ON_WRITE_ERROR on linux/win32).
Fixes: https://ci.appveyor.com/project/nmathewson/libevent/build/2.1.5.216#L499 (win32) Fixes: 680742e1665b85487f10c0ef3df021e3b8e98634 ("http: read server response
even after server closed the connection")
v2: v0 was just removing that flag, i.e. make it deprecated and set_flags() will return -1
This patch set fixes and covers http client and "Expect: 100-continue"
functionality (plus increase coverage under some related options, to avoid
further regressions).
* http-client-fix-expect-100-continue-v6:
http: fix leaking of response_code_line
http: fix "Expect: 100-continue" client side
test/http: separate coverage for EVHTTP_CON_READ_ON_WRITE_ERROR
test/http: cover "Expect: 100-continue" client-server interaction
test/http: *lingering tests shouldn't have "Expect: 100-continue"
Azat Khuzhin [Fri, 11 Mar 2016 17:40:52 +0000 (20:40 +0300)]
http: fix leaking of response_code_line
Since now evhttp_parse_response_line() can be called twice because after
"HTTP/1.1 100 Continue" we can have "HTTP/1.1 200"
==29162== 9 bytes in 1 blocks are definitely lost in loss record 1 of 1
==29162== at 0x4C29C0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==29162== by 0x5CBF0A9: strdup (in /lib/x86_64-linux-gnu/libc-2.21.so)
==29162== by 0x4AA3AC: event_mm_strdup_ (event.c:3493)
==29162== by 0x4BD843: evhttp_parse_response_line (http.c:1680)
==29162== by 0x4BE333: evhttp_parse_firstline_ (http.c:2013)
==29162== by 0x4BEA4F: evhttp_read_firstline (http.c:2243)
==29162== by 0x4BC5F8: evhttp_read_cb (http.c:1136)
==29162== by 0x4993F1: bufferevent_run_readcb_ (bufferevent.c:233)
==29162== by 0x49FBC0: bufferevent_trigger_nolock_ (bufferevent-internal.h:392)
==29162== by 0x49FF10: bufferevent_readcb (bufferevent_sock.c:208)
==29162== by 0x4A474A: event_persist_closure (event.c:1580)
==29162== by 0x4A49F5: event_process_active_single_queue (event.c:1639)
projects / libevent / log