Daniel Stenberg [Tue, 19 Feb 2019 09:02:27 +0000 (10:02 +0100)]
curl: remove MANUAL from -M output
... and remove it from the dist tarball. It has served its time, it
barely gets updated anymore and "everything curl" is now convering all
this document once tried to include, and does it more and better.
In the compressed scenario, this removes ~15K data from the binary,
which is 25% of the -M output.
It remains in the git repo for now for as long as the web site builds a
page using that as source. It renders poorly on the site (especially for
mobile users) so its not even good there.
Daniel Stenberg [Mon, 18 Feb 2019 08:10:01 +0000 (09:10 +0100)]
http2: verify :athority in push promise requests
RFC 7540 says we should verify that the push is for an "authoritative"
server. We make sure of this by only allowing push with an :athority
header that matches the host that was asked for in the URL.
Fixes #3577 Reported-by: Nicolas Grekas
Bug: https://curl.haxx.se/mail/lib-2019-02/0057.html
Closes #3581
Daniel Stenberg [Mon, 18 Feb 2019 07:14:52 +0000 (08:14 +0100)]
http: make adding a blank header thread-safe
Previously the function would edit the provided header in-place when a
semicolon is used to signify an empty header. This made it impossible to
use the same set of custom headers in multiple threads simultaneously.
This approach now makes a local copy when it needs to edit the string.
Reported-by: d912e3 on github
Fixes #3578
Closes #3579
Jay Satiro [Sat, 16 Feb 2019 23:55:40 +0000 (18:55 -0500)]
easy: fix win32 init to work without CURL_GLOBAL_WIN32
- Change the behavior of win32_init so that the required initialization
procedures are not affected by CURL_GLOBAL_WIN32 flag.
libcurl via curl_global_init supports initializing for win32 with an
optional flag CURL_GLOBAL_WIN32, which if omitted was meant to stop
Winsock initialization. It did so internally by skipping win32_init()
when that flag was set. Since then win32_init() has been expanded to
include required initialization routines that are separate from
Winsock and therefore must be called in all cases. This commit fixes
it so that CURL_GLOBAL_WIN32 only controls the optional win32
initialization (which is Winsock initialization, according to our doc).
The only users affected by this change are those that don't pass
CURL_GLOBAL_WIN32 to curl_global_init. For them this commit removes the
risk of a potential crash.
The draft-ietf-httpbis-rfc6265bis-02 draft, specify a set of prefixes
and how they should affect cookie initialization, which has been
adopted by the major browsers. This adds support for the two prefixes
defined, __Host- and __Secure, and updates the testcase with the
supplied examples from the draft.
Closes #3554 Reviewed-by: Daniel Stenberg <daniel@haxx.se>
If mbedtls_ssl_get_session() fails, it may still have allocated
memory that needs to be freed to avoid leaking. Call the library
API function to release session resources on this errorpath as
well as on Curl_ssl_addsessionid() errors.
Closes: #3574 Reported-by: Michał Antoniak <M.Antoniak@posnet.com> Reviewed-by: Daniel Stenberg <daniel@haxx.se>
It was discovered a month ago that before checking whether to extract a
dead connection that that connection should be associated with a "live"
transfer for the check (ie original conn->data ignored and set to the
passed in data). A fix was landed in 54b201b which did that and also
cleared conn->data after the check. The original conn->data was not
restored, so presumably it was thought that a valid conn->data was no
longer needed.
Several days later it was discovered that a valid conn->data was needed
after the check and follow-up fix was landed in bbae24c which partially
reverted the original fix and attempted to limit the scope of when
conn->data was changed to only when pruning dead connections. In that
case conn->data was not cleared and the original conn->data not
restored.
A month later it was discovered that the original fix was somewhat
correct; a "live" transfer is needed for the check in all cases
because original conn->data could be null which could cause a bad deref
at arbitrary points in the check. A fix was landed in 38d8e1b which
expanded the scope to all cases. conn->data was not cleared and the
original conn->data not restored.
A day later it was discovered that not restoring the original conn->data
may lead to busy loops in applications that use the event interface, and
given this observation it's a pretty safe assumption that there is some
code path that still needs the original conn->data. This commit is the
follow-up fix for that, it restores the original conn->data after the
connection check.
Patrick Monnerat [Thu, 14 Feb 2019 15:03:24 +0000 (16:03 +0100)]
transfer.c: do not compute length of undefined hex buffer.
On non-ascii platforms, the chunked hex header was measured for char code
conversion length, even for chunked trailers that do not have an hex header.
In addition, the efective length is already known: use it.
Since the hex length can be zero, only convert if needed.
Frank Gevaerts [Fri, 8 Feb 2019 09:55:45 +0000 (10:55 +0100)]
tests: add stderr comparison to the test suite
The code is more or less copied from the stdout comparison code, maybe
some better reuse is possible.
test 1457 is adjusted to make the output actually match (by using --silent)
test 506 used <stderr> without actually needing it, so that <stderr> block is removed
Patrick Monnerat [Mon, 11 Feb 2019 18:10:41 +0000 (19:10 +0100)]
cli tool: do not use mime.h private structures.
Option -F generates an intermediate representation of the mime structure
that is used later to create the libcurl mime structure and generate
the --libcurl statements.
Reported-by: Daniel Stenberg
Fixes #3532
Closes #3546
There is no benefit to holding the data sharelock when freeing the
addrinfo in case it fails, so ensure releaseing it as soon as we can
rather than holding on to it. This also aligns the code with other
consumers of sharelocks.
Closes #3516 Reviewed-by: Daniel Stenberg <daniel@haxx.se>
Daniel Stenberg [Fri, 8 Feb 2019 11:08:42 +0000 (12:08 +0100)]
scripts/singleuse: script to use to track single-use functions
That is functions that are declared global but are not used from outside
of the file in which it is declared. Such functions should be made
static or even at times be removed.
It also verifies that all used curl_ prefixed functions are "blessed"
Dan Fandrich [Sat, 9 Feb 2019 09:04:39 +0000 (10:04 +0100)]
cirrus: Added FreeBSD builds using Cirrus CI.
The build logs will be at https://cirrus-ci.com/github/curl/curl
Some tests are currently failing and so disabled for now. The SSH server
isn't starting for the SSH tests due to unsupported options used in its
config file. The DICT server also is failing on startup.
zsh.pl: update regex to better match curl -h output
The current regex fails to match '<...>' arguments properly (e.g. those
with spaces in them), which causes an completion script with wrong
descriptions for some options.
Here's a diff of the generated completion script, comparing the previous
version to the one with this fix:
_arguments -C -S \
--happy-eyeballs-timeout-ms'[How long to wait in milliseconds for IPv6 before trying IPv4]':'<milliseconds>' \
+ --resolve'[Resolve the host+port to this address]':'<host:port:address[,address]...>' \
{-c,--cookie-jar}'[Write cookies to <filename> after operation]':'<filename>':_files \
{-D,--dump-header}'[Write the received headers to <filename>]':'<filename>':_files \
{-y,--speed-time}'[Trigger '\''speed-limit'\'' abort after this time]':'<seconds>' \
--proxy-cacert'[CA certificate to verify peer against for proxy]':'<file>':_files \
- --tls13-ciphers'[of TLS 1.3 ciphersuites> TLS 1.3 cipher suites to use]':'<list' \
+ --tls13-ciphers'[TLS 1.3 cipher suites to use]':'<list of TLS 1.3 ciphersuites>' \
{-E,--cert}'[Client certificate file and password]':'<certificate[:password]>' \
--libcurl'[Dump libcurl equivalent code of this command line]':'<file>':_files \
--proxy-capath'[CA directory to verify peer against for proxy]':'<dir>':_files \
- --proxy-negotiate'[HTTP Negotiate (SPNEGO) authentication on the proxy]':'Use' \
--proxy-pinnedpubkey'[FILE/HASHES public key to verify proxy with]':'<hashes>' \
--crlfile'[Get a CRL list in PEM format from the given file]':'<file>':_files \
- --proxy-insecure'[HTTPS proxy connections without verifying the proxy]':'Do' \
- --proxy-ssl-allow-beast'[security flaw for interop for HTTPS proxy]':'Allow' \
+ --proxy-negotiate'[Use HTTP Negotiate (SPNEGO) authentication on the proxy]' \
--abstract-unix-socket'[Connect via abstract Unix domain socket]':'<path>' \
--pinnedpubkey'[FILE/HASHES Public key to verify peer against]':'<hashes>' \
+ --proxy-insecure'[Do HTTPS proxy connections without verifying the proxy]' \
--proxy-pass'[Pass phrase for the private key for HTTPS proxy]':'<phrase>' \
+ --proxy-ssl-allow-beast'[Allow security flaw for interop for HTTPS proxy]' \
{-p,--proxytunnel}'[Operate through an HTTP proxy tunnel (using CONNECT)]' \
--socks5-hostname'[SOCKS5 proxy, pass host name to proxy]':'<host[:port]>' \
--proto-default'[Use PROTOCOL for any URL missing a scheme]':'<protocol>' \
- --proxy-tls13-ciphers'[list> TLS 1.3 proxy cipher suites]':'<ciphersuite' \
+ --proxy-tls13-ciphers'[TLS 1.3 proxy cipher suites]':'<ciphersuite list>' \
--socks5-gssapi-service'[SOCKS5 proxy service name for GSS-API]':'<name>' \
--ftp-alternative-to-user'[String to replace USER \[name\]]':'<command>' \
- --ftp-ssl-control'[SSL/TLS for FTP login, clear for transfer]':'Require' \
{-T,--upload-file}'[Transfer local FILE to destination]':'<file>':_files \
--local-port'[Force use of RANGE for local port numbers]':'<num/range>' \
--proxy-tlsauthtype'[TLS authentication type for HTTPS proxy]':'<type>' \
{-R,--remote-time}'[Set the remote file'\''s time on the local output]' \
- --retry-connrefused'[on connection refused (use with --retry)]':'Retry' \
- --suppress-connect-headers'[proxy CONNECT response headers]':'Suppress' \
- {-j,--junk-session-cookies}'[session cookies read from file]':'Ignore' \
- --location-trusted'[--location, and send auth to other hosts]':'Like' \
+ --ftp-ssl-control'[Require SSL/TLS for FTP login, clear for transfer]' \
--proxy-cert-type'[Client certificate type for HTTPS proxy]':'<type>' \
{-O,--remote-name}'[Write output to a file named as the remote file]' \
+ --retry-connrefused'[Retry on connection refused (use with --retry)]' \
+ --suppress-connect-headers'[Suppress proxy CONNECT response headers]' \
--trace-ascii'[Like --trace, but without hex output]':'<file>':_files \
--connect-timeout'[Maximum time allowed for connection]':'<seconds>' \
--expect100-timeout'[How long to wait for 100-continue]':'<seconds>' \
{-g,--globoff}'[Disable URL sequences and ranges using {} and \[\]]' \
+ {-j,--junk-session-cookies}'[Ignore session cookies read from file]' \
{-m,--max-time}'[Maximum time allowed for the transfer]':'<seconds>' \
--dns-ipv4-addr'[IPv4 address to use for DNS requests]':'<address>' \
--dns-ipv6-addr'[IPv6 address to use for DNS requests]':'<address>' \
- --ignore-content-length'[the size of the remote resource]':'Ignore' \
{-k,--insecure}'[Allow insecure server connections when using SSL]' \
+ --location-trusted'[Like --location, and send auth to other hosts]' \
--mail-auth'[Originator address of the original email]':'<address>' \
--noproxy'[List of hosts which do not use proxy]':'<no-proxy-list>' \
--proto-redir'[Enable/disable PROTOCOLS on redirect]':'<protocols>' \
@@ -62,18 +62,19 @@
--socks5-basic'[Enable username/password auth for SOCKS5 proxies]' \
--cacert'[CA certificate to verify peer against]':'<file>':_files \
{-H,--header}'[Pass custom header(s) to server]':'<header/@file>' \
+ --ignore-content-length'[Ignore the size of the remote resource]' \
{-i,--include}'[Include protocol response headers in the output]' \
--proxy-header'[Pass custom header(s) to proxy]':'<header/@file>' \
--unix-socket'[Connect through this Unix domain socket]':'<path>' \
{-w,--write-out}'[Use output FORMAT after completion]':'<format>' \
- --http2-prior-knowledge'[HTTP 2 without HTTP/1.1 Upgrade]':'Use' \
{-o,--output}'[Write to file instead of stdout]':'<file>':_files \
- {-J,--remote-header-name}'[the header-provided filename]':'Use' \
+ --preproxy'[\[protocol://\]host\[:port\] Use this proxy first]' \
--socks4a'[SOCKS4a proxy on given host + port]':'<host[:port]>' \
{-Y,--speed-limit}'[Stop transfers slower than this]':'<speed>' \
{-z,--time-cond}'[Transfer based on a time condition]':'<time>' \
--capath'[CA directory to verify peer against]':'<dir>':_files \
{-f,--fail}'[Fail silently (no output at all) on HTTP errors]' \
+ --http2-prior-knowledge'[Use HTTP 2 without HTTP/1.1 Upgrade]' \
--proxy-tlspassword'[TLS password for HTTPS proxy]':'<string>' \
{-U,--proxy-user}'[Proxy user and password]':'<user:password>' \
--proxy1.0'[Use HTTP/1.0 proxy on given port]':'<host[:port]>' \
@@ -81,52 +82,49 @@
{-A,--user-agent}'[Send User-Agent <name> to server]':'<name>' \
--egd-file'[EGD socket path for random data]':'<file>':_files \
--fail-early'[Fail on first transfer error, do not continue]' \
- --haproxy-protocol'[HAProxy PROXY protocol v1 header]':'Send' \
- --preproxy'[Use this proxy first]':'[protocol://]host[:port]' \
+ {-J,--remote-header-name}'[Use the header-provided filename]' \
--retry-max-time'[Retry only within this period]':'<seconds>' \
--socks4'[SOCKS4 proxy on given host + port]':'<host[:port]>' \
--socks5'[SOCKS5 proxy on given host + port]':'<host[:port]>' \
- --socks5-gssapi-nec'[with NEC SOCKS5 server]':'Compatibility' \
- --ssl-allow-beast'[security flaw to improve interop]':'Allow' \
--cert-status'[Verify the status of the server certificate]' \
- --ftp-create-dirs'[the remote dirs if not present]':'Create' \
{-:,--next}'[Make next URL use its separate set of options]' \
--proxy-key-type'[Private key file type for proxy]':'<type>' \
- --remote-name-all'[the remote file name for all URLs]':'Use' \
{-X,--request}'[Specify request command to use]':'<command>' \
--retry'[Retry request if transient problems occur]':'<num>' \
- --ssl-no-revoke'[cert revocation checks (WinSSL)]':'Disable' \
--cert-type'[Certificate file type (DER/PEM/ENG)]':'<type>' \
--connect-to'[Connect to host]':'<HOST1:PORT1:HOST2:PORT2>' \
--create-dirs'[Create necessary local directory hierarchy]' \
+ --haproxy-protocol'[Send HAProxy PROXY protocol v1 header]' \
--max-redirs'[Maximum number of redirects allowed]':'<num>' \
{-n,--netrc}'[Must read .netrc for user name and password]' \
+ {-x,--proxy}'[\[protocol://\]host\[:port\] Use this proxy]' \
--proxy-crlfile'[Set a CRL list for proxy]':'<file>':_files \
--sasl-ir'[Enable initial response in SASL authentication]' \
- --socks5-gssapi'[GSS-API auth for SOCKS5 proxies]':'Enable' \
+ --socks5-gssapi-nec'[Compatibility with NEC SOCKS5 server]' \
+ --ssl-allow-beast'[Allow security flaw to improve interop]' \
+ --ftp-create-dirs'[Create the remote dirs if not present]' \
--interface'[Use network INTERFACE (or address)]':'<name>' \
--key-type'[Private key file type (DER/PEM/ENG)]':'<type>' \
--netrc-file'[Specify FILE for netrc]':'<filename>':_files \
{-N,--no-buffer}'[Disable buffering of the output stream]' \
--proxy-service-name'[SPNEGO proxy service name]':'<name>' \
- --styled-output'[styled output for HTTP headers]':'Enable' \
+ --remote-name-all'[Use the remote file name for all URLs]' \
+ --ssl-no-revoke'[Disable cert revocation checks (WinSSL)]' \
--max-filesize'[Maximum file size to download]':'<bytes>' \
--negotiate'[Use HTTP Negotiate (SPNEGO) authentication]' \
--no-keepalive'[Disable TCP keepalive on the connection]' \
{-#,--progress-bar}'[Display transfer progress as a bar]' \
- {-x,--proxy}'[Use this proxy]':'[protocol://]host[:port]' \
- --proxy-anyauth'[any proxy authentication method]':'Pick' \
{-Q,--quote}'[Send command(s) to server before transfer]' \
- --request-target'[the target for this request]':'Specify' \
+ --socks5-gssapi'[Enable GSS-API auth for SOCKS5 proxies]' \
{-u,--user}'[Server user and password]':'<user:password>' \
{-K,--config}'[Read config from a file]':'<file>':_files \
{-C,--continue-at}'[Resumed transfer offset]':'<offset>' \
--data-raw'[HTTP POST data, '\''@'\'' allowed]':'<data>' \
- --disallow-username-in-url'[username in url]':'Disallow' \
--krb'[Enable Kerberos with security <level>]':'<level>' \
--proxy-ciphers'[SSL ciphers to use for proxy]':'<list>' \
--proxy-digest'[Use Digest authentication on the proxy]' \
--proxy-tlsuser'[TLS username for HTTPS proxy]':'<name>' \
+ --styled-output'[Enable styled output for HTTP headers]' \
{-b,--cookie}'[Send cookies from string/file]':'<data>' \
--data-urlencode'[HTTP POST data url encoded]':'<data>' \
--delegation'[GSS-API delegation permission]':'<LEVEL>' \
@@ -134,7 +132,10 @@
--post301'[Do not switch to GET after following a 301]' \
--post302'[Do not switch to GET after following a 302]' \
--post303'[Do not switch to GET after following a 303]' \
+ --proxy-anyauth'[Pick any proxy authentication method]' \
+ --request-target'[Specify the target for this request]' \
--trace-time'[Add time stamps to trace/verbose output]' \
+ --disallow-username-in-url'[Disallow username in url]' \
--dns-servers'[DNS server addrs to use]':'<addresses>' \
{-G,--get}'[Put the post data in the URL and use GET]' \
--limit-rate'[Limit transfer speed to RATE]':'<speed>' \
@@ -148,21 +149,21 @@
--metalink'[Process given URLs as metalink XML file]' \
--tr-encoding'[Request compressed transfer encoding]' \
--xattr'[Store metadata in extended file attributes]' \
- --ftp-skip-pasv-ip'[the IP address for PASV]':'Skip' \
--pass'[Pass phrase for the private key]':'<phrase>' \
--proxy-ntlm'[Use NTLM authentication on the proxy]' \
{-S,--show-error}'[Show error even when -s is used]' \
- --ciphers'[of ciphers> SSL ciphers to use]':'<list' \
+ --ciphers'[SSL ciphers to use]':'<list of ciphers>' \
--form-string'[Specify multipart MIME data]':'<name=string>' \
--login-options'[Server login options]':'<options>' \
--tftp-blksize'[Set TFTP BLKSIZE option]':'<value>' \
- --tftp-no-options'[not send any TFTP options]':'Do' \
{-v,--verbose}'[Make the operation more talkative]' \
+ --ftp-skip-pasv-ip'[Skip the IP address for PASV]' \
--proxy-key'[Private key for HTTPS proxy]':'<key>' \
{-F,--form}'[Specify multipart MIME data]':'<name=content>' \
--mail-from'[Mail from this address]':'<address>' \
--oauth2-bearer'[OAuth 2 Bearer Token]':'<token>' \
--proto'[Enable/disable PROTOCOLS]':'<protocols>' \
+ --tftp-no-options'[Do not send any TFTP options]' \
--tlsauthtype'[TLS authentication type]':'<type>' \
--doh-url'[Resolve host names over DOH]':'<URL>' \
--no-sessionid'[Disable SSL session-ID reusing]' \
@@ -173,14 +174,13 @@
--ftp-ssl-ccc'[Send CCC after authenticating]' \
{-4,--ipv4}'[Resolve names to IPv4 addresses]' \
{-6,--ipv6}'[Resolve names to IPv6 addresses]' \
- --netrc-optional'[either .netrc or URL]':'Use' \
--service-name'[SPNEGO service name]':'<name>' \
{-V,--version}'[Show version number and quit]' \
--data-ascii'[HTTP POST ASCII data]':'<data>' \
--ftp-account'[Account data string]':'<data>' \
- --compressed-ssh'[SSH compression]':'Enable' \
--disable-eprt'[Inhibit using EPRT or LPRT]' \
--ftp-method'[Control CWD usage]':'<method>' \
+ --netrc-optional'[Use either .netrc or URL]' \
--pubkey'[SSH Public key file name]':'<key>' \
--raw'[Do HTTP "raw"; no transfer decoding]' \
--anyauth'[Pick any authentication method]' \
@@ -189,6 +189,7 @@
--no-alpn'[Disable the ALPN TLS extension]' \
--tcp-nodelay'[Use the TCP_NODELAY option]' \
{-B,--use-ascii}'[Use ASCII/text transfer]' \
+ --compressed-ssh'[Enable SSH compression]' \
--digest'[Use HTTP Digest Authentication]' \
--proxy-tlsv1'[Use TLSv1 for HTTPS proxy]' \
--engine'[Crypto engine to use]':'<name>' \
Marcel Raad [Wed, 6 Feb 2019 13:59:15 +0000 (14:59 +0100)]
tool_operate: fix typecheck warning
Use long for CURLOPT_HTTP09_ALLOWED to fix the following warning:
tool_operate.c: In function 'operate_do':
../include/curl/typecheck-gcc.h:47:9: error: call to
'_curl_easy_setopt_err_long' declared with attribute warning:
curl_easy_setopt expects a long argument for this option [-Werror]
Chris Araman [Wed, 6 Feb 2019 05:56:36 +0000 (21:56 -0800)]
url: close TLS before removing conn from cache
- Fix potential crashes in schannel shutdown.
Ensure any TLS shutdown messages are sent before removing the
association between the connection and the easy handle. Reverts
@bagder's previous partial fix for #3412.
Commit 7a09b52c98ac8d840a8a9907b1a1d9a9e684bcf5 introduced support
for the draft-ietf-httpbis-cookie-alone-01 cookie draft, and while
the entry was removed from the TODO it was mistakenly left here.
Fix by removing and rewording the entry slightly.
Closes #3530 Reviewed-by: Daniel Stenberg <daniel@haxx.se>
But on line 3868, that option is omitted. This caused problems for me,
as the symbol-scan.pl script in particular couldn't find its
dependencies properly:
If the incoming len 5, but the buffer does not have a termination
after 5 bytes, the strtol() call may keep reading through the line
buffer until is exceeds its boundary. Fix by ensuring that we are
using a bounded read with a temporary buffer on the stack.
Bug: https://curl.haxx.se/docs/CVE-2019-3823.html Reported-by: Brian Carpenter (Geeknik Labs)
CVE-2019-3823
georgeok [Tue, 29 Jan 2019 17:26:31 +0000 (18:26 +0100)]
spnego_sspi: add support for channel binding
Attempt to add support for Secure Channel binding when negotiate
authentication is used. The problem to solve is that by default IIS
accepts channel binding and curl doesn't utilise them. The result was a
401 response. Scope affects only the Schannel(winssl)-SSPI combination.
John Marshall [Thu, 31 Jan 2019 11:52:51 +0000 (11:52 +0000)]
doc: use meaningless port number in CURLOPT_LOCALPORT example
Use an ephemeral port number here; previously the example had 8080
which could be confusing as the common web server port number might
be misinterpreted as suggesting this option affects the remote port.
Jay Satiro [Tue, 29 Jan 2019 05:33:14 +0000 (00:33 -0500)]
TODO: WinSSL: 'Add option to disable client cert auto-send'
By default WinSSL selects and send a client certificate automatically,
but for privacy and consistency we should offer an option to disable the
default auto-send behavior.
Jeremie Rapin [Wed, 23 Jan 2019 14:35:46 +0000 (15:35 +0100)]
sigpipe: if mbedTLS is used, ignore SIGPIPE
mbedTLS doesn't have a sigpipe management. If a write/read occurs when
the remote closes the socket, the signal is raised and kills the
application. Use the curl mecanisms fix this behavior.
projects / curl / log