PEM_def_callback(): use same parameter names as for pem_password_cb

Add a bit more commentary to explain what's going on.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6080)

(cherry picked from commit d6d94d339756332bbabe2a1032ac511ae31b3fdc)

PEM_def_callback(): don't loop because of too short password given

That error is already caught by EVP_read_pw_string_min, and causes
this function to return -1, so the code detecting too short passwords
in this function is practically dead.

Fixes #5465

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6080)

(cherry picked from commit 4977b4e9281c981efcf6a8b31378b8bbd6a8a96f)

openssl rehash: exit 0 on warnings, same as c_rehash

Fixes #6083

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/6084)

(cherry picked from commit e6a833cb97ed762408b57ea3efa83bd10c1d2a78)

ms/uplink-x86.pl: close the file handle that was opened

Fixes #5656

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6079)

(cherry picked from commit 3625f6c977beccd7bb0d80a1a670c19f4338a27a)

Fix documentation for the -showcerts s_client option

This option shows the certificates as sent by the server. It is not the
full verified chain.

Fixes #4933

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6068)

Fix the MAX_CURVELIST definition

The MAX_CURVELIST macro defines the total number of in-built SSL/TLS curves
that we support. However it has not been updated as new curves are added.

Fixes #5232

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/6065)

(cherry picked from commit ca50cd911ca3c9dc9ec8dd956f8eb45557585a98)

In a reneg use the same client_version we used last time

In 1.0.2 and below we always send the same client_version in a reneg
ClientHello that we sent the first time around, regardless of what
version eventually gets negotiated. According to a comment in
statem_clnt.c this is a workaround for some buggy servers that choked if
we changed the version used in the RSA encrypted premaster secret.

In 1.1.0+ this behaviour no longer occurs. This restores the original
behaviour.

Fixes #1651

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6060)

Document when a session gets removed from cache

Document the fact that if a session is not closed down cleanly then the
session gets removed from the cache and marked as non-resumable.

Fixes #4720

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6054)

Allow intermediate CAs to use RSA PSS in 1.1.0

In 1.1.0 and above we check the digest algorithm used to create signatures
in intermediate CA certs. If it is not sufficiently strong then we reject
the cert. To work out what digest was used we look at the OID for the
signature. This works for most signatures, but not for RSA PSS where the
digest is stored as parameter of the SignatureAlgorithmIdentifier. This
results in the digest look up routines failing and the cert being rejected.

PR #3301 added support for doing this properly in master. So in that
branch this all works as expected. It also works properly in 1.0.2 where we
don't have the digest checks at all. So the only branch where this fails is
1.1.0.

PR #3301 seems too significant to backport to 1.1.0. Instead we simply skip
the signature digest algorithm strength checks if we detect RSA PSS.

Fixes #3558.

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/6052)

Improve backwards compat with 1.0.2 for ECDHParameters

In 1.0.2 you could configure automatic ecdh params by using the
ECDHParameters config directive and setting it to the value
"+Automatic" or just "Automatic". This is no longer required in 1.1.0+
but we still recognise the "+Automatic" keyword for backwards compatibility.
However we did not recognise just "Automatic" without the leading "+" which
is equally valid. This commit fixes that omission.

Fixes #4113

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6036)

Remove superfluous NULL checks. Add Andy's BN_FLG comment.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6009)

(cherry picked from commit 39df51522ba2e3773ae2f1d4df5a6031ef41c1ba)

Move up check for EC_R_INCOMPATIBLE_OBJECTS and for the point at infinity case

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6009)

(cherry picked from commit 736b31e5ea33166d89d5cff5774697d0c15d96bd)

Pass through

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6009)

(cherry picked from commit f4675379275c304dbfa593cc573b4e4c4eb54bd4)

ladder description: why it works

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6009)

(cherry picked from commit a067a8705a654c85d43b942e0d1616e282667969)

Address code style comments

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6009)

(cherry picked from commit 36bed230b580f92d2e10d13e4ba472236e622562)

Elliptic curve scalar multiplication with timing attack defenses

Co-authored-by: Nicola Tuveri <nic.tuv@gmail.com>
Co-authored-by: Cesar Pereida Garcia <cesar.pereidagarcia@tut.fi>
Co-authored-by: Sohaib ul Hassan <soh.19.hassan@gmail.com>
Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6009)

(cherry picked from commit 40e48e54582e46c1a01e184ecf5bd31f4f7f8294)

ARM assembly pack: make it work with older assembler.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/6043)

(cherry picked from commit 198a2ed791e8f4f00d0b92272987f564ca1d9783)

Fix openssl ca, to correctly make output file binary when using -spkac

On Unix, this doesn't matter, but on other platforms, it may.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6050)

(cherry picked from commit 25642ad29e6a2c15c10ceb5e4f029638f73a879e)

Fix late opening of output file

For 'openssl dhparams', the output file was opened after calculations
were made, which is a waste of cycles and time if the output file
turns out not to be writable.

Fixes #3404

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6051)

(cherry picked from commit 10b37541dc7f27b0ca74102970691ce4636cc0bd)

Ensure the thread keys are always allocated in the same order

Back-port of #5911
Fixes: #5899
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6037)

Fix a warning about missing prototype on arm

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6028)

Fix a gcc-8 warning -Wcast-function-type

Casting to the generic function type "void (*)(void)"
prevents the warning.

Backport of #5816

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6027)

Fix building linux-armv4 with --strict-warnings

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6026)

(cherry picked from commit 0e0f8116e247912f5c48f8b3786e543f37fc1f87)

Add a test for SSL_pending()

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6021)

Fix SSL_pending() for DTLS

DTLS was not correctly returning the number of pending bytes left in
a call to SSL_pending(). This makes the detection of truncated packets
almost impossible.

Fixes #5478

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6021)

Fix the alert sent if no shared sig algs

We were sending illegal parameter. This isn't correct. The parameters are
legal, we just don't have an overlap. A more appropriate alert is
handshake failure.

Fixes #2919

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6010)

Don't distribute team internal config targets

Configurations/90-team.conf isn't for public consumption, so we rename
it to 90-team.norelease.conf and make sure 'make dist' and 'make tar'
don't include it in the tarball.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6016)

Add a test for a NULL X509_STORE in X509_STORE_CTX_init

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6002)

Don't crash if there are no trusted certs

The X509_STORE_CTX_init() docs explicitly allow a NULL parameter for the
X509_STORE. Therefore we shouldn't crash if we subsequently call
X509_verify_cert() and no X509_STORE has been set.

Fixes #2462

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6002)

Fix ocsp app exit code

If we run the ocsp command line app and the responder returns a
non-successful status code then the app should exit with a failure code.

Based on an original patch by Tatsuhiro Tsujikawa.

Fixes #2387

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5999)

X509_cmp_time: only return 1, 0, -1.

The behaviour of X509_cmp_time used to be undocumented.

The new behaviour, documented in master, is to return only 0, 1, or -1.
Make the code in the other branches to adhere to this behaviour too,
to reduce confusion. There is nothing to be gained from returning
other values.

Fixes GH#4954

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4955)

Update EVP_DigestSignInit() docs

Explicitly state which digests can be used with which algorithms.

Fixes #5854

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5995)

poly1305/asm/poly1305-armv4.pl: remove unintentional relocation.

Branch to global symbol results in reference to PLT, and when compiling
for THUMB-2 - in a R_ARM_THM_JUMP19 relocation. Some linkers don't
support this relocation (ld.gold), while others can end up truncating
the relocation to fit (ld.bfd).

Convert this branch through PLT into a direct branch that the assembler
can resolve locally.

See https://github.com/android-ndk/ndk/issues/337 for background.

The current workaround is to disable poly1305 optimization assembly,
which is not optimal and can be reverted after this patch:
https://github.com/freedesktop/gstreamer-cerbero/commit/beab607d2b1ff23c41b7e01aa9c64be5e247d1e6

CLA: trivial

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5949)

(cherry picked from commit 5bb1cd2292b388263a0cc05392bb99141212aa53)

Check the return from EVP_PKEY_get0_DH()

Fixes #5934

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5993)

Fix assertion failure in SSL_set_bio()

If SSL_set_bio() is called with a NULL wbio after a failed connection then
this can trigger an assertion failure. This should be valid behaviour and
the assertion is in fact invalid and can simply be removed.

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5966)

(cherry picked from commit bd7775e14a19c326d3720f2345c2ae324409e979)

Update fingerprints.txt

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5987)

(cherry picked from commit 2bdeffefddd8e8a65a51a7b020f8d51a4a3b1602)

Ignore the status_request extension in a resumption handshake

We cannot provide a certificate status on a resumption so we should
ignore this extension in that case.

Fixes #1662

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/5897)

p5_scrypt.c: fix error check of RAND_bytes() call

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/5977)

Revert "Add OPENSSL_VERSION_AT_LEAST"

Fixes #5961

This reverts commit d8adfdcd2e5de23f3e1d1a1d10c2fda6f4a65c4a.

The macros OPENSSL_MAKE_VERSION() and OPENSSL_VERSION_AT_LEAST() contain
errors and don't work as designed. Apart from that, their introduction
should be held back until a decision has been mad about the future
versioning scheme.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5968)

RSA key generation: ensure BN_mod_inverse and BN_mod_exp_mont both get called with BN_FLG_CONSTTIME flag set.

CVE-2018-0737

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>

Custome built dladdr() for AIX.

Implemented a stripped down dladdr()-implementation using AIX' own
loadquery()-function. Following the SGI example in the same code, the
DL_info only has the dli_fname member. As the scope of
dlfcn_pathbyaddr() is the filename, this implementation does not
consider archive members, which can be dlopen()ed in AIX.
Added DATA segment checking to catch ptrgl virtual addresses.

Added test case for DSO_dsobyaddr(), but only for DSO_DLFCN.
Added PIC-flag to aix*-cc build targets.

Signed-off-by: Matthias Kraft <makr@gmx.eu>
Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5626)

test/recipes/test_genrsa.t : don't fail because of size limit changes

There is a test to check that 'genrsa' doesn't accept absurdly low
number of bits.  Apart from that, this test is designed to check the
working functionality of 'openssl genrsa', so instead of having a hard
coded lower limit on the size key, let's figure out what it is.

Partially fixes #5751

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/5754)

Docs for OpenSSL_init_crypto: there is no way to specify another file

The documentation erroneously stated that one can change the default
configuration file name.

Fixes #5939

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5941)

(cherry picked from commit 0320e8e2869fb6cde4579375e65f6d576bbec95e)

openssl rehash: use libcrypto variables for default dir

X509_get_default_cert_dir_env() returns the default environment
variable to check for certificate directories.
X509_get_default_cert_dir() returns the default configured certificate
directory.

Use these instead of hard coding our own values, and thereby be more
integrated with the rest of OpenSSL.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5937)

(cherry picked from commit 362ff3c39703633ac53ba9130c7acc02177bedc5)

openssl rehash: document -compat

Fixes #5902

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5937)

(cherry picked from commit adaf3cfad2319fe5c0d6fe179c726cc85eebf2d3)

Clarify default section in config.pod

This is a minor update which hopefully makes these particular lines
read a little easier.

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5938)

(cherry picked from commit 0652e8a7fe6dd8cbdf4516b221642b10bbfc68fe)

Fix bugs in X509_NAME_ENTRY_set

The wrong "set" field was incremented in the wrong place and would
create a new RDN, not a multi-valued RDN.
RDN inserts would happen after not before.
Prepending an entry to an RDN incorrectly created a new RDN

Anything which built up an X509_NAME could get a messed-up structure,
which would then be "wrong" for anyone using that name.

Thanks to Ingo Schwarze for extensive debugging and the initial
fix (documented in GitHub issue #5870).

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/5882)

(cherry picked from commit bbf27cd58337116c57a1c942153330ff83d5540a)

bio/b_addr.c: resolve HP-UX compiler warnings.

The warning reads "[cast] may cause misaligned access". Even though
this can be application-supplied pointer, misaligned access shouldn't
happen, because structure type is "encoded" into data itself, and
application would customarily pass correctly aligned pointer. But
there is no harm in resolving the warning...

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5894)

(cherry picked from commit 55bd917bc4213bc668f48b87d8c6feb9918fef8f)

Configurations/10-main.conf: further HP-UX cleanups/unifications.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Configurations/10-main.conf: add magic macros to hpux targets.

HP-UX provides sockets symbols with incompatible prototypes under same
names. Additional macros force unambitious symbols with unambitious
prototypes.

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/5862)

Updated to CONTRIBUTING to reflect GitHub, etc.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5889)
(cherry picked from commit 2876872ffe5dd53ec1c446656e924ff463e5d4bf)

Set biom->type in BIO_METH_new

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5812)

Fix an error code to be consistent with master

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5892)

Add test/versions to gitignore

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5881)

Update the genpkey documentation

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5883)

Pick a q size consistent with the digest for DSA param generation

There are two undocumented DSA parameter generation options available in
the genpkey command line app:
dsa_paramgen_md and dsa_paramgen_q_bits.

These can also be accessed via the EVP API but only by using
EVP_PKEY_CTX_ctrl() or EVP_PKEY_CTX_ctrl_str() directly. There are no
helper macros for these options.

dsa_paramgen_q_bits sets the length of q in bits (default 160 bits).
dsa_paramgen_md sets the digest that is used during the parameter
generation (default SHA1). In particular the output length of the digest
used must be equal to or greater than the number of bits in q because of
this code:

            if (!EVP_Digest(seed, qsize, md, NULL, evpmd, NULL))
                goto err;
            if (!EVP_Digest(buf, qsize, buf2, NULL, evpmd, NULL))
                goto err;
            for (i = 0; i < qsize; i++)
                md[i] ^= buf2[i];

            /* step 3 */
            md[0] |= 0x80;
            md[qsize - 1] |= 0x01;
            if (!BN_bin2bn(md, qsize, q))
                goto err;

qsize here is the number of bits in q and evpmd is the digest set via
dsa_paramgen_md. md and buf2 are buffers of length SHA256_DIGEST_LENGTH.
buf2 has been filled with qsize bits of random seed data, and md is
uninitialised.

If the output size of evpmd is less than qsize then the line "md[i] ^=
buf2[i]" will be xoring an uninitialised value and the random seed data
together to form the least significant bits of q (and not using the
output of the digest at all for those bits) - which is probably not what
was intended. The same seed is then used as an input to generating p. If
the uninitialised data is actually all zeros (as seems quite likely)
then the least significant bits of q will exactly match the least
significant bits of the seed.

This problem only occurs if you use these undocumented and difficult to
find options and you set the size of q to be greater than the message
digest output size. This is for parameter generation only not key
generation. This scenario is considered highly unlikely and
therefore the security risk of this is considered negligible.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5883)

Don't crash if an unrecognised digest is used with dsa_paramgen_md

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5883)

Move the loading of the ssl_conf module to libcrypto

The GOST engine needs to be loaded before we initialise libssl. Otherwise
the GOST ciphersuites are not enabled. However the SSL conf module must
be loaded before we initialise libcrypto. Otherwise we will fail to read
the SSL config from a config file properly.

Another problem is that an application may make use of both libcrypto and
libssl. If it performs libcrypto stuff first and OPENSSL_init_crypto()
is called and loads a config file it will fail if that config file has
any libssl stuff in it.

This commit separates out the loading of the SSL conf module from the
interpretation of its contents. The loading piece doesn't know anything
about SSL so this can be moved to libcrypto. The interpretation of what it
means remains in libssl. This means we can load the SSL conf data before
libssl is there and interpret it when it later becomes available.

Fixes #5809

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5879)

fix build error in 32 bit debug build

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/5861)

Don't use CPP in Configurations/unix-Makefile.tmpl

We started using $(CPP) instead of $(CC) -E, with the assumption that
CPP would be predefined.  This is, however, not always true, and
rather depends on the 'make' implementation.  Furthermore, on
platforms where CPP=cpp or something else other than '$(CC) -E',
there's a risk that it won't understand machine specific flags that we
pass to it.  So it turns out that trying to use $(CPP) was a mistake,
and we therefore revert that use back to using $(CC) -E directly.

Fixes #5867

Note: this affects config targets that use Alpha, ARM, IA64, MIPS,
s390x or SPARC assembler modules.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5871)

Fix some errors in the mem leaks docs

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/5795)

Fix a text canonicalisation bug in CMS

Where a CMS detached signature is used with text content the text goes
through a canonicalisation process first prior to signing or verifying a
signature. This process strips trailing space at the end of lines, converts
line terminators to CRLF and removes additional trailing line terminators
at the end of a file. A bug in the canonicalisation process meant that
some characters, such as form-feed, were incorrectly treated as whitespace
and removed. This is contrary to the specification (RFC5485). This fix
could mean that detached text data signed with an earlier version of
OpenSSL 1.1.0 may fail to verify using the fixed version, or text data
signed with a fixed OpenSSL may fail to verify with an earlier version of
OpenSSL 1.1.0. A workaround is to only verify the canonicalised text data
and use the "-binary" flag (for the "cms" command line application) or set
the SMIME_BINARY/PKCS7_BINARY/CMS_BINARY flags (if using CMS_verify()).

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5791)

Change the "offset too large" message to more generic wording

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/5826)

(cherry picked from commit 1518c55a796b058eff01f3cbf177f4b726c01d7c)

Remove an unnecessary cast in the param to BUF_MEM_grow

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/5826)

(cherry picked from commit 21db0e1afdcf17a189ebe49af3506551b108a2f2)

Fix range checks with -offset and -length in asn1parse

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/5826)

(cherry picked from commit 16e1eea6a67c85c9d786f3c4448182b1aca101b8)

Use strtol instead of atoi in asn1parse

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/5826)

(cherry picked from commit 18ada952d1dcea8dfc5008df9f317b3eb47c13fa)

Improve diagnostics for invalid arguments in asn1parse -strparse

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/5826)

(cherry picked from commit b998745a4596c05f673ed8acdcaedcb4c5e208ad)

Fix a crash in the asn1parse command

Thanks to Sem Voigtländer for reporting this issue.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/5826)

(cherry picked from commit 752837e0664e990b5edf6f0b69e1b4612efadce0)

EVP,KDF: Add more error code along some return 0

in methods :
- EVP_PBE_scrypt
- EVP_PKEY_meth_add0
- EVP_PKEY_meth_new
- EVP_PKEY_CTX_dup

Backport of 3484236d8d7afedd3e5c7771bd49d3385340e3bf

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/5803)

Remove import/use of File::Spec::Function

It looks like the usage of these functions were removed in
in commit 0a4edb931b883b9973721ae012e60c028387dd50 ("Unified - adapt
the generation of cpuid, uplink and buildinf to use GENERATE").

This commit removes the import/use of File::Spec::Functions module as it
is no longer needed by crypto/build.info.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5832)

(cherry picked from commit 0e34f37fb1b7474c86ac9a170dfda5226351ecc9)

VMS: Copy DECC inclusion epi- and prologues to internals

Because many of our test programs use internal headers, we need to make
sure they know how, exactly, to mangle the symbols.  So far, we've done
so by specifying it in the affected test programs, but as things change,
that will develop into a goose chase.  Better then to declare once and
for all how symbols belonging in our libraries are meant to be treated,
internally as well as publically.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3259)

(cherry picked from commit f46f69f4092768ec4b911ced45c8cc73cc008739)

Refuse to run test_cipherlist unless shared library matches build

test/cipherlist_test.c is an internal consistency check, and therefore
requires that the shared library it runs against matches what it was
built for.  test/recipes/test_cipherlist.t is made to refuse running
unless library version and build version match.

This adds a helper program test/versions.c, that simply displays the
library and the build version.

Partially fixes #5751

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5753)

.travis.yml: with fast fuzz testing, there is no point avoiding it

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/5788)

Faster fuzz test: teach the fuzz test programs to handle directories

Instead of invoking the fuzz test programs once for every corpora
file, we invoke them once for each directory of corpora files.  This
dramatically reduces the number of program invokations, as well as the
time 90-test_fuzz.t takes to complete.

fuzz/test-corpus.c was enhanced to handle directories as well as
regular files.

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/5788)

Revert "util/dofile.pl: only quote stuff that actually needs quoting"

This wasn't a good solution, too many things depend on the quotes being
there consistently.

This reverts commit 49cd47eaababc8c57871b929080fc1357e2ad7b8.

Fixes #5772

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5773)

(cherry picked from commit 00701e5ea84861b74d9d624f21a6b3fcb12e8acd)

Tolerate a Certificate using a non-supported group on server side

If a server has been configured to use an ECDSA certificate, we should
allow it regardless of whether the server's own supported groups list
includes the certificate's group.

Fixes #2033

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/5607)

Adds multiple checks to avoid buffer over reads

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/5686)

o_time.c: use gmtime_s with MSVC
ts/ts_rsp_sign.c: change to OPENSSL_gmtime.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5720)

(cherry picked from commit 98c03302fb7b855647aa14022f61f5fb272e514a)

Fix typo in OPENSSL_LH_new compat API

CLA: trivial

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3292)
(cherry picked from commit b3c42fc2503a685a9e51427c1a83c8f09487389d)

Don't write out a bad OID

If we don't have OID data for an object then we should fail if we
are asked to encode the ASN.1 for that OID.

Fixes #5723

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5725)

(cherry picked from commit 53c9818e970fc0c22d77e19fda3b3e6f6c9e759d)

Prepare for 1.1.0i-dev

Reviewed-by: Richard Levitte <levitte@openssl.org>

Prepare for 1.1.0h release

Reviewed-by: Richard Levitte <levitte@openssl.org>

Update copyright year

Reviewed-by: Richard Levitte <levitte@openssl.org>

Update CHANGES and NEWS for the new release

Reviewed-by: Richard Levitte <levitte@openssl.org>

pariscid.pl: fix nasty typo in CRYPTO_memcmp.

Comparison was effectively reduced to least significant bits.

CVE-2018-0733

Reviewed-by: Matt Caswell <matt@openssl.org>

Add fuzz corpora file that found the ASN.1 stack depth issue

Reviewed-by: Rich Salz <rsalz@openssl.org>

Limit ASN.1 constructed types recursive definition depth

Constructed types with a recursive definition (such as can be found in
PKCS7) could eventually exceed the stack given malicious input with
excessive recursion. Therefore we limit the stack depth.

CVE-2018-0739

Credit to OSSFuzz for finding this issue.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Copy the produced .dll files to fuzz/ as well (Cygwin & mingw)

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5759)

Previously this x509 command line was working, restore that

openssl x509 -in server.pem -signkey privkey.pem -out server.pem

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5747)

Fix dsaparam -genkey with DER outform

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5744)

(cherry picked from commit 5281435258b5d8201a00b4a9781bb724d99630f0)

Fix ecparam -genkey with point compression or DER outform

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5741)

(cherry picked from commit 4bdc25b07f007d9c383fbad159f81543f2e95965)

Handle partial messages in TLSProxy

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5726)

The default conv_form is uncompressed

Fixes #5711

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5712)

(cherry picked from commit ab0a3914a64d8f1fce22795c02269e1288df52b1)

consttime flag changed

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5170)

used ERR set/pop mark

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5170)

Replaced variable-time GCD with consttime inversion to avoid side-channel attacks on RSA key generation

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5170)

Cleanup the s_time command.

Various code-cleanups.
Use SSL_CTX_set_mode(ctx, SSL_MODE_AUTO_RETRY) insead of handling
SSL_ERROR_WANT_READ everywhere.
Turn off the linger option on connected sockets to avoid failure.
Add BIO_set_conn_mode(conn, BIO_SOCK_NODELAY) to improve thruput.
Continue test even without -cipher option as in 1.0.2.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5698)

Don't wait for dry at the end of a handshake

For DTLS/SCTP we were waiting for a dry event during the call to
tls_finish_handshake(). This function just tidies up various internal
things, and after it completes the handshake is over. I can find no good
reason for waiting for a dry event here, and nothing in RFC6083 suggests
to me that we should need to. More importantly though it seems to be
wrong. It is perfectly possible for a peer to send app data/alerts/new
handshake while we are still cleaning up our handshake. If this happens
then we will never get the dry event and so we cannot continue.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5085)

Check for alerts while waiting for a dry event

At a couple of points in a DTLS/SCTP handshake we need to wait for a dry
event before continuing. However if an alert has been sent by the peer
then we will never receive that dry event and an infinite loop results.

This commit changes things so that we attempt to read a message if we
are waiting for a dry event but haven't got one yet. This should never
succeed, but any alerts will be processed.

Fixes #4763

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5085)

Do not cache sessions with zero sid_ctx_length when SSL_VERIFY_PEER

The sid_ctx is something of a "certificate request context" or a
"session ID context" -- something from the application that gives
extra indication of what sort of thing this session is/was for/from.
Without a sid_ctx, we only know that there is a session that we
issued, but it could have come from a number of things, especially
with an external (shared) session cache.  Accordingly, when resuming,
we will hard-error the handshake when presented with a session with
zero-length sid_ctx and SSL_VERIFY_PEER is set -- we simply have no
information about the peer to verify, so the verification must fail.

In order to prevent these future handshake failures, proactively
decline to add the problematic sessions to the session cache.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5175)

(cherry picked from commit d316cdcf6d8d6934663278145fe0a8191e14a8c5)