Handle SSL_shutdown while in init more appropriately #2

Previous commit f73c737c7 attempted to "fix" a problem with the way
SSL_shutdown() behaved whilst in mid-handshake. The original behaviour had
SSL_shutdown() return immediately having taken no action if called mid-
handshake with a return value of 1 (meaning everything was shutdown
successfully). In fact the shutdown has not been successful.

Commit f73c737c7 changed that to send a close_notify anyway and then
return. This seems to be causing some problems for some applications so
perhaps a better (much simpler) approach is revert to the previous
behaviour (no attempt at a shutdown), but return -1 (meaning the shutdown
was not successful).

This also fixes a bug where SSL_shutdown always returns 0 when shutdown
*very* early in the handshake (i.e. we are still using SSLv23_method).

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>

if no comparison function set make sk_sort no op

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit 0ca2e82ab1575f9c4eed259c4d66ca9372a45bf5)

Fix missing ok=0 with locally blacklisted CAs

Also in X509_verify_cert() avoid using "i" not only as a loop
counter, but also as a trust outcome and as an error ordinal.

Finally, make sure that all "goto end" jumps return an error, with
"end" renamed to "err" accordingly.

[ The 1.1.0 version of X509_verify_cert() is major rewrite,
  which addresses these issues in a more systemic way. ]

Reviewed-by: Rich Salz <rsalz@openssl.org>

Restore xmm7 from the correct address on win64

Reviewed-by: Richard Levitte <levitte@openssl.org>
RT: #4288, MR: #1831
(cherry picked from commit df057ea6c8a20e4babc047689507dfafde59ffd6)

Fix pkeyutl/rsautl empty encrypt-input/decrypt-output handling

Also fix option processing in pkeyutl to allow use of (formerly)
"out-of-order" switches that were needless implementation limitations.

RT2018

Reviewed-by: Rich Salz <rsalz@openssl.org>

GH611: s_client help message bug

Reviewed-by: Emilia Käsper <emilia@openssl.org>

GH554: Improve pkeyutl doc

Signed-off-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Dr. Stephen Henson <steve@openssl.org>
(cherry picked from commit 53619f9f40740ae8b256998574758aa191635db8)

RT4129: BUF_new_mem_buf should take const void *

Signed-off-by: Rich Salz <rsalz@akamai.com>
Reviewed-by: Dr. Stephen Henson <steve@openssl.org>
(cherry picked from commit 8ab31975bacb9c907261088937d3aa4102e3af84)

GH102: Add volatile to CRYPTO_memcmp

Can't hurt and seems to prevent problems from some over-aggressive
(LTO?) compilers.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 98ab57644f44d2d83595c2d0f69138a284d6096b)

Backport SHA2 support for capi engine

Reviewed-by: Andy Polyakov <appro@openssl.org>

Add have_precompute_mult tests

Add tests for have_precompute_mult for the optimised curves (nistp224,
nistp256 and nistp521) if present

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 8ce4e7e605577cb5818de068e2c6da60901cddba)

Fix bug in nistp224/256/521 where have_precompute_mult always returns 0

During precomputation if the group given is well known then we memcpy a
well known precomputation. However we go the wrong label in the code and
don't store the data properly. Consequently if we call have_precompute_mult
the data isn't there and we return 0.

RT#3600

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 615614c8862fb89dcf1551a4e113be0789dddf5f)

Add missing return value checks

The function DH_check_pub_key() was missing some return value checks in
some calls to BN functions.

RT#4278

Reviewed-by: Andy Polyakov <appro@openssl.org>
(cherry picked from commit f5a12207eccfd814bde68b880a96910dfa25f164)

Correct value of DH_CHECK_PUBKEY_INVALID

A new return value for DH_check_pub_key was recently added:
DH_CHECK_PUBKEY_INVALID. As this is a flag which can be ORed with other
return values it should have been set to the value 4 not 3.

RT#4278

Reviewed-by: Andy Polyakov <appro@openssl.org>
(cherry picked from commit cb389fe80462e20daba30835a9e86354451bd14f)

Fix CHANGES entry about DSA_generate_parameters_ex

Reviewed-by: Viktor Dukhovni <openssl-users@dukhovni.org>

Correct number of arguments in BIO_get_conn_int_port macro

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit 41a28cb2944a4e1c9d13889757a3bd9f72abeca1)

Prepare for 1.0.2g-dev

Reviewed-by: Richard Levitte <levitte@openssl.org>

Prepare for 1.0.2f release

Reviewed-by: Richard Levitte <levitte@openssl.org>

Update CHANGES and NEWS for release

Add details about the latest issues into CHANGES and NEWS ready for the
next release.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add a test for small subgroup attacks on DH/DHE

Following on from the previous commit, add a test to ensure that
DH_compute_key correctly fails if passed a bad y such that:

y^q (mod p) != 1

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>

Always generate DH keys for ephemeral DH cipher suites

Modified version of the commit ffaef3f15 in the master branch by Stephen
Henson. This makes the SSL_OP_SINGLE_DH_USE option a no-op and always
generates a new DH key for every handshake regardless.

CVE-2016-0701 (fix part 2 or 2)

Issue reported by Antonio Sanso

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>

Prevent small subgroup attacks on DH/DHE

Historically OpenSSL only ever generated DH parameters based on "safe"
primes. More recently (in version 1.0.2) support was provided for
generating X9.42 style parameter files such as those required for RFC
5114 support. The primes used in such files may not be "safe". Where an
application is using DH configured with parameters based on primes that
are not "safe" then an attacker could use this fact to find a peer's
private DH exponent. This attack requires that the attacker complete
multiple handshakes in which the peer uses the same DH exponent.

A simple mitigation is to ensure that y^q (mod p) == 1

CVE-2016-0701 (fix part 1 of 2)

Issue reported by Antonio Sanso.

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>

Better SSLv2 cipher-suite enforcement

Based on patch by: Nimrod Aviram <nimrod.aviram@gmail.com>

CVE-2015-3197

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>

pkeyutl: allow peerkey for EC_DERIVE to reside on a hardware token (public key for now)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>

RT3863 ECC: Add missing NULL check. Set a flag

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(cherry picked from commit dd67493c344d9d98413d2ee7fd2b6fa9411d975c)

GH issue 572: Error in help message

Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Viktor Dukhovni <viktor@openssl.org>

Handle SSL_shutdown while in init more appropriately

Calling SSL_shutdown while in init previously gave a "1" response, meaning
everything was successfully closed down (even though it wasn't). Better is
to send our close_notify, but fail when trying to receive one.

The problem with doing a shutdown while in the middle of a handshake is
that once our close_notify is sent we shouldn't really do anything else
(including process handshake/CCS messages) until we've received a
close_notify back from the peer. However the peer might send a CCS before
acting on our close_notify - so we won't be able to read it because we're
not acting on CCS messages!

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>

Fix BSD -rpath parameter

For BSD systems, Configure adds a shared_ldflags including a reference
to the Makefile variable LIBRPATH, but since it must be passed down to
Makefile.shared, care must be taken so the value of LIBRPATH doesn't
get expanded too early, or it ends up giving an empty string.

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>
(cherry picked from commit c64879d3f3cc4c7f1c436a9fe3bd109847a23629)

Validate ClientHello session_id field length and send alert on failure

RT#4080

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>

Update license year range to 2016

Signed-off-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 02f70372470b4dd3b21443bb615292175f5d2c88)

Don't use "grep -q", "-q" is not POSIX, and fails on Solaris.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>

Empty SNI names are not valid

While empty inputs to SSL_set1_host() clear the reference identifier
list.

(cherry-picked from 1.1.0-dev)

Reviewed-by: Rich Salz <rsalz@openssl.org>

NGX-2040 - fix wildcard match on punycode/IDNA DNS names

    - bugfix: should not treat '--' as invalid domain substring.
    - '-' should not be the first letter of a domain

Signed-off-by: Viktor Dukhovni <viktor@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>

To avoid possible time_t overflow use X509_time_adj_ex()

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>
(cherry picked from commit 9aa00b187a65b1f30789d6274ec31ea86efe7973)

Conflicts:
apps/x509.c

Root cause discovered and fixed, this fix became unnecessary

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>

Changed help output

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>

Pass engine=NULL to EVP_PKEY_CTX_new(), unless "-engine_impl" was given

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>

Remove unnecessary debugging fprintf

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>

Fixed crash (SIGSEGV) when freeing of ex_data stumbles upon a NULL-pointer.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>

Fixed a bug preventing pkeyutl from accessing keys directly on the token via engine_pkcs11

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>

Wrong definition of the macro SSL_set1_sigalgs in ssl.h

Error in the definition of the macro SSL_set1_sigalgs(ctx, slist,
slistlen): the third parameter 'slistlen' not used in the substitution
code; used 'clistlen' instead. As a result of this, compilation error
occurs when any application uses this macro.

Signed-off-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Kurt Roeckx <kurt@openssl.org>

Change minimum DH size from 768 to 1024

Reviewed-by: Viktor Dukhovni <openssl-users@dukhovni.org>

File is about s_time, not s_client

Reviewed-by: Dr. Stephen Henson <steve@openssl.org>
(cherry picked from commit 1918e01c9f915e2eba31a5e2f86f0a5daa4fafb6)

Fix X509_STORE_CTX_cleanup()

Reviewed-by: Dr. Stephen Henson <steve@openssl.org>

RT4202: Update rt URL's.

Reviewed-by: Matt Caswell <matt@openssl.org>
(cherry picked from commit 41977c53cd04f52b2b5e56d31ace782577620ac3)

Add some documentation for the OCSP callback functions

Describe the usage of the OCSP callback functions on both the client and
the server side.

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>

Ensure we don't call the OCSP callback if resuming a session

It makes no sense to call the OCSP status callback if we are resuming a
session because no certificates will be sent.

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>

Fix error when server does not send CertificateStatus message

If a server sends the status_request extension then it may choose
to send the CertificateStatus message. However this is optional.
We were treating it as mandatory and the connection was failing.

Thanks to BoringSSL for reporting this issue.

RT#4120

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>

The functions take a SSL *, not a SSL_CTX *

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Rich Salz <rsalz@openssl.org>
RT: #4192, MR: #1533
(cherry picked from commit d938e8dfee16e6bb5427eac7bda32337634ce130)

Fix memory leak in DSA redo case.

Found by clang scan-build.

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Richard Levitte <levitte@openssl.org>
RT: #4184, MR: #1496
(cherry picked from commit 679d87515d23ca31491effdc264edc81c695a72a)

Don't check RSA_FLAG_SIGN_VER.

Reviewed-by: Richard Levitte <levitte@openssl.org>

BIO_s_datagram() ctrl doesn't support SEEK/TELL, so don't pretend it does

Reviewed-by: Rich Salz <rsalz@openssl.org>

Correct or add comments indicating what controls belong to what

Reviewed-by: Rich Salz <rsalz@openssl.org>

Document how BIO_get_conn_ip and BIO_get_conn_int_port actually work

No dummy arguments.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Have BIO_get_conn_int_port use BIO_ctrl instead BIO_int_ctrl

BIO_int_ctrl isn't made for the purpose BIO_get_conn_int_port used it
for.

This also changes BIO_C_GET_CONNECT to actually return the port
instead of assigning it to a pointer that was never returned back to
the caller.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Fix more URLs mangled by reformat

Fix some more URLs mangled by indent in the reformat. These ones don't exist
in master so we have a separate commit. Based on a patch supplied by Arnaud
Lacombe <al@aerilon.ca>

Reviewed-by: Richard Levitte <levitte@openssl.org>

Fix URLs mangled by reformat

Some URLs in the source code ended up getting mangled by indent. This fixes
it. Based on a patch supplied by Arnaud Lacombe <al@aerilon.ca>

Reviewed-by: Richard Levitte <levitte@openssl.org>

Remove the "eay" c-file-style indicators

Since we don't use the eay style any more, there's no point tryint to
tell emacs to use it.

Reviewed-by: Ben Laurie <ben@openssl.org>

Provide better "make depend" warning.

Reviewed-by: Matt Caswell <matt@openssl.org>
(cherry picked from commit 2e31ef0366d368ac8cf7f5ecc9052bff27337799)

Fix a ** 0 mod 1 = 0 for real this time.

Commit 2b0180c37fa6ffc48ee40caa831ca398b828e680 attempted to do this but
only hit one of many BN_mod_exp codepaths. Fix remaining variants and add
a test for each method.

Thanks to Hanno Boeck for reporting this issue.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Dr. Stephen Henson <steve@openssl.org>
(cherry picked from commit d911097d7c93e4cfeab624b34d73fe51da158b69)

Fix tarball production to keep test/bctest and util/pod2mantest

Reviewed-by: Matt Caswell <matt@openssl.org>

x86_64 assembly pack: tune clang version detection even further.

RT#4171

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
(cherry picked from commit b9749432346f69b29d82070041e71b237d718ce7)

ec/ecp_nistz256_table.c: fix potential misalignment problem with Sun C.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Ensure |rwstate| is set correctly on BIO_flush

A BIO_flush call in the DTLS code was not correctly setting the |rwstate|
variable to SSL_WRITING. This means that SSL_get_error() will not return
SSL_ERROR_WANT_WRITE in the event of an IO retry.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 67f60be8c9ae5ff3129fcd6238baf124385a41d8)

Fix DTLS handshake fragment retries

If using DTLS and NBIO then if a second or subsequent handshake message
fragment hits a retry, then the retry attempt uses the wrong fragment
offset value. This commit restores the fragment offset from the last
attempt.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 2ad226e88bee97847496e542d63c67997d5beda6)

Add a return value check

If the call to OBJ_find_sigid_by_algs fails to find the relevant NID then
we should set the NID to NID_undef.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 330dcb09b2df7e1e6d1d3d14a5df7269aebd9a68)

Don't use applink for static builds.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Refer to website for acknowledgements.

Reviewed-by: Steve Marquess <marquess@openssl.com>
(cherry picked from commit ab29c82a55f3583a490733dd521ea6c486e8e2fb)

Not all 'find's know -xtype, use -type instead

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit 3cd7aef34d0d414d27ab00abadb99265a2cffde9)

Adapt the OS X build to use the OS X tar

As part of this, move release creation to a script to be called from
.travis.yml.  That makes it much easier to test outside of travis.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit 382af61f6213e975b4c2a50fd8b9fedd23d86ab5)

Make it possible to affect the way dists are made

Introducing DISTTARVARS to propagate changed variables down to the
tar-making target.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit 4d3c30a1799bf7b4dc7223b84417c4de992a6b9c)

Cleanup the EVP_MD_CTX before exit rather than after

Reviewed-by: Matt Caswell <matt@openssl.org>

Change tar owner and group to just 0

It seems like some tar versions don't like the name:id form for
--owner and --group.  The closest known anonymous user being 0 (root),
that seems to be the most appropriate user/group to assign ownership
to.  It matters very little when unpacking either way.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit b91dd150d2b9b5ddca37722e7f52ea59ba7f80da)

Do not add symlinks in the source release

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit 451a5bdf0386d7acf091c3e3b39107e5ed8be25d)

In travis, build from a "source release" rather than from the build tree

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit 475fc3d8729190fd12b4ff23d6ec488439fb78f9)

Small changes to creating dists

Make TARFILE include ../ instead of having that hard coded all over the place.
When transforming file names in TAR_COMMAND, use $(NAME) instead of openssl-$(VERSION)

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit 4a544810f08539f1549eea9be36bd878c67c8e26)

Prepare for 1.0.2f-dev

Reviewed-by: Richard Levitte <levitte@openssl.org>

Prepare for 1.0.2e release

Reviewed-by: Richard Levitte <levitte@openssl.org>

make update

Reviewed-by: Richard Levitte <levitte@openssl.org>

Update CHANGES and NEWS

Update the CHANGES and NEWS files for the new release.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Add test for CVE-2015-3194

Reviewed-by: Richard Levitte <levitte@openssl.org>

Add PSS parameter check.

Avoid seg fault by checking mgf1 parameter is not NULL. This can be
triggered during certificate verification so could be a DoS attack
against a client or a server enabling client authentication.

Thanks to Loïc Jonas Etienne (Qnective AG) for discovering this bug.

CVE-2015-3194

Reviewed-by: Richard Levitte <levitte@openssl.org>

bn/asm/x86_64-mont5.pl: fix carry propagating bug (CVE-2015-3193).

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit e7c078db57908cbf16074c68034977565ffaf107)

Fix leak with ASN.1 combine.

When parsing a combined structure pass a flag to the decode routine
so on error a pointer to the parent structure is not zeroed as
this will leak any additional components in the parent.

This can leak memory in any application parsing PKCS#7 or CMS structures.

CVE-2015-3195.

Thanks to Adam Langley (Google/BoringSSL) for discovering this bug using
libFuzzer.

PR#4131

Reviewed-by: Richard Levitte <levitte@openssl.org>

Add cleanup of *.s

Reviewed-by: Matt Caswell <matt@openssl.org>

perlasm/ppc-xlate.pl: comply with ABIs that specify vrsave as reserved.

RT#4162

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit b5516cfbd65de9331d827012fc5bdace1953613e)

_BSD_SOURCE is deprecated, use _DEFAULT_SOURCE instead

The feature_test_macros(7) manual tells us that _BSD_SOURCE is
deprecated since glibc 2.20 and that the compiler will warn about it
being used, unless _DEFAULT_SOURCE is defined as well.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit f9fd35248c9a3b1125d9ab82ffb19d62e86533ac)

crypto/sparcv9cap.c: add SIGILL-free feature detection for Solaris.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit 2238e0e45d03ec9ba15e9547f7739cc6987a8f57)

Remove cookie validation return value trick

In the DTLS ClientHello processing the return value is stored in |ret| which
by default is -1. |ret| is only updated to a positive value once we are past
all points where we could hit an error. We wish to return 1 on success or 2
on success *and* we have validated the DTLS cookie. Previously on successful
validation of the cookie we were setting |ret| to -2, and then once we were
past all error points we set |ret = -ret|. This is non-obvious behaviour and
could be error prone. This commit tries to make this a bit more intuitive.

Reviewed-by: Andy Polyakov <appro@openssl.org>

x86_64 assembly pack: tune clang version detection.

RT#4142

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 76eba0d94bb418325be6409b272eac5e2bd4a0a9)

Resolved conflicts:
crypto/bn/asm/x86_64-mont.pl
crypto/bn/asm/x86_64-mont5.pl

Reviewed-by: Kurt Roeckx <kurt@openssl.org>

Use both sun and __sun

Reviewed-by: Andy Polyakov <appro@openssl.org>

mark openssl configuration as loaded at end of OPENSSL_config

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Rich Salz <rsalz@akamai.com>
GH: #466
(cherry picked from commit 434b58457c8cca4d09f77a1774d6d2c446604bae)

ssl3_free(): Return if it wasn't created

If somewhere in SSL_new() there is a memory allocation failure, ssl3_free() can
get called with s->s3 still being NULL.

Patch also provided by Willy Tarreau <wtarreau@haproxy.com>

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Viktor Dukhovni <openssl-users@dukhovni.org>
(cherry picked from commit 3e7bd2ce0b16f8611298175d6dc7cb35ee06ea6d)

Set reference count earlier

Backport of 0e04674e964b905e67e3d215bcf888932c92765f

Reviewed-by: Steve Henson <steve@openssl.org>
RT #4047, #4110, MR #1356

Fix uninitialised p error.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit 63eb10a07ee29a312e50a227f5b3a290b1ad22b4)

Limit depth of ASN1 parse printing.

Thanks to Guido Vranken <guidovranken@gmail.com> for reporting this issue.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(cherry picked from commit 158e5207a794603f5d64ffa95e0247c7808ab445)

Conflicts:
crypto/asn1/asn1_par.c

Makefile.org: add LC_ALL=C to unify error [and other] messages.

RT#4138

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 193ed2d74b1a643b5f32bd46bf50a858fcdd2da4)

Use defined(__sun) instead of defined(sun)

Strict ISO confirming C compilers only define __sun

Reviewed-by: Viktor Dukhovni <openssl-users@dukhovni.org>
RT #4144, MR #1353

(cherry picked from commit 3d32218812e87221344f2985512e42e4aaa88745)

Add initial AppVeyor configuration

Original patch by Frank Morgner.

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Rich Salz <rsalz@akamai.com>
GH: #456
(cherry picked from commit 68db80e2d1accdd4c4a6b4763559c6cfe9663820)

Good hygiene with size_t output argument.

Though the callers check the function return value and ignore the
size_t output argument on failure, it is still often not ideal to
store -1 in a size_t on error.  That might signal an unduly large
buffer.  Instead set the size_t to 0, to indicate no space.

Reviewed-by: Richard Levitte <levitte@openssl.org>