Use more flexible method of determining output length, by setting &outlen

Use more flexible method of determining output length, by setting &outlen
value of the passed output buffer is NULL.

The old method of using EVP_PKEY_size(pkey) isn't flexible enough to cover all
cases where the output length may depend on the operation or the parameters
associated with it.