not call those functions directly! :c:type:`PyOS_sighandler_t` is a typedef
alias for :c:type:`void (\*)(int)`.
+.. c:function:: wchar_t* Py_DecodeLocale(const char* arg, size_t *size)
+
+ Decode a byte string from the locale encoding with the :ref:`surrogateescape
+ error handler <surrogateescape>`: undecodable bytes are decoded as
+ characters in range U+DC80..U+DCFF. If a byte sequence can be decoded as a
+ surrogate character, escape the bytes using the surrogateescape error
+ handler instead of decoding them.
+
+ Return a pointer to a newly allocated wide character string, use
+ :c:func:`PyMem_RawFree` to free the memory. If size is not ``NULL``, write
+ the number of wide characters excluding the null character into ``*size``
+
+ Return ``NULL`` on decoding error or memory allocation error. If *size* is
+ not ``NULL``, ``*size`` is set to ``(size_t)-1`` on memory error or set to
+ ``(size_t)-2`` on decoding error.
+
+ Decoding errors should never happen, unless there is a bug in the C
+ library.
+
+ Use the :c:func:`Py_EncodeLocale` function to encode the character string
+ back to a byte string.
+
+ .. seealso::
+
+ The :c:func:`PyUnicode_DecodeFSDefaultAndSize` and
+ :c:func:`PyUnicode_DecodeLocaleAndSize` functions.
+
+ .. versionadded:: 3.5
+
+
+.. c:function:: char* Py_EncodeLocale(const wchar_t *text, size_t *error_pos)
+
+ Encode a wide character string to the locale encoding with the
+ :ref:`surrogateescape error handler <surrogateescape>`: surrogate characters
+ in the range U+DC80..U+DCFF are converted to bytes 0x80..0xFF.
+
+ Return a pointer to a newly allocated byte string, use :c:func:`PyMem_Free`
+ to free the memory. Return ``NULL`` on encoding error or memory allocation
+ error
+
+ If error_pos is not ``NULL``, ``*error_pos`` is set to the index of the
+ invalid character on encoding error, or set to ``(size_t)-1`` otherwise.
+
+ Use the :c:func:`Py_DecodeLocale` function to decode the bytes string back
+ to a wide character string.
+
+ .. seealso::
+
+ The :c:func:`PyUnicode_EncodeFSDefault` and
+ :c:func:`PyUnicode_EncodeLocale` functions.
+
+ .. versionadded:: 3.5
+
+
.. _systemfunctions:
System Functions
*errors* is ``NULL``. *str* must end with a null character but
cannot contain embedded null characters.
+ Use :c:func:`PyUnicode_DecodeFSDefaultAndSize` to decode a string from
+ :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at
+ Python startup).
+
.. seealso::
- Use :c:func:`PyUnicode_DecodeFSDefaultAndSize` to decode a string from
- :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at
- Python startup).
+ The :c:func:`Py_DecodeLocale` function.
.. versionadded:: 3.3
*errors* is ``NULL``. Return a :class:`bytes` object. *str* cannot
contain embedded null characters.
+ Use :c:func:`PyUnicode_EncodeFSDefault` to encode a string to
+ :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at
+ Python startup).
+
.. seealso::
- Use :c:func:`PyUnicode_EncodeFSDefault` to encode a string to
- :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at
- Python startup).
+ The :c:func:`Py_EncodeLocale` function.
.. versionadded:: 3.3
If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the
locale encoding.
+ :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the
+ locale encoding and cannot be modified later. If you need to decode a string
+ from the current locale encoding, use
+ :c:func:`PyUnicode_DecodeLocaleAndSize`.
+
.. seealso::
- :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the
- locale encoding and cannot be modified later. If you need to decode a
- string from the current locale encoding, use
- :c:func:`PyUnicode_DecodeLocaleAndSize`.
+ The :c:func:`Py_DecodeLocale` function.
.. versionchanged:: 3.2
Use ``"strict"`` error handler on Windows.
If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the
locale encoding.
+ :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the
+ locale encoding and cannot be modified later. If you need to encode a string
+ to the current locale encoding, use :c:func:`PyUnicode_EncodeLocale`.
+
.. seealso::
- :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the
- locale encoding and cannot be modified later. If you need to encode a
- string to the current locale encoding, use
- :c:func:`PyUnicode_EncodeLocale`.
+ The :c:func:`Py_EncodeLocale` function.
.. versionadded:: 3.2
encodings.
+.. _surrogateescape:
.. _codec-base-classes:
Codec Base Classes
.. versionchanged:: 3.1
On some systems, conversion using the file system encoding may fail. In this
- case, Python uses the ``surrogateescape`` encoding error handler, which means
- that undecodable bytes are replaced by a Unicode character U+DCxx on
- decoding, and these are again translated to the original byte on encoding.
+ case, Python uses the :ref:`surrogateescape encoding error handler
+ <surrogateescape>`, which means that undecodable bytes are replaced by a
+ Unicode character U+DCxx on decoding, and these are again translated to the
+ original byte on encoding.
The file system encoding must guarantee to successfully decode all bytes
PyAPI_FUNC(PyObject *) _Py_device_encoding(int);
-PyAPI_FUNC(wchar_t *) _Py_char2wchar(
+PyAPI_FUNC(wchar_t *) Py_DecodeLocale(
const char *arg,
size_t *size);
-PyAPI_FUNC(char*) _Py_wchar2char(
+PyAPI_FUNC(char*) Py_EncodeLocale(
const wchar_t *text,
size_t *error_pos);
Core and Builtins
-----------------
+- Issue #18395: Rename ``_Py_char2wchar()`` to :c:func:`Py_DecodeLocale`,
+ rename ``_Py_wchar2char()`` to :c:func:`Py_EncodeLocale`, and document
+ these functions.
+
- Issue #20179: Apply Argument Clinic to bytes and bytearray.
Patch by Tal Einat.
}
/* Python/fileutils.c */
-wchar_t *_Py_char2wchar(const char* arg, size_t *size)
+wchar_t *Py_DecodeLocale(const char* arg, size_t *size)
{
wchar_t *w;
__coverity_tainted_data_sink__(arg);
joinpath(prefix, L"Modules/Setup");
if (isfile(prefix)) {
/* Check VPATH to see if argv0_path is in the build directory. */
- vpath = _Py_char2wchar(VPATH, NULL);
+ vpath = Py_DecodeLocale(VPATH, NULL);
if (vpath != NULL) {
wcsncpy(prefix, argv0_path, MAXPATHLEN);
prefix[MAXPATHLEN] = L'\0';
wchar_t *_pythonpath, *_prefix, *_exec_prefix;
wchar_t *lib_python;
- _pythonpath = _Py_char2wchar(PYTHONPATH, NULL);
- _prefix = _Py_char2wchar(PREFIX, NULL);
- _exec_prefix = _Py_char2wchar(EXEC_PREFIX, NULL);
- lib_python = _Py_char2wchar("lib/python" VERSION, NULL);
+ _pythonpath = Py_DecodeLocale(PYTHONPATH, NULL);
+ _prefix = Py_DecodeLocale(PREFIX, NULL);
+ _exec_prefix = Py_DecodeLocale(EXEC_PREFIX, NULL);
+ lib_python = Py_DecodeLocale("lib/python" VERSION, NULL);
if (!_pythonpath || !_prefix || !_exec_prefix || !lib_python) {
Py_FatalError(
}
if (_path) {
- path_buffer = _Py_char2wchar(_path, NULL);
+ path_buffer = Py_DecodeLocale(_path, NULL);
path = path_buffer;
}
** be running the interpreter in the build directory, so we use the
** build-directory-specific logic to find Lib and such.
*/
- wchar_t* wbuf = _Py_char2wchar(modPath, NULL);
+ wchar_t* wbuf = Py_DecodeLocale(modPath, NULL);
if (wbuf == NULL) {
Py_FatalError("Cannot decode framework location");
}
if (_rtpypath && _rtpypath[0] != '\0') {
size_t rtpypath_len;
- rtpypath = _Py_char2wchar(_rtpypath, &rtpypath_len);
+ rtpypath = Py_DecodeLocale(_rtpypath, &rtpypath_len);
if (rtpypath != NULL)
bufsz += rtpypath_len + 1;
}
/* Used by Mac/Tools/pythonw.c to forward
* the argv0 of the stub executable
*/
- wchar_t* wbuf = _Py_char2wchar(pyvenv_launcher, NULL);
+ wchar_t* wbuf = Py_DecodeLocale(pyvenv_launcher, NULL);
if (wbuf == NULL) {
Py_FatalError("Cannot decode __PYVENV_LAUNCHER__");
char *cfilename_buffer;
const char *cfilename;
int err = errno;
- cfilename_buffer = _Py_wchar2char(filename, NULL);
+ cfilename_buffer = Py_EncodeLocale(filename, NULL);
if (cfilename_buffer != NULL)
cfilename = cfilename_buffer;
else
/* "surrogateescape" error handler */
char *str;
- str = _Py_wchar2char(wstr, &error_pos);
+ str = Py_EncodeLocale(wstr, &error_pos);
if (str == NULL) {
if (error_pos == (size_t)-1) {
PyErr_NoMemory();
if (errmsg != NULL) {
size_t errlen;
- wstr = _Py_char2wchar(errmsg, &errlen);
+ wstr = Py_DecodeLocale(errmsg, &errlen);
if (wstr != NULL) {
reason = PyUnicode_FromWideChar(wstr, errlen);
PyMem_RawFree(wstr);
if (surrogateescape) {
/* "surrogateescape" error handler */
- wstr = _Py_char2wchar(str, &wlen);
+ wstr = Py_DecodeLocale(str, &wlen);
if (wstr == NULL) {
if (wlen == (size_t)-1)
PyErr_NoMemory();
error_pos = mbstowcs_errorpos(str, len);
if (errmsg != NULL) {
size_t errlen;
- wstr = _Py_char2wchar(errmsg, &errlen);
+ wstr = Py_DecodeLocale(errmsg, &errlen);
if (wstr != NULL) {
reason = PyUnicode_FromWideChar(wstr, errlen);
PyMem_RawFree(wstr);
setlocale(LC_ALL, "");
for (i = 0; i < argc; i++) {
- argv_copy[i] = _Py_char2wchar(argv[i], NULL);
+ argv_copy[i] = Py_DecodeLocale(argv[i], NULL);
if (!argv_copy[i]) {
PyMem_RawFree(oldloc);
fprintf(stderr, "Fatal Python error: "
Values of force_ascii:
- 1: the workaround is used: _Py_wchar2char() uses
- encode_ascii_surrogateescape() and _Py_char2wchar() uses
+ 1: the workaround is used: Py_EncodeLocale() uses
+ encode_ascii_surrogateescape() and Py_DecodeLocale() uses
decode_ascii_surrogateescape()
- 0: the workaround is not used: _Py_wchar2char() uses wcstombs() and
- _Py_char2wchar() uses mbstowcs()
+ 0: the workaround is not used: Py_EncodeLocale() uses wcstombs() and
+ Py_DecodeLocale() uses mbstowcs()
-1: unknown, need to call check_force_ascii() to get the value
*/
static int force_ascii = -1;
/* Decode a byte string from the locale encoding with the
- surrogateescape error handler (undecodable bytes are decoded as characters
- in range U+DC80..U+DCFF). If a byte sequence can be decoded as a surrogate
+ surrogateescape error handler: undecodable bytes are decoded as characters
+ in range U+DC80..U+DCFF. If a byte sequence can be decoded as a surrogate
character, escape the bytes using the surrogateescape error handler instead
of decoding them.
- Use _Py_wchar2char() to encode the character string back to a byte string.
+ Return a pointer to a newly allocated wide character string, use
+ PyMem_RawFree() to free the memory. If size is not NULL, write the number of
+ wide characters excluding the null character into *size
- Return a pointer to a newly allocated wide character string (use
- PyMem_RawFree() to free the memory) and write the number of written wide
- characters excluding the null character into *size if size is not NULL, or
- NULL on error (decoding or memory allocation error). If size is not NULL,
- *size is set to (size_t)-1 on memory error and (size_t)-2 on decoding
- error.
+ Return NULL on decoding error or memory allocation error. If *size* is not
+ NULL, *size is set to (size_t)-1 on memory error or set to (size_t)-2 on
+ decoding error.
- Conversion errors should never happen, unless there is a bug in the C
- library. */
+ Decoding errors should never happen, unless there is a bug in the C
+ library.
+
+ Use the Py_EncodeLocale() function to encode the character string back to a
+ byte string. */
wchar_t*
-_Py_char2wchar(const char* arg, size_t *size)
+Py_DecodeLocale(const char* arg, size_t *size)
{
#ifdef __APPLE__
wchar_t *wstr;
#endif /* __APPLE__ */
}
-/* Encode a (wide) character string to the locale encoding with the
- surrogateescape error handler (characters in range U+DC80..U+DCFF are
- converted to bytes 0x80..0xFF).
+/* Encode a wide character string to the locale encoding with the
+ surrogateescape error handler: surrogate characters in the range
+ U+DC80..U+DCFF are converted to bytes 0x80..0xFF.
- This function is the reverse of _Py_char2wchar().
+ Return a pointer to a newly allocated byte string, use PyMem_Free() to free
+ the memory. Return NULL on encoding or memory allocation error.
- Return a pointer to a newly allocated byte string (use PyMem_Free() to free
- the memory), or NULL on encoding or memory allocation error.
+ If error_pos is not NULL, *error_pos is set to the index of the invalid
+ character on encoding error, or set to (size_t)-1 otherwise.
- If error_pos is not NULL: *error_pos is the index of the invalid character
- on encoding error, or (size_t)-1 otherwise. */
+ Use the Py_DecodeLocale() function to decode the bytes string back to a wide
+ character string. */
char*
-_Py_wchar2char(const wchar_t *text, size_t *error_pos)
+Py_EncodeLocale(const wchar_t *text, size_t *error_pos)
{
#ifdef __APPLE__
Py_ssize_t len;
{
int err;
char *fname;
- fname = _Py_wchar2char(path, NULL);
+ fname = Py_EncodeLocale(path, NULL);
if (fname == NULL) {
errno = EINVAL;
return -1;
errno = EINVAL;
return NULL;
}
- cpath = _Py_wchar2char(path, NULL);
+ cpath = Py_EncodeLocale(path, NULL);
if (cpath == NULL)
return NULL;
f = fopen(cpath, cmode);
int res;
size_t r1;
- cpath = _Py_wchar2char(path, NULL);
+ cpath = Py_EncodeLocale(path, NULL);
if (cpath == NULL) {
errno = EINVAL;
return -1;
return -1;
}
cbuf[res] = '\0'; /* buf will be null terminated */
- wbuf = _Py_char2wchar(cbuf, &r1);
+ wbuf = Py_DecodeLocale(cbuf, &r1);
if (wbuf == NULL) {
errno = EINVAL;
return -1;
wchar_t *wresolved_path;
char *res;
size_t r;
- cpath = _Py_wchar2char(path, NULL);
+ cpath = Py_EncodeLocale(path, NULL);
if (cpath == NULL) {
errno = EINVAL;
return NULL;
if (res == NULL)
return NULL;
- wresolved_path = _Py_char2wchar(cresolved_path, &r);
+ wresolved_path = Py_DecodeLocale(cresolved_path, &r);
if (wresolved_path == NULL) {
errno = EINVAL;
return NULL;
if (getcwd(fname, Py_ARRAY_LENGTH(fname)) == NULL)
return NULL;
- wname = _Py_char2wchar(fname, &len);
+ wname = Py_DecodeLocale(fname, &len);
if (wname == NULL)
return NULL;
if (size <= len) {
setlocale(LC_ALL, "");
for (i = 0; i < argc; i++) {
- argv_copy[i] = _Py_char2wchar(argv[i], NULL);
+ argv_copy[i] = Py_DecodeLocale(argv[i], NULL);
argv_copy2[i] = argv_copy[i];
if (!argv_copy[i]) {
fprintf(stderr, "Unable to decode the command line argument #%i\n",
projects / python / commitdiff