[pull] main from python:main

Doc/library/contextlib.rst

@@ -327,10 +327,10 @@ Functions and classes provided:
 .. function:: redirect_stdout(new_target)
 
    Context manager for temporarily redirecting :data:`sys.stdout` to
-   another file or file-like object.
+   another :term:`file object`.
 
    This tool adds flexibility to existing functions or classes whose output
-   is hardwired to stdout.
+   is hardwired to :data:`sys.stdout`.
 
    For example, the output of :func:`help` normally is sent to *sys.stdout*.
    You can capture that output in a string by redirecting the output to an
@@ -366,8 +366,8 @@ Functions and classes provided:
 
 .. function:: redirect_stderr(new_target)
 
-   Similar to :func:`~contextlib.redirect_stdout` but redirecting
-   :data:`sys.stderr` to another file or file-like object.
+   Similar to :func:`~contextlib.redirect_stdout` but redirecting the global
+   :data:`sys.stderr` to another :term:`file object`.
 
    This context manager is :ref:`reentrant <reentrant-cms>`.
 

Doc/library/subprocess.rst

@@ -649,7 +649,7 @@ functions.
 
       If specified, *env* must provide any variables required for the program to
       execute.  On Windows, in order to run a `side-by-side assembly`_ the
-      specified *env* **must** include a valid :envvar:`SystemRoot`.
+      specified *env* **must** include a valid ``%SystemRoot%``.
 
    .. _side-by-side assembly: https://en.wikipedia.org/wiki/Side-by-Side_Assembly
 
@@ -1473,7 +1473,7 @@ handling consistency are valid for these functions.
 
    Return ``(exitcode, output)`` of executing *cmd* in a shell.
 
-   Execute the string *cmd* in a shell with :meth:`Popen.check_output` and
+   Execute the string *cmd* in a shell with :func:`check_output` and
    return a 2-tuple ``(exitcode, output)``.
    *encoding* and *errors* are used to decode output;
    see the notes on :ref:`frequently-used-arguments` for more details.

Doc/reference/lexical_analysis.rst

@@ -10,12 +10,76 @@ Lexical analysis
 A Python program is read by a *parser*.  Input to the parser is a stream of
 :term:`tokens <token>`, generated by the *lexical analyzer* (also known as
 the *tokenizer*).
-This chapter describes how the lexical analyzer breaks a file into tokens.
+This chapter describes how the lexical analyzer produces these tokens.
 
-Python reads program text as Unicode code points; the encoding of a source file
-can be given by an encoding declaration and defaults to UTF-8, see :pep:`3120`
-for details.  If the source file cannot be decoded, a :exc:`SyntaxError` is
-raised.
+The lexical analyzer determines the program text's :ref:`encoding <encodings>`
+(UTF-8 by default), and decodes the text into
+:ref:`source characters <lexical-source-character>`.
+If the text cannot be decoded, a :exc:`SyntaxError` is raised.
+
+Next, the lexical analyzer uses the source characters to generate a stream of tokens.
+The type of a generated token generally depends on the next source character to
+be processed. Similarly, other special behavior of the analyzer depends on
+the first source character that hasn't yet been processed.
+The following table gives a quick summary of these source characters,
+with links to sections that contain more information.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Character
+     - Next token (or other relevant documentation)
+
+   * - * space
+       * tab
+       * formfeed
+     - * :ref:`Whitespace <whitespace>`
+
+   * - * CR, LF
+     - * :ref:`New line <line-structure>`
+       * :ref:`Indentation <indentation>`
+
+   * - * backslash (``\``)
+     - * :ref:`Explicit line joining <explicit-joining>`
+       * (Also significant in :ref:`string escape sequences <escape-sequences>`)
+
+   * - * hash (``#``)
+     - * :ref:`Comment <comments>`
+
+   * - * quote (``'``, ``"``)
+     - * :ref:`String literal <strings>`
+
+   * - * ASCII letter (``a``-``z``, ``A``-``Z``)
+       * non-ASCII character
+     - * :ref:`Name <identifiers>`
+       * Prefixed :ref:`string or bytes literal <strings>`
+
+   * - * underscore (``_``)
+     - * :ref:`Name <identifiers>`
+       * (Can also be part of :ref:`numeric literals <numbers>`)
+
+   * - * number (``0``-``9``)
+     - * :ref:`Numeric literal <numbers>`
+
+   * - * dot (``.``)
+     - * :ref:`Numeric literal <numbers>`
+       * :ref:`Operator <operators>`
+
+   * - * question mark (``?``)
+       * dollar (``$``)
+       *
+         .. (the following uses zero-width space characters to render
+         .. a literal backquote)
+
+         backquote (``​`​``)
+       * control character
+     - * Error (outside string literals and comments)
+
+   * - * other printing character
+     - * :ref:`Operator or delimiter <operators>`
+
+   * - * end of file
+     - * :ref:`End marker <endmarker-token>`
 
 
 .. _line-structure:
@@ -120,6 +184,8 @@ If an encoding is declared, the encoding name must be recognized by Python
 encoding is used for all lexical analysis, including string literals, comments
 and identifiers.
 
+.. _lexical-source-character:
+
 All lexical analysis, including string literals, comments
 and identifiers, works on Unicode text decoded using the source encoding.
 Any Unicode code point, except the NUL control character, can appear in

Doc/tools/.nitignore

@@ -35,7 +35,6 @@ Doc/library/smtplib.rst
 Doc/library/socket.rst
 Doc/library/ssl.rst
 Doc/library/stdtypes.rst
-Doc/library/subprocess.rst
 Doc/library/termios.rst
 Doc/library/test.rst
 Doc/library/tkinter.rst

Doc/tools/templates/download.html

@@ -42,22 +42,22 @@ <h1>{% trans %}Download Python {{ dl_version }} documentation{% endtrans %}</h1>
   </tr>
   <tr>
     <td>{% trans %}HTML{% endtrans %}</td>
-    <td>{% trans download_size="13" %}<a href="{{ dl_base }}/python-{{ dl_version }}-docs-html.zip">Download</a> (ca. {{ download_size }} MiB){% endtrans %}</td>
-    <td>{% trans download_size="8" %}<a href="{{ dl_base }}/python-{{ dl_version }}-docs-html.tar.bz2">Download</a> (ca. {{ download_size }} MiB){% endtrans %}</td>
+    <td>{% trans %}<a href="{{ dl_base }}/python-{{ dl_version }}-docs-html.zip">Download</a>{% endtrans %}</td>
+    <td>{% trans %}<a href="{{ dl_base }}/python-{{ dl_version }}-docs-html.tar.bz2">Download</a>{% endtrans %}</td>
   </tr>
   <tr>
     <td>{% trans %}Plain text{% endtrans %}</td>
-    <td>{% trans download_size="4" %}<a href="{{ dl_base }}/python-{{ dl_version }}-docs-text.zip">Download</a> (ca. {{ download_size }} MiB){% endtrans %}</td>
-    <td>{% trans download_size="3" %}<a href="{{ dl_base }}/python-{{ dl_version }}-docs-text.tar.bz2">Download</a> (ca. {{ download_size }} MiB){% endtrans %}</td>
+    <td>{% trans %}<a href="{{ dl_base }}/python-{{ dl_version }}-docs-text.zip">Download</a>{% endtrans %}</td>
+    <td>{% trans %}<a href="{{ dl_base }}/python-{{ dl_version }}-docs-text.tar.bz2">Download</a>{% endtrans %}</td>
   </tr>
   <tr>
     <td>{% trans %}Texinfo{% endtrans %}</td>
-    <td>{% trans download_size="9" %}<a href="{{ dl_base }}/python-{{ dl_version }}-docs-texinfo.zip">Download</a> (ca. {{ download_size }} MiB){% endtrans %}</td>
-    <td>{% trans download_size="7" %}<a href="{{ dl_base }}/python-{{ dl_version }}-docs-texinfo.tar.bz2">Download</a> (ca. {{ download_size }} MiB){% endtrans %}</td>
+    <td>{% trans %}<a href="{{ dl_base }}/python-{{ dl_version }}-docs-texinfo.zip">Download</a>{% endtrans %}</td>
+    <td>{% trans %}<a href="{{ dl_base }}/python-{{ dl_version }}-docs-texinfo.tar.bz2">Download</a>{% endtrans %}</td>
  </tr>
  <tr>
     <td>{% trans %}EPUB{% endtrans %}</td>
-    <td>{% trans download_size="6" %}<a href="{{ dl_base }}/python-{{ dl_version }}-docs.epub">Download</a> (ca. {{ download_size }} MiB){% endtrans %}</td>
+    <td>{% trans %}<a href="{{ dl_base }}/python-{{ dl_version }}-docs.epub">Download</a>{% endtrans %}</td>
     <td></td>
   </tr>
 </table>
@@ -71,6 +71,9 @@ <h1>{% trans %}Download Python {{ dl_version }} documentation{% endtrans %}</h1>
 and run <code>make dist-pdf</code> in the <code>Doc/</code> directory of a copy of the CPython repository.
 {% endtrans %}</p>
 
+<p>{% trans %}
+See the <a href="https://docs.python.org/{{ version }}/archives/">directory listing</a>
+for file sizes.{% endtrans %}</p>
 
 <h2>{% trans %}Unpacking{% endtrans %}</h2>
 
@@ -83,9 +86,8 @@ <h2>{% trans %}Unpacking{% endtrans %}</h2>
 <p>{% trans %}Windows users can use the ZIP archives since those are customary on that
 platform. These are created on Unix using the Info-ZIP zip program.{% endtrans %}</p>
 
-
 <h2>{% trans %}Problems{% endtrans %}</h2>
-
-<p>{% trans %}If you have comments or suggestions for the Python documentation, please send
-email to <a href="mailto:docs@python.org">docs@python.org</a>.{% endtrans %}</p>
+{% set bugs = pathto('bugs') %}
+<p>{% trans bugs = bugs %}<a href="{{ bugs }}">Open an issue</a>
+if you have comments or suggestions for the Python documentation.{% endtrans %}</p>
 {% endblock %}

Include/internal/pycore_unicodeobject.h

@@ -11,6 +11,44 @@ extern "C" {
 #include "pycore_fileutils.h"     // _Py_error_handler
 #include "pycore_ucnhash.h"       // _PyUnicode_Name_CAPI
 
+// Maximum code point of Unicode 6.0: 0x10ffff (1,114,111).
+#define _Py_MAX_UNICODE 0x10ffff
+
+
+static inline void
+_PyUnicode_Fill(int kind, void *data, Py_UCS4 value,
+                Py_ssize_t start, Py_ssize_t length)
+{
+    assert(0 <= start);
+    switch (kind) {
+    case PyUnicode_1BYTE_KIND: {
+        assert(value <= 0xff);
+        Py_UCS1 ch = (unsigned char)value;
+        Py_UCS1 *to = (Py_UCS1 *)data + start;
+        memset(to, ch, length);
+        break;
+    }
+    case PyUnicode_2BYTE_KIND: {
+        assert(value <= 0xffff);
+        Py_UCS2 ch = (Py_UCS2)value;
+        Py_UCS2 *to = (Py_UCS2 *)data + start;
+        const Py_UCS2 *end = to + length;
+        for (; to < end; ++to) *to = ch;
+        break;
+    }
+    case PyUnicode_4BYTE_KIND: {
+        assert(value <= _Py_MAX_UNICODE);
+        Py_UCS4 ch = value;
+        Py_UCS4 * to = (Py_UCS4 *)data + start;
+        const Py_UCS4 *end = to + length;
+        for (; to < end; ++to) *to = ch;
+        break;
+    }
+    default: Py_UNREACHABLE();
+    }
+}
+
+
 /* --- Characters Type APIs ----------------------------------------------- */
 
 extern int _PyUnicode_IsXidStart(Py_UCS4 ch);
@@ -240,21 +278,6 @@ extern PyObject* _PyUnicode_XStrip(
     );
 
 
-/* Using explicit passed-in values, insert the thousands grouping
-   into the string pointed to by buffer.  For the argument descriptions,
-   see Objects/stringlib/localeutil.h */
-extern Py_ssize_t _PyUnicode_InsertThousandsGrouping(
-    _PyUnicodeWriter *writer,
-    Py_ssize_t n_buffer,
-    PyObject *digits,
-    Py_ssize_t d_pos,
-    Py_ssize_t n_digits,
-    Py_ssize_t min_width,
-    const char *grouping,
-    PyObject *thousands_sep,
-    Py_UCS4 *maxchar,
-    int forward);
-
 /* Dedent a string.
    Behaviour is expected to be an exact match of `textwrap.dedent`.
    Return a new reference on success, NULL with exception set on error.

Lib/argparse.py

@@ -280,7 +280,7 @@ def add_argument(self, action):
         if action.help is not SUPPRESS:
 
             # find all invocations
-            get_invocation = self._format_action_invocation
+            get_invocation = lambda x: self._decolor(self._format_action_invocation(x))
             invocation_lengths = [len(get_invocation(action)) + self._current_indent]
             for subaction in self._iter_indented_subactions(action):
                 invocation_lengths.append(len(get_invocation(subaction)) + self._current_indent)

Lib/test/test_argparse.py

@@ -7311,11 +7311,11 @@ def custom_formatter(prog):
             {heading}usage: {reset}{prog}PROG{reset} [{short}-h{reset}] [{short}+f {label}FOO{reset}] {pos}spam{reset}
 
             {heading}positional arguments:{reset}
-                 {pos_b}spam{reset}               spam help
+                 {pos_b}spam{reset}           spam help
 
             {heading}options:{reset}
-                 {short_b}-h{reset}, {long_b}--help{reset}         show this help message and exit
-                 {short_b}+f{reset}, {long_b}++foo{reset} {label_b}FOO{reset}      foo help
+                 {short_b}-h{reset}, {long_b}--help{reset}     show this help message and exit
+                 {short_b}+f{reset}, {long_b}++foo{reset} {label_b}FOO{reset}  foo help
         '''))
 
     def test_custom_formatter_class(self):
@@ -7348,11 +7348,11 @@ def __init__(self, prog):
             {heading}usage: {reset}{prog}PROG{reset} [{short}-h{reset}] [{short}+f {label}FOO{reset}] {pos}spam{reset}
 
             {heading}positional arguments:{reset}
-                 {pos_b}spam{reset}               spam help
+                 {pos_b}spam{reset}           spam help
 
             {heading}options:{reset}
-                 {short_b}-h{reset}, {long_b}--help{reset}         show this help message and exit
-                 {short_b}+f{reset}, {long_b}++foo{reset} {label_b}FOO{reset}      foo help
+                 {short_b}-h{reset}, {long_b}--help{reset}     show this help message and exit
+                 {short_b}+f{reset}, {long_b}++foo{reset} {label_b}FOO{reset}  foo help
         '''))
 
 

Lib/test/test_clinic.py

@@ -2980,6 +2980,8 @@ def test_cli_converters(self):
             "uint64",
             "uint8",
             "unicode",
+            "unicode_fs_decoded",
+            "unicode_fs_encoded",
             "unsigned_char",
             "unsigned_int",
             "unsigned_long",

Lib/test/test_compile.py

@@ -651,6 +651,21 @@ def test_compile_filename(self):
                 compile('pass', filename, 'exec')
         self.assertRaises(TypeError, compile, 'pass', list(b'file.py'), 'exec')
 
+    def test_compile_filename_refleak(self):
+        # Regression tests for reference leak in PyUnicode_FSDecoder.
+        # See https://github.com/python/cpython/issues/139748.
+        mortal_str = 'this is a mortal string'
+        # check error path when 'mode' AC conversion failed
+        self.assertRaises(TypeError, compile, b'', mortal_str, mode=1234)
+        # check error path when 'optimize' AC conversion failed
+        self.assertRaises(OverflowError, compile, b'', mortal_str,
+                          'exec', optimize=1 << 1000)
+        # check error path when 'dont_inherit' AC conversion failed
+        class EvilBool:
+            def __bool__(self): raise ValueError
+        self.assertRaises(ValueError, compile, b'', mortal_str,
+                          'exec', dont_inherit=EvilBool())
+
     @support.cpython_only
     def test_same_filename_used(self):
         s = """def f(): pass\ndef g(): pass"""

Lib/test/test_symtable.py

@@ -579,6 +579,13 @@ def test_nested_genexpr(self):
         self.assertEqual(sorted(st.get_identifiers()), [".0", "y"])
         self.assertEqual(st.get_children(), [])
 
+    def test__symtable_refleak(self):
+        # Regression test for reference leak in PyUnicode_FSDecoder.
+        # See https://github.com/python/cpython/issues/139748.
+        mortal_str = 'this is a mortal string'
+        # check error path when 'compile_type' AC conversion failed
+        self.assertRaises(TypeError, symtable.symtable, '', mortal_str, 1)
+
 
 class ComprehensionTests(unittest.TestCase):
     def get_identifiers_recursive(self, st, res):

Lib/test/test_zipfile/_path/test_path.py

@@ -274,7 +274,8 @@ def test_pathlike_construction(self, alpharep):
         """
         zipfile_ondisk = self.zipfile_ondisk(alpharep)
         pathlike = FakePath(str(zipfile_ondisk))
-        zipfile.Path(pathlike)
+        root = zipfile.Path(pathlike)
+        root.root.close()
 
     @pass_alpharep
     def test_traverse_pathlike(self, alpharep):
@@ -373,6 +374,7 @@ def test_root_on_disk(self, alpharep):
         root = zipfile.Path(self.zipfile_ondisk(alpharep))
         assert root.name == 'alpharep.zip' == root.filename.name
         assert root.stem == 'alpharep' == root.filename.stem
+        root.root.close()
 
     @pass_alpharep
     def test_suffix(self, alpharep):
@@ -574,11 +576,13 @@ def test_inheritance(self, alpharep):
     )
     def test_pickle(self, alpharep, path_type, subpath):
         zipfile_ondisk = path_type(str(self.zipfile_ondisk(alpharep)))
-
-        saved_1 = pickle.dumps(zipfile.Path(zipfile_ondisk, at=subpath))
+        root = zipfile.Path(zipfile_ondisk, at=subpath)
+        saved_1 = pickle.dumps(root)
+        root.root.close()
         restored_1 = pickle.loads(saved_1)
         first, *rest = restored_1.iterdir()
         assert first.read_text(encoding='utf-8').startswith('content of ')
+        restored_1.root.close()
 
     @pass_alpharep
     def test_extract_orig_with_implied_dirs(self, alpharep):
@@ -590,6 +594,7 @@ def test_extract_orig_with_implied_dirs(self, alpharep):
         # wrap the zipfile for its side effect
         zipfile.Path(zf)
         zf.extractall(source_path.parent)
+        zf.close()
 
     @pass_alpharep
     def test_getinfo_missing(self, alpharep):