Engineering
Exploring Python 3.10
On October 4th, the new 3.10 version of Python was released. According to the TIOBE Index for October 2021, Python is the most popular programming language in the world. Naturally, many people are interested in the new features added in the new version. As Backend.AI is a program written primarily in Python, we are always keeping a close eye on these developments. In this blog post, we'd like to take a look together at the major changes in Python 3.10.
Structural Pattern Matching and Enhancements to the with statement
The highlight of this release, and the most noteworthy change in recent years, is the addition of structural pattern matching syntax.
It is also the first new syntax introduced since the async and await keywords were added in Python 3.5 and the walrus operator (:=) for assignment expressions was added in Python 3.8.
On a simple level, structural pattern matching can serve the same role as the switch-case statement in languages like C or Java. However, it enables the creation of much more sophisticated conditional branches by considering both the value's type and its structural form.
The most striking differences compared to the switch statement in other languages are its ability to execute different branches based on structural information, such as the number and type of elements in a tuple or the types of keys in a dictionary, and the ability to capture parts of a matched pattern as variables within the branch using the as clause.
For example, when implementing something like a parser, it's possible to express the Abstract Syntax Tree (AST) generation code much more concisely, in a way that resembles EBNF itself, by using pattern matching syntax.
def simplify_expr(tokens):
match tokens:
case [('(' | '[') as left, *expr, (')' | ']') as right] if (left + right) in ('()', '[]'):
return simplify_expr(expr)
case [0, ('+' | '-') as op, right]:
return UnaryOp(op, right)
case [
(int() | float() as left) | Num(left),
'+',
(int() | float() as right) | Num(right),
]:
return Num(left + right)
case [(int() | float()) as value]:
return Num(value)Detailed syntax and examples can be found in a series of PEP documents (PEP-634, PEP-635, PEP-636).
Another welcome change from a practical systems programming perspective is that it is now possible to group multiple context managers in a single with statement using parentheses.
Previously, this was not possible, so one had to either bundle them with a separate contextlib.ExitStack object, create unnecessarily nested indentations, or use backslashes to connect multiple lines, which is not recommended by the PEP-8 code style.
with (
Session(engine).begin() as session,
contextlib.closing(some_file) as f,
tempfile.NamedTemporaryFile() as t,
):
# do something...Both the match statement for structural pattern matching and the enhancement to the with statement were made possible by the new PEG parser introduced in Python 3.9 and made the default in 3.10. We can look forward to seeing more diverse changes and improvements in Python's syntax in the future.
Standard Library Improvements
This release also includes the contextlib.aclosing() function, to which I made a small contribution myself.
It is a very simple patch that corresponds to the async version of contextlib.closing().
The original related issue, bpo-41229, was a report about a problem where async generator objects would remain in memory until the next context switch of the event loop if the code was executed without an explicit close.
In reality, for long-running asyncio code, a context switch would occur very quickly, so it wasn't a very critical issue. However, I thought it would be good to add a "standard recommended way" to explicitly close async generators to the standard library, so I created the PR.
Enhanced Type Hinting Support
In addition, new features related to type hinting, which are being improved with the continuous participation of Guido van Rossum, have been added.
A representative example is that Union types, which allow multiple types, can now be written concisely as X | Y instead of typing.Union[X, Y] (PEP-604).
This was usable in previous Python versions by setting the from __future__ import annotations flag, but from 3.10 onwards, it can be used without it.
Furthermore, typing.TypeAlias has been added, allowing one to explicitly express that a name defined to refer to a complex type definition is an alias for an existing type, not a new type definition (PEP-613).
At first glance, it may not seem like a big difference, but it helps the type checker generate clearer error messages and can prevent you from writing code that accidentally executes a type definition.
Finally, typing.ParamSpec and typing.Concatenate have been added to allow for strict type definitions of decorators and higher-order functions (functions that take or return functions) (PEP-612).
Patterns like *args, **kwargs, which previously could only be expressed with typing.Any, can now be defined as a kind of generic and treated as a type argument. This allows for precise expression of patterns such as forcing the input and return functions of a decorator to have the same argument signature or adding an argument of a specific type.
What's Next?
As we've seen, Python 3.10 is a release focused on syntax and type improvements, centered around structural pattern matching.
It appears that performance improvements will be the main enhancement for version 3.11, scheduled for release next October, and subsequent versions. In particular, the Python core developer community has recently been experimenting with a variety of ideas. For example, they are considering introducing sub-interpreters to multiplex the GIL within a single process, or investigating the introduction of biased reference counting to eliminate the GIL, based on the insight that a lock is not actually required for most object reference count changes. Of course, these are still in the experimental stage, and it is not certain when or how they will be reflected in the final version. However, since the creator of the Python language, Guido van Rossum, recently moved to Microsoft and is leading a performance improvement team full-time with Microsoft's support, we can expect significant improvements in 1-2 years.
In the future, Backend.AI will also actively incorporate these improvements to become an even more stable and faster solution.
