Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PEP 764: Inlined typed dictionaries #4082

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions peps/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@
"typing": ("https://typing.readthedocs.io/en/latest/", None),
"trio": ("https://trio.readthedocs.io/en/latest/", None),
"devguide": ("https://devguide.python.org/", None),
"mypy": ("https://mypy.readthedocs.io/en/latest/", None),
"py3.11": ("https://docs.python.org/3.11/", None),
"py3.12": ("https://docs.python.org/3.12/", None),
"py3.13": ("https://docs.python.org/3.13/", None),
Expand Down
218 changes: 218 additions & 0 deletions peps/pep-9995.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,218 @@
PEP: 9995
Viicos marked this conversation as resolved.
Show resolved Hide resolved
Title: Inlined typed dictionaries
Author: Victorien Plot <[email protected]>
Discussions-To:
Status: Draft
Type: Standards Track
Topic: Typing
Created: 23-Oct-2024
Viicos marked this conversation as resolved.
Show resolved Hide resolved
Post-History:
Python-Version: 3.14
Resolution:

.. highlight:: python


Abstract
========

:pep:`589` defines a `class-based <https://typing.readthedocs.io/en/latest/spec/typeddict.html#class-based-syntax>`_ and a
:ref:`functional syntax <typing:typeddict-functional-syntax>` to create typed
dictionaries. In both scenarios, it requires defining a class or assigning to
a value. In some situations, this can add unnecessary boilerplate, especially
if the typed dictionary is only used once.

This PEP proposes the addition of a new inlined syntax, by subscripting the
:class:`~typing.TypedDict` type::

from typing import TypedDict

def get_movie() -> TypedDict[{'name': str, 'year': int}]:
return {
'name': 'Blade Runner',
'year': 1982,
}

Motivation
==========

Python dictionaries are an essential data structure of the language. Many
times, it is used to return or accept structured data in functions. However,
it can get tedious to define :class:`~typing.TypedDict` classes:

* A typed dictionary requires a name, which might not be relevant.
* Nested dictionaries requires more than one class definition.

Taking a simple function returning some nested structured data as an example::

from typing import TypedDict

class ProductionCompany(TypedDict):
name: str
location: str

class Movie(TypedDict):
name: str
year: int
production: ProductionCompany


def get_movie() -> Movie:
return {
'name': 'Blade Runner',
'year': 1982,
'production': {
'name': 'Warner Bros.',
'location': 'California',
}
}


Rationale
=========

The new inlined syntax can be used to resolve these problems::

Viicos marked this conversation as resolved.
Show resolved Hide resolved
def get_movie() -> TypedDict[{'name': str, year: int, 'production': {'name': str, 'location': str}}]:
Copy link
Contributor

@erictraut erictraut Oct 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should support nested inlined TypedDict definitions as shown above. We should require that the type annotation for each item be a valid type expression, and a dictionary expression not enclosed by a TypedDict is an invalid type expression. The above example should therefore be changed to:

def get_movie() -> TypedDict[{"name": str, "year": int, "production": TypedDict[{"name": str, "location": str}]}]: ...

The grammar definition proposed above by Jelle already implies this, but the spec should make it clear that using nested dictionary expressions (like in the current code sample) is not allowed and should result in a type checker error.

I understand this might seem needlessly verbose, but if we allow dictionary expressions in type expressions outside of a TyepdDict type argument, it will create a cascade of exceptions and future composability issues. For example, we'd need to support raw dictionary expressions within Required, NotRequired, and ReadOnly.

Copy link
Contributor Author

@Viicos Viicos Oct 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed using a nested inlined typed dictionary is more consistent! Changed.

(Using typing.Dict instead of TypedDict might help reducing verbosity here).

...

It is recommended to *only* make use of inlined typed dictionaries when the
structured data isn't too large, as this can quickly get hard to read.

While less useful (as the functional or even the class-based syntax can be
Viicos marked this conversation as resolved.
Show resolved Hide resolved
used), inlined typed dictionaries can be defined as a type alias::

type InlinedDict = TypedDict[{'name': str}]
Viicos marked this conversation as resolved.
Show resolved Hide resolved

Specification
=============

The :class:`~typing.TypedDict` class is made subscriptable, and accepts a
single type argument which must be a :class:`dict`, following the same
semantics as the :ref:`functional syntax <typing:typeddict-functional-syntax>`
(the dictionary keys are strings representing the field names, and values are
valid :ref:`annotation expressions <typing:annotation-expression>`).

Inlined typed dictionaries can be referred as being *anonymous*, meaning they
Viicos marked this conversation as resolved.
Show resolved Hide resolved
don't have a name. For this reason, their :attr:`~type.__name__` attribute
will be set to an empty string.

It is not possible to specify any class arguments such as ``total``.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that Required, NotRequired and ReadOnly should be permitted here. Absent any mention of this, the typing spec could be interpreted as disallowing this. It's better to make it clear.

I presume that all inlined TypedDicts are implicitly "total" (i.e. all items are assumed to be Required unless explicitly NotRequired). This makes sense because it's consistent with precedent, but it would be good to spell it out in the spec.

I think it's also worth considering making all inlined TypedDicts implicitly "closed" (as defined in draft PEP 728). This would deviate from precedent, but I think one could make a good argument for why inlined TypedDicts should be closed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that Required, NotRequired and ReadOnly should be permitted here. Absent any mention of this, the typing spec could be interpreted as disallowing this. It's better to make it clear.

I presume that all inlined TypedDicts are implicitly "total" (i.e. all items are assumed to be Required unless explicitly NotRequired). This makes sense because it's consistent with precedent, but it would be good to spell it out in the spec.

Updated.

I think it's also worth considering making all inlined TypedDicts implicitly "closed" (as defined in draft PEP 728). This would deviate from precedent, but I think one could make a good argument for why inlined TypedDicts should be closed.

Sounds reasonable. I have to say I'm still a bit confused with PEP 728 and closed=True. If we disallow subclassing inlined typed dictionaries, it means that every inlined typed dictionary is implicitly @final. What's the purpose of having closed=True then? I guess I should ask my question in the PEP 728 discussion thread instead.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am I correct in assuming that there is no way for an inlined TypedDict to "extend" another (named) TypedDict? One could imagine supporting this using dictionary expansion syntax, but that adds complexity that may not be justifiable. It would also impose additional runtime requirements on TypedDict classes (they'd need to be iterable).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A good way to support inheritance could be to allow more than one argument to the subscript: TypedDict[Base, {"a": int}].


Runtime behavior
----------------

Although :class:`~typing.TypedDict` is commonly referred as a class, it is
implemented as a function at runtime. To be made subscriptable, it will be
changed to be a class.

Creating an inlined typed dictionary results in a new class, so both syntaxes
return the same type::

from typing import TypedDict

T1 = TypedDict('T1', {'a': int})
T2 = TypedDict[{'a': int}]


Backwards Compatibility
=======================

Apart from the :class:`~typing.TypedDict` internal implementation change, this
PEP does not bring any backwards incompatible changes.


Security Implications
=====================

There are no known security consequences arising from this PEP.


How to Teach This
=================

The new inlined syntax will be documented both in the :mod:`typing` module
documentation and the :ref:`typing specification <typing:typed-dictionaries>`.

As mentioned in the `Rationale`_, it should be mentioned that inlined typed
dictionaries should be used for small structured data to not hurt readability.


Reference Implementation
========================

Mypy supports a similar syntax as an :option:`experimental feature <mypy:mypy.--enable-incomplete-feature>`::

def test_values() -> {"int": int, "str": str}:
return {"int": 42, "str": "test"}

Pyright has added support in version `1.1.297`_ (using :class:`dict`), but was later
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pyright 1.1.387, which will be published in the next day, includes experimental support for the latest draft of this PEP.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing, updated to reference the future release.

removed in version `1.1.366`_.

.. _1.1.297: https://github.com/microsoft/pyright/releases/tag/1.1.297
.. _1.1.366: https://github.com/microsoft/pyright/releases/tag/1.1.366

Runtime implementation
----------------------

A draft implementation is available `here <https://github.com/Viicos/cpython/commit/49e5a83f>`_.


Rejected Ideas
==============

Using the functional syntax in annotations
------------------------------------------

The alternative functional syntax could be used as an annotation directly::

def get_movie() -> TypedDict('Movie', {'title': str}): ...

However, call expressions are currently unsupported in such a context for
various reasons (expensive to process, evaluating them is not standardized).

This would also require a name which is sometimes not relevant.

Using ``dict`` with a single type argument
------------------------------------------

We could reuse :class:`dict` with a single type argument to express the same
concept::

def get_movie() -> dict[{'title': str}]: ...

While this would avoid having to import :class:`~typing.TypedDict` from
:mod:`typing`, this solution has several downsides:

* For type checkers, :class:`dict` is a regular class with two type variables.
Allowing :class:`dict` to be parametrized with a single type argument would
require special casing from type checkers, as there is no way to express
parametrization overloads. On ther other hand, :class:`~typing.TypedDict` is
already a :term:`special form <typing:special form>`.

* If fufure work extends what inlined typed dictionaries can do, we don't have
Viicos marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you consider reusing typing.Dict here? I can see some potential advantages. It's less verbose, doesn't have the baggage of builtins.dict, and the fact that this is a "typed" dict is already implicit given that it is imported from typing and involves type annotations.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Showing my TypeScript bias here 😅 but I'm wondering how sacrilegious it would be to avoid the subscription all together and simply use the dictionary itself?

def get_movie() -> {'name': str, 'year': int}:
    return {
        'name': 'Blade Runner',
        'year': 1982,
    }

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This causes problems with the | operator at least. I've been thinking of writing up something like "Why can't we ...?" discussing why certain "obvious" ways to make typing more ergonomic may not work well in practice.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that such writeup would be very useful!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added an extra section in the rejected ideas regarding the plain dict syntax.

On using typing.Dict, I kind of like the idea for the reasons you mentioned @erictraut. Dict[{...}] implicitly spells out as something like:

Dict[{...}]
|      |------> ...with some typed fields
|--> a dict... 

However, I still think this is debatable for a couple reasons:

  • typing.Dict is currently marked as deprecated in the Python documentation (although not scheduled for removal). It might be confusing to undeprecate it.
  • The same subscripting syntax — with the only difference being the number of type arguments — means two different things. Can be confusing as well.

I'll add an entry to the open issues so that we can further discuss this option.

to worry about impact of sharing the symbol with :class:`dict`.


Open Issues
===========

Subclassing an inlined typed dictionary
---------------------------------------

Should we allow the following?::
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this shouldn't be allowed by a static type checker. In this example, InlinedTD is a type alias.

If you change this to InlinedTD = TypedDict('InlinedTD', {'a': int}), then it should be allowed (as it is today) because InlinedTD is now a class, which can be used as a base class.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think we should also discuss whether an inlined typed dictonary is a proper class or an instance of some new internal typing._InlinedTypedDict class. As specified currently in this PEP, an inlined typed dictionary is a also a class, with an empty string for __name__ (which isn't ideal to be fair). Initially, I wanted inlined typed dictionaries to be an instance of some _InlinedTypedDict class, but this complicates the runtime behavior. For example, should we allow accessing the introspection attributes?

InlinedTD = TypedDict[{'a': int}]
InlinedTD.__required_keys__
InlinedTD.__total__
# etc


from typing import TypedDict

InlinedTD = TypedDict[{'a': int}]


class SubTD(InlinedTD):
pass


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
Loading