When we add type hints, we find that our desire for rigor contradicts Python’s flexibility. In this article, we’ll explore three sets of functions from the standard library that I naively expected to use narrow types, but use Any instead because of some edge cases.

1.operatorfunction

The operator module provides wrapper functions for Python operators. For example, operator.gt(a, b) wraps the “greater than” operator, so it equals a > b.

We usually expect operators to have well-defined types. For example, comparison operators like > return the value of bool, as stated in the syntax documentation.

But Python allows operator overloading, which allows custom operator functions to return arbitrary types. Pathlib. Path does this, to great effect, using the division operator/to join.

This flexibility forces the type of the operator module function to accept and return Any. Here is the current typeshed stubs for the operator comparison function.

def lt(__a: Any, __b: Any) - >Any:.def le(__a: Any, __b: Any) - >Any:.def eq(__a: Any, __b: Any) - >Any:.def ne(__a: Any, __b: Any) - >Any:.def ge(__a: Any, __b: Any) - >Any:.def gt(__a: Any, __b: Any) - >Any:.Copy the code

Given operator’s flexibility, we might find it better to re-implement narrow functions for our particular use case. For example.

def gt(a: int, b: int) - >bool:
    return a > b
Copy the code

2.loggingThe module

Python’s logging module is logged to receive _ string _ log messages.

MSG is the message format string, and args is the parameter, which is merged into MSG using the string formatting operator.

So we can reasonably expect logger.debug () and co. All type prompts define MSG as STR. But virtually all methods define MSG as Any. Why is that?

Typeshed PR #1776 changes STR to Any and explains why. The core Logger method uses STR (MSG) to force MSG to be a string, which means it allows any type. Guido van Rossum regrets on typeshed PR that using a non-STR type can represent an error, rendering log information useless.

In this particular case, I’m still sad to see the log MSG parameter change from STR to Any, because this reduces the chance of catching an error. In my experience, * usually * this is a coding error.

Oh, no.

3.json.loads()And friends

JSON has a particularly limited set of types, which seems to translate well into json.loads() type hints. There are four atomic types.

  • null– Is loaded asNone
  • Boolean — loaded asbool
  • Digital –intS orfloatS form load
  • String – asstrS load

. There are also two container types.

  • Array — loaded aslists
  • Object –dictS form load, and withstrThe key.

Container types can contain any atomic type _ or _ other containers.

This container-can container-recursion is our first problem with representing JSON in type hints. We need to use recursive type hints, which unfortunately Mypy does not currently support. If we try to recursively define JSON types like this.

from typing import Dict.List.Union

_PlainJSON = Union[
    None.bool.int.float.str.List["_PlainJSON"].Dict[str."_PlainJSON"]
]
JSON = Union[_PlainJSON, Dict[str."JSON"].List["JSON"]]
Copy the code

. Mypy will report a “possible loop definition” error.

$ mypy example.py
example.py:3: error: Cannot resolve name "_PlainJSON" (possible cyclic definition)
example.py:4: error: Cannot resolve name "_PlainJSON" (possible cyclic definition)
example.py:6: error: Cannot resolve name "JSON" (possible cyclic definition)
Found 3 errors in 1 file (checked 1 source file)
Copy the code

Support for recursive types is available in Mypy and is tracked in its issue #731.

But even though Mypy has added support for recursive types, json.loads() still needs to use Any return type. This, in turn, is due to the additional flexibility of its API.

Json.loads () accepts several extra parameters that can be used to change the type of json to load into different Python types. It is worth noting that the CLS parameter allows for a complete substitution of the loading mechanism, so we can have JSON parse to _ any _. So json.loads() always needs a return type of Any.

Libraries in other formats, such as PyYAML, follow the same pattern. So they also use the return type Any.

conclusion

As we’ve seen, the pesky Any type can be “hidden” in apis that are usually well-typed, but provide some flexibility. We need to be careful when using these functions.

As type hints spread through the Python ecosystem, we might see such apis changed to allow for stricter typing in common cases. For example, json.loads() can be split into two functions: one providing a well-defined return type with no flexibility and the other providing all custom return types of Any.

Finn

May your type tips take you where you want to go Any.

Dan