Hello everyone, I am Jiejie, today I would like to introduce a very mysterious magic method.

This method is so obscure and so narrow that I almost never pay attention to it, but when IT turns out that it may be the only exception to the above “law,” I think it’s worth writing another article to examine it in detail.

This paper mainly focuses on the following issues:

(1) What is __missing__()?

(2) What’s so special about __missing__()? Good at make-live magic?

(3) Is __missing__() really an exception to the above findings? If so, why the exception?

1. __missing__()

The key may not exist when the value is taken from a common dictionary:

Dd = {'name':'PythonCat'} dd. Get ('age') # Dd.__getitem__('age') # equivalent to dd['age']Copy the code

In the case of the get() method, it returns a value, and a second argument can be passed as a return that key does not exist, so it is acceptable. However, the other two will return an error.

To solve the latter two problems, use the __missing__() magic method.

Now, suppose we have a request that takes the value of a key from a dictionary, returns the value if there is a value, inserts the key if there is no value, and gives it a default value (such as an empty list).

It’s not easy to implement with native dict, but Python provides a very useful extension class called Collections.defaultdict:

As shown in the figure, KeyError is not reported when a nonexistent key is fetched, but is saved to the dictionary by default.

Why does DefaultDict do this?

The reason is that defaultdict, in addition to inheriting the built-in dict, also defines a __missing__() method that calls the factory function passed in when __getitem__ takes a nonexistent value (in this case, calling list() to create an empty list).

As a typical example, DefaultDict writes in the documentation comment:

In short, the main purpose of __missing__() is to be called by __getitem__ when a key is missing, thus avoiding keyerrors.

Another typical use example is collections.counter, which is also a dict subclass that returns the count 0 when fetching an uncounted key:

2. __missing__()

From the above, __missing__() is called when __getitem__() has no value, but I also inadvertently noticed that __getitem__() does not necessarily call __missing__() when __getitem__() has no value.

This is because it is not a required property of the built-in type and is not predefined in the dictionary base class.

If you take the attribute value directly from the dict type, the attribute does not exist: AttributeError: type object’ object’ has no attribute ‘__missing__’.

Use dir() to check that it does not exist:

The same result is found if you look at dict’s parent, Object.

What’s going on here? Why is there no __missing__ attribute in dict and object?

However, looking at the latest official documentation, object clearly contains this property:

Reference: docs.python.org/3/reference… _

That is, __missing__ is theoretically predefined in the object class, and its documentation proves this, but in practice it is not defined! Documentation deviates from reality!

Thus, when dict subclasses (such as Defaultdict and Counter) define __missing__, the magic method actually belongs only to that subclass, that is, it is a magic method born in that subclass!

Based on this, I have an immature guess: __getitem__() will determine if the current object is a dict subclass and has __missing__() before calling it (if the method is also in the parent class, it will be called).

I made this conjecture in the chat group, and some students quickly found verification in CPython source code:

And here’s the interesting thing: ** Magic methods exist only in subclasses of built-in types, ** it’s hard to find a second example in the Python world.

I suddenly had an association: This spooky __missing__() is like a magician who is good at “metasomes,” making the viewer see him through the glass outside (the official documentation), but when he opens the door, he’s not inside (the built-in type), changes the props, and he reappears intact (a dict subclass).

3. Enchanted __missing__()

The magic of __missing__(), in addition to its own “magic”, it also needs a strong “magic” to drive.

In the last article, I discovered that the native magic methods are independent of each other. They may have the same core logic in C, but there is no call relationship in Python:

This “dead-cut” behavior of the magic method violates general code reuse principles and is responsible for some strange behavior of subclasses of built-in types.

The only logical explanation for Python’s willingness to provide new subclasses of UserString, UserList, and UserDict rather than reuse magic methods seems to be that it is too expensive to make magic methods call each other.

However, with __missing__(), Python has to compromise and pay the price!

__missing__() is a “second-class citizen” of the magic method. It has no independent call entry and can only be passively called by __getitem__(), that is, __missing__() depends on __getitem__().

Unlike first-class citizens, such as __init__(), __enter__(), __len__(), __eq__(), and so on, which are fired either at a node in the object’s life cycle or execution, or by a built-in function or operator, these are relatively independent events that have no dependencies.

__missing__() depends on __getitem__() to implement method calls; __getitem__() also relies on __missing__() for full functionality.

To do this, __getitem__() creates a back door in the interpreter code, folding from the C interface back to the Python interface to call that particular method called “__missing__”.

And this is real “magic”, __missing__() seems to be the only magic method that has enjoyed such treatment so far!

4, summary

Python dictionaries provide built-in methods for two values, __getitem__() and get(), which handle values differently when they do not exist: the former returns KeyError, while the latter returns None.

Why does Python provide two different methods? Or rather, why did Python make these two methods behave differently?

There could be a very complicated (or very simple) explanation for this, but I’ll leave it at that.

One thing is certain, though: The crude tossing of keyErrors by native dict types falls short.

To give dictionary types a more powerful representation (or to make __getitem__() behave like get()), Python lets subclasses of dictionaries define __missing__() for __getitem__() lookup calls.

In this paper, the implementation principle of __missing__() is reviewed, which reveals that __missing__() is not an insignificant existence, on the contrary, it is the only one that breaks the barrier between magic methods and supports being called by other magic methods.

Python takes great pains to keep magic methods independent by introducing derived classes such as UserString, UserList, and UserDict, but compromises __missing__().

This article reveals the mystery of this magic method, I wonder how you feel after reading? Welcome to comment.