The SYS module in Python is very basic and important. It mainly provides variables that the interpreter uses (or maintains) and functions that strongly interact with the interpreter.

getsizeof()

  • This method is used to get the size of an object in bytes.

  • It only counts the memory directly occupied, not the memory of the objects referenced within the object

Here’s an intuitive example:

Import sysa = [1, 2]b = [a, a] # i.e. [1, 2], [1, 2]]# sysa = [1, 2]b = [a, a] # 80sys.getsizeof(b) # result: 80Copy the code

The above example illustrates one thing: a statically created list containing only two elements takes up 80 bytes of memory on its own, regardless of the object to which its elements refer.

Now, with this measurement tool in hand, let’s explore what Python’s built-in objects are hiding.

1. Empty objects are not empty!

Do empty objects take up no memory? If it takes up memory, how much? Why is it distributed this way?

Let’s look at the size of empty objects for some basic data structures:

import syssys.getsizeof("") # 49sys.getsizeof([]) # 64sys.getsizeof(()) # 48sys.getsizeof(set()) # 224sys.getsizeof(dict()) # 240# for reference: sys.getsizeof(1) # 28sys.getsizeof(True) # 28Copy the code

As you can see, these objects are empty, but they are not “empty” in memory allocation, and they are allocated quite large (remember these numbers, we will check later).

In order: base number < empty tuple < empty string < empty list < empty set < empty dictionary.

How to explain this little secret?

Because these empty objects are containers, we can understand them abstractly: some of their memory is used to create the skeleton of the container, to record information about the container (reference counts, usage information, and so on), and some of their memory is pre-allocated.

2, memory expansion is not uniform!

Empty objects are not empty, in part because the Python interpreter preallocates some initial space for them. Each new element uses the existing memory without exceeding the original memory, thus avoiding the need to apply for new memory.

So, if the initial memory is allocated, how is the new memory allocated?

import sysletters = "abcdefghijklmnopqrstuvwxyz"a = []for i in letters:    a.append(i)    print(f'{len(a)}, sys.getsizeof(a) = {sys.getsizeof(a)}')b = set()for j in letters:    b.add(j)    print(f'{len(b)}, sys.getsizeof(b) = {sys.getsizeof(b)}')c = dict()for k in letters:    c[k] = k    print(f'{len(c)}, sys.getsizeof(c) = {sys.getsizeof(c)}')Copy the code

Add 26 elements to each of the three mutable objects and see what happens:

  • Overallocation mechanism: New memory is allocated not on demand, but more, so that when a few more elements are added, there is no need to immediately allocate new memory

  • Non-uniform allocation mechanism: The three types of objects apply for new memory at different frequencies, but the over-allocated memory of the same type of objects is not uniform, but gradually expanded

A list is not a list!

The above variable objects have a similar allocation mechanism when expanding, and the effect can be clearly seen in dynamic expansion.

Does this work for statically created objects? Is it different from the dynamic expansion ratio?

Let’s start with sets and dictionaries:

# static object creation set_1 = {1, 2, 3, 4} set_2 = {1, 2, 3, 4, 5} dict_1 = {' a ', 1, 'b' : 2, 'c' : 3, 'd', 4, 'e' : 5} dict_2 = {' a ', 1, 'b' : 2, 'c':3, 'd':4, 'e':5, 'f':6}sys.getsizeof(set_1) # 224sys.getsizeof(set_2) # 736sys.getsizeof(dict_1) # 240sys.getsizeof(dict_2) # 368Copy the code

Looking at this result and comparing the screenshot from the previous section, you can see that statically created collections/dictionaries take up exactly the same amount of memory as dynamically expanded collections with the same number of elements.

Does this apply to list objects? Take a look:

list_1 = ['a', 'b']list_2 = ['a', 'b', 'c']list_3 = ['a', 'b', 'c', 'd']list_4 = ['a', 'b', 'c', 'd', 'e']sys.getsizeof(list_1)  # 80sys.getsizeof(list_2)  # 88sys.getsizeof(list_3)  # 96sys.getsizeof(list_4)  # 104Copy the code

The screenshot from the previous section shows that the list is 96 bytes for each of the first four elements and 128 bytes for each of the five elements, which clearly contradicts this.

So, the secret is clear: with the same number of elements, statically created lists can take up less memory than dynamically expanded lists!

In other words, the two lists look the same but are actually different! A list is not a list!

Eliminating elements does not free memory!

As mentioned earlier, when scaling mutable objects, it is possible to request new memory.

So, if you shrink the mutable object in reverse, after subtracting a few elements, will the newly allocated memory be automatically reclaimed?

Import sysa = [1, 2, 3, 4]sys.getsizeof(a) # default value: 96a. appEnd (5) # [1, 2, 3, 4]sys.getsizeof(a) #Copy the code

As the code shows, the list expands and shrinks, but the memory space is not automatically freed up. The same goes for other mutable objects.

Here’s the Python secret: It’s easy for skinny people to get fat and lose weight, but they can’t lose weight


Empty dictionary is not equal to empty dictionary!

Using the pop() method only shrinks the elements in the mutable object, but it does not free up allocated memory.

There is also the clear() method, which clears all elements of the mutable object. Let’s try it:

import sysa = [1, 2, 3]b = {1, 2, 3}c = {'a':1, 'b':2, 'c':3}sys.getsizeof(a) # 88sys.getsizeof(b) # 224sys.getsizeof(c) # 240a.clear() # clear: Set ()c.clear() # empty: {}, dict()Copy the code

By calling the clear() method, we get several empty objects.

Their memory size was checked in the first section. (Previously said the examination, please write back to read)

However, if you check again at this point, you will be surprised to find that the empty objects are not exactly the same size!

Getsizeof (a) # 64sys.getsizeof(b) # 224sys.getsizeof(c) # 72Copy the code

Empty lists and empty tuples remain the same size, but the empty dictionary (72) is much smaller than the previous empty dictionary (240)!

In other words, lists and tuples, after being emptied, return to the original point of view, whereas dictionaries throw up what they ate and lose what they had.

The dictionary is a pretty deep secret. To be honest, I have just learned it, and I can’t figure it out…

These are some of the secrets of allocating memory in Python. Do you feel enlightened after reading them?

How many have you figured out? How many new mysteries have you created? Welcome to leave a message to exchange oh ~

For those little secrets that haven’t been fully explained, we’ll reveal them later…

About the author: