What do you do when your Python program uses too much memory? How do you find memory allocation points in your code, especially large chunks? As it turns out, there’s usually no easy answer to these questions, but there are tools that can help you figure out where your code allocates memory. In this article, I will highlight one of them. memory_profiler.

The Memory_profiler tool is similar in spirit to (and inspired by) the Line_profiler tool, which I have also written about. Line_profiler tells you how much time each row took, while memory_profiler tells you how much memory was allocated (or freed) per row. This allows you to see the true impact of each line of code and get a sense of memory usage. While this tool is helpful, there are a few things you need to know to use it effectively. I’ll cover some of the details in this article.

The installation

Memory_profiler is written in Python and can be installed with PIP. The package will include libraries, as well as some command-line utilities.

pip install memory_profiler
Copy the code

It uses the Psutil library (or perhaps tracemalloc or POSIX) to access process information in a cross-platform manner, so it can be used on Windows, Mac, and Linux.

Basic anatomy

Memory_profiler is a set of tools for profiling The memory usage of Python programs, and the documentation provides a good overview of these tools. The tool that provides the most detail is the line-by-line memory usage that the module reports as it dissects individual functions. You can get this information by running the module against a Python file on the command line. It is also available via Juypyter/IPython Magics, or in your own code. I’ll cover all of these options in this article.

I’ve expanded on the sample code in the documentation to show several ways you might see memory grow and reclaim in Python code, as well as line by line output on my computer. You can follow along by running the configuration file yourself, using the sample code (performance_memory_profiler.py) saved in the source file below.

from functools import lru_cache from memory_profiler import profile import pandas as pd import numpy as np @profile def simple_function(): a = [1] * (10 ** 6) b = [2] * (2 * 10 ** 7) del b return a @profile def simple_function2(): a = [1] * (10 ** 6) b = [2] * (2 * 10 ** 8) del b return a @lru_cache def caching_function(size): return np.ones(size) @profile def test_caching_function(): for i in range(10_000): Caching_function (I) for I in range(10_000,0,-1): caching_function(I) if __name__ == '__main__': simple_function() simple_function() simple_function2() test_caching_function()Copy the code

runmemory_profiler

To provide line-by-line output, memory_profiler needs to decorate a method with the @profile decorator. Just add it to the method you want to parse, and I’ve done this with the three methods above. Then, you need a way to actually execute those methods, such as a command-line script. Running a unit test is fine, as long as you can run it from the command line. You can do this by running the Memory_profiler module and providing Python scripts to drive your code. You can give it a minus h and look at help.

$ python -m memory_profiler -h
usage: python -m memory_profiler script_file.py

positional arguments:
  program               python script or module followed by command line arguements to run

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  --pdb-mmem MAXMEM     step into the debugger when memory exceeds MAXMEM
  --precision PRECISION
                        precision of memory output in number of significant digits
  -o OUT_FILENAME       path to a file where results will be written
  --timestamp           print timestamp instead of memory measurement for decorated functions
  --include-children    also include memory used by child processes
  --backend {tracemalloc,psutil,posix}
                        backend using for getting memory info (one of the {tracemalloc, psutil, posix})
Copy the code

To see the results of the sample program, simply run it with the default values. Since we marked three of the functions with the @profile decorator, all three calls will be printed out. Be careful when profiling a method or function that is called multiple times; it prints a result for each call. Here’s the result on my computer, and I’ll explain how it works in detail. For each function, we get the source line number on the left, the actual Python source code on the right, and three metrics for each line. The first is the memory usage of the entire process when executing the line of code, how much the line’s memory was incremented (positive) or decayed (negative), and how many times the line was executed.

$ python -m memory_profiler performance_memory_profiler.py Filename: performance_memory_profiler.py Line # Mem usage Increment Occurences Line Contents = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 8 67.2 67.2 MiB MiB 1 @ profile 9 def simple_function () : 10 74.8 MiB 7.6 MiB 1 A = [1] * (10 ** 6) 11 227.4 MiB 152.6 MiB 1 b = [2] * (2 * 10 ** 7) 12 227.4 MiB 0.0 MiB 1 del b 13 227.4 MiB 0.0 MiB 1 return a Filename: performance_memory_profiler.py Line # Mem usage Increment Occurences Line Contents = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 8 227.5 227.5 MiB MiB 1 @ profile 9 def simple_function () : 10 235.1 MiB 7.6 MiB 1 A = [1] * (10 ** 6) 11 235.1 MiB 0.0 MiB 1 b = [2] * (2 * 10 ** 7) 12 235.1 MiB 0.0 MiB 1 del b 13 235.1 MiB 0.0 MiB 1 return a Filename: performance_memory_profiler.py Line # Mem usage Increment Occurences Line Contents = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 15 MiB MiB 1 @ 235.1 235.1 profile 16 def simple_function2(): 17 235.1 MiB 0.0 MiB 1 A = [1] * (10 ** 6) 18 1761.0 MiB 1525.9 MiB 1 B = [2] * (2 * 10 ** 8) 19 235.1 MiB -1525.9 MiB 1 Del b 20 235.1 MiB 0.0 MiB 1 return a Filename: performance_memory_profiler.py Line # Mem usage Increment Occurences Line Contents = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 27 MiB MiB 1 @ 235.1 235.1 profile 28 def Test_caching_function (): 29 275.6 MiB 0.0 MiB 10001 for I in range(10_000): 30 275.6 MiB 40.5 MiB 10000 Caching_Function (I) 31 32 280.6 MiB 0.0 MiB 10001 for I in range(10_000,0,-1): 33 280.6 MiB 5.0 MiB 10000 Caching_function (I)Copy the code

Explain the results

If you look at the official documentation, you’ll see slightly different results in their sample output than when I executed simple_function. For example, in my first two calls to the function, del seemed to have no effect, and their example showed that memory was freed. This is because Python is a garbage collection language, so del is different from freeing memory in a language like C or C ++. As you can see, on the first call to the method, memory explodes, but on the second call, b is created the second time and no new memory is required. To clarify this, I’ve added another method, simple_function2, to create a larger list, and this time we see memory freed that the garbage collector decides to reclaim. This is just one example of how profiling code may require multiple runs with different input data to get a true result for your code. Also consider the hardware you’re using; Production problems may not match development workstations. Creating a good test program can take as much time as interpreting the results and deciding how to improve.

The second thing to notice from my results is for caching_function. Notice that the test driver runs the function with 10,000 values, but then runs it again in reverse. The cache will be occupied for the first 128 calls (the default size of the funcTools.lru_cache function decorator). We see much less memory growth the second time around (both because the cache was hit and because the garbage collector did not reclaim previously allocated memory). In general, look for continuous or large memory increments without decrement. Also look for memory increases, even small amounts, each time a function is called.

Profiling in regular code

If you import the function decorator (as described above) into your code and it works, the profiling data will be sent to STdout. This can be a convenient way to quickly dissect individual methods. You can annotate any function, just run your code with the script you normally use. Note that you can either send this output to a file or use the logging module to record it. See the documentation for details.

Jupyter/IPython magic

The Memory_Profiler project also includes Jupyter/IPython Magics, which can be useful. It is important to note that in order to get line-by-line output (as of this writing, the latest version — V0.58), the code must be saved in a native Python source file and not read directly from a notebook or IPython interpreter. But this magic can still be useful for debugging memory problems. To use them, load the extension.

%load_ext memory_profiler
Copy the code

mprun

The %mprun magic is similar to running the above function, but you can do some more AD hoc checks. First, just import the functions and run them. Please note that I found it didn’t seem to work well with AutoReload so your mileage may vary when trying to modify the code and test without having to do a full kernel reboot.

from performance_memory_profiler import test_caching_function, simple_function
Copy the code
%mprun -f simple_function simple_function() Filename: /Users/mcw/projects/python_blogposts/performance/performance_memory_profiler.py Line # Mem usage Increment Occurences Line Contents = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 8 MiB MiB 1 @ 76.4 76.4 profile 9 def simple_function(): 10 84.0 MiB 7.6 MiB 1 A = [1] * (10 ** 6) 11 236.6 MiB 152.6 MiB 1 B = [2] * (2 * 10 ** 7) 12 236.6 MiB 0.0 MiB 1 del b 13 236.6 MiB 0.0 MiB 1 Return ACopy the code

memit

The %memit and %%memit magic help to check what the memory spikes and memory increments are for the code being executed. You won’t get line-by-line output, but this allows for interactive debugging and testing.

%%memit Range (1000) Peak Memory: 237.00 MiB, increment: 0.32 MiBCopy the code

Observe specific objects without memory_profiler

Let’s take a quick look at the Numpy and PANDAS objects and how we can see the memory usage of these objects. These two libraries and their objects are likely to be large for many use cases. For newer versions of the library, you can use sys.get_size_of to check its memory usage. Under the hood, pandas objects call their memory_Usage method directly, which you can also use directly. Note that if you also want to see the memory usage of the objects in the PANDAS container, you need to specify deep=True.

import sys

import numpy as np
import pandas as pd

def make_big_array():
    x = np.ones(int(1e7))
    return x

def make_big_string_array():
    x = np.array([str(i) for i in range(int(1e7))])
    return x

def make_big_series():
    return pd.Series(np.ones(int(1e7)))

def make_big_string_series():
    return pd.Series([str(i) for i in range(int(1e7))])

arr = make_big_array()
arr2 = make_big_string_array()
ser = make_big_series()
ser2 = make_big_string_series()

print("arr: ", sys.getsizeof(arr), arr.nbytes)
print("arr2: ", sys.getsizeof(arr2), arr2.nbytes)
print("ser: ", sys.getsizeof(ser))
print("ser2: ", sys.getsizeof(ser2))
print("ser: ", ser.memory_usage(), ser.memory_usage(deep=True))
print("ser2: ", ser2.memory_usage(), ser2.memory_usage(deep=True))
Copy the code
arr: 80000096 80000000 arr2: 280000096 280000000 ser: 80000144 ser2: 638889034 ser: 80000128 80000128 ser2: 80000128, 638889018,Copy the code
%memit make_big_string_series()
Copy the code
Peak Memory: 1883.11 MiB, increment: 780.45 MiBCopy the code
%%memit
x = make_big_string_series()
del x
Copy the code
Peak Memory: 1883.14 MiB, increment: 696.07 MiBCopy the code

There are two things to point out here. Int First, you can see that the size of the Series object is the same whether you use deep=True or not. For string objects, the size is the same as int Series, but the underlying object is much larger. You can see that our Series of string objects is over 600MB, and using %memit, we can see an increment when we call the function. This tool helps you narrow down which functions allocate the most memory and should be investigated further with a line-by-line analysis.

Further investigation

The Memory_Profile project also has tools for investigating long-running programs to see how memory grows over time. See the mprof command for this feature. It also supports memory tracking for fork processing in a multi-process environment.

conclusion

Debugging memory problems can be a very difficult and laborious process, but there are tools available to help understand where memory is allocated, which can be very helpful in facilitating the debugging process. When used with other profiling tools, such as Line_profiler or py-Spy, you can get a better idea of where your code needs improvement.

The postProfiling Python code with memory_profilerappeared first onwrighters.io.