With the rapid development of mobile Internet, artificial intelligence has been applied more and more widely in mobile terminals. The group’s internal intelligence plays an important role in image recognition, video detection, data calculation and other core scenes. During the development phase, Python was the language of choice for algorithms. However, on the mobile terminal, the deployment, debugging and verification of algorithms are still in the era of “slash-and-burn”. At present, algorithms mainly verify the running logic and results of programs by inserting logs into the code.
Logging can also be used to verify results and locate problems, but once the project gets a little more complicated, productivity is very low. As a result, end-to-end Python debugging capabilities are embedded in the MNN workbench. The PDB module is an interactive code debugger provided by the Official Python standard library. Like any other debugging capabilities provided by a language, PDB provides routine debugging capabilities such as setting breakpoints and single step execution at the source code level. It is a very important tool module for Python development.
Today let us focus on the analysis of the official PDB module source code, see its debugging function of the underlying technical principle.
The principle of
As you can see from the CPython source code, the PDB module is not a built-in module of the C implementation, but a pure Python implementation and encapsulation module. The core file is db.py, which inherits from the BDB and CMD modules:
class Pdb(bdb.Bdb, cmd.Cmd): ...
Copy the codeBasic principle: use CMD module to define and realize a series of interactive input of debugging commands, based on sys. setTrace peg tracking code running stack frame, for different debugging commands to control the code running and breakpoint state, and output the corresponding information to the console.
The CMD module provides console command interaction, waits for input through raw_input/readline blocking methods, and then passes the command to subclasses to decide whether to continue the loop, just like its main method name, Runloop.
CMD is a common module, not specifically designed for THE PDB, which uses the framework of CMD to implement interactive custom debugging.
BDB core framework provides the debugging, rely on the sys. Settrace to run the code of single-step tracking, and then distribute the corresponding event (call/line/return/exception) to subclasses (PDB). The core logic of BDB is in the interrupt control of debugging commands, such as entering a single step “S” command, deciding whether to continue tracing or interrupt to wait for interactive input, which frame to interrupt to, etc.
The basic flow
- PDB starts and the current frame binding trace function trace_dispatch
def trace_dispatch(self, frame, event, arg):
if self.quitting:
return # None
if event == 'line':
return self.dispatch_line(frame)
if event == 'call':
return self.dispatch_call(frame, arg)
if event == 'return':
return self.dispatch_return(frame, arg)
if event == 'exception':
...
Copy the code- The processing of different events in each frame will go through the interrupt control logic, mainly stop_here (line events will also go through break_here) function, after processing to determine whether the code is interrupted, which line to interrupt
- If an interrupt is needed, the subclass method user_#event is triggered. The subclass uses Interaction to update the stack frame information and print the corresponding information on the console. Then cmdloop is executed to leave the console waiting for interactive input
def interaction(self, frame, traceback): self.setup(frame, Traceback) # Vars self.print_stack_entry(self.stack[self.curindex]) self.cmdloop() self.forget()Copy the code- When the user enters a debugging command such as “next” and press enter, the set_# command will be called first to set stopFrame, returnFrame and stoplineno, which will affect the interrupt control ‘ ‘stop_here’ ‘logic and thus determine the interrupt result to run to the next frame.
def _set_stopinfo(self, stopframe, returnframe, stoplineno=0):
self.stopframe = stopframe
self.returnframe = returnframe
self.quitting = 0
# stoplineno >= 0 means: stop at line >= the stoplineno
# stoplineno -1 means: don't stop at all
self.stoplineno = stoplineno
Copy the code- For debug process control commands, the do_# command generally returns 1, so that the runloop ends immediately. The next time a frame is run, the runloop will start again (see Step 3). The do_# command does not return a value for any of the commands of the information retrieval class, leaving the current interrupt state.
- The code runs to the next frame and repeats step 3
Interrupt control
Interrupt control means that after input of different debugging commands, the code execution can stop at the correct position and wait for user input. For example, the console should stop at the next frame code when input “S”, and the console should run at the next break point when output “C”. The interrupt control occurs at each step of the sys. setTrace trace and is the core logic of the debug run.
The PDB tracks four frame events:
- Line: Sequential execution events within the same frame
- Call: a function call occurs, jumps to the next level of frame, and generates a Call event on the first line of the function
- Return: after the function completes the last line (line), the result is returned. It is about to jump out of the current frame and return to the upper frame. A return event is generated on the last line of the function
- Exception: An exception occurs during the execution of the function, an exception is generated on the exception line, and a return (a return event) is generated on that line, and then an Exception and return are generated on the frame level up until the underlying frame is returned.
These are different node types for code trace, and the PDB will perform interrupt control at each step of frame trace based on debugging commands entered by the user, deciding whether to interrupt next and to which line. The main method of interrupt control is stop_here:
def stop_here(self, frame): # (CT) stopframe may now also be None, see dispatch_call. # (CT) the former test for None is therefore removed from here. if self.skip and \ self.is_skipped_module(frame.f_globals.get('__name__')): return False # next if frame is self.stopframe: # stoplineno >= 0 means: stop at line >= the stoplineno # stoplineno -1 means: don't stop at all if self.stoplineno == -1: Return False return frame.f_lineno >= self.stoplineno # step: Currently only trace back to botFrame and wait for execution. while frame is not None and frame is not self.stopframe: if frame is self.botframe: return True frame = frame.f_back return FalseCopy the codeThere are generally two types of debugging commands:
- Process control: the execution of code immediately following execution of setP, next, continue, etc
- Information obtaining/setting: Obtaining the current information, such as args, P, and list, does not affect the CMD status
The following highlights some of the most common debugging commands for process control interrupt control implementation principles:
S (step)
1 Command Definition
Execute the next command. If this is a function call, s will execute to the first line of the function.
2 Code Analysis
The implementation logic in PDB is to execute each frame sequentially and wait for execution, which has the same execution granularity as setTrace.
def stop_here(self, frame): ... # stopFrame None if frame is self. Stopframe:... While frame is not None and frame is not self.stopFrame: if frame is self.botFrame: return True frame = frame.f_back return FalseCopy the codeStep sets stopFrame to None, so as long as the current frame can be traced back to the underlying frame (botFrame), it is ready to execute, i.e. the PDB is in an interactive wait state.
Because step has the same granularity as setTrace, it waits for execution at every frame.
N (next)
1 Command Definition
Executes the next statement, or if this is a function call, the function, followed by the next statement of the currently executed statement.
2 Code Analysis
The implementation logic in the PDB is that the next trace interrupt running to the current frame is not interrupted when entering the next frame (function call).
def stop_here(self, frame): ... # Return true if frame is self. stopFrame: if self.stoplineno == -1: F_lineno >= self. Stoplineno # If the frame jumps out of stopFrame and goes to the next frame, execution is not interrupted until it jumps out of stopFrame # While frame is not None and frame is not self.stopFrame: if frame is self.botframe: return True frame = frame.f_back return FalseCopy the codeNext sets stopFrame to the current frame, which means that no interrupts are performed on other frames unless they are inside the current frame.
c
1 Command Definition
Execution continues until the next breakpoint is reached
2 Code Analysis
Stopframe is set to botFrame and stoplineno is set to -1. Stop_here always returns false and runs without interruption until a breakpoint is reached (break_here condition holds)
def stop_here(self, frame): ... # if in botFrame, stoplineno returns false if frame is self.stopFrame: if self.stoplineno == -1: F_lineno >= self. Stoplineno # If in a non-botFrame, stopFrame will be traced first, Return false while frame is not None and frame is not self.stopFrame: if frame is self.botFrame: return True frame = frame.f_back return FalseCopy the codeR (return)
1 Command Definition
Execute the currently running function to the end.
2 Code Analysis
The return command is interrupted only at the end of the frame (function call), when a return event is encountered. PDB sets stopFrame to last frame and ReturnFrame to current frame. If it is a non-return event, stop_here always returns false without breaking;
def stop_here(self, frame): ... # If the current frame code is executed sequentially, lineno of the next frame ==stoplineno # If the last line of the for loop is executed, lineno<stoplineno of the next frame (the first line of the for loop) will not break. Until the end of the for loop, lineno==stoplineno on the next line is interrupted if frame is self.stopFrame: if self.stoplineno == -1: F_lineno >= self. Stoplineno # If in a non-botFrame, stopFrame will be traced first and return False, Next while frame is not None and frame is not self.stopframe: if frame is self.botframe: return True frame = frame.f_back return FalseCopy the codeIf it is a return event, stop_here will still return false, but returnFrame will interrupt if the current frame is true.
def dispatch_return(self, frame, arg):
if self.stop_here(frame) or frame == self.returnframe:
self.user_return(frame, arg)
if self.quitting: raise BdbQuit
return self.trace_dispatch
Copy the codeWithout (until)
1 Command Definition
To the next line, the difference from next is that the for loop only traces once
2 Code Analysis
Set stopFrame and ReturnFrame to current frame, and stoplineno to current lineno+1.
def stop_here(self, frame): ... # If the current frame code is executed sequentially, lineno of the next frame ==stoplineno # If the last line of the for loop is executed, lineno<stoplineno of the next frame (the first line of the for loop) will not break. Until the end of the for loop, lineno==stoplineno on the next line is interrupted if frame is self.stopFrame: if self.stoplineno == -1: F_lineno >= self. Stoplineno # If in a non-botFrame, stopFrame will be traced first and return False, Next while frame is not None and frame is not self.stopframe: if frame is self.botframe: return True frame = frame.f_back return FalseCopy the codeIf there is a for loop in the current frame, it only executes once from the top down. If the function returns a return event, the lineno of the next frame may be less than stoplineno, so set the returnFrame to the current frame so that the function behaves like next.
U (up)/D (down)
1 Command Definition
Switch to up/down stack frame
2 Code Analysis
The stack frame information
The stack frame contains the frame information of each level on the invocation path of the code, which will be refreshed every time the command execution is interrupted. The frame can be switched up and down by u/ D command. The get_stack method is used to obtain the stack frame. The first parameter is frame and the second parameter is traceback object. The traceback object is generated in an exception event, which takes an arG argument:
exc_type, exc_value, exc_traceback = arg
(<type 'exceptions.IOError'>, (2, 'No such file or directory', 'wdwrg'), <traceback object at 0x10bd08a70>)
Copy the codeThe Traceback object has several common attributes:
- Tb_frame: The frame in which the current exception occurs
- Tb_lineno: The line number of the frame in which the current exception occurs, frame.tb_lineno
- Tb_next: Points to exc_traceback (Traceback object) called at the next level of the stack, or None if it is the topmost
The stack frame information consists of two parts, frame call stack and exception stack (if any), in the order: botFrame -> frame1 -> frame2 -> TB1 -> TB2 (error TB)
Def get_stack(self, f, t): stack = [] if t and t.tb_frame is f: t = t.tb_next stack.append((f, f.f_lineno)) if f is self.botframe: Break f = f.f_back stack.reverse() I = Max (0, len(stack) -1) # while t is not None: stack.append((t.tb_frame, t.tb_lineno)) t = t.tb_next if f is None: i = max(0, len(stack) - 1) return stack, iCopy the codePDB updates the stack frame table of the call and the current stack frame information with each execution interruption. Switching the stack can be done by switching the index up/down.
def setup(self, f, t): self.forget() self.stack, self.curindex = self.get_stack(f, t) self.curframe_locals = self.curframe.f_locals ... . def do_up(self, arg): if self.curindex == 0: print >>self.stdout, '*** Oldest frame' else: self.curindex = self.curindex - 1 self.curframe = self.stack[self.curindex][0] self.curframe_locals = self.curframe.f_locals self.print_stack_entry(self.stack[self.curindex]) self.lineno = NoneCopy the codeB (break)
Unlike the debugging command for process control, the break command is used to set a breakpoint, which does not affect the program interrupt status immediately, but may affect subsequent interrupts. When the line event occurs, in addition to stop_here will increase the break_here condition judgment, set breakpoint implementation is relatively simple, here mainly introduces the function to set breakpoint, how to make the code execution to the first line of the function interrupt.
When setting a breakpoint, lineno of the breakpoint is for the first line of the function:
Def do_break(self, arg, temporary = 0): break func def do_break(self, arg, temporary = 0):... if hasattr(func, 'im_func'): func = func.im_func funcname = code.co_name lineno = code.co_firstlineno filename = code.co_filenameCopy the codeWhen the line event is executed on the firstline of the function, the breakpoint has not been actively set, but the firstline of the function, co_firstlineno, hits the breakpoint, so the breakpoint validity continues.
def break_here(self, frame):
...
lineno = frame.f_lineno
if not lineno in self.breaks[filename]:
lineno = frame.f_code.co_firstlineno
if not lineno in self.breaks[filename]:
return False
# flag says ok to delete temp. bp
(bp, flag) = effective(filename, lineno, frame)
Copy the codeThe validity of breakpoints is determined by the effective method, which handles configurations such as ignore and enabled, and the validity of function breakpoints is determined by the checkfuncName method:
def checkfuncname(b, frame): """Check whether we should break here because of `b.funcname`.""" ... # Breakpoint set via function name. ... # We are in the right frame. if not b.func_first_executable_line: # The function is entered for the 1st time. b.func_first_executable_line = frame.f_lineno if b.func_first_executable_line ! = frame.f_lineno: # But we are not at the first line number: don't break. return False return TrueCopy the codeFunc_first_executable_line is not yet func_first_executable_line when the line event occurs on the first line of the function, so the current line number is set, and the breakpoint is in effect, so the function is interrupted on the first line. When line goes to the next line after the number of rows, because func_first_executable_line already has a value and is definitely not equal to the current line number, break_here is judged to be invalid and will not break.
The example analysis
A simple Python code debugging example is used to review how these commands work:
In the console, the command line executes the snapshot:
The python code is actually executed from the first line, but because pdb.set_trace() is called from __main__, it is actually mounted to the PDB trace function from the next line of set_trace to start frame interrupt control.
This Python code will pass through three frames:
- The bottom root of the frame0, __main__, which contains a broken for loop, has a back frame of None
- On the second layer frame1, enter the frame where the func method is located. The back frame of frame1 is frame0
- Top frame2, enter frame2 where add is located, and the back frame of frame2 is frame1
Debugging process:
- Trace the frame (root frame0) on which __main__ is located, raising the line event at line 20
- Frame0 triggers the line event on line 21. The line number is equal to the last traced line number +1. Stop_here is established and the wait is interrupted
- The user enters unt, same as 2, and breaks at line 22
- When the user enters the unt command enter, the code is traced to frame0 and the line number is less than the last traced line number +1 (23). The stop_here is not established and the execution continues
- Trigger the line event at line 24, the line number is greater than the last trace line number +1 (23), stop_here is established, interrupt wait
- The user enters the s command enter, the code traces to Frame1 and fires the call event at line 12. Step execution is the same granularity as sys. setTrace, and is interrupted and waits at line 12
- When the user sets a breakpoint for the add function, the breakpoint on line 1 (line 7) is added to the breakpoint list
- The user enters c, stop_here always returns false, and continues tracing until the line event is triggered at line 8. Although line 8 is no longer in the breakpoint list, the current function frame firstlineno is in and valid, so it interrupts and waits at line 8
- When the user enters the r command enter, stop_here returns false in the following line event processing until the return event is triggered at line 10, at which point the returnFrame is the current frame, and the wait is interrupted at line 10
- When the user types up, the stack frame switches its index forward to the previous frame frame1, where add is called on line 13 func
- When the user enters the down command, the index of the stack frame is switched back and forth to return to the current frame
- The user enters the n command and runs until the next trace on line 14. This trace is on frame1 and can be traced back to botFrame, so it is interrupted on line 14
- The user enters the n command and runs to the next trace on line 14 (return event), still in the current frame1, interrupted
- The user enters the n command and runs until the next trace 24 lines (return event). This trace is botFrame (frame0) and is interrupted
- The user enters the n command, and the frame0 command is executed.
summary
The implementation of the PDB provided by the Python standard library is not complicated. This article explains the core logic in the source code. If you know how it works, you can customize or rewrite a Python debugger yourself. In fact, many common ides in the industry, such as pycharm and vscode, do not use the standard PDB and have developed their own Python debugger to better accommodate the IDE. However, understanding the PDB and adapting and customizing debuggers on the PDB to meet your debugging needs is a cost-effective way to do so.
The end-to-end debugging capability of MNN workbench is also realized based on native PDB, and supports various R&D scenarios of alibaba Group’s inner end computing, which greatly improves the efficiency of algorithm r&d deployment. Go to www.mnn.zone to download MNN Workbench.
