Introduction

pymontrace is a production oriented debugger for python.

It enables you to attach to running python programs an inspect their activity without stopping them.

Unlike many other tools it does not require any upfront preparation such as installing a special package in the application in advance.

Note: it's not possible that pymontrace be 100% safe. Use against your production system at your own peril. However, if any lockups or crashes do occur, do let us know.

Quick Start

Installation

pymontrace may be installed from PyPI either using pip or pipx as the situation calls.

pip install pymontrace

pipx install pymontrace

It's also possible to run without an explicit install using pipx run or uvx.

Simple Examples

The following are pretty naïve but illustrative. We make the assumption you are able to find the process ID (PID) of a process you wish to trace and we use the dummy value 1234 here.

Listing probes

List every top level python function and class method of every loaded module. The points at which the func probe may attach.

pymontrace -p 1234 -l 'func:*:start'

Note: on macOS you'll need to use sudo to trace a running process.

List all probe sites of a module:

pymontrace -p 1234 -l 'func:mymodule.*:'

List every line of every loaded module:

pymontrace -p 1234 -l 'line:'

Note: You'll likely see the warning WARN: dropped buffer(s) whizz past and not see many files with names starting near the start of the alphabet. pymontrace drops trace data if too much is produced too quickly. It's best to try to employ some filters.

List the first line of every loaded module

pymontrace -p 1234 -l 'line::1'

List every line of the contextlib module:

pymontrace -p 1234 -l 'line:*/contextlib.py:'

Tracing

Observe the entrance to every python function call. Use CTRL-C to end.

pymontrace -p 1234 -e 'func:*:start {{ print(qualname()) }}'

Observe the entry and exit to every python function from a module, including the arguments it was called with.

pymontrace -p 1234 -e '
func:mymodule.*:start {{ print("->", funcname(), args()) }}
func:mymodule.*:return {{ print("<-", funcname()) }}
'

Count the number of times each function is called during the duration of the trace.

pymontrace -p 1234 -e 'func:*:start {{ maps.calls[qualname()] = agg.count() }}'

Print the minimum and maximum value that a given function was called with

pymontrace -p 1234 -e '
func:mymodule.myfunc:start {{
    arg0 = next(iter(args().values()))
    vars.maxval = agg.max(arg0)
    vars.minval = agg.min(arg0)
}}
pymontrace::END {{
    print("max =", vars.maxval)
    print("min =", vars.minval)
}}
'

Plot a histogram of the ms durations of a top-level function called g:

pymontrace -p 1234 -e '
pymontrace::BEGIN {{
    import time
    import threading
    vars.monotime = time.monotonic_ns
    vars.threads = {}
    vars.get_ident = threading.get_ident
}}

func:__main__.g:start {{
  vars.threads[vars.get_ident()] = vars.monotime()
}}

func:__main__.g:return {{
    end = vars.monotime()
    i = vars.get_ident()
    if (start := vars.threads.pop(i, 0)) != 0:
        maps.calls[i] = agg.quantize((end - start) / 1000 / 1000)
}}
'

Which may output something like the following:

Waiting for process to reach safepoint...
Probes installed. Hit CTRL-C to end...
^CRemoving probes...
Waiting for process to reach safepoint...
calls

  8562249600:
               value  ------------- Distribution ------------- count
                 128 |                                         0
                 256 |@@@@@@@@@@@@@@@@@@@@@@@                  4
                 512 |@@@@@@@@@@@                              2
                1024 |                                         0
                2048 |                                         0
                4096 |                                         0
                8192 |                                         0
               16384 |                                         0
               32768 |                                         0
               65536 |                                         0
              131072 |                                         0
              262144 |                                         0
              524288 |                                         0
             1048576 |                                         0
             2097152 |                                         0
             4194304 |                                         0
             8388608 |                                         0
            16777216 |                                         0
            33554432 |                                         0
            67108864 |                                         0
           134217728 |                                         0
           268435456 |                                         0
           536870912 |                                         0
          1073741824 |                                         0
          2147483648 |@@@@@@                                   1
          4294967296 |                                         0

Tracing Scripts

Visualize the execution flow of a script

pymontrace -c 'myscript.py 1 7' -e '

pymontrace::BEGIN {{
    vars.prefix = ""
}}

func:__main__.*:start {{
    print(vars.prefix, "->", funcname())
    vars.prefix += "  "
}}

func:__main__.*:return {{
    vars.prefix = vars.prefix[:-2]
    print(vars.prefix, "<-", funcname())
}}
func:__main__.*:unwind {{
    vars.prefix = vars.prefix[:-2]
    print(vars.prefix, "<*", funcname())
}}
'

Which could output something similar to:

Probes installed. Hit CTRL-C to end...
 -> <module>
   -> main
     -> f
       -> g
       <- g
       -> g
       <- g
       -> g
       <- g
       -> g
       <- g
       -> g
^C       <* g
     <* f
   <* main
 <* <module>

The pymontrace Command

Name

pymontrace - A production oriented Python debugger.

Synopsis

pymontrace [-h] (-c pyprog | -p pid) (-e prog_text | -l probe_filter)

Description

The pymontrace utility attaches to a running python program or starts ones and injects debugging statements into selected probe sites.

Options

Bugs

The -l option is unable to find nested functions nor functions in modules that have not yet been imported by the target program.

pymontrace Programs

The pymontrace expression language

The pymontrace language is heavily inspired by the D language of DTrace and by bpftrace.

In general it follows the form

program ::= ( probe-spec probe-action )*

probe-spec = probe-name ":" probe-arg1 ":" probe-arg2

probe-action = "{{" python-program-text "}}"

probe-name = "line" | ...

Here is an example pymontrace program:

line:*/some-file.py:123 {{
    print("a =", ctx.a)
    if b is not None:
        vars.b = agg.count()
}}

pymontrace::END {{
    print("b =", vars.b)
}}

The python blocks are run in the context of the probe site. The local and global variables are available on ctx and ctx.globals respectively.

Special Variables

There are a couple of variables that act as namespaces for storing data between the executions of probe actions.

Functions

A number of utility functions are made available in probe context to facilitate common debugging and tracing scenarios. These functions are divided up into two kinds, standard functions and aggregation functions.

Standard Functions

Aggregation Functions

Aggregation functions are special and must be assigned to pymontrace variables or into map entries.

Probes

pymontrace::BEGIN

BEGIN can be useful to set up initial values of variables.

Since it runs in the context of the traced target, it can also be used to do simple state checks.

pymontrace::BEGIN {{ import gc; print(gc.get_stats()); exit() }}

Another use for BEGIN is to import modules and define helper functions. Since importing in python can be very expensive you'll want to avoid that in a tight loop. A way around that would be to import and assign to a variable on vars.

Example showing import in BEGIN:

pymontrace::BEGIN {{
    import base64
    vars.b64decode = base64.b64decode
}}

line:*/target.py:123 {{
    print(vars.b64decode(some_base64_encoded_value))
}}

pymontrace::END

END fires at the end of tracing, including when you hit CTRL+C.

It can be used to print values that were saved in the vars namespace.

line:filepath:lineno

It corresponds to sys.monitoring.events.LINE when tracing Python 3.12 and later. It corresponds to the 'line' trace event when tracing Python 3.11 and earlier.

For example, given a target:

target.py:

import time         # 1
                    # 2
while True:         # 3
    time.sleep(1)   # 4

The following pymontrace program would fire on the just before the time.sleep call:

line:*/target.py:4 {{ ... }}

func:qpath:...

func probes are able to monitor the entry and exit points of any python function.

The qpath segment is the module qualified function path.

To give an example, let's state the qpaths for if the following was imported as import helpers.helpful

helpers/helpful.py:

class Helper:
    def help(self):  # helpers.helpful.Helper.help
        pass

def make_helper():  # helpers.helpful.make_helper
    class Elf:
        def help(self):  # helpers.helpful.make_helper.<locals>.Elf.help
            pass
    return Elf().help()

Note: Using a module path based on a re-export will not match.

For example, assuming the next two files are part of the traced process, the probe spec func:requests.client.exceptions.ClientException.__init__:start will match when ClientException is constructed, whereas func:requests.exceptions.ClientException.__init__:start will not.

requests/exceptions.py:

from client.exceptions import ClientException
__all__ = ("ClientException",)

requests/clients/exceptions.py:

class ClientException(Exception):
    def __init__(*args):
        ...

Probe Sites

The following shows the positions of the probe sites in a representative function

def example():
    # start
    ...
    # yield
    yield
    # resume

    if ...:
        # unwind
        raise Exception

    # return
    return

async def coro():
    ...
    # yield
    await other()
    # resume
    ...

Note: Tracking the unwind event causes some overhead when any exception is raised within the target. Whereas, on Python 3.12 and later, tracking for example start only causes overhead in matching functions.

Note: yield and resume only match on Python 3.12 and later.

Known Issues

macOS

• Tracing a python process on macOS which has either its binary or shared objects under a system path is not possible unless SIP is disabled. This includes

  1. The system python (/usr/bin/python3)
  2. Python installed via the macOS universal installer found on https://python.org

  Versions installed via Homebrew should work

• Attaching to uv builds of Python 3.11 and 3.12 may never succeed. These builds appear to have inlined calls to PyEval_SaveThread.

Supported Platforms

• Linux with kernel major version 6, with glibc 2.28 and above.

  Architectures:

  • x86_64
  • aarch64
  • riscv64

• macOS version 13 and later.

  Architectures:

  • x86_64
  • arm64