Python profilers

CPython’s thread state has a profiler callback hook c_profilefunc. If this callback is set, then it will be called when a function enters and exits. See relevant code here. Node this callback is only invoked for PyTrace_CALL, PyTrace_RETURN, PyTrace_C_CALL, PyTrace_C_EXCEPTION and PyTrace_C_RETURN. CPython also has a tracer callback hook c_tracefunc. This one has more overhead because it is invoked at PyTrace_LINE as well.

sys module function sys.setprofile sets the profiler for current thread. At this moment, cProfile is the builtin deterministic profiler. Check relevant code here. The official documentation claims that cProfile should have low overhead, but according to pyinstrument’s benchmark, cProfile’s overhead can be as high as 84%.

There are quite a few Python profilers out there. cProfile, pyinstrument and py-spy. In my opinion, for a profiler to be useful, it should have below features.

1. Low overhead. Usually statistical profiler has lower overhead than statistical profiler.
2. Async support. It should be able to profile asyncio function as well.
3. Be able to attach to a running process.