Tuning Python applications can dramatically increase performance

Traditional Python profiling tools have limitations. Standard tools like cProfile and most all third party tools (like Python Tools plugin for Microsoft Visual Studio) suffer from common flaws. First, the profiling overhead is high (up to 50%). Second, the information provided is “function-level” i.e. the tool shows how much time was spent within a function, but not actionable “line-level” information to show which exact lines are the bottleneck in a function. Adding “line-level” information to most tools causes the application to run even slower. Third, some tools require modification of the application source code to enable profiling thus disrupting development.

This talk presents an experimental Python profiler. It typically has less than 15% overhead, shows line-level information and does not require modification of application source code. Experiments using it resulted in performance gains of 2x and more. Of course results vary by application, but in a typical application there may be quick optimizations easily identified by this type of profiler.

The talk will briefly describe the basics of what, why and how to profile. The profiler‘s use and results will be shown in the presentation with examples based on real-life applications. Previous experience of working with profilers and trying to optimize an application is a plus, but not required, to gain a better appreciation of the work presented.