I recently watched a talk given by Tim Bunce called 'Performance Profiling with Devel::NYTProf'. The talk is from 2015 but it is still very relevant for Perl profiling today.
More computing sins are committed in the name of efficiency (without necessarily achieving it) than for any other single reason - including blind stupidity.
“A Case Against the GOTO,” Proceedings of the 25th National ACM Conference, August 1972, pp. 791-97.
The complete talk is worth a watch, but if you’re just after some hints and tips then skip to 21:38. I particularly enjoyed the example just after this point in the video about 2 being slower than 1. Spoiler alert, it’s not really slower, but it does highlight the importance of understanding the context within which a line of code executes.
What if the problem you want to profile is intermittent?
You can trigger NYTProf on demand. That means loading it at start-up but then using a signal or similar to tell your program to start benchmarking. Shout out to jes for this tip.
The NYProf docs talk about how to do this. It is along the lines of:
$SIG{USR1} = sub {
DB::enable_profile();
$SIG{ALRM} = sub { DB::finish_profile() };
alarm(60); # Profile for a minute
};
What if profiling at any point isn’t an option?
It might be that you can’t easily make the load problem occur again so even on-demand profiling isn’t super useful. If you’ve got structured log files then they might already have the required hint.
Let’s say your log files are JSON lines. If you’ve got an attribute for package name and line number then you can probably get a hint at the go-slow area:
cat foo.log | jq .message,.line | sort | uniq -c | sort -rn
As you may have guessed, the log file method has a bunch of caveats, you’re program’s slow point might not log at the same rate as your other components if at all, or it might be a slow subroutine, rather than a subroutine that’s called a large number of times, but when other options aren’t available it can be surprisingly helpful.
As an aside, logtop is a really handy utility that will give you counts of lines seen from a tail of an active log file. That might be another useful tool to have in your toolbox for understanding the program performance whilst it’s running.
To wrap up this short meandering post here is a favourite quote of mine about performance optimisation:
The best performance improvement is the transition from the nonworking state to the working state.
Very true!