I recently wrote a blog post about the performance testing in LibreOffice. This post will give a short update and show how our performance testing has helped to find a number of performance problems in LibreOffice Calc.
If we look at the performance testing page for the calc tests we can see some interesting graphs. To explain the changes that we see we will concentrate on the sum_numbers_column test case.
As can be seen in the screenshot we have 7 interesting points. The rest is just the normal variation in the test results that we have to ignore. We will look through each of them as they show different aspects of the performance testing.
The first and most obvious point is the huge jump at point 1 from about 10 million to about 1.2 billion IR cycles. That looks like a horrible performance regression as it seems that we have a 120x performance regression. Looking at the commit range however shows that I made a mistake during review and we changed the test file while adding a new test.
This shows that you should never change the test document used for your performance test as it leads to results that are not comparable anymore. As a general rule it is safer and simpler to use one test document per test case and treat the test files as immutable.
The second spike from 1.2 billion to 1.4 billion happened after switching the memory allocator from our internal allocator to the system allocator. This was done as part of a test to see if we still need our internal allocator or if we could use the system allocator. As it is obvious from looking at the graph our internal memory allocator seems to perform quite a bit better than the system allocator so we switched back to the internal allocator in point 4.
On the positive side while inspecting the first issue I noticed that about 60% of our calls are into the configuration layer reading the OpenCL setting. Luckily after talking to Michael, Tor started looking into it and came up with a simple fix for this problem. This explains the big drop from 1.4 billion to 460 million IR cycles (number 3). The quick spike (number 5 and 6) is the follow-up commit that tried to improve that patch and the following revert after it was noticed that this introduced a performance problem.
The last small drop at number 7 of about 8% is a simple fix for unnecessary memory allocations on the heap that I noticed while looking through the callgrind logs.
I hope that this story shows how useful our in-tree performance tests are and how they help us discover existing performance problems and prevent performance regressions. So the blog post shows how we fixed a serious performance problem, used the performance testing to check the performance impact of a change and how the performance tests help us prevent performance regressions.
Hopefully this motivates more people to add in-tree performance tests to LibreOffice. A special thank goes to Łukasz Hryniuk who has been adding a number of calc performance tests recently.