Someone following the LibreOffice development closely might know that we have actually two types of performance tests. I’ll will talk mostly about the newer in-tree tests but will also give a short explanation of the out-of-tree tests to explain the difference and when to use which of the two.
Out-of-tree performance tests
The out-of-tree tests have been written by Matúš Kukan and Michael Meeks quite some time ago and they have been running regularly on a LibreOffice server. Sadly until now they have not been as visible as necessary to alert us of newly introduced regressions. Thanks to some awesome work by Norbert we have now a nice website that shows the results of all performance tests.
Technically these tests are taking a full LibreOffice (built with optimization and symbols) and execute it under callgrind. We do this for import and export of some sample documents and later analyse the generated callgrind log files.
This allows to quickly generate numbers for import and export tasks. However it always includes a full LibreOffice start-up and shut down. Additionally it tests a whole bunch of unrelated parts of the import and export and makes it nearly impossible to test the performance of any dynamic operation. The “new” tests (they are already more than half a year old) help to solve a number of these problems.
In-tree performance tests
Matúš together with Laurent Godard added another kind of performance tests to solve some of the limitations mentioned above. They can be run in our build as part of make perfcheck and use the LibreOffice testing framework.
Instead of testing a whole LibreOffice start-up, import, export, shut down cycle under callgrind these tests just test some calls that you annotate in the code. It allows us to quickly write tests that just measure the performance of the small part of code that we are really interested in. It also allows us to test more or less user input by calling the methods, that the user input would call, directly in the test. Obviously this gives us more flexibility in testing the performance of LibreOffice and opens the possibility to find performance regressions in code outside of the import and export filters.
These tests in itself are sadly not that useful unless they are executed regularly and the results are easily accessible. Norbert has been working recently on making these results more visible and run them regularly on TDF hardware. You can find the latest results now also at perf.libreoffice.org together with the out-of-tree tests.
To show you how easy it is to add a new performance test I’ll document here the steps necessary for adding a test case to sc/qa/perf/scperfobj.cxx.
- add a new class method
- register the new class method inside the CPPUNIT_TEST_SUITE/CPPUNIT_TEST_SUITE_END block
- Use callgrindStart(); before the part that you want to include in your performance test
- Use callgrindDump(“module:a_nice_id”); after the last call that should be included in your performance test
You can even use several callgrindStart(), callgrindDump(“”) pairs in the same cppunit test. The new test case should appear automatically on the status page once the tests are run the next time.
As can be seen it is quite easy to write a new performance test and hopefully this motivates people to add new tests.
When should I use an out-of-tree test now?
Reading my section about the in-tree tests I might sound that you should always use an in-tree test. As a general rule you should put tests where you want to test the performance of the whole import or the export into the out-of-tree tests and all other tests into the in-tree tests.
However try to test as close to user interactions as possible. Testing some code that is never exposed directly to the user might give misleading results as some of its code might have been moved to another function that is always called with it, or some code was moved from a related method to the one that you are testing.
Helping with the web development
On a only slightly related note: We are still actively looking for an experienced web developer who helps us improve the results page and the new devcentral webpage. If you are interested in that task please contact Norbert or myself.
moggi's blog about Libreoffice hacking 
Thanks for the report! Just a (slightly stupid) question: What do the y axis represent? Bytes (memory consumption) or seconds (time it takes to load)?
They represent callgrind IR. They are a much better way to measure performance as they are less affected by the load of the machine and in an ideal world would show the same result for the same build.
We actually log in master also the time spent in user space and kernel space on linux (see workdir/CppunitTest/*.resource.log)
“You can even use several callgrindStart(), callgrindDump(“”) pairs in the same cppunit test.”
Can multiple callgrindDump calls use the same “module:a_nice_id” (and the cycle counts for the multiple blocks accumulated)?
No. The module is used to group the results on the website and the a_nice_id (best without spaces) is used as file name and as test name in the perf.libreoffice.org website. I think using the same module:a_nice_id would overwrite the previous dumps but we might be able to fix that in the script that analyses the callgrind output.
actually the underlying database has a unique index on suite_name(1) + test_name + sha
so the second and successive ones would be dropped for ‘duplicate keys’
(1) for cppu driven tests, the suite name is generated as cppu_
Pingback: Short update about the performance testing | moggi's blog about Libreoffice hacking
That’s a very good joke! Libreoffice has tons of memory leaks, blocks input for many seconds, without any hint what’s going on! Regular performance tests are a good idea … but we are waiting for effective results. While I write these lines I see a ODS-Document with 200 sheets blocking my whole system … eating 10 GB of memory within 2 hours. Where are the results of the performance tests?