Someone following the LibreOffice development closely might know that we have actually two types of performance tests. I’ll will talk mostly about the newer in-tree tests but will also give a short explanation of the out-of-tree tests to explain the difference and when to use which of the two.
Out-of-tree performance tests
The out-of-tree tests have been written by Matúš Kukan and Michael Meeks quite some time ago and they have been running regularly on a LibreOffice server. Sadly until now they have not been as visible as necessary to alert us of newly introduced regressions. Thanks to some awesome work by Norbert we have now a nice website that shows the results of all performance tests.
Technically these tests are taking a full LibreOffice (built with optimization and symbols) and execute it under callgrind. We do this for import and export of some sample documents and later analyse the generated callgrind log files.
This allows to quickly generate numbers for import and export tasks. However it always includes a full LibreOffice start-up and shut down. Additionally it tests a whole bunch of unrelated parts of the import and export and makes it nearly impossible to test the performance of any dynamic operation. The “new” tests (they are already more than half a year old) help to solve a number of these problems.
In-tree performance tests
Matúš together with Laurent Godard added another kind of performance tests to solve some of the limitations mentioned above. They can be run in our build as part of make perfcheck and use the LibreOffice testing framework.
Instead of testing a whole LibreOffice start-up, import, export, shut down cycle under callgrind these tests just test some calls that you annotate in the code. It allows us to quickly write tests that just measure the performance of the small part of code that we are really interested in. It also allows us to test more or less user input by calling the methods, that the user input would call, directly in the test. Obviously this gives us more flexibility in testing the performance of LibreOffice and opens the possibility to find performance regressions in code outside of the import and export filters.
These tests in itself are sadly not that useful unless they are executed regularly and the results are easily accessible. Norbert has been working recently on making these results more visible and run them regularly on TDF hardware. You can find the latest results now also at perf.libreoffice.org together with the out-of-tree tests.
To show you how easy it is to add a new performance test I’ll document here the steps necessary for adding a test case to sc/qa/perf/scperfobj.cxx.
- add a new class method
- register the new class method inside the CPPUNIT_TEST_SUITE/CPPUNIT_TEST_SUITE_END block
- Use callgrindStart(); before the part that you want to include in your performance test
- Use callgrindDump(“module:a_nice_id”); after the last call that should be included in your performance test
You can even use several callgrindStart(), callgrindDump(“”) pairs in the same cppunit test. The new test case should appear automatically on the status page once the tests are run the next time.
As can be seen it is quite easy to write a new performance test and hopefully this motivates people to add new tests.
When should I use an out-of-tree test now?
Reading my section about the in-tree tests I might sound that you should always use an in-tree test. As a general rule you should put tests where you want to test the performance of the whole import or the export into the out-of-tree tests and all other tests into the in-tree tests.
However try to test as close to user interactions as possible. Testing some code that is never exposed directly to the user might give misleading results as some of its code might have been moved to another function that is always called with it, or some code was moved from a related method to the one that you are testing.
Helping with the web development
On a only slightly related note: We are still actively looking for an experienced web developer who helps us improve the results page and the new devcentral webpage. If you are interested in that task please contact Norbert or myself.