Anyone who follows my LibreOffice work knows that I spent quite some time on the automated testing frameworks for LibreOffice. As part of this blog I want to use the LibreOffice 6.2 branch-off as a chance to look at current automated testing related statistics. All of the numbers were generated on 2018-11-1, so might already be slightly outdated.
We currently have 4804 different C++ based test cases in 357 different test suites and contain a total of 26215 test asserts. The largest test suite is ucalc (a test suite in Calc that links statically against the main calc library) with nearly 25000 lines of code in several files and 291 test cases.
In addition to our normal C++ based tests, we also have 409 UI tests in 39 test suites. As part of the UI tests we have another 2282 assert statements. A special thanks here to Zdeněk Crhonek who has written 154 commits adding UI tests in 2018. Everyone else together wrote about 45 patches this year touching the UI testing code.
Less well known test concepts in LibreOffice include our callgrind based performance testing (26 out-of-tree test cases and 25 in-tree test cases) and the automatic import and export crash testing with nearly 100000 documents. We managed to run the automatic import and export crash testing, which also generates more than 200000 documents for the export tests, a total of 73 times this year.
Another interesting statistics of this year is that the bug report with the most linked commits is related to the automated testing. As part of the Bug tdf#45904 several brave LibreOffice developers, including Jens Carl and Rahul Gurung, have converted more than 40000 lines of old Java based API tests to C++. In total they have produced more than 60 commits that have been linked to the bug report in 2018 alone and 132 since 2016.
I’m most likely forgetting some additional test frameworks but wanted to give a short overview of all the work that goes into LibreOffice’s automated testing framework. If you are interested in joining the effort please talk to us on #libreoffice-dev or mail the LibreOffice developer mailing list at firstname.lastname@example.org. We have tasks in the automated testing area in C++, python, java and some web related tasks.