I have been working recently on finishing the work on a python script for Libreoffice that automatically imports documents and tests if we crash there. The plan is to run this script automatically against all our bugzilla documents on a regular basis.
I have been running similar tests for calc files(the TEST_BUG_FILES) case already before the 3.5 and the 4.0 releases and fixed these crashes with Eike and Kohei before the releases. However this work was done half manually inside of an “unit” test and as soon as it crashed I had to restart the test. As a result of this complex setup it took me between 4 and 6 days to import all 6000+ calc documents. Back then I already had the idea that this task could be automated and moved to a TDF server.
I already tried to convince someone at the Munich hackfest to write such a script as an Easy Hack but had to wait till December until Joren picked up the task. Based on convwatch.py he implemented the first version that has undergone several iterations now and can be found in the Libreoffice dev-tools repository. The script is still looking quite ugly as I have been only adding code and it still contains a large number of debug output for me but the current version should work fine against current Libreoffice master.
After several toolchain problems, one needs a libstdc++ created with at least Linux binutils 18.104.22.168.1 or newer, I finally published the results of the current test run at the Libreoffice developer mailing list. In my latest test run I had a collection of a bit more than 24500 documents with the file extensions cdr, doc, docx, fodg, fodp, fods, fodt, odg, odp, ods, odt, ppt, pptx, pub, rtf, vdx, vsd, wpd, xls, xlsx. While 60 crashes might sound like a lot one has to remember that many of these crashes will never be seen by users. The test is run with a dbgutil build which means that we enforce the exception specification, we switch the standard library to the gcc debug library which has additional assertions and some crashes are related to the special set up of the test. Nevertheless we are planning to fix all these crashes and use the script as part of our automatic testing.
And finally a special thanks to the amazing Libreoffice community who was incredibly supporting in realizing this crazy concept.