I wrote a blog post last year reporting about our import crash testing with a python script and how we use these results to improve our quality. Since last year we have extended the script and use it regularly on a TDF server.
Export crash testing
The largest change to the script was the new support for export crash testing. Every document that we successfully import is now exported to a number of formats depending on the application that opens it. Similar to the import testing crashes are logged into a file and are made available together with the import crash testing logs.
File Format Validation testing
Based on the exported files we started to run validators against the exported files. Right now we use officeotron for validating exported OOXML files and ODF Validator for validating ODF files. The logs for each document are written to an own file and published on a TDF server. Additionally to prevent introducing validation errors we started recently to use the same validators in our build to validate the files generated by our automated tests. Building with –with-export-validation and two scripts similar to the ones found here a validation error in the exported files will generate a test failure.
Increased document pool
At the time of my last blog post we were using a bit less than 25000 documents for the import testing. Since then we increased that number to about 54000 documents in many more formats. Together with the export testing which generates about 120000 documents with about 90GB of generated files the tests need about 3 days to run.
The reports have been incomplete recently as we have been hit by a bug currently suspected to be in the kernel. Around the 10000th document the load of the server increases without doing any actual work. We are currently trying to determine if it is a single document that is responsible or if it is a combination of a more complex setup. It has been limited to the 18000 writer documents already.
As always I’m looking for people who either want to fix one of the issues or improve the script.