Statistics about automated testing in LibreOffice

Anyone who follows my LibreOffice work knows that I spent quite some time on the automated testing frameworks for LibreOffice. As part of this blog I want to use the LibreOffice 6.2 branch-off as a chance to look at current automated testing related statistics. All of the numbers were generated on 2018-11-1, so might already be slightly outdated.

We currently have 4804 different C++ based test cases in 357 different test suites and contain a total of 26215 test asserts. The largest test suite is ucalc (a test suite in Calc that links statically against the main calc library) with nearly 25000 lines of code in several files and 291 test cases.

In addition to our normal C++ based tests, we also have 409 UI tests in 39 test suites. As part of the UI tests we have another 2282 assert statements. A special thanks here to Zdeněk Crhonek who has written 154 commits adding UI tests in 2018. Everyone else together wrote about 45 patches this year touching the UI testing code.

Less well known test concepts in LibreOffice include our callgrind based performance testing (26 out-of-tree test cases and 25 in-tree test cases) and the automatic import and export crash testing with nearly 100000 documents. We managed to run the automatic import and export crash testing, which also generates more than 200000 documents for the export tests, a total of 73 times this year.

Another interesting statistics of this year is that the bug report with the most linked commits is related to the automated testing. As part of the Bug tdf#45904 several brave LibreOffice developers, including Jens Carl and Rahul Gurung, have converted more than 40000 lines of old Java based API tests to C++. In total they have produced more than 60 commits that have been linked to the bug report in 2018 alone and 132 since 2016.

I’m most likely forgetting some additional test frameworks but wanted to give a short overview of all the work that goes into LibreOffice’s automated testing framework. If you are interested in joining the effort please talk to us on #libreoffice-dev or mail the LibreOffice developer mailing list at libreoffice@lists.freedesktop.org. We have tasks in the automated testing area in C++, python, java and some web related tasks.

 

Advertisement

Announcing automatically updating daily Windows Libreoffice builds

I announced my work on automatically updating builds and the support for Linux daily builds a while ago. The plan has always been to get this feature ready for Windows release builds in time for the 6.0 release. We are by now about halfway through the feature development cycle for the release and I’m finally able to announce the accomplishment of the next milestone: automatically updating daily Windows builds.

You can fetch the daily builds from the @38-updater build directory. The builds are currently produced on my TDF-owned Windows 7 build bot and will normally be updated during the European night. Currently the builds are en-US only but I plan to add additional languages and a few more disabled features over time.

Note that the archives contain one extra level of directories compared to Linux archives with long directory names which can cause problems with the Windows path length limit. If your build does not start, move the installation up in the directory hierarchy and shorten the directory names.

Limitations

After many comments related to my last blog post about the automatic updater were by people not reading the limitations and plans (or missing plans) of our Linux updater builds (we don’t plan to provide updater enabled release builds on Linux!!) I’m going to list them first this time.

The support that has been finished by now is limited to daily builds that are “installed” in user writeable locations. This is not yet a complete updater for our release builds and is mainly for the QA team and power users that want to test the current daily master builds.

We can not yet update builds installed through MSI and can not handle the user account control which will be necessary for correctly installed LibreOffice. However, similar to the automatically updating Linux builds, this is one step on the way to automatically updating Windows release builds.

Implementation

Many parts of the original blog post about the updater are also true for the Windows updater code. The biggest change is the switch from a two step update, first downloading and creating an updated copy and later replacing the installation with the updated copy, to a one step update, first downloading and later applying the update in place. This change was necessary as Windows prevents modifying files that are already open. Additionally, this makes it significantly easier to implement a secure updater for MSI/MSP installations.

One small additional feature managed to creep into the code while I was waiting for builds to finish. I managed to enable the updater executable GUI so that we can now show a dialog with a progress bar while the update is running. The GUI is already working on Windows and I’m close to finishing the work on the Linux version.

The new LibreOffice Updater GUI.

Apart from these two larger changes, I was mostly fixing the Windows integration. This includes handling windows paths, integrating the updater with the windows crypto system, and making sure that my tooling handles the special cases of our Windows updates. For anyone interested in the details I try to tag all commits with the updater prefix.

Future

As I mentioned in the beginning the overall goal is still an automatic updater for our 6.0 release builds. On the way to this goal I still have two big milestones:

1. Generation of MSP updates and infrastructure to apply them through our MAR file based updater. The Mozilla developers were once working on integrating MSI support into their updater code as part of moz#231062 but abandoned that work.
2. Integration of the updater service to allow installing in the normal Program Files directory and integration with the UAC. Most of the code is already there and already built as part of the updater builds but the integration into our (horrible Perl mess of) packaging and installer code is still missing.

If I manage to finish these two milestones by mid of November in time for the feature freeze and the release engineering team and ESC are satisfied we will have automatically updating builds in time for the LibreOffice 6.0 release. By using the current updater builds and reporting any problems to me, you can help making sure that this will become reality.

As always additional help is welcome. Just ping me (moggi) on IRC (#libreoffice-dev channel on Freenode) and talk to me how you can help. We also always have tasks that don’t require any C++ skills and I’m currently looking especially for interested developers with python and django skills (The whole server side of the automatic updater as well as a large part of the crash reporter is written in python with django). If you know another programming language and want to help with LibreOffice please contact us on IRC or on our developer mailing list (libreoffice@lists.freedesktop.org).

 

 

Announcing automatically updating Linux LibreOffice builds

I’m finally ready to announce LibreOffice daily builds for Linux that integrate our new automatic updater. The work on the automatic updater has been going on for nearly a year now and is finally in a shape that we produce builds on TDF hardware that will automatically update using delta updates.

The current builds are 64-bit Linux builds created on SLES 12.2 and should run on most Linux distros. These builds are .tar.gz based archives that you can extract and just run. Note that we can’t update builds that are placed into locations that are not writeable by the current user (and due to missing support for signing executables and libraries on Linux there are no plans to change that).

Below you will find a short summary how the updating works and some of my open items until this feature can be shipped in release builds (hopefully already in the next major release). If you are not interested in the technical details, head to the daily builds directory and download the build (currently the only way to see that an update happened is through the version string in About dialog but future builds will open my test wiki page after a successful build).

LibreOffice crash reporting – An update

Nearly a year ago I wrote a blog post describing the LibreOffice crash reporting setup and how the crash reporting code works. Since then we have released two minor versions with the crash reporter enabled (5.2 and 5.3) including many bug fix releases and release candidates. According to the crash reporting server a total of 27 versions are recognized and it is time to list some of the statistics surrounding the crash reporter.

During the one year of operation we had about 900,000 uploaded crash reports with around 152,000 of them reported in the last 30 days. The number of crash reports might sound like a lot, but a huge amount (around 430,000) have been reported for the 5.2.4.2 release. We introduced a regression in the shutdown code during the 5.2.4.2 development window which resulted in a crash if the clipboard was not empty during shutdown. Due to the crash reporter we were able to fix the crash quickly and release a fixed 5.2.5 version ahead of schedule. Without the crash reporter we would not have noticed the problem that quickly.

During the early 5.3 releases we saw a spike of random crashes in the window generation code that we could not really explain. After some careful analysis, it was concluded that these crashes happen as a result of GDI resource leaks. Amazingly, on windows you are only able to hold references to 10,000 GDI handles (fonts, bitmaps, …). Some of the GDI leaks have been fixed ([1], [2]) already but we are still chasing some more. At the time of writing, running out of GDI handles is still the number one source of crashes in the 5.3 releases. In an attempt to make it easier to spot these crashes the number of open GDI handles are now reported as part of the metadata by the crash reporter (example in a recent 5.3.2.2 crash). Starting with the 5.3 release the metadata also contains the localization of the installation as we found a few crashes that appear to be locale dependent.

As the transformation from a stack and memory dump to a human readable stacktrace happens on the server we need symbol information to map stack information to source code files, method names and line numbers. These symbol information currently occupy 97GB of data on the server with many of the symbol information being for system libraries. Despite this large amount of symbol information we have still recorded around 100,000 libraries without associated symbol information. These range from system libraries without corresponding debug information, libraries for multimedia codecs, OpenGL and OpenCL drivers to antivirus solutions that inject code into LibreOffice. As part of the symbolization we received around 11,000 different signatures for crash reports. However many of them are actually crashes caused by the same problem and just crash in different places. As an example the 5.2.4.2 crash during shutdown is known with many dozens if not even hundreds of different crash signatures.

Additionally, 179 bug reports have been connected to the crash reporter and 42 commits to the LibreOffice master branch reference the crash reporter. Most of these are attempts to fix reported crashes. Sadly we don’t have a common way to reference the crash reporter yet so there are surely more commits that I could not find with a quick search.

With the next minor release (5.4), it is expected that the Linux version of our signal handler allows crash report generation even when the LibreOffice signal handler overwrites our crash reporting signal handler. All in all the crash reporter works amazingly well and we have been able to fix a number of of serious crashes already. With more time we hope to be able to fix more of the reported crashes and therefore improve the quality of LibreOffice even more.

 

As always, help in fixing and researching some of the reported crashes is highly appreciated. Especially the more obscure crash reports require detective work to trace them back to their cause.

Additionally help with the crash reporting server is always welcome. The server is written in python with django and some javascript. No C++ or LibreOffice code knowledge required 😉

cppunit 1.14 released

It has been quite some time since the last cppunit release – about 3.5 years according to wikipedia – despite having enough changes for quite some time. After seeing some recent LibreOffice commits that would have benefited from the existing unreleased commits I finished the work. Here is a short summary of the changes to cppunit 1.14:

Removed features and platform support

• Removed special support for BeOS
• Removed QT test runner
• Removed MSVC6 test runner
• Removed MSVC6 plugin
• Removed support for pre-C++11 compilers
• Removed support for compilers that don’t support RTTI and default template arguments in standard containers

 

New features

• Support for enum class in template<typename T> CppUnit::assertEquals, and therefore in CPPUNIT_ASSERT_EQUAL
  • An example can be found in the internal examples directory
• New assertion macros: CPPUNIT_ASSERT_LESS, CPPUNIT_ASSERT_GREATER, CPPUNIT_ASSERT_LESSEQUAL, CPPUNIT_ASSERT_GREATEREQUAL
  • Examples for the new assertion macros can be found in the internal examples directory
  • Note that the first argument is always the reference value, so CPPUNIT_ASSERT_LESS(x, y) will check that y is less than x. Sadly this confusing behavior is required to keep compatibility with CPPUNIT_ASSERT_EQUAL.
• TestCaller now takes as a constructor argument any callable that can be put into std::function<void()>
• Based on the TestCaller change, CPPUNIT_TEST_PARAMETERIZED has been added that works similar to CPPUNIT_TEST but takes as a second argument an iteratable whose items can be passed to the test function. std::bind of the test method with each element of the iteratable needs to result in an object that can be stored in std::function<void()>.
  • An example can be seen in the examples directory.
• Various warning and bug fixes
• New and improved build system (thanks to Tomas Chvatal and David Tardon)

 

With the removal of the old features the code now much smaller and the next release will most likely move into the directory of modernizing the code base. Thanks to all contributors who provided patches for this release.

As always bug reports are welcome at the LibreOffice bug tracker.

 

LibreOffice UI test tutorial (part 2): Improve the introspection library

The last part of the UI test tutorial introduced the python part of the UI testing framework. In general most times you only need to care about the python part of the test but sometimes you actually need to extend the introspection part of the framework inside of the LibreOffice code. Many UI elements are already supported in the introspection library either directly or through one of its parent classes.

This blog post will give a short summary of how the introspection code works and how support for new elements can be added. In contrast to the other three parts this blog post requires some understanding of the LibreOffice source code and an idea about VCL, our window toolkit.

Overview

The introspection library provides two simple UNO interfaces to the outside world. The XUITest interface provides access to static data in VCL. Currently this is limited to executing UNO commands and getting access to the current top level window. Access to this object is provided by UITestCase.xUITest in the python test code. In the future we might extend this interface to provide access to more VCL data that is not otherwise available and is useful for the UI testing.

The more interesting one and the one that abstracts all the different UI objects is the XUIObject interface with the corresponding implementation on the LibreOffice side through UIObjectUnoObj, which just wraps a virtual class called UIObject.

The two important methods are get_state and execute. The get_type and get_name methods should also be overridden but the other methods can often just be taken from the base class.

Adding support for a new object

For now we are just going to talk about UI elements that inherit from vcl::Window, so basically most of our GUI elements. On the introspection side the corresponding class is WindowUIObject, which provides the base class for all vcl::Window based introspection abstraction. All classes inheriting from vcl::Window provide the virtual method GetUITestFactory that returns a factory function for the introspection library.

Adding support for a previously not well covered UI object – all have basic coverage through at least the WindowUIObject – normally requires three steps: First adding a GetUITestFactory method to the UI object class, secondly adding the corresponding factory method and finally implementing the introspection wrapper class.

Most of the time the factory method just casts the passed in vcl::Window pointer to the correct type and create the introspection wrapper object. The actual work is in implementing the actual introspection class for the UI element, which should expose the properties that the tests need as well as the operations that can be done on the UI element.

The inheritance from WindowUIObject or one of its subclasses already provides a number of properties as well as some common operations like typing text.

An example for adding support for a previously unsupported object can be found at [1]. This one has a slightly more complicated factory method but does not add more than a basic property.

Non vcl::Window UI objects

Sometimes there are non-vcl::Window UI objects that need to be wrapped which makes everything a bit more complicated. Examples for this are e.g. tree lists or tables where we want to expose elements of the vcl::Window UI object like single tree entries as own objects. As these objects don’t correspond to actual UI objects in our code we have to employ some tricks to provide them to the introspection library. The basic idea behind supporting such pseudo objects is to hold a pointer to the corresponding vcl::Window based UI object through a VclPtr and a way to get the correct property that represents the object we want to cover. Additionally as these objects don’t have an ID we need to override the get_child and get_children methods inside of the the wrapper class for the vcl::Window. The get_children method should return all the IDs of descendants, and the get_child method should return for each of these IDs the corresponding wrapper object.

Handling missing IDs

As has been mentioned in the first part we identify UI elements through a locally unique ID. Normally this ID is loaded from the UI files that describe our dialogs. However for UI elements that are dynamically generated in the code we need to set the ID in the code. For this case the vcl::Window provides the set_id method.

Next

Hopefully this gave a short overview over the C++ part of the UI testing. The next tutorial will cover the directory layout of the python code together with some information about adding normal UNO code to the python tests.

LibreOffice UI test tutorial (part 1): Adding a simple test.

After writing my last blog post about adding a simple Calc UI test I was thinking about the future direction of my documentation effort. As this documentation has to move to the TDF wiki at some point I was looking for a way to document every step in a cohesive way. After looking through the conditional format bugs I think I found one that allows me to write several independent documentation pieces building on each other.

As a result this is the first of four tutorials showing how to write an UI test for tdf#96453. The test will test the interaction of the “Manage Conditional Format” dialog with the “Add Conditional Format” dialog witha  focus on tdf#96453 and some related bugs that I found on the way.

The test document that we will use will have already 2 conditional formats defined. For the final test we want to open the “Manage Conditional Format” dialog, delete one entry and assert that we have only one entry in the table but still two in the document. After that we want to add a new entry through the “Add” button and the corresponding dialog, close that new dialog, assert that we have again two entries in the table and still only 2 entries in the document. Finally close the “Manage Conditional Format” dialog and assert that there are still exactly 2 conditional formats in the document.

Note that I picked one of the more complicated tasks and that it is normally less work to write a test. Also we are still at the early stages of the UI testing framework so we still have to add common code from time to time. As can be seen with the other LibreOffice test frameworks the more common code we add the easier it becomes to add new tests.

The different tutorials cover:

1. Add test, load test file, open dialog, close dialog, close LibreOffice (basically a summary of the previous blog post)
2. Add support for previously unsupported UI elements to the introspection library (C++)
3. Add python code to assert document properties and a short summary of the directory structure
4. Combine everything and make it work

Most of the time step 2 and to some degree step 3 should not be necessary for adding a new test. Over time more UI elements will be covered by the introspection library and more shared code for asserting properties on the document model will be available for reuse.

The Test

This is basically a short extension of the previous blog post. A detailed explanation of the code can be found there. The code for this tutorial can be found in the corresponding commit.

We don’t need to register the new file as we place it in the uitest/calc_tests directory which is registered inside of the calc_demo makefile. The actual test discovery happens through introspection by the test framework.

In contrast to the previous test we are going to load an existing document as that is much faster than creating a conditional format through the UI. Loading a document is already supported by the UITest class. Similar to creating a new document in the start center the method to load the document waits until the document is ready. So we can basically replace the UITest.create_doc_in_start_center method with the UITest.load_file method. This method expects as the argument the URL to the file so we generate the URL through a short helper methods. Better support for document URLs will be added to the UI testing framework in the future.

The conditional format manager is a modal dialog so we use UITest.execute_modal_dialog_through_command with the command ID “.uno:ConditionalFormatManagerDialog“. An easy way to find the correct command ID is to check the definition of the menu (for calc in the menubar file or the popupmenu directory) and find the correct entry.

As we are not yet planning to do any work we can just close the dialog again through the Cancel button and close the document. Both of these actions have been shown already in the older blog post.

Next blog post

We have not done any useful work in the test yet. The problem is that the table used in the “Manage Conditional Format” dialog is not well supported in the introspection library. So for the next tutorial we will improve this support and show how the introspection library is organized and how to add support for a previously unsupported elements.

Writing a LibreOffice Calc UI test

After the blog post describing the design of the UI testing framework I wanted to quickly follow up with a post showing how to write a simple test.

As is a bit expected (in the end I’m still a calc developer 😉 ) currently the support for calc tests is better than for other applications. There is already support for writer, impress and math with different degree of completeness. Especially the special main windows have only a limited support for special operations right now. A future blog post will show how to add introspection support to a special UI element.

In this blog post I will show how to add a simple Calc UI test that creates a range name.

Basics

We will start with a new file that contains just the minimal boilerplate code to have a new test that is discovered by the test framework. We place the file in the uitest/calc_tests directory as this one is listed in uitest/UITest_calc_demo.mk to contain test code.

Inside of any class derived from uitest.framework.TestCase any method with a name starting with test will be executed by the test framework. We will simply name our method for now test_number_format

 

Writing the test

Let us start with the code and explain on the code what each part does.

1  from libreoffice.uno.propertyvalue import mkPropertyValues
2
3  from uitest.framework import UITestCase
4
5  class CreateRangeNameTest(UITestCase):
6
7      def test_create_range_name(self):
8
9          self.ui_test.create_doc_in_start_center("calc")
10
11         self.ui_test.execute_modeless_dialog_through_command(".uno:AddName")
12
13         xAddNameDlg = self.xUITest.getTopFocusWindow()
14         xEdit = xAddNameDlg.getChild("edit")
15
16         actionProps = mkPropertyValues({"TEXT":"simpleRangeName"})
17         xEdit.executeAction("TYPE", actionProps)
18
19         xAddBtn = xAddNameDlg.getChild("add")
20         xAddBtn.executeAction("CLICK", tuple())
21
22         self.ui_test.close_doc()

 

Obviously line 1 and 3 just import the method and class that we need.

Line 5 creates a class that inherits from our UITestCase class and therefore all methods in it will be executed as test cases.

Line 7 defines the test method. Note that the method name needs to start with test to be picked up by the test framework.

The test framework starts LibreOffice and opens the start center so line 9 is a helper method that clicks on the Calc button in the start center and waits until the document is ready.

By line 11 we have a working calc document and need to open the range name dialog. There are a few helper methods for this in the UITest class. All dialogs are opened by sending an UNO command ID (in our case “.uno:AddName“) instead of going through the menus. Currently there are two separate methods for modal and modeless dialogs but the plan is to combine the two methods into one. As soon as this method returns the dialog is ready and we can work with the dialog.

Line 13 and 14 select different UI elements. In line 13 we get a reference to the current top window that has the focus. This is the window that represents the whole dialog. From this UI element we ask for the child with the ID “edit”. The ID is imported from the ui file of the dialog. xEdit now contains a reference to the name edit of the dialog.

Line 16 and 17 send a string that should be typed to the edit. mkPropertyValues in line 16 creates the required com.sun.star.beans.PropertyValue sequence from a dict. In line 17 we send this sequence as argument to executeAction. The first argument to the method is the action that should be executed on the UI element. For the TYPE action the framework generates keyboard events that are sent to the element.

Line 19 and 20 select the “add” button and send a “CLICK” action closing the dialog. At this point we could add any code we want to assert that the range name has actually been created correctly.

Finally in line 22 we close the document. The UITest.close_doc method correctly handles the query dialog requesting confirmation that we want to discard all changes.

Running the test

Before we push the test to master we should obviously check that the test is working. A simple way is to execute all UI tests in the uitest module through make uitest.uicheck. However we would like to see the UI while the test is being executed and the tests are run headless by default.

The solution that I currently use is a simple script starting a test. Obviously the plan is to have a better solution in the framework at some point.

When you execute the test with the visible UI you’ll notice that the test is too fast to see what is going on. A quick and dirty solution is to add python’s time.sleep which stops execution for a bit. The other solution that you can actually leave in the code (only leave a few useful ones in a test) in interesting places is uitest.debug.sleep. This method is ignored unless the test is started in debug mode by passing the --debug parameter to the test.

More

You can see the test in the repository in the uitest/calc_tests/create_range_name.py file. The second test in that file shows how to create a local range name by selecting the second entry in the combo box.

More blog posts and wiki pages documenting the UI test framework are going to follow soon.

If you have any question or suggestion please contact me.

UI testing in LibreOffice

Anyone familiar with testing in LibreOffice knows that for a long time we have had one huge weak point in our testing infrastructure. Until now we were more or less unable to do any UI testing which means we had no chances to test a fairly big part of our code base. We have existing tests for import and export, for performance problems, for normal low level classes and even two frameworks for our UNO API. More or less for everything except for our UI we already have a solution that works quite well.

This post will outline why it took until now to add the UI testing framework and how the design tries to solve some of the weak points of existing UI testing approaches.

Difficulties of UI testing

I have been discussing different designs for an UI testing framework for a few years with other developers at various conferences but never felt like one of the designs would not cause more problems than it solves. I’ll outline below how the new UI testing framework solves some of the problems that I have with other concepts.

My list of requirements for a good (UI-)testing framework are:

• Easy to add a test: This is even more important in an UI testing framework where we ideally want the UX team to add test cases as well.
• Stable across unrelated changes: I don’t want to see failing tests because someone changed an unrelated part of the code. In the UI case this also includes that the test should not fail because someone renames a button or moves an UI element. This requirement is meant to reduce the maintenance costs of the tests.
• As close to the code paths triggered by an user as possible: We want to test the code that is used in production and not some random test code.
• Does not need random sleeps: Needing to add sleeps to deal with asynchronous events makes executing the tests brittle and includes random false positives.
• Low number of false positives: Obviously if the number of false positives is too high developers start to ignore test failures.

 

This is a quite complex list and even the proposed solution might not always be able to fulfill all the requirements but at least it should improve compared to the other considered solutions. The most common solutions for UI testing that people propose are using the accessibility framework or hard coding the path to UI objects. However in my opinion both solutions have some problems that make them unsuitable as testing concepts in LibreOffice.

The hard coded path approach obviously is not stable across UI changes and becomes brittle across different OS versions. The big advantage compared to many other approaches is that you can in theory just record an user interaction and replay it during the tests. Also it would work with any version of LibreOffice without any changes but you pay with increased maintenance costs.

The accessibility based approach suffers from our poor quality of the accessibility code (some people will argue that it is a chance to clean it up) and that the code is not that close to what happens if a non-accessibility user does an action. Sadly the code is not yet stable so at least for a long time until we manage to improve the accessibility code significantly we would suffer from regularly failing tests which in the long run means that developers start to ignore the tests.

Compared to the hard coded path approach the accessibility idea is already an improvement from the maintainability based approach and is used in quite a few programs. However it still does not completely solve the problem of changing dialogs and it is much mroe complicated to write a test.

Design

To solve the problems outlined above the new framework trades upfront costs for easier maintenance later. From a mile away the UI testing approach is based on a bit of newly written introspection code interacting with a testing framework in python through a simple mostly un-typed UNO interface. To identify objects we use the ids that we introduced for loading dialogs from UI files.

Introspection code

The introspection code is a small bit of code around UI objects providing a simple standard interface to each UI object. The interface provides methods to get state and properties of the UI objects as well as sending commands to the objects. For a button a property could be a label and an action could be sending a click command. The introspection library would then call the method on the VCL button object that is called by a real button click.

Obviously we are still not directly taking the same code paths with this introspection code as a real mouse click but we can get as close as possible as any automation framework that is not sending real mouse events can get.

Introspection code for many commonly used UI objects is already available and adding new code for UI objects that are not yet covered is simple. More details on how to add such introspection objects will be added to the wiki page soon.

To solve the problem that many UI operations happen asynchronously we have added some events (for now only for opening dialogs) and will use events as a way to make the tests more reliable. Existing events that get reused are the events for signaling that a document is ready and that the document has been closed.

Finding an object is solved through the ID that we most of the time get from the ui file. Nearly all of them are locally unique so they make for some great identifiers that stay unchanged even when a dialog is reorganized or a new element is introduced. Obviously this would still cause issues when a button is moved from one container widget to another one and we ask the old one for a child with the ID. However as we assume locally unique IDs we just always walk up to the top level window (either of the dialog or the whole document) and search for there.

Python testing framwork

The actual tests are not written in C++ and while I’m normally not a fan of using different languages for writing tests it seems like the right decision here. The tests only interact with LibreOffice through the UNO interface and we want to lower the barrier for adding UI tests compared to our complicated internal C++ tests (which are still much more useful for anything that you need to debug regularly). Additionally debugging of the UI tests can normally happen by invoking the UI operations manually whereas debugging one of our other tests is bound to some function calls.

The testing framework is split into three parts: the test cases, the framework code specific to UI testing and useful UNO code. Splitting the last part out from the rest allows us to collect useful code snippets that can be used in other places or even by external developers interacting with LibreOffice. An example that is already in the framework code is the code used to start LibreOffice and create a socket connection.

Adding a new test mainly requires to create a class that derives from the main test class and through introspection each method whose names starts with test will be executed (an example; the sleeps are just for visual inspection).

To make the life easier there is a UITest object that provides access to some common operations like opening a dialog and waiting for the event that the dialog is ready.

 

Open issues

The implemented test framework trades stability of the tests against less code. For now only the most common UI objects have introspection code, which still covers around 90% of the objects, but there are still corner cases where we need to add more code to the introspection library. Nevertheless you can already write tests for many of the dialogs and can already inspect some of the demo code that I have written.

I’m also still chasing two race conditions that make the tests deadlock or fail. The deadlock case is already handled by a patch that is in gerrit and just not yet pushed to the repository, but the case that Impress tests fail with the new template chooser is still under investigation(the time.sleep in [1] are currently mandatory).

I would also like to move some of the tests([2],[3],[4],[5]) from the uitest module into the correct application module (sc for calc, sw for writer, …) but that needs some more makefile code.

Another open issue is still how to best organize running the uitests under gdb. There is some support already but sadly it is not working as well as it should.

Conclusion

I’m going to add more documentation in form of blog posts and wiki pages soon but you can already get an idea of how the tests work by inspecting the code in the uitest module.

Executing the tests should mainly work (except for the two problems mentioned above) with a make uicheck or make uitest.uicheck after a successful make.

Please report all problems and suggestions to me so that I can integrate them into the new framework.

Crash reporting for LibreOffice

Starting with LibreOffice 5.2 the LibreOffice project will have an automated crash reporting tool with server side analysis of the reports. This has been active in the builds since 5.2.0.0.beta1 and was really working since beta 2.

You can find some of the current results at the 5.2.0 beta result page. A typical report that provides a lot of useful information can be found here.

Client side

On the client we are using breakpad to generate a minidump whenever we are hitting a crash. As we are not wanting to do more than necessary in the crashing process we are checking for crash reports during the next start and ask the user if he wants to upload the crash report. Together with the minidump, which only contains a backtrace and some system information (OS, CPU, loaded modules and time) we are also collecting some information that are helpful in fixing crashes.

Currently this is limited to information about OpenGL with information whether our OpenGL backend was activated, which OpenGL driver, device and vendor where used and hopefully in a bit whether the crash happened in our OpenGL render path. These info will help to quickly detect broken drivers and put them onto our blacklist.

We might be collecting more information in the future but of course there always needs to be a balance between the privacy of the user and the necessary information to identify the causes for crashes.

Server side

Originally my plan was to use Socorro from the Mozilla project but the code is quite heavily connected to their release process. So back in December I naively decided to write an own small django web service that takes the ideas from Socorro and adapts them to the LibreOffice release process. It turned out to be a lot more work but we are finally at a place where the server is usable and provides all the information that we might need for now. There are still a number of open issues and there is still a lot of room for improvements (if you are interested in django web development and want to help the LibreOffice project please contact me).

Also the UX is horrible until now and my lacking skills in CSS are obvious while looking at the web site. Help in fixing many of these issues are greatly appreciated.

Despite these short comings we have already found a few nasty issues in the beta2builds  with the reported crashes and we hope to fix more in the future.

 

Other parts

Currently the crash reporting is only enabled and useful in official TDF builds but I hope to get a working master daily build with enabled crash reporting working at some point.

There is some initial connection support to LibreOffice’s bugzilla instance but we are working on improving this both on the Bugzilla side and on the crash reporting server side. Look forward to some improvements there.

Additionally as I have no Mac and nobody else stepped up there is currently only support for Windows and Linux. Together with support for Android this is something that should be implemented at some point in the future.

Thanks

A special thanks to all the people who helped making this possible, especially Norbert Thiébaud who does an amazing job maintaining the LibreOffice development infrastructure and Christian Lohmaier who has the difficult task of producing the builds and uploading the symbols. Additionally Riccardo Magliocchetti and Aybüke Oezdemir have contributed to the server side.

Of course this would also not possibly without the amazing code produced by the Socorro developers who provided some valuable code and information around the Windows SEH exception handling (especially on 64 bit) and the server side development.