LibreOffice UI test tutorial (part 2): Improve the introspection library

The last part of the UI test tutorial introduced the python part of the UI testing framework. In general most times you only need to care about the python part of the test but sometimes you actually need to extend the introspection part of the framework inside of the LibreOffice code. Many UI elements are already supported in the introspection library either directly or through one of its parent classes.

This blog post will give a short summary of how the introspection code works and how support for new elements can be added. In contrast to the other three parts this blog post requires some understanding of the LibreOffice source code and an idea about VCL, our window toolkit.

Overview

The introspection library provides two simple UNO interfaces to the outside world. The XUITest interface provides access to static data in VCL. Currently this is limited to executing UNO commands and getting access to the current top level window. Access to this object is provided by UITestCase.xUITest in the python test code. In the future we might extend this interface to provide access to more VCL data that is not otherwise available and is useful for the UI testing.

The more interesting one and the one that abstracts all the different UI objects is the XUIObject interface with the corresponding implementation on the LibreOffice side through UIObjectUnoObj, which just wraps a virtual class called UIObject.

The two important methods are get_state and execute. The get_type and get_name methods should also be overridden but the other methods can often just be taken from the base class.

Adding support for a new object

For now we are just going to talk about UI elements that inherit from vcl::Window, so basically most of our GUI elements. On the introspection side the corresponding class is WindowUIObject, which provides the base class for all vcl::Window based introspection abstraction. All classes inheriting from vcl::Window provide the virtual method GetUITestFactory that returns a factory function for the introspection library.

Adding support for a previously not well covered UI object – all have basic coverage through at least the WindowUIObject – normally requires three steps: First adding a GetUITestFactory method to the UI object class, secondly adding the corresponding factory method and finally implementing the introspection wrapper class.

Most of the time the factory method just casts the passed in vcl::Window pointer to the correct type and create the introspection wrapper object. The actual work is in implementing the actual introspection class for the UI element, which should expose the properties that the tests need as well as the operations that can be done on the UI element.

The inheritance from WindowUIObject or one of its subclasses already provides a number of properties as well as some common operations like typing text.

An example for adding support for a previously unsupported object can be found at [1]. This one has a slightly more complicated factory method but does not add more than a basic property.

Non vcl::Window UI objects

Sometimes there are non-vcl::Window UI objects that need to be wrapped which makes everything a bit more complicated. Examples for this are e.g. tree lists or tables where we want to expose elements of the vcl::Window UI object like single tree entries as own objects. As these objects don’t correspond to actual UI objects in our code we have to employ some tricks to provide them to the introspection library. The basic idea behind supporting such pseudo objects is to hold a pointer to the corresponding vcl::Window based UI object through a VclPtr and a way to get the correct property that represents the object we want to cover. Additionally as these objects don’t have an ID we need to override the get_child and get_children methods inside of the the wrapper class for the vcl::Window. The get_children method should return all the IDs of descendants, and the get_child method should return for each of these IDs the corresponding wrapper object.

Handling missing IDs

As has been mentioned in the first part we identify UI elements through a locally unique ID. Normally this ID is loaded from the UI files that describe our dialogs. However for UI elements that are dynamically generated in the code we need to set the ID in the code. For this case the vcl::Window provides the set_id method.

Next

Hopefully this gave a short overview over the C++ part of the UI testing. The next tutorial will cover the directory layout of the python code together with some information about adding normal UNO code to the python tests.

Posted in Libreoffice | Tagged | Leave a comment

LibreOffice UI test tutorial (part 1): Adding a simple test.

After writing my last blog post about adding a simple Calc UI test I was thinking about the future direction of my documentation effort. As this documentation has to move to the TDF wiki at some point I was looking for a way to document every step in a cohesive way. After looking through the conditional format bugs I think I found one that allows me to write several independent documentation pieces building on each other.

As a result this is the first of four tutorials showing how to write an UI test for tdf#96453. The test will test the interaction of the “Manage Conditional Format” dialog with the “Add Conditional Format” dialog witha┬á focus on tdf#96453 and some related bugs that I found on the way.

The test document that we will use will have already 2 conditional formats defined. For the final test we want to open the “Manage Conditional Format” dialog, delete one entry and assert that we have only one entry in the table but still two in the document. After that we want to add a new entry through the “Add” button and the corresponding dialog, close that new dialog, assert that we have again two entries in the table and still only 2 entries in the document. Finally close the “Manage Conditional Format” dialog and assert that there are still exactly 2 conditional formats in the document.

Note that I picked one of the more complicated tasks and that it is normally less work to write a test. Also we are still at the early stages of the UI testing framework so we still have to add common code from time to time. As can be seen with the other LibreOffice test frameworks the more common code we add the easier it becomes to add new tests.

The different tutorials cover:

  1. Add test, load test file, open dialog, close dialog, close LibreOffice (basically a summary of the previous blog post)
  2. Add support for previously unsupported UI elements to the introspection library (C++)
  3. Add python code to assert document properties and a short summary of the directory structure
  4. Combine everything and make it work

Most of the time step 2 and to some degree step 3 should not be necessary for adding a new test. Over time more UI elements will be covered by the introspection library and more shared code for asserting properties on the document model will be available for reuse.

The Test

This is basically a short extension of the previous blog post. A detailed explanation of the code can be found there. The code for this tutorial can be found in the corresponding commit.

We don’t need to register the new file as we place it in the uitest/calc_tests directory which is registered inside of the calc_demo makefile. The actual test discovery happens through introspection by the test framework.

In contrast to the previous test we are going to load an existing document as that is much faster than creating a conditional format through the UI. Loading a document is already supported by the UITest class. Similar to creating a new document in the start center the method to load the document waits until the document is ready. So we can basically replace the UITest.create_doc_in_start_center method with the UITest.load_file method. This method expects as the argument the URL to the file so we generate the URL through a short helper methods. Better support for document URLs will be added to the UI testing framework in the future.

The conditional format manager is a modal dialog so we use UITest.execute_modal_dialog_through_command with the command ID “.uno:ConditionalFormatManagerDialog“. An easy way to find the correct command ID is to check the definition of the menu (for calc in the menubar file or the popupmenu directory) and find the correct entry.

As we are not yet planning to do any work we can just close the dialog again through the Cancel button and close the document. Both of these actions have been shown already in the older blog post.

Next blog post

We have not done any useful work in the test yet. The problem is that the table used in the “Manage Conditional Format” dialog is not well supported in the introspection library. So for the next tutorial we will improve this support and show how the introspection library is organized and how to add support for a previously unsupported elements.

Posted in Uncategorized | 1 Comment

Writing a LibreOffice Calc UI test

After the blog post describing the design of the UI testing framework I wanted to quickly follow up with a post showing how to write a simple test.

As is a bit expected (in the end I’m still a calc developer­čśë ) currently the support for calc tests is better than for other applications. There is already support for writer, impress and math with different degree of completeness. Especially the special main windows have only a limited support for special operations right now. A future blog post will show how to add introspection support to a special UI element.

In this blog post I will show how to add a simple Calc UI test that creates a range name.

Basics

We will start with a new file that contains just the minimal boilerplate code to have a new test that is discovered by the test framework. We place the file in the uitest/calc_tests directory as this one is listed in uitest/UITest_calc_demo.mk to contain test code.

Inside of any class derived from uitest.framework.TestCase any method with a name starting with test will be executed by the test framework. We will simply name our method for now test_number_format

 

Writing the test

Let us start with the code and explain on the code what each part does.


1  from libreoffice.uno.propertyvalue import mkPropertyValues
2
3  from uitest.framework import UITestCase
4
5  class CreateRangeNameTest(UITestCase):
6
7      def test_create_range_name(self):
8
9          self.ui_test.create_doc_in_start_center("calc")
10
11         self.ui_test.execute_modeless_dialog_through_command(".uno:AddName")
12
13         xAddNameDlg = self.xUITest.getTopFocusWindow()
14         xEdit = xAddNameDlg.getChild("edit")
15
16         actionProps = mkPropertyValues({"TEXT":"simpleRangeName"})
17         xEdit.executeAction("TYPE", actionProps)
18
19         xAddBtn = xAddNameDlg.getChild("add")
20         xAddBtn.executeAction("CLICK", tuple())
21
22         self.ui_test.close_doc()

 

Obviously line 1 and 3 just import the method and class that we need.

Line 5 creates a class that inherits from our UITestCase class and therefore all methods in it will be executed as test cases.

Line 7 defines the test method. Note that the method name needs to start with test to be picked up by the test framework.

The test framework starts LibreOffice and opens the start center so line 9 is a helper method that clicks on the Calc button in the start center and waits until the document is ready.

By line 11 we have a working calc document and need to open the range name dialog. There are a few helper methods for this in the UITest class. All dialogs are opened by sending an UNO command ID (in our case “.uno:AddName“) instead of going through the menus. Currently there are two separate methods for modal and modeless dialogs but the plan is to combine the two methods into one. As soon as this method returns the dialog is ready and we can work with the dialog.

Line 13 and 14 select different UI elements. In line 13 we get a reference to the current top window that has the focus. This is the window that represents the whole dialog. From this UI element we ask for the child with the ID “edit”. The ID is imported from the ui file of the dialog. xEdit now contains a reference to the name edit of the dialog.

Line 16 and 17 send a string that should be typed to the edit. mkPropertyValues in line 16 creates the required com.sun.star.beans.PropertyValue sequence from a dict. In line 17 we send this sequence as argument to executeAction. The first argument to the method is the action that should be executed on the UI element. For the TYPE action the framework generates keyboard events that are sent to the element.

Line 19 and 20 select the “add” button and send a “CLICK” action closing the dialog. At this point we could add any code we want to assert that the range name has actually been created correctly.

Finally in line 22 we close the document. The UITest.close_doc method correctly handles the query dialog requesting confirmation that we want to discard all changes.

Running the test

Before we push the test to master we should obviously check that the test is working. A simple way is to execute all UI tests in the uitest module through make uitest.uicheck. However we would like to see the UI while the test is being executed and the tests are run headless by default.

The solution that I currently use is a simple script starting a test. Obviously the plan is to have a better solution in the framework at some point.

When you execute the test with the visible UI you’ll notice that the test is too fast to see what is going on. A quick and dirty solution is to add python’s time.sleep which stops execution for a bit. The other solution that you can actually leave in the code (only leave a few useful ones in a test) in interesting places is uitest.debug.sleep. This method is ignored unless the test is started in debug mode by passing the --debug parameter to the test.

More

You can see the test in the repository in the uitest/calc_tests/create_range_name.py file. The second test in that file shows how to create a local range name by selecting the second entry in the combo box.

More blog posts and wiki pages documenting the UI test framework are going to follow soon.

If you have any question or suggestion please contact me.

Posted in Uncategorized | 1 Comment

UI testing in LibreOffice

Anyone familiar with testing in LibreOffice knows that for a long time we have had one huge weak point in our testing infrastructure. Until now we were more or less unable to do any UI testing which means we had no chances to test a fairly big part of our code base. We have existing tests for import and export, for performance problems, for normal low level classes and even two frameworks for our UNO API. More or less for everything except for our UI we already have a solution that works quite well.

This post will outline why it took until now to add the UI testing framework and how the design tries to solve some of the weak points of existing UI testing approaches.

Difficulties of UI testing

I have been discussing different designs for an UI testing framework for a few years with other developers at various conferences but never felt like one of the designs would not cause more problems than it solves. I’ll outline below how the new UI testing framework solves some of the problems that I have with other concepts.

My list of requirements for a good (UI-)testing framework are:

  • Easy to add a test: This is even more important in an UI testing framework where we ideally want the UX team to add test cases as well.
  • Stable across unrelated changes: I don’t want to see failing tests because someone changed an unrelated part of the code. In the UI case this also includes that the test should not fail because someone renames a button or moves an UI element. This requirement is meant to reduce the maintenance costs of the tests.
  • As close to the code paths triggered by an user as possible: We want to test the code that is used in production and not some random test code.
  • Does not need random sleeps: Needing to add sleeps to deal with asynchronous events makes executing the tests brittle and includes random false positives.
  • Low number of false positives: Obviously if the number of false positives is too high developers start to ignore test failures.

 

This is a quite complex list and even the proposed solution might not always be able to fulfill all the requirements but at least it should improve compared to the other considered solutions. The most common solutions for UI testing that people propose are using the accessibility framework or hard coding the path to UI objects. However in my opinion both solutions have some problems that make them unsuitable as testing concepts in LibreOffice.

The hard coded path approach obviously is not stable across UI changes and becomes brittle across different OS versions. The big advantage compared to many other approaches is that you can in theory just record an user interaction and replay it during the tests. Also it would work with any version of LibreOffice without any changes but you pay with increased maintenance costs.

The accessibility based approach suffers from our poor quality of the accessibility code (some people will argue that it is a chance to clean it up) and that the code is not that close to what happens if a non-accessibility user does an action. Sadly the code is not yet stable so at least for a long time until we manage to improve the accessibility code significantly we would suffer from regularly failing tests which in the long run means that developers start to ignore the tests.

Compared to the hard coded path approach the accessibility idea is already an improvement from the maintainability based approach and is used in quite a few programs. However it still does not completely solve the problem of changing dialogs and it is much mroe complicated to write a test.

Design

To solve the problems outlined above the new framework trades upfront costs for easier maintenance later. From a mile away the UI testing approach is based on a bit of newly written introspection code interacting with a testing framework in python through a simple mostly un-typed UNO interface. To identify objects we use the ids that we introduced for loading dialogs from UI files.

Introspection code

The introspection code is a small bit of code around UI objects providing a simple standard interface to each UI object. The interface provides methods to get state and properties of the UI objects as well as sending commands to the objects. For a button a property could be a label and an action could be sending a click command. The introspection library would then call the method on the VCL button object that is called by a real button click.

Obviously we are still not directly taking the same code paths with this introspection code as a real mouse click but we can get as close as possible as any automation framework that is not sending real mouse events can get.

Introspection code for many commonly used UI objects is already available and adding new code for UI objects that are not yet covered is simple. More details on how to add such introspection objects will be added to the wiki page soon.

To solve the problem that many UI operations happen asynchronously we have added some events (for now only for opening dialogs) and will use events as a way to make the tests more reliable. Existing events that get reused are the events for signaling that a document is ready and that the document has been closed.

Finding an object is solved through the ID that we most of the time get from the ui file. Nearly all of them are locally unique so they make for some great identifiers that stay unchanged even when a dialog is reorganized or a new element is introduced. Obviously this would still cause issues when a button is moved from one container widget to another one and we ask the old one for a child with the ID. However as we assume locally unique IDs we just always walk up to the top level window (either of the dialog or the whole document) and search for there.

Python testing framwork

The actual tests are not written in C++ and while I’m normally not a fan of using different languages for writing tests it seems like the right decision here. The tests only interact with LibreOffice through the UNO interface and we want to lower the barrier for adding UI tests compared to our complicated internal C++ tests (which are still much more useful for anything that you need to debug regularly). Additionally debugging of the UI tests can normally happen by invoking the UI operations manually whereas debugging one of our other tests is bound to some function calls.

The testing framework is split into three parts: the test cases, the framework code specific to UI testing and useful UNO code. Splitting the last part out from the rest allows us to collect useful code snippets that can be used in other places or even by external developers interacting with LibreOffice. An example that is already in the framework code is the code used to start LibreOffice and create a socket connection.

Adding a new test mainly requires to create a class that derives from the main test class and through introspection each method whose names starts with test will be executed (an example; the sleeps are just for visual inspection).

To make the life easier there is a UITest object that provides access to some common operations like opening a dialog and waiting for the event that the dialog is ready.

 

Open issues

The implemented test framework trades stability of the tests against less code. For now only the most common UI objects have introspection code, which still covers around 90% of the objects, but there are still corner cases where we need to add more code to the introspection library. Nevertheless you can already write tests for many of the dialogs and can already inspect some of the demo code that I have written.

I’m also still chasing two race conditions that make the tests deadlock or fail. The deadlock case is already handled by a patch that is in gerrit and just not yet pushed to the repository, but the case that Impress tests fail with the new template chooser is still under investigation(the time.sleep in [1] are currently mandatory).

I would also like to move some of the tests([2],[3],[4],[5]) from the uitest module into the correct application module (sc for calc, sw for writer, …) but that needs some more makefile code.

Another open issue is still how to best organize running the uitests under gdb. There is some support already but sadly it is not working as well as it should.

Conclusion

I’m going to add more documentation in form of blog posts and wiki pages soon but you can already get an idea of how the tests work by inspecting the code in the uitest module.

Executing the tests should mainly work (except for the two problems mentioned above) with a make uicheck or make uitest.uicheck after a successful make.

Please report all problems and suggestions to me so that I can integrate them into the new framework.

Posted in Uncategorized | 3 Comments

Crash reporting for LibreOffice

Starting with LibreOffice 5.2 the LibreOffice project will have an automated crash reporting tool with server side analysis of the reports. This has been active in the builds since 5.2.0.0.beta1 and was really working since beta 2.

You can find some of the current results at the 5.2.0 beta result page. A typical report that provides a lot of useful information can be found here.

Client side

On the client we are using breakpad to generate a minidump whenever we are hitting a crash. As we are not wanting to do more than necessary in the crashing process we are checking for crash reports during the next start and ask the user if he wants to upload the crash report. Together with the minidump, which only contains a backtrace and some system information (OS, CPU, loaded modules and time) we are also collecting some information that are helpful in fixing crashes.

Currently this is limited to information about OpenGL with information whether our OpenGL backend was activated, which OpenGL driver, device and vendor where used and hopefully in a bit whether the crash happened in our OpenGL render path. These info will help to quickly detect broken drivers and put them onto our blacklist.

We might be collecting more information in the future but of course there always needs to be a balance between the privacy of the user and the necessary information to identify the causes for crashes.

Server side

Originally my plan was to use Socorro from the Mozilla project but the code is quite heavily connected to their release process. So back in December I naively decided to write an own small django web service that takes the ideas from Socorro and adapts them to the LibreOffice release process. It turned out to be a lot more work but we are finally at a place where the server is usable and provides all the information that we might need for now. There are still a number of open issues and there is still a lot of room for improvements (if you are interested in django web development and want to help the LibreOffice project please contact me).

Also the UX is horrible until now and my lacking skills in CSS are obvious while looking at the web site. Help in fixing many of these issues are greatly appreciated.

Despite these short comings we have already found a few nasty issues in the beta2builds  with the reported crashes and we hope to fix more in the future.

 

Other parts

Currently the crash reporting is only enabled and useful in official TDF builds but I hope to get a working master daily build with enabled crash reporting working at some point.

There is some initial connection support to LibreOffice’s bugzilla instance but we are working on improving this both on the Bugzilla side and on the crash reporting server side. Look forward to some improvements there.

Additionally as I have no Mac and nobody else stepped up there is currently only support for Windows and Linux. Together with support for Android this is something that should be implemented at some point in the future.

Thanks

A special thanks to all the people who helped making this possible, especially Norbert Thi├ębaud who does an amazing job maintaining the LibreOffice development infrastructure and Christian Lohmaier who has the difficult task of producing the builds and uploading the symbols. Additionally Riccardo Magliocchetti and Ayb├╝ke Oezdemir have contributed to the server side.

Of course this would also not possibly without the amazing code produced by the Socorro developers who provided some valuable code and information around the Windows SEH exception handling (especially on 64 bit) and the server side development.

Posted in Uncategorized | 1 Comment

Short update about the performance testing

I recently wrote a blog post about the performance testing in LibreOffice. This post will give a short update and show how our performance testing has helped to find a number of performance problems in LibreOffice Calc.

If we look at the performance testing page for the calc tests we can see some interesting graphs. To explain the changes that we see we will concentrate on the sum_numbers_column test case.

perf_test

As can be seen in the screenshot we have 7 interesting points. The rest is just the normal variation in the test results that we have to ignore. We will look through each of them as they show different aspects of the performance testing.

The first and most obvious point is the huge jump at point 1 from about 10 million to about 1.2 billion IR cycles. That looks like a horrible performance regression as it seems that we have a 120x performance regression. Looking at the commit range however shows that I made a mistake during review and we changed the test file while adding a new test.

This shows that you should never change the test document used for your performance test as it leads to results that are not comparable anymore. As a general rule it is safer and simpler to use one test document per test case and treat the test files as immutable.

The second spike from 1.2 billion to 1.4 billion happened after switching the memory allocator from our internal allocator to the system allocator. This was done as part of a test to see if we still need our internal allocator or if we could use the system allocator. As it is obvious from looking at the graph our internal memory allocator seems to perform quite a bit better than the system allocator so we switched back to the internal allocator in point 4.

On the positive side while inspecting the first issue I noticed that about 60% of our calls are into the configuration layer reading the OpenCL setting. Luckily after talking to Michael, Tor started looking into it and came up with a simple fix for this problem. This explains the big drop from 1.4 billion to 460 million IR cycles (number 3). The quick spike (number 5 and 6) is the follow-up commit that tried to improve that patch and the following revert after it was noticed that this introduced a performance problem.

The last small drop at number 7 of about 8% is a simple fix for unnecessary memory allocations on the heap that I noticed while looking through the callgrind logs.

I hope that this story shows how useful our in-tree performance tests are and how they help us discover existing performance problems and prevent performance regressions. So the blog post shows how we fixed a serious performance problem, used the performance testing to check the performance impact of a change and how the performance tests help us prevent performance regressions.

Hopefully this motivates more people to add in-tree performance tests to LibreOffice. A special thank goes to Łukasz Hryniuk who has been adding a number of calc performance tests recently.

Posted in Libreoffice | Tagged , , | Leave a comment

Performance tests are now executed regularly in LibreOffice

Someone following the LibreOffice development closely might know that we have actually two types of performance tests. I’ll will talk mostly about the newer in-tree tests but will also give a short explanation of the out-of-tree tests to explain the difference and when to use which of the two.

Out-of-tree performance tests

The out-of-tree tests have been written by Matúš Kukan and Michael Meeks quite some time ago and they have been running regularly on a LibreOffice server. Sadly until now they have not been as visible as necessary to alert us of newly introduced regressions. Thanks to some awesome work by Norbert we have now a nice website that shows the results of all performance tests.

Technically these tests are taking a full LibreOffice (built with optimization and symbols) and execute it under callgrind. We do this for import and export of some sample documents and later analyse the generated callgrind log files.

This allows to quickly generate numbers for import and export tasks. However it always includes a full LibreOffice start-up and shut down. Additionally it tests a whole bunch of unrelated parts of the import and export and makes it nearly impossible to test the performance of any dynamic operation. The “new” tests (they are already more than half a year old) help to solve a number of these problems.

 

In-tree performance tests

Matúš together with Laurent Godard added another kind of performance tests to solve some of the limitations mentioned above. They can be run in our build as part of make perfcheck and use the LibreOffice testing framework.

Instead of testing a whole LibreOffice start-up, import, export, shut down cycle under callgrind these tests just test some calls that you annotate in the code. It allows us to quickly write tests that just measure the performance of the small part of code that we are really interested in. It also allows us to test more or less user input by calling the methods, that the user input would call, directly in the test. Obviously this gives us more flexibility in testing the performance of LibreOffice and opens the possibility to find performance regressions in code outside of the import and export filters.

These tests in itself are sadly not that useful unless they are executed regularly and the results are easily accessible. Norbert has been working recently on making these results more visible and run them regularly on TDF hardware. You can find the latest results now also at perf.libreoffice.org together with the out-of-tree tests.

To show you how easy it is to add a new performance test I’ll document here the steps necessary for adding a test case to sc/qa/perf/scperfobj.cxx.

  1. add a new class method
  2. register the new class method inside the CPPUNIT_TEST_SUITE/CPPUNIT_TEST_SUITE_END block
  3. Use callgrindStart(); before the part that you want to include in your performance test
  4. Use callgrindDump(“module:a_nice_id”); after the last call that should be included in your performance test

You can even use several callgrindStart(), callgrindDump(“”) pairs in the same cppunit test. The new test case should appear automatically on the status page once the tests are run the next time.

As can be seen it is quite easy to write a new performance test and hopefully this motivates people to add new tests.

 

When should I use an out-of-tree test now?

Reading my section about the in-tree tests I might sound that you should always use an in-tree test. As a general rule you should put tests where you want to test the performance of the whole import or the export into the out-of-tree tests and all other tests into the in-tree tests.

However try to test as close to user interactions as possible. Testing some code that is never exposed directly to the user might give misleading results as some of its code might have been moved to another function that is always called with it, or some code was moved from a related method to the one that you are testing.

 

Helping with the web development

On a only slightly related note: We are still actively looking for an experienced web developer who helps us improve the results page and the new devcentral webpage. If you are interested in that task please contact Norbert or myself.

 

 

Posted in Libreoffice | Tagged , | 7 Comments