The Fast, Furious, and Flaky: Continuous Integration with Xcode 10 Parallel Tests

Apple is truly investing in testing tools and technologies, which is a great thing for app developers. In fact, Apple announced parallel testing support for the XCTest framework at WWDC 2018 in its platform state of the union session. There was a separate session on What’s New in Testing, describing new features in Code Coverage, XCTest, and XCUITests, which included parallel testing.

Apple actually announced other parallel testing features last year at WWDC 2017, including the ability to specify multiple destinations to trigger on different simulators or devices.

However, with Xcode 10, we can split tests between different clones of the same simulator. Xcode creates a different runner process under the hood, and each process is assigned specific tests.

This is intended to reduce test execution time dramatically. In this post, we’ll investigate what works in this realm and what areas still need improvements.

Parallel Testing in Xcode 10

With Xcode 10, parallel testing can easily be enabled by editing the UI Test scheme. We can select the Test option and select Options against the test bundle to choose the parallelization option.

By default, there are only two runner processes assigned, but we can increase the number when running from the command line. The various options to run XCTest in parallel has been added to the xcodebuild tool.

These include -parallel-testing-enabled YES|NO; -maximum-parallel-testing-workers NUMBER; and many more. These options allow us to execute tests in parallel.

Does Parallel Testing Work Locally?

Using Xcode 10, let’s try this feature on a pilot project. Let’s create a very simple iOS demo app that has an “Enter” button on the home screen. In our UI test. we’ll launch an app and assert that the “Enter” button is there.

A pretty simple UI test, right? It shouldn’t fail or shouldn’t be flaky. I have created a demo app, available on this Xcode10-ParallelTest-CI GitHub repo. In order to run the test locally, we can edit the scheme and enable Parallel testing from the test action as shown above.

You can access the demo source code and try it yourself by downloading this repository from GitHub. In Xcode, use CMD+U to launch the tests and two clones of the simulator that will run in parallel. There won’t be any issue running them locally.

Yes, parallel testing works well on a local machine!

Does Parallel Testing Works on CI?

Now that we’ve confirmed that this parallel testing feature works on a local machine, let’s configure parallel tests on a Continuous Integration server using Travis CI. In order to do this, create a .travis.ymlfile at the root of the project with the below code:

language: objective-c
osx_image: xcode10
install: travis_wait 30 mvn install
install: true
matrix:
  include:
  - osx_image: xcode10
    env: iPhone X
    script: xcodebuild -project Xcode10-ParallelTest-CI.xcodeproj/ -scheme Xcode10-ParallelTest-CI -destination 'platform=iOS Simulator,OS=12.0,name=iPhone X' clean build test CODE_SIGN_IDENTITY="" CODE_SIGNING_REQUIRED=NO -parallel-testing-enabled YES -parallel-testing-worker-count 4 -quiet
  - osx_image: xcode10
    env: iPhone 6
    script: xcodebuild -project Xcode10-ParallelTest-CI.xcodeproj/ -scheme Xcode10-ParallelTest-CI -destination 'platform=iOS Simulator,OS=12.0,name=iPhone 6' clean build test CODE_SIGN_IDENTITY="" CODE_SIGNING_REQUIRED=NO -parallel-testing-enabled YES -parallel-testing-worker-count 4 -quiet

This will create a build matrix for iPhone X and iPhone 6 simulators to run our tests. When the build is triggered on Travis CI, we can see that all tests passed in just 3 minutes.

You can check out the entire build logs on Travis here.

This confirmed that parallel testing works on Continuous Integration servers as well. So what’s the problem?

Xcode 10 Parallel Testing Issues

Although, parallel testing seems to work well in the above examples, there are still some issues worth considering. In fact, I’ve found that for real-world projects, parallel testing isn’t quite there yet. The major blockers adopting parallel testing are:

• Parallel tests introduced flakiness in the tests. It’s not guaranteed that the test will always pass when run in parallel. It’s true for the example mentioned above in the demo app.
• When one of the tests fails, the parallel test waits for a long time, resulting in the long build time. I intentionally made a test to fail, and it resulted in a 7 minute build. The build log for the failed test can be found here.
• Debugging the test becomes harder, as we can’t reproduce the failures on a local machine.
• Test reporting tools like xcpretty don’t work well with parallel testing, so we lose nice outputs in the build logs. Perhaps this is a limitation of xcpretty, but it still limits the value we can obtain from parallel testing.
• Fastlane scan doesn’t have built-in options to run the test in parallel, so we have to add additional parameters, such as xcargs, and disable the xcpretty formatter to get this working on CI.
• While running the test with xcodebuild, we need to a add -quiet parameter on TravisCI. Otherwise, we get massive build logs. On Travis CI, they have a limit of 4 MB logs. Also, when the test fails, the build logs waits for more than 10 minutes, causing Travis to build to terminate.

These are some of the common issues observed while trying parallel testing features, but the main blocker is flakiness. Until flakiness is resolved, it doesn’t make sense to adopt Xcode 10’s parallel testing feature.

Back to Apple

With this experiment, it’s clear that the parallel test feature is flaky and can not be used until we can resolve the flakiness. Apple engineers are working on developer tools that might clarify the root cause of flakiness and identifying some of the issues that make the tests unstable. Do companies need to buy expensive iMac Pros to run UI tests, as shown in the WWDC demo? Or something else to get rid of the flakiness?

There is a demo repo on GitHub to reproduce the issues mentioned in this tutorial:

$ git clone https://github.com/Shashikant86/Xcode10-ParallelTest-CI 

$ cd Xcode10-ParallelTest-CI 

$ xcodebuild -project Xcode10-ParallelTest-CI.xcodeproj/ -scheme Xcode10-ParallelTest-CI -destination 'platform=iOS Simulator,OS=12.0,name=iPhone X' clean build test CODE_SIGN_IDENTITY="" CODE_SIGNING_REQUIRED=NO -parallel-testing-enabled YES -parallel-testing-worker-count 4 -quiet

Conclusion

Parallel testing clearly has a lot of potential to bring various benefits to development, in terms of reducing build times, but it introduced flakiness into the testing process, especially in UI tests. Hopefully, Apple will fix or provide workarounds to provide more stable testing results. What are your experiences with Xcode 10 parallel tests? Share in the comments below!

Like this post from XCBlog By XCTEQ ? You may also like some of our open source projects on Github or Follow us on Twitter and LinkedIn

Discuss this post on Hacker News.