I’m convinced that any time we want to build a large application we need automated testing. If we can’t run our tests automatically, we can’t scale development. The testing time will increase for each feature we add and eventually our testing cycle will take longer than the worth we get out of the new features we add.
With automated testing, we can increase our confidence that both our new and old functionality works. This frees up time from testing that we can spend on delivering value, which is why we are in this to begin with.
Unit testing is great to quickly iterate quickly on the component level but it doesn’t help us know if our components work together. If we don’t run integration tests, things might break when we connect them no matter how good unit tests coverage we have.
This is the reason I tried the flutter integration test framework almost immediately after getting started with Flutter.
Testing in CI
The sooner we detect an issue the easier it is to connect to the cause. Imagine that you got an error right after you write a new line of code. This is what makes our modern IDEs so powerful, they statically analyse our syntax and alerts us to errors. Unfortunately, it cannot (yet) run our code live and give us error reports on integration issues.
Well, this is where continuous integration comes in. If we do small commits and push often, we can at least get feedback for every couple of small changes. It’s not immediate but better than spending two weeks coding only to find out that something you’ve done past two weeks breaks the build.
A company called NeverCode provides a CI system called CodeMagic specifically catering for Flutter builds. The reason I mention them specifically is that they solve the issue of where to run your tests.
The biggest problem running integration tests in CI is that we need a device to run it on and the CI won’t be running inside an iPhone or Android device but rather on a Linux machine somewhere in the cloud. CodeMagic seems to smoothly integrate AWS Device Farm to solve this problem.
AWS Device Farm and Sylph
AWS Device Farms allows you to upload iOS and Android apps to AWS and execute an automated test script on the app on real devices.
The AWS Device farm also comes with a free first 1000 minutes. That’s enough to get you started before investing hard cash.
$0.17/device minute is pretty cheap compared to the overhead and mental drain, not to mention actual lost time, of having someone manually do the same job. But the big win also comes with the speed of feedback.
The problem with Flutter is, it doesn’t allow testing with Appium or similar libraries. Sylph fixes this comes in. Sylph wraps flutter driver inside something that looks like Appium and provides an easy interface for us to run our flutter driver tests inside AWS Device Farm.
How to run Sylph with AWS
Before we start you need to:
- Create a flutter driver test (see flutter integration guide)
- Create an AWS IAM account (service account to authenticate with)
- Install AWS CLI and authenticate
- Install Sylph package globally
The Sylph package page has good detailed instructions for this. CodeMagic also has a good guide to get started although it’s geared more towards running it in their CI rather than locally.
The first time I authenticated I couldn’t list the projects properly though and had to authenticate again with us-west-2 as the region.
When everything is configured, we simply run sylph to run the tests.
Version incompatibility
Unfortunately, my first run failed. I’m building my app on flutter v1.17.5 right now and will be upgrading soon enough again. The Sylph package, however, triggers the test run using a very old v1.12.13 build of flutter. For me this simply didn’t work as my flutter driver version requires other dependencies and I got this error:
Because every version of flutter_driver from sdk depends on intl 0.16.0 and japanese_reading depends on intl ^0.16.1, flutter_driver from sdk is forbidden.
The flutter version is unfortunately hard-coded in the Sylph test_spec.yaml configuration.
To get around this, I forked sylph and registered from my local folder. I had to double-check the release number on the Flutter SDK release list to make sure the version format I used matched the available releases.
# - FLUTTER_VERSION='v1.12.13+hotfix.8-stable' # replace this row - FLUTTER_VERSION='1.17.5-stable'
~/code/sylph$ pub global activate --source path $(pwd)
This got the tests to run using the later version of flutter and it passed gallantly!
Test across devices
Once I got it working, it was on to the real goal, verifying that it works across a range of different devices. Running sylph --devices=android will give you a list of available devices that you can then list in your device pool.
device_pools:
- pool_name: android pool
pool_type: android
devices:
- name: HTC One M8 (AT&T)
model: 6268A
os: 4.4.4
- name: Samsung Galaxy J2
model: SM-J260M
os: 8.1.0
- name: Samsung Galaxy J7 (2018)
model: SM-J737U
os: 8.0.0
- name: Sony Xperia XZ1
model: "{G8342,G8343}"
os: 8.0.0
Having added this in and re-running sylph I got my app successfully tested across the various devices. Alas, I ran into the isolates issue but at least allowed me to verify that the app could be started alright on the four selected devices.
Available devices
I had hoped to try on a wider range of manufacturers but most devices are Google or Samsung ones. They have a few Sony as well. See the full device list.
Final words
I did the above to try and address some cross-device crashes I’ve had on Android in JReader lately and now I can rest assured that it works at least on a subset of the known devices out there. Unfortunately, since November 2018 there is a github issue on the flutter driver causing the app to halt during tests which I got stuck on this time around as well. It looks as though it will be resolved though, so fingers crossed we can do some real integration testing soon enough.
Do you use something like this already? Do you use local devices? I’ve been wanting to do the above and use Flutter Driver more since I started to safely be able to keep up a quick release cycle. Any tips or experiences you can share would be great, do drop a line below! Happy testing.