Embedded Testing with PlatformIO - Part 2

In our first part of this testing series we outlined how to set up unit test cases for an embedded project using PlatformIO’s testing capabilities. Our first test cases have been running on the native platform, that is, a developer notebook or a CI server. Now we’re going to look at bringing test code onto devices.

The first and foremost question of course it wether it makes sense to run test code, especially unit tests, on an embedded device. Your embedded device is typically the only place where device characteristics such as interrupts, GPIO and limited RAM/speed play a role, so it might sense to test your device-dependent code there as well. It definitely allows for better level of process repeatability, which we value important when working in an agile way: fail early. If there are bugs within the code, you’d want to know soon instead of too late in the development process.

The big pro about automated firmware unit testing compared to debugging firmware is that one is able to specify testing conditions and automatically test them. When debugging code using an Embedded IDE, one is able to look at all the MCU registers, but each debugging session is an individual setup, so repeatability is definitely limited there. So let’s take a look at some of the unit testing automation the PlatformIO offers.

Support both native and embedded test code

In this post we’re going to show how to run unit test code on a embedded device using PlatformIO’s built-in testing capabilities. We build this second step upon the code of the first blog post. If you followed the steps of the first part, you’re ready - if not, please check out the code from github at the tag blogpost1:

$ git clone --branch blogpost1 https://github.com/thingforward/unit-testing-with-platformio.git

At that point in our code, we have a test/ directory with a first unit test test_first.cpp, which can be run on the native environment. This way, it is not meant for an embedded platform, but only for a CI server or a developer’s notebook. Let’s start to refactor the test/ folder, so that it contains both test code for native and embedded platforms. We shove our first unit test file in a new folder called native:

$ cd test
$ mkdir native
$ mv test_first.cpp native/
$ cd ..

Remembering from the first blog post, we can point PlatformIO’s command line client to execute only for specific environments using -e. Additionally we may supply a filter for selecting only specific file to test. In this case we’d like to execute test cases from only our new native/ folder:

$ pio test -e native -f native                                                             
PlatformIO Plus (https://pioplus.com) v0.10.1
Verbose mode can be enabled via `-v, --verbose` option
Collected 1 items

=============== [test::native] Building... (1/2) ====================
Please wait...

=============== [test::native] Testing... (2/2) ===================
test/desktop/test_first.cpp:21:test_mod1    [PASSED]

-----------------------
1 Tests 0 Failures 0 Ignored
OK

As expected, it found our unit test file, built and executed it. Now it’s time to extend the sample to run on an embedded device as well.

Creating a test skeleton for an embedded device

We’ve chose nodemcuv2 as the environment for our NodeMCU board, but a plethora of other boards are supported by PlatformIO as well, so you might want to put in your board choice here (perhaps you already did in part 1). Here we’ll continue with nodemcuv2, we’re going to create a new folder under test/:

$ mkdir test/nodemcuv2

and put in a file called test_main.cpp with the following content:

// compile only if in correct env/testing situation
#if defined(ARDUINO) && defined(UNIT_TEST)

#include <Arduino.h>
#include "unity.h"

// setup connects serial, runs test cases (upcoming)
void setup() {
  delay(2000);

  //
  UNITY_BEGIN();

  // calls to tests will go here

  UNITY_END();
}

void loop() {
  // nothing to be done here.
}

#endif

In the end, the above snippet is an Arduino sketch with setup() and loop() functions. The file is surrounded with

#if defined(ARDUINO) && defined(UNIT_TEST)
(…)
#endif 

This makes sure that the code is only compiled under the Ardunio environment (and not the native on), and only for unit-testing purposes. This is important, otherwise calls to pio run might fail because compilation for different environments and testing/non-testing code block get messed up.

So now we’re able to run our unit tests for both environments, native and nodemcuv2:

$ pio test -e native -f native
(…)
test/native/test_first.cpp:21:test_mod1 [PASSED]

-----------------------
1 Tests 0 Failures 0 Ignored
OK
(…)
test:native/env:native  [PASSED]
test:nodemcuv2/env:native   [IGNORED]

and:

$ pio test -e nodemcuv2 -f nodemcuv2

PlatformIO Plus (https://pioplus.com) v0.10.1
Verbose mode can be enabled via `-v, --verbose` option
Collected 2 items

=========== [test::nodemcuv2] Building... (1/3) =========================
Please wait...
Detected non-PlatformIO `test_filter` option in `[env:nodemcuv2]` section

============= [test::nodemcuv2] Uploading... (2/3) ========================
Please wait...
Detected non-PlatformIO `test_filter` option in `[env:nodemcuv2]` section

============= [test::nodemcuv2] Testing... (3/3) ===========================
If you don't see any output for the first 10 secs, please reset board (press reset button)

-----------------------
0 Tests 0 Failures 0 Ignored

======================= [TEST SUMMARY] ==============================
test:native/env:nodemcuv2   [IGNORED]
test:nodemcuv2/env:nodemcuv2    [PASSED]
======================== [PASSED] Took 13.33 seconds ======

Calling pio test for NodeMCU takes significantly longer, because PlatformIO also uploads the test sketch onto the board (step 2/3), then reboots the device and connects its serial output to the board to evaluate the output of the unit test reports.

Add unit test code

It says 0 Tests, because we did not include any test code yet. This would go between UNITY_BEGIN and UNITY_END section in our sketch. Lets add the unit test to test_main.cpp:

#if defined(ARDUINO) && defined(UNIT_TEST)

#include <Arduino.h>
#include "unity.h"

#include "mod1.h"

void test_mod1() {
        mod1_s  o;
        mod1_init(&o);

        mod1_set_a(&o, 17);
        TEST_ASSERT_EQUAL(mod1_get_a(&o), 17);

        mod1_process(&o);
        TEST_ASSERT_EQUAL(mod1_get_a(&o), 18);
}

void setup() {
  delay(2000);

  UNITY_BEGIN();

  RUN_TEST(test_mod1);

  UNITY_END();
}

void loop() {
  // nothing to be done here.
}

#endif

Now we have the same unit test function as from the first blog post, test_mod1 to test our module code. setup() calls RUN_TEST(test_mod1); to have it executed by the Unity Test Runner. Let’s try:

$ pio test -e nodemcuv2 -f nodemcuv2

(…)
test/nodemcuv2/test_main.cpp:24:test_mod1   [PASSED]
-----------------------
1 Tests 0 Failures 0 Ignored

==================== [TEST SUMMARY] ================================
test:native/env:nodemcuv2   [IGNORED]
test:nodemcuv2/env:nodemcuv2    [PASSED]

That worked out. The test runner picked up our test_mod1 function and executed it. On the embedded device side, we’d also be able to use all board features, i.e. controlling GPIO etc.

Connecting environments and test targets

In the above example, environment names and testing target names are identical, so the -e and -f parameters are identical as well. We can tell PlatformIO to automatically choose the right unit tests, depending on the environment. For this to work, we need to update platformio.ini and set a test_filter variable to both enviromments:

[env:nodemcuv2]
platform = espressif8266
board = nodemcuv2
framework = arduino

; add:
test_filter = nodemcuv2

[env:native]
platform = native

; add:
test_filter = native

From now on it’s sufficient to select an environment using -e for the test command, PlatformIO will look only for matching unit tests. Furthermore, we’re able to run pio test and pio run without environments, and the commands will be executed for all environments:

$ pio test
(…)
test:native/env:nodemcuv2   [IGNORED]
test:native/env:native  [PASSED]
test:nodemcuv2/env:nodemcuv2    [PASSED]
test:nodemcuv2/env:native   [IGNORED]

This concludes the second part. If you’ve been following to this point you have the basis for unit testing your firmware both on a PC development platform, and on an embedded device as well. In the next step we’re going to integrate this in a small continuous integration pipeline. Stay tuned! :)

Andreas

Series