Start Embedded Testing with PlatformIO

This blog post is the first part of a series of post on how to set up a test-driven development environment for embedded device testing. We started using PlatformIO for our educational track’s prototyping workshops, and PlatformIO offers integrated testing mechanisms which we’ve found really useful.

These mechanisms are a great surplus compared to other embedded IDEs, but can be a bit tricky at the same time. This is what this post is for, get you acquainted to unit testing on PlatformIO, step-by-step. However it requires some basic knowledge about PlatformIO, and basic C coding capabilities.

Testing can come in different flavors, which serve different purposes:

  • You might want to unit-test the functionality of your C code in terms of device-independent business logic. This can be done on your workstation, or within a CI service, possibly cloud-based.
  • You might want to run unit tests on the device as well, possibly including features which depend on hardware.
  • Additionally, end-to-end test show the integration of hardware and firmware, so that’s another valid testing scenario.

Regarding these scenarios and the structure of the code modules, we have to clearly separate „environments“:

  • Unit tests that run on a server/workstation (typically x86, a „native“ platform), but do not contain device-dependent code (i.e. ARM Cortex M or Arduino Framework code, because that does not compile on a native platform)
  • Unit tests that run on a device as a test harness, i.e. some tests that are processed sequentially, possibly use GPIO and other device features and output results.
  • The final firmware that does not contain any unit tests or testing code.

Luckily, PlatformIO offers mechanisms to distinguish between environments and Testing/Non-Testing code parts. Let’s start with a blank project:

Create a skeleton project, ready for testing

Create a new directory, enter it, and initialize a PlatformIO project there. In our case, we’re using NodeMCU as a board, but of course you may choose others boards. The following code references Arduino-style Code elements, so make sure to choose a board with support for the Arduino Framework.

$ platformio init -b nodemcuv2

Looking at platformio.ini we find an environment definition for nodemcuv2. We add another environment definition, a native environment. Add the following lines to platformio.ini:

[env:native]
platform = native

This makes PlatformIO use the compiler suite native for your OS (i.e. gcc or clang or llvm-gcc). Add a file src/main.cpp and put the following contents in. It’s a blank arduino-sketch with setup() and loop() functions. Note the #ifndef macros around it. This causes our main sketch functions to compile only if we’re NOT unit-testing. Vice versa, in case we’re running unit tests, we do not want our main sketch functions to be compiled.

main.cpp:

#ifndef UNIT_TEST

#include <Arduino.h>

void setup() {
}

void loop() {
}

#endif

This should build and be flashable to a device (albeit it does nothing useful):

$ pio run -e nodemcuv2

Add Testing structure

We want to be able to run unit-test, so the next step is to create some structure to make this testable. PlatformIO already integrates unity, a small c-based unit-testing framework from ThrowTheSwitch.

PlatformIO’s test function runs all code that is located under test/ as unit-tests. We are free to structure this directory with further subdirectories, so that we are able to distinguish sub units.

Let’s make some directories and create a unit test cpp file there:

$ mkdir test
$ mkdir test/test_desktop
$ touch test/test_desktop/test_empty.cpp

Put the following contents into test/test_desktop/test_empty.cpp. It includes unity (which is added to the include path automagically) and defines a main function with a BEGIN/END block for Unit (doing unit test setup etc.) Note the #ifdef UNIT_TEST around it: This should only be compiled if we’re unit-testing, and should NOT be compiled if we’re building the final firmware.

#ifdef UNIT_TEST

#include <unity.h>

int main( int argc, char **argv) {
    UNITY_BEGIN();

    UNITY_END();
}

#endif

PlatformIO includes a test command, which will compile and run our test cases. We pass it an option -e to specify the native environment, because we’d like to run our unit test only native, but not (yet) on our device. Let’s try it:

$ pio test -e native
PlatformIO Plus (https://pioplus.com) v0.9.3
Verbose mode can be enabled via `-v, --verbose` option
Collected 1 items

======= [test::test_desktop] Building... (1/2) ==================
Please wait...

======= [test::test_desktop] Testing... (2/2) ===================

-----------------------
0 Tests 0 Failures 0 Ignored
OK

============ [TEST SUMMARY] =============================
test:test_desktop/env:native    [PASSED]
============ [PASSED] Took 1.53 seconds =====================

Building and running worked out. Of course it did not do anything useful because we don’t have test cases yet. But you might want to execute pio test without the environment option, or other options, or pio run with/without environment to see what happens and how the #if(n)defs have influence on this.

Some sample business logic

We need some sample foo business logic to write our test cases against. Let’s add a module mod1 to our project:

$ mkdir lib/mod1

$ touch lib/mod1/mod1.h
$ touch lib/mod1/mod1.cpp

Place this in lib/mod1/mod1.h. It declares a struct and some sample functions for accessing and processing it:

#ifndef __MOD1_H
#define __MOD1_H

typedef struct mod1_s {
    int a;
} mod1_s;

void mod1_init(mod1_s *obj);

void mod1_set_a(mod1_s *obj, int a);
int  mod1_get_a(mod1_s *obj);

void mod1_process(mod1_s *obj);

#endif

Put the following in lib/mod1/mod1.cpp:

#include <mod1.h>

void mod1_init(mod1_s *obj) {
    obj->a = 0;
}

void mod1_set_a(mod1_s *obj, int a) {
    obj->a = a;
}

int  mod1_get_a(mod1_s *obj) {
    return obj->a;
}

void mod1_process(mod1_s *obj) {
}

That does not do much, but suffices to write unit tests against.

Adding a test case

Our testing source file is empty so far, let’s enhance it by writing a first test case for mod1. Put the following into test/test_desktop/test_empty.cpp:

#ifdef UNIT_TEST

#include <unity.h>

#include "mod1.h"

void test_mod1() {
    mod1_s  o;
    mod1_init(&o);

    mod1_set_a(&o, 17);
    TEST_ASSERT_EQUAL(mod1_get_a(&o), 17);

    mod1_process(&o);
    TEST_ASSERT_EQUAL(mod1_get_a(&o), 18);
}

int main( int argc, char **argv) {
    UNITY_BEGIN();

    RUN_TEST(test_mod1);

    UNITY_END();
}

#endif

This introduces a first unit test function test_mod1(). It sets up a mod1_s struct, puts a value in, checks the value, calls the mod1_process function and checks whether it has increased. For a complete reference to the TEST_ASSERT_… functions available in Unity, please have a look at http://www.throwtheswitch.org/unity/

Caveat: As of now (07/2017, PlatformIO 3.4.0) we need to #include our module in our main sketch too. That seems to be necessary, because the dependency management looks at source files under src/ and includes modules from lib/ automagically, but not if they’re referenced under test/ only. So add this line to src/main.cpp:

#include "mod1.h"

Additionally, we need to call the test case function within main, using the RUN_TEST macro.

Let’s run the unit test:

$ pio test -e native

PlatformIO Plus (https://pioplus.com) v0.9.3
Verbose mode can be enabled via `-v, --verbose` option
Collected 1 items

========= [test::*] Building... (1/2) ============
Please wait...

========= [test::*] Testing... (2/2) =============
test/test_first.cpp:15:test_mod1:FAIL: Expected 17 Was 18   [FAILED]

-----------------------
1 Tests 1 Failures 0 Ignored
FAIL

Our test failed! After calling on mod1_process we expect the value to have incremented from 17 to 18, but it did not. This is due to the mod1_process function being not implemented. Update lib/mod1/mod1.cpp :

void mod1_process(mod1_s *obj) {
    obj->a++; 
}

and test again:

$ pio test -e native

PlatformIO Plus (https://pioplus.com) v0.9.3
Verbose mode can be enabled via `-v, --verbose` option
Collected 1 items

========= [test::*] Building... (1/2) ============
Please wait...

========= [test::*] Testing... (2/2) =============
test/test_first.cpp:21:test_mod1    [PASSED]

-----------------------
1 Tests 0 Failures 0 Ignored
OK

Great, that worked out. So we’re able to work in a test-driven way, that is, first coding (empty) structures and unit tests, and implementing them afterwards, following by refactorings. As often as we like we’re able to run test cases to see if we’re still „green“.

It’s important to keep code parts separated by their purpose:

  • Library code that is independent of embedded device capabilities goes to lib/. It should be unit-tested on the native environment.
  • Embedded code goes into src/. It may use libraries from lib/but it does NOT contain any test code.
  • Test code goes into test/.

This concludes the first part of our PlatformIO testing series. In the next post, we’re going to bring unit tests to the device!