PIO Unit Testing

PIO Unit Testing allows segregating each part of the firmware/program and testing that the individual parts are working correctly. Using PIO Unit Testing Engine you can execute the same tests on the local host machine (native), on the multiple local embedded devices/boards (connected to local host machine), or on both. When testing both, PlatformIO builds firmware on the host machine, uploads into a target device, starts tests, and collects the test results into test reports. The final information will be shown on the host side with informative output and statistic.

Using PIO Remote you can start unit tests on the Remote Device from anywhere in the world or integrate with Continuous Integration systems.

Demo

This is a demo of Local & Embedded: Calculator, which demonstrates running embedded tests on physical hardware (Arduino Uno) and native tests on host machine (desktop).

Learn more about platformio test command.

../_images/pioplus-unit-testing-demo.png

Test Types

Desktop

PIO Unit Testing Engine builds a test program for a host machine using Native development platform. This test could be run only with the desktop or Continuous Integration VM instance.

Note

PlatformIO does not install any toolchains automatically for Native and requires GCC toolchain to be installed on your host machine. Please open Terminal and check that the gcc command is installed.

Embedded

PIO Unit Testing Engine builds a special firmware for a target device (board) and programs it. Then, it connects to this device using configured Serial test_port and communicates via test_transport. Finally, it runs tests on the embedded side, collects results, analyzes them, and provides a summary on a host machine side (desktop).

Note

Please note that the PIO Unit Testing Engine uses the first available Serial/UART implementation (depending on a framework) as a communication interface between the PIO Unit Testing Engine and target device. If you use Serial in your project libraries, please wrap/hide Serial-based blocks with #ifndef UNIT_TEST macro.

Also, you can create custom test_transport and implement the base interface.

Test Runner

Test Runner allows you to process specific environments or ignore a test using “Glob patterns”. You can also ignore a test for specific environments using a test_ignore option from “platformio.ini” (Project Configuration File).

Local

Allows you to run a test on a host machine or on a target device (board), which is directly connected to the host machine. In this case, you need to use the platformio test command.

Remote

Allows you to run test on a remote machine or remote target device (board) without having to depend on OS software, extra software, SSH, VPN or opening network ports. Remote Unit Testing works in pair with PIO Remote. In this case, you need to use the special command platformio remote test.

PlatformIO supports multiple Continuous Integration systems where you can run unit tests at each integration stage. See real PlatformIO Remote Unit Testing Example.

Test Transport

PIO Unit Testing Engine engine uses different transports to communicate with a target device. By default, it uses Serial/UART transport provided by a framework. For example, when “framework = arduino”, the first available Serial will be used. When Native dev-platform is used a native transport will be activated automatically. See example below.

Default baudrate/speed is set to test_speed.

You can also define custom transport and implement its interface:

  • unittest_uart_begin();

  • unittest_uart_putchar(char c);

  • unittest_uart_flush();

  • unittest_uart_end();

Examples

  1. Custom transport for Native platform

[env:mycustomtransport]
platform = native
test_transport = custom

  • Create unittest_transport.h file in project/test directory and implement prototypes above

#ifndef UNITTEST_TRANSPORT_H
#define UNITTEST_TRANSPORT_H

#include <stdio.h>

void unittest_uart_begin() {

}

void unittest_uart_putchar(char c) {
  putchar(c);
}

void unittest_uart_flush() {
  fflush(stdout);
}

void unittest_uart_end() {

}

#endif

  1. STM32Cube HAL and Nucleo-F401RE: debugging and unit testing

Workflow

  1. Create PlatformIO project using the platformio project init command. For Desktop Unit Testing (on a host machine), you need to use Native.

    ; PlatformIO Project Configuration File
;
;   Build options: build flags, source filter, extra scripting
;   Upload options: custom port, speed and extra flags
;   Library options: dependencies, extra library storages
;
; Please visit documentation for the other options and examples
; https://docs.platformio.org/page/projectconf.html

;
; Embedded platforms
;

[env:uno]
platform = atmelavr
framework = arduino
board = uno

[env:nodemcu]
platform = espressif8266
framework = arduino
board = nodemcuv2

;
; Desktop platforms (Win, Mac, Linux, Raspberry Pi, etc)
; See https://platformio.org/platforms/native
;

[env:native]
platform = native

  2. Create a test folder in a root of your project. See test_dir.

  3. Write a test using API. Each test is a small independent program/firmware with its own main() or setup()/loop() functions. Test should start with UNITY_BEGIN() and finish with UNITY_END() calls.

    Warning

    If your board does not support software resetting via Serial.DTR/RTS, you should add at least 2 seconds delay before UNITY_BEGIN(). That time is needed to establish a Serial connection between a host machine and a target device.

    delay(2000); // for Arduino framework
wait(2);     // for ARM mbed framework
UNITY_BEGIN();

  4. Place a test in the test directory. If you have more than one test, split them into sub-folders. For example, test/test_1/*.[c,cpp,h], test_N/*.[c,cpp,h], etc. If there is no such directory in the test folder, then PIO Unit Testing Engine will treat the source code of test folder as SINGLE test.

  5. Run tests using the platformio test command.

Shared Code

PIO Unit Testing Engine does not build source code from src_dir folder by default. If you have a shared/common code between your “main” and “test” programs, you have 2 options:

  1. RECOMMENDED. We recommend splitting the source code into multiple components and placing them into lib_dir (project’s private libraries and components). Library Dependency Finder (LDF) will find and include these libraries automatically in the build process. You can include any library/component header file in your test or program source code via #include <MyComponent.h>.

    See Local & Embedded: Calculator for an example, where we have a “calculator” component in lib_dir folder and include it in tests and the main program using #include <calculator.h>.

  2. Manually instruct PlatformIO to build source code from src_dir folder using test_build_project_src option in “platformio.ini” (Project Configuration File):

    [env:myenv]
platform = ...
test_build_project_src = true

    This is very useful if you unit test independent libraries where you can’t split source code.

    Warning

    Please note that you will need to use #ifdef UNIT_TEST and #endif guard to hide non-test related source code. For example, own main() or setup() / loop() functions.

API

Summary of the Unity Test API:

  • Running Tests

    • RUN_TEST(func)

  • Ignoring Tests

    • TEST_IGNORE()

    • TEST_IGNORE_MESSAGE (message)

  • Aborting Tests

    • TEST_PROTECT()

    • TEST_ABORT()

  • Basic Validity Tests

    • TEST_ASSERT_TRUE(condition)

    • TEST_ASSERT_FALSE(condition)

    • TEST_ASSERT(condition)

    • TEST_ASSERT_UNLESS(condition)

    • TEST_FAIL()

    • TEST_FAIL_MESSAGE(message)

  • Numerical Assertions: Integers

    • TEST_ASSERT_EQUAL_INT(expected, actual)

    • TEST_ASSERT_EQUAL_INT8(expected, actual)

    • TEST_ASSERT_EQUAL_INT16(expected, actual)

    • TEST_ASSERT_EQUAL_INT32(expected, actual)

    • TEST_ASSERT_EQUAL_INT64(expected, actual)

    • TEST_ASSERT_EQUAL_UINT(expected, actual)

    • TEST_ASSERT_EQUAL_UINT8(expected, actual)

    • TEST_ASSERT_EQUAL_UINT16(expected, actual)

    • TEST_ASSERT_EQUAL_UINT32(expected, actual)

    • TEST_ASSERT_EQUAL_UINT64(expected, actual)

    • TEST_ASSERT_EQUAL_HEX(expected, actual)

    • TEST_ASSERT_EQUAL_HEX8(expected, actual)

    • TEST_ASSERT_EQUAL_HEX16(expected, actual)

    • TEST_ASSERT_EQUAL_HEX32(expected, actual)

    • TEST_ASSERT_EQUAL_HEX64(expected, actual)

    • TEST_ASSERT_EQUAL_HEX8_ARRAY(expected, actual, elements)

    • TEST_ASSERT_EQUAL(expected, actual)

    • TEST_ASSERT_INT_WITHIN(delta, expected, actual)

  • Numerical Assertions: Bitwise

    • TEST_ASSERT_BITS(mask, expected, actual)

    • TEST_ASSERT_BITS_HIGH(mask, actual)

    • TEST_ASSERT_BITS_LOW(mask, actual)

    • TEST_ASSERT_BIT_HIGH(mask, actual)

    • TEST_ASSERT_BIT_LOW(mask, actual)

  • Numerical Assertions: Floats

    • TEST_ASSERT_FLOAT_WITHIN(delta, expected, actual)

    • TEST_ASSERT_EQUAL_FLOAT(expected, actual)

    • TEST_ASSERT_EQUAL_DOUBLE(expected, actual)

  • String Assertions

    • TEST_ASSERT_EQUAL_STRING(expected, actual)

    • TEST_ASSERT_EQUAL_STRING_LEN(expected, actual, len)

    • TEST_ASSERT_EQUAL_STRING_MESSAGE(expected, actual, message)

    • TEST_ASSERT_EQUAL_STRING_LEN_MESSAGE(expected, actual, len, message)

  • Pointer Assertions

    • TEST_ASSERT_NULL(pointer)

    • TEST_ASSERT_NOT_NULL(pointer)

  • Memory Assertions

    • TEST_ASSERT_EQUAL_MEMORY(expected, actual, len)