Looking inside PlatformIO's Build System

Today we are going to continue our blogpost series about PlatformIO with an explaination about how PlatformIO works under the bonnet, specifically the integration of SCONS, the underlying build system.

A Software construction tool

To give you a quick overview, here is a quote about what SCONS actually is:

  • “SCons is an Open Source software construction tool—that is, a next-generation build tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software.”

As you already know, PlatformIO supports tons of different development boards and therefor can compile for multiple targets from single source without the need of manual construction of Makefiles or similar. SCONS is the part which provides this functionality to PlatformIO.

Once PlatformIO and a project are set up (which you learned about in our previous posts), you may start to build and run your project. This is the time for SCONS.

Integration in PlatformIO

It is integrated and called via Python code which is a big advantage because SCONS itself is also making heavy usage of Python, for example its configuration files are written in Python, too. Think of it as a programmable build system which comes into existence when PlatformIO runs. This can be very helpful to solve complex building tasks. PlatformIO starts by constructing the necessary SCONS command by using variables which are already defined by you. These typically come from definitions in platformio.ini and may look like the following example:

{
  'pioframework': 'arduino', 
  'board': 'nodemcuv2', 
  'pioplatform': 'espressif8266', 
  'pioenv': 'nodemcuv2'
}

After parsing the variables, PlatformIO comes up with the “ready to be executed” command (in this case on a windows box):

python.exe C:\\User\\.platformio\\packages\\tool-scons\\script\\scons 
  -Q -j 4 --warn=no-no-parallel-support 
  -f c:\\user\\.platformio\\penv\\lib\\site-packages\\platformio\\builder\\main.py 
  PIOVERBOSE=0 
  PIOFRAMEWORK=YXJkdWlubw== 
  PLATFORM_MANIFEST=QzpcVXNlcnNcY2Fzc2luaVwucGxhdGZvcm1pb1xwbGF0Zm9ybXNcZXNwcmVzc2lmODI2NlxwbGF0Zm9ybS5qc29u 
  PIOPLATFORM=ZXNwcmVzc2lmODI2Ng== 
  PIOENV=bm9kZW1jdXYy 
  BOARD=bm9kZW1jdXYy 
  BUILD_SCRIPT=QzpcVXNlcnNcY2Fzc2luaVwucGxhdGZvcm1pb1xwbGF0Zm9ybXNcZXNwcmVzc2lmODI2NlxidWlsZGVyXG1haW4ucHk=

For unknown reasons PlatformIO encodes some values with Base64. So, in cleartext it would look like:

python.exe C:\\User\\.platformio\\packages\\tool-scons\\script\\scons 
  -Q -j 4 --warn=no-no-parallel-support 
  -f c:\\user\\.platformio\\penv\\lib\\site-packages\\platformio\\builder\\main.py 
  PIOVERBOSE=0 
  PIOFRAMEWORK=arduino 
  PLATFORM_MANIFEST=C:\User\.platformio\platforms\espressif8266\platform.json 
  PIOPLATFORM=espressif8266 
  PIOENV=nodemcuv2 
  BOARD=nodemcuv2 
  BUILD_SCRIPT=C:\User\.platformio\platforms\espressif8266\builder\main.py

For your interest, you may take a look at the platform.json and main.py file here:

platform.json for our testcase

Now comes the most important part, which is that the non-platform specific main.py which was passed via the -f argument, is set as SCONS’ SConstruct file. This main.py is part of PlatformIO. You may imagine it as the equivalent of a common Makefile, but written in Python which gives it the power to handle complicated build tasks very dynamically. PlatformIO also provides the possibility to execute your own custom Python files before and/or after this step! You will receive more information about this in the next PlatformIO related blogpost. main.py takes the variables listed above as command line parameters, downloads necessary code and tools which are defined in the platform.json (see above) and furthermore sets up a build environment with necessary paths and variables for the actual compilation. If any optional variables are passed, these are also shifted to the environment. It basically generates a common C Makefile, but for now just as skeleton for the upcoming step. After preparing the environment, the SConstruct file is switched to the now platform-specific main.py, i.e. \\.platformio\\platforms\\espressif8266\\builder\\main.py which was set as “BUILD_SCRIPT” earlier on and is executed after. It replaces variables and fills the earlier mentioned Makefile skeleton with data required for platform specific compilation. Once done, there should be a ready-to-be-compiled Makefile for your specific target and parameters. In our case it looked like the following:

bp2.PNG

The above is JSON, but includes all define statements you'd typically find in a Makefile.

Last thing to do is the compilation which PlatformIO now starts with by using the common compilation way :) In the upcoming posts we're going to see how to include own program calls into the build process, i.e. to run a linter or a code coverage. Stay tuned!

Dominik