HTTPS on the ESP32 - Server and Client Side

This post first appeared at THNG:STRUCTION and is CC-BY-SA 4.0

Many embedded maker projects involve HTTP or MQTT communication and more often the question arises if one can secure that communication in an easy way. The answer can be tricky and heavily depends on the hardware and the OS or embedded framework being used. In this series we'll take a look at the ESP32 using the Arduino framework, and its capabilities regarding TLS.

Why Not Just Prepend https:// and Be Done?

In regular languages or frameworks for web or desktop development, we're used to just making HTTP calls using https://, or including small code snippets for HTTPS listeners with a server key and certificate. That's (almost) it - securing an API on the transport level isn't that hard, sometime putting an HTTPS-offloading proxy in front also solves the problem.
On embedded devices, however, things look much different:

  • Sometimes MCUs do not offer encryption in silicon. The MCU itself of course can calculate encryption and signatures, but is typically (too) slow at it.
  • Some networking/transceiver chip sets include encryption (e.g. all WiFi chip sets have to, otherwise they could not connect to WiFi..), and they may make this functionality accessible to the outside
  • Some boards feature a secure element - that is a separate security chip for crypto primitives, encryption and storage of keys etc.

Whatever the situation for a given board/framework/OS is, the communication libraries above have to make use of specific features, and the majority of them remain specific to the board/framework/OS.

But what is needed for HTTPS? This is determined by the cipher specification, and involves a bunch of cryptographic functions:

  • A cryptographic hash function, typically SHA256
  • A symmetric cipher: AES, Camellia
  • An asymmetric cipher/signature algorithm, e.g. RSA, DSA, ECDSA, ...
  • on top of that, key exchange: DH, ECDH, at best in ephemeral variants.

Crypto on the ESP32

Luckily, the ESP32 includes some on-chip crypto functionality. Beside what's necessary for all WiFi-related crypto, it has built in AES, RSA, SHA-2, ECC and a random number generator. The Arduino Core of ESP32 includes a port of Arm Mbed TLS (see in tools/sdk/include/mbedtls) and also OpenSSL. These functions can be addressed directly, DFRobot has an example for AES-128-ECB. So the foundation is there.

HTTPS server

As a first step, I'd like my ESP32 to be a web server for a REST API, but using HTTPS. Now most of HTTP server code for Arduino works with EthernetServer or WiFiServer, but there's no TLS or link to the mbedtls port beneath. A sample HTTPS implementation can be found at github.com/fhessel/esp32_https_server. It is marked as work-in-progress, but i gave this a try. It uses OpenSSL to handle TLS and offers a HTTP server API to create endpoints and resources etc. I'm using PlatformIO with a NodeMCU ESP32s to set up the example code:

$ mkdir pio-https
$ cd pio-https
$ pio init -b nodemcu-32s --ide vscode

Clone the code from fhessel's repo, arrange it under /src. We going to need data/, https/, tools/ and the example .cpp/.h under src/.

$ mkdir suppl
$ git -C suppl clone https://github.com/fhessel/esp32_https_server.git
$ cp -rp suppl/esp32_https_server/{data,https,tools,https_server.*} src/

The repo's README has a description of the library, its capabilities and how to set it up. In essence, what needs to be done is:

  • Add WiFi credentials in /data/wifi
  • Create a X509 key and certificate, reformat it as C code so it can be included/compiled into our firmware
$ cd src/data/wifi/
$ mv wifi.example.h wifi.h

Edit wifi.h and put in some valid WiFi WPA2 credentials:

#define WIFI_SSID "<your ssid goes here>"
#define WIFI_PSK  "<your pre-shared key goes here>"
$ cd ../../tools/cert

Edit create_cert.sh and update the X.509 location parts that will make up the DN of the certificate. Run create_cert.sh:

$ . create_cert.sh
Generating RSA private key, 1024 bit long modulus
..........................++++++
...........................................................++++++
e is 65537 (0x10001)
Generating RSA private key, 1024 bit long modulus
..................................................++++++
..................++++++
e is 65537 (0x10001)
Signature ok
subject=/C=DE/ST=NRW/CN=esp32.local
Getting CA Private Key
example.crt: OK
writing RSA key

It creates a CA, then a server key and a request for a certificate, to be signed by the CA. In the end, xxd is used to format the DER-encoded certificate/key into C source code, which gets written to data/cert/cert.h and data/cert/private_key.h. These files, in turn are included by https_server.cpp and used.

Let's compile this:

$ cd ../../..
$ pio run
(...)
Linking .pioenvs/nodemcu-32s/firmware.elf
Building .pioenvs/nodemcu-32s/firmware.bin
Retrieving maximum program size .pioenvs/nodemcu-32s/firmware.elf
Checking size .pioenvs/nodemcu-32s/firmware.elf
Memory Usage -> http://bit.ly/pio-memory-usage
DATA:    [=         ]  13.5% (used 39880 bytes from 294912 bytes)
PROGRAM: [======    ]  57.9% (used 758310 bytes from 1310720 bytes)

Works out well. I'd like to add one more thing which is printing out the IP address on the serial port to see how to reach the device. Edit src/https_server.cpp, go to line #197 and add:

        Serial.println(WiFi.localIP());

Compile, flash and watch the serial port:

$ pio run -t upload && pio device monitor -b 115200
(...)
............................... connected.
192.168.0.101
Creating server task...
Beginning to loop()...
Configuring Server...
Starting Server...
Server Socket fid=0x1000
Server started.

Looks good. We could use a web browser, point to the IP and inspect the certificate. The browser will complain about the certificate being not valid because obviously it's self-signed and thus not trusted. But nevertheless it works!
On the command line, either curl or openssl are nice alternatives to find out more about the encryption:

$ curl -vvv --insecure https://192.168.0.101/
(...)
*   Trying 192.168.0.101...
* TCP_NODELAY set
* Connected to 192.168.0.101 (192.168.0.101) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=DE; ST=HE; L=Darmstadt; O=MyCompany; CN=esp32.local
*  start date: Jul  4 11:53:12 2018 GMT
*  expire date: Jul  1 11:53:12 2028 GMT
*  issuer: C=DE; ST=HE; L=Darmstadt; O=MyCompany; CN=myca.local
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET / HTTP/1.1

curl with a -vvv prints out TLS handshake messages as well as the server certificate. Openssl does the same, with additional levels of details using -debug and -msg.

$ openssl s_client -connect 192.168.0.101:443 -showcerts
(...)
---
Certificate chain
 0 s:/C=DE/ST=HE/L=Darmstadt/O=MyCompany/CN=esp32.local
   i:/C=DE/ST=HE/L=Darmstadt/O=MyCompany/CN=myca.local
-----BEGIN CERTIFICATE-----
MIICHjCCAYcCAQIwDQYJKoZIhvcNAQEFBQAwVzELMAkGA1UEBhMCREUxCzAJBgNV
BAgMAkhFMRIwEAYDVQQHDAlEYXJtc3RhZHQxEjAQBgNVBAoMCU15Q29tcGFueTET
MBEGA1UEAwwKbXljYS5sb2NhbDAeFw0xODA3MDQxMTUzMTJaFw0yODA3MDExMTUz
MTJaMFgxCzAJBgNVBAYTAkRFMQswCQYDVQQIDAJIRTESMBAGA1UEBwwJRGFybXN0
YWR0MRIwEAYDVQQKDAlNeUNvbXBhbnkxFDASBgNVBAMMC2VzcDMyLmxvY2FsMIGf
MA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC3AaSEXqVR+GIPTBA92hkoHViO3J0q
84/ysIBauA3f0b/YZdvl+rDa4CbwDw9UiHvGWZ5uMmiXXIWeF+B3A5iI7u/DOQSh
E1zE5SlTFjgID3gAOcBauY0XatbdYPKDcjq5r3TxPWyWtCr9Y5C0YslFIiB5LMRa
Cf/WHxti3rCJJQIDAQABMA0GCSqGSIb3DQEBBQUAA4GBAKKxgpT/7gX605TqTFWW
nO6BuCUvwgmrXqegMfZlUqRYXLS7cWzKMl8wlqGl5UJBNwJtPXcbjzqs+cnD/RH3
9UgTqpCeFH06XiQgtVQi83XisHxgMytVHTaoLoIKJLRMgNbDFzWpexq6reYJU7Jh
dYgEVq6aVFPd9boeGEe23myQ
-----END CERTIFICATE-----
---
Server certificate
subject=/C=DE/ST=HE/L=Darmstadt/O=MyCompany/CN=esp32.local
issuer=/C=DE/ST=HE/L=Darmstadt/O=MyCompany/CN=myca.local
---
No client certificate CA names sent
---
SSL handshake has read 991 bytes and written 512 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: B852C6DB374B8682DBC8761176128C78F2934A449B7F9F0EB0D4D5FAEAF913CD
    Session-ID-ctx:
    Master-Key: 4D09B700856D8A6019FFA9D38B8E24B01CB531EEBFAF582444F605D460C64435D06FDB189D309E70DE47B848B61F8769
    Start Time: 1530785865
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---
closed

Good to see: TLS 1.2 is used, the Cipher is up-to-date. On my ESP32, it crashed sometimes with a panic. I did not investigate this further but as the README says: it's work in progress. What i wanted to now is the amount of time necessary to set up the TLS connection. A small patch to https/HTTPSServer.cpp does the job:

line #182

  // Start to accept data on the socket
  long d1 = millis();
  int socketIdentifier = _connections[freeConnectionIdx]->initialize(_socket, _sslctx, &_defaultHeaders);
  long d2 = millis();
  HTTPS_DLOG((d2-d1));

This measures the time needed to run HTTPSConnection::initialize, and it prints out:

HTTPSServer->debug: [-->] New connection. Socket fid is:  0x1001
HTTPSServer->debug: 1483

So it's ~1.5 seconds to set up the TLS connection, and that can roughly be measured on the client side as well:

$ time $(echo '' | openssl s_client -connect 192.168.0.101:443 )
(...)
real    0m1.516s
user    0m0.034s
sys 0m0.007s

That's quite some work for the ESP. For single calls to e.g. an API this can be acceptable; for subsequent calls to access a web site on the ESP, it might be too much.

HTTPS client

The Arduino Core for ESP32 contains ports of Arm's Mbed TLS and openssl but they're buried deeper within the SDK, so they typically do not come into appearance when coding for the Arduino Framework with all these WiFiClients, digitalRead, loop and setup. Regarding HTTPS as a client, this isn't necessary. Wherever you'd make a HTTP call (or in general open a tcp connection, you can use the WiFiClient class. Additionally, the ESP32 libraries include a WiFiClientSecure, which interfaces with mbedtls to establish a secure TLS-Connection.

On the surface, WiFiClientSecure is used equal to WiFiClient, as it inherits from this class. What's necessary in addition to this is to set the CA certificate of the CA that signed the server's certificate. This is so that the device can be sure to connect to a server which has received a certificate from the respective CA. (at least better than nothing..)
One important aspect is that - at least in my tests - the ESP seems to be unable to deal with self-signed certificates or a self-created CA. So for tests I've been using a server certificate signed by LetsEncrypt.

There's demo code available for WiFiClientSecure, which makes connection to howsmyssl.com, and it works well. To play around with it, I've taken the code and redirected it my own HTTP server.

So first we need a server to connect to.

Server side

The server code is written in go, packaged as a docker container, and given a server key and certificate. server.go has a single endpoint which prints out some basic TLS information it can obtain from the http.Request:

package main

import (
    "fmt"
    "strings"
    "net/http"
    "log"
)

func TLSInfoServer(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "text/plain")

    if r.TLS != nil {
    fmt.Fprintf(w, "TLS Version: 0x%x\n", r.TLS.Version)
    fmt.Fprintf(w, "Cipher Suite: 0x%x\n", r.TLS.CipherSuite)

    if len(r.TLS.PeerCertificates) > 0 {
        cn := strings.ToLower(r.TLS.PeerCertificates[0].Subject.CommonName)
        fmt.Fprintf(w , "CN: %s\n", cn)
    }
     }

}

func main() {
    http.HandleFunc("/", TLSInfoServer)
    err := http.ListenAndServeTLS(":443", "cert.pem", "privkey.pem", nil)
    if err != nil {
        log.Fatal("ListenAndServe: ", err)
    }
}

Let's put this in a simple (yes, unoptimized) Dockerfile:

FROM debian:9

RUN apt-get update -y -q && apt-get install -y golang

RUN mkdir /app
WORKDIR /app

ADD server.go /app/server.go

CMD [ "go", "run", "server.go" ]

It assumes that the server's key and cert are on the host and can be mounted into the container:

$ docker build -t gohttps:latest .
$ docker run -ti --mount type=bind,source=privkey.pem,target=/app/privkey.pem --mount type=bind,source=cert.pem,target=/app/cert.pem -p 0.0.0.0:6565:443 gohttps:latest

The code does not produce output, just errors to stderr. Let's look at the client code, i take the WiFiClientSecure.ino demo sketch from arduino-esp32 as a basis:

ESP32 https client

$ pio init -b nodemcu-32s --ide vscode
$ curl -L https://raw.githubusercontent.com/espressif/arduino-esp32/master/libraries/WiFiClientSecure/examples/WiFiClientSecure/WiFiClientSecure.ino >src/main.cpp

To get more debug messages on the serial console of the ESP32, add the following to platformio.ini:

build_flags = -DCORE_DEBUG_LEVEL=ARDUHAL_LOG_LEVEL_VERBOSE

Within src/main.cpp, change lines #11 and #12 to match your WiFi SSID and password. If you happen a have a server certificate signed by LetsEncrypt and its recent X3 CA, you do not need to change lines 21-47 which contain the CA certificate in PEM format. Otherwise, from the CA certificate of your servers' CA, openssl x509 -in <file> -text and copy/paste the base64-encoded certificate part into the source code.

Change line #14 to match your server name, change line #80 to match your port (it's 6565 in my case). Change lines #85 and #86 to match the server name, or delete #86 and change #85 to GET / HTTP/1.1 which should do as well. Compiling, flashing and testing yields:

$ pio run -t upload
$ pio device monitor -b 115200

(...)

Attempting to connect to SSID: <YourWiFi>
.Connected to <YourWiFi>

Starting connection to server...
[V][ssl_client.cpp:52] start_ssl_client(): Free heap before TLS 153488
[V][ssl_client.cpp:54] start_ssl_client(): Starting socket
[V][ssl_client.cpp:90] start_ssl_client(): Seeding the random number generator
[V][ssl_client.cpp:99] start_ssl_client(): Setting up the SSL/TLS structure...
[V][ssl_client.cpp:112] start_ssl_client(): Loading CA cert
[V][ssl_client.cpp:147] start_ssl_client(): Setting hostname for TLS session...
[V][ssl_client.cpp:162] start_ssl_client(): Performing the SSL/TLS handshake...
[D][ssl_client.cpp:173] start_ssl_client(): Protocol is TLSv1.2 Ciphersuite is TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256
[D][ssl_client.cpp:175] start_ssl_client(): Record expansion is 29
[V][ssl_client.cpp:181] start_ssl_client(): Verifying peer X.509 certificate...
[V][ssl_client.cpp:190] start_ssl_client(): Certificate verified.
[V][ssl_client.cpp:205] start_ssl_client(): Free heap after TLS 114320
Connected to server!
headers received
TLS Version: 303
Cipher Suite: c02f

The last three lines have been returned by the server, so it successfully connected using HTTPS. The hex constants can be looked up in the golang package docs, and 303 as a TLS Version means TLSv1.2, c02f as Cipher Suite equals to TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, which is identical to the one within debug output. Feel free to include other fields from the tls.ConnectionState struct in the go code.

Adding client side certificates

The go ouput does not list peer certificates, as the client did not present any - as the server did not request any! Let's try to change that by first generating a key and self-signed certificate, to be used by the ESP32 device:

$ openssl genrsa -out server.key 2048
Generating RSA private key, 2048 bit long modulus
.....................+++
..............................................................................................+++
e is 65537 (0x10001)

$ openssl req -new -x509 -sha256 -key server.key -out server.crt -days 3650
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) []:DE
State or Province Name (full name) []:NRW
Locality Name (eg, city) []:Somewhere
Organization Name (eg, company) []:My
Organizational Unit Name (eg, section) []:Own
Common Name (eg, fully qualified host name) []:esp32.local
Email Address []:

The code needs both key and certificate in PEM format, as a single string. I use sed to bring it into shape:

$ cat server.crt | sed -e 's/\(.*\)/\"\1\\n\" \\/g'
$ cat server.key | sed -e 's/\(.*\)/\"\1\\n\" \\/g'

Paste the output to src/main.cpp to lines #50 and #51. Go to #76/#77 to activate them. Caution, the example code passes the cert as the key and vice versa. So correct this to:

  client.setCertificate(test_client_cert); // for client verification
  client.setPrivateKey(test_client_key);    // for client verification

Now, back again to the go server part. We have to make the HTTPS server request certificates, so change main() to:

func main() {
    tlsConfig := &tls.Config{
      // NoClientCert
      // RequestClientCert
      // RequireAnyClientCert
      // VerifyClientCertIfGiven
      // RequireAndVerifyClientCert
      ClientAuth: tls.RequestClientCert,
    }
    server := &http.Server{
      Addr:      ":443",
      TLSConfig: tlsConfig,
    }

    http.HandleFunc("/", TLSInfoServer)
    err := server.ListenAndServeTLS("cert.pem", "privkey.pem")
    if err != nil {
        log.Fatal("ListenAndServe: ", err)
    }
}

This sets up a tls.Config struct with ClientAuth set to tls.RequestClientCert. The snippet contains other options, such as RequireAndVerifyClientCert (which would require to additionally set up CA certificates). We'll start with the simple setting to ask the client for (any) certificate.

Build and run the container again, compile and flash the ESP code. It should output something similar to this:

Starting connection to server...
[V][ssl_client.cpp:52] start_ssl_client(): Free heap before TLS 153488
[V][ssl_client.cpp:54] start_ssl_client(): Starting socket
(..)
[V][ssl_client.cpp:244] send_ssl_data(): Writing HTTP request...
[V][ssl_client.cpp:244] send_ssl_data(): Writing HTTP request...
headers received
TLS Version: 0x303
Cipher Suite: 0xc02f
CN: esp32.local
[V][ssl_client.cpp:213] stop_ssl_socket(): Cleaning SSL connection.
[V][ssl_client.cpp:213] stop_ssl_socket(): Cleaning SSL connection.

Now it includes the line CN: esp32.local, which means that on the server side we can check for valid certificates and get information such as users and/or device IDs out of certificates. What still needs to be done is some investigation into self-signed certificates, which in my case caused Mbed TLS errors during connection initialization.

Wrapping up

For HTTPS server and client code, things technically work out. A TLS handshake still takes a significant amount of time. Placing calls to millis() in the ESP32 sketch yields ~2.2 seconds necessary for the client.connect() function call.

  • If you really want to work with HTTPS on ESP as a server, make sure to check out github.com/fhessel/esp32_https_server
  • Given the time, a deep dive into openssl and Mbed tls could be worth it to further understand the inner workings
  • Without optimization, TLS is probably too slow to make an ESP a secure web server.
  • As a client, the initial handshake time can be acceptable if the connection is maintained and e.g. upgraded to WebSockets

I'm curious about throughput, i still have to test that. But it's a valid basis to start further experiments from :)

Andreas

Follow ThingForward on Twitter, Facebook and Linkedin!