Hacking on the decentralized and distributed edge, Part 2

In the first post of this series we’ve been looking into architecture styles and explored both centralized and decentralized application architectures, and highlighted the differences between those and a truly distributed application architecture. Both decentralized and distributed architectures are really interested within the domain of IoT, because the projected fleet of billions of devices will probably be communicating in these ways. That’s a good reason to explore this a bit more.

Now we’d like to add some tech bits to the ideas from the last post, that is:

  • private networking between nodes,
  • asynchronous messaging,
  • file exchange,
  • database services,
  • distributed ledgers and
  • compute.


The power of open, distributed applications (such as Blockchains) relies in parts upon their openness. New nodes can join the network, because networked services are reachable on public, static IP addresses. However some companies might want to start within a more closed environment, so we’d like to look at how to build a private network.

One (among many other) solutions for setting up virtual private networks is Wireguard. It’s relatively new (started 2015, according to website) but has a number of appealing features. Its code size is quite small, thus minimizing the attack surface. It offers modern crypto: Noise Protocol, Curve25519, ChaCha20, Poly1305, and more, which are fixed in their combination, so users do not have to choose among Cipher suites etc. It has an easy setup and peering model using public keys of peers, making it quite easy to configure.

We’re going to use Wireguard to build a VPN, ideally using IPv6 within the overlay network.

Discovery and Node management

If you have a static definition of all nodes, connected by an overlay network, how to control them? How to enumerate all nodes, send commands to all (or individual ones)? Serf from Hashicorp is a solution for decentralized cluster management, and forms the basis for their other products such as Vault. Serf uses a gossip protocol to deliver messages and events across a cluster of nodes, and it has mechanisms for failure detection. It looks like a good solution to our ideas, so we’re adding this to the mix.

Asynchronous messaging

The world of IoT is highly influenced by asynchronous processing. You do not know when nodes are available or when their sleeping (and saving precious battery power), so communicating with them in an asynchronous fashion, i.e. using MQTT is helpful. Quite some MQTT brokers are built on Erlang/OTP. Erlang offers distributed messaging on the language level, which makes it a reasonable choice for programming distributed system.

We’re using VerneMQ, an open source MQTT broker to support messaging throughout our network.

File Exchange

Sometimes we need to distribute plain files to other nodes, i.e. configuration data or even applications. IPFS, a „peer-to-peer hypermedia protocol“ aims to be the file system for the distributed world. Looks like it’s configurable for private networks as well, so we’ll give it a try!

Database Services

Most applications need some form of structured data storage, and by now quite some database products come with a distribution mechanism. Not in the sense of clustering, i.e. duplicating state to just another node, so in case of a failure the other can can resume operations, but in the sense of really distributing state and database content among a larger number of nodes. CockroachDB is one example for this, the Interplanetary Database another interesting development. From the installation point of view, we’re planning with CockroachDB first.

Distributed Ledgers

BigChainDB is another „distributed database“, but with added blockchain capabilities. This brings us to the next topic of decentralized & distributed architectures: distributed ledgers as a way of secure and immutable storage with transactions-based characteristics. From Hyperledger to Iota, new ideas and products pop up. Not sure right now how this is going to evolve other the next months, but our plans include to run some form of distributed ledger (or tangle? :) on our network.


Finally, we need to run applications within our distributed stack. Every node is able to receive, send, process and store data, all through services running locally, but being connected to (all) other nodes in the network. What’s missing is an application as a frontend to users, whether this is going to be an API or a full Web UI. Comparable to the other services from above, the application should be running on all nodes, in an identical fashion. We’re going to use Linux containers for this and need some form of balancing/forwarding solutions: Apps must be reachable from the outside (internet side), at the same time be able to reach the services of the private network.

The Docker Engine is well-known and supported, so we’ll be running our application modules on it.

Next up

That makes up for a nice todo list :) Let’s start to implement this, we’ll keep you posted!

Your ThingForward Team