I decided to build a Raspberry Pi cluster, this is what I ended up with.

Goals and motivation

  • play with some physical hardware again
  • run highly available services (DHCP, DNS, NTP, HTTP)
  • learning, experimenting and having fun with a wide variety of software (over the whole stack)
  • to have a small, somewhat portable cluster I can also use offline. Bonus if it looks cool

alt text

Setup

I have 8 Raspberry Pi’s, of various models, arranged into the following groups:

  • x2 - LanServices main (model 1)
  • x1 - LanServices misc (model 2)
  • x1 - Admin/Deployer (model 2)
  • x4 - Compute/Worker nodes (model 3)

Some were gifted on from others old projects (ta!), the model 3’s I bought new.

All provisioning starts off from a Virtual Machine, “vagrant up”. All of the provisioning tools/code (bash, ansible, fabric, serverspec etc) can also run from the Admin/Deployer R-Pi.

Then I can configure the cluster of 8 nodes, all I need is a Vanilla Raspbian installation with SSH enabled (with default credentials still, ansible will change them later).

alt text

Overview of groups

What the cluster is doing, more or less. The software stack for each group.

Deployer

The Deployer role runs on x1 R-Pi, and in a Virtual Machine. This configures all of the other hosts.

  • Fabric (http://www.fabfile.org/)
  • Ansible (https://www.ansible.com/)
  • ServerSpec (http://serverspec.org/)
  • Redis DB for Ansible fact cache (https://redis.io/)

LanServices - Main

To provide redundant essential services for the LAN.

  • DHCP Server (in high availability)
  • DNS (Bind with zone replication between master/slave)
  • NTPD Server
  • FTP Daemon
  • BusyBox httpd (running in chroot)

LanServices - Misc

For miscellaneous, non-essential, net services.

  • Used for dev, reporting, testing
  • Nginx + PHP-fpm
  • Haproxy
  • Hugo website generator
  • NFS server

Compute/Worker

To play with services. See the diagram below for the hosting.

  • consul (x2 clients, x2 server)
  • Keepalived (floating IP over x2 nodes)
  • Haproxy
  • NFS Client
  • DistCC (for distributed compiling)
  • Hadoop (to do)
  • Docker cluster (x2 frontend, x2 backend) (to do)
  • Python MPICH (partly done)

alt text

cost

From a more financial perspective I am not sure its “worth” building a R-Pi cluster when you need to buy all of the parts new. Laptops are powerful enough to have a number of virtual machines, cloud providers are cheap (and whole environments that only last hours can be created).

If you plan on playing with the hardware often, and powering down a board to re-image it with Raspbian from time-to-time while everything is running or otherwise experimenting with physical systems, then the R-Pi clusters are very much worth it.

Next I need an x86 cluster.

code

The project, including better documentation, is hosted here:

github.com/craig-m/rpi_cluster

There is still a fair amount left to do…