I decided to build a Raspberry Pi cluster, this is what I ended up with.
Goals and motivation
- play with some physical hardware again
- run highly available services (DHCP, DNS, NTP, HTTP)
- learning, experimenting and having fun with a wide variety of software (over the whole stack)
- to have a small, somewhat portable cluster I can also use offline. Bonus if it looks cool
I have 8 Raspberry Pi’s, of various models, arranged into the following groups:
- x2 - LanServices main (model 1)
- x1 - LanServices misc (model 2)
- x1 - Admin/Deployer (model 2)
- x4 - Compute/Worker nodes (model 3)
Some were gifted on from others old projects (ta!), the model 3’s I bought new.
All provisioning starts off from a Virtual Machine, “vagrant up”. All of the provisioning tools/code (bash, ansible, fabric, serverspec etc) can also run from the Admin/Deployer R-Pi.
Then I can configure the cluster of 8 nodes, all I need is a Vanilla Raspbian installation with SSH enabled (with default credentials still, ansible will change them later).
Overview of groups
What the cluster is doing, more or less. The software stack for each group.
The Deployer role runs on x1 R-Pi, and in a Virtual Machine. This configures all of the other hosts.
- Fabric (http://www.fabfile.org/)
- Ansible (https://www.ansible.com/)
- ServerSpec (http://serverspec.org/)
- Redis DB for Ansible fact cache (https://redis.io/)
LanServices - Main
To provide redundant essential services for the LAN.
- DHCP Server (in high availability)
- DNS (Bind with zone replication between master/slave)
- NTPD Server
- FTP Daemon
- BusyBox httpd (running in chroot)
LanServices - Misc
For miscellaneous, non-essential, net services.
- Used for dev, reporting, testing
- Nginx + PHP-fpm
- Hugo website generator
- NFS server
To play with services. See the diagram below for the hosting.
- consul (x2 clients, x2 server)
- Keepalived (floating IP over x2 nodes)
- NFS Client
- DistCC (for distributed compiling)
- Hadoop (to do)
- Docker cluster (x2 frontend, x2 backend) (to do)
- Python MPICH (partly done)
From a more financial perspective I am not sure its “worth” building a R-Pi cluster when you need to buy all of the parts new. Laptops are powerful enough to have a number of virtual machines, cloud providers are cheap (and whole environments that only last hours can be created).
If you plan on playing with the hardware often, and powering down a board to re-image it with Raspbian from time-to-time while everything is running or otherwise experimenting with physical systems, then the R-Pi clusters are very much worth it.
Next I need an x86 cluster.
The project, including better documentation, is hosted here:
There is still a fair amount left to do…