My New Server Rack: Mieshu

After saving up for a long while and thinking about what I want in my new home, I have finally taken the leap and gotten myself a server rack for home use. Its has a 15U capacity, which should be plenty to get started, but this same brand has larger racks too, in case I do want to upgrade it and keep the same style.

That said, for now there’s only two 4U units in them, one for (file) storage, and one for database purposes. I sadly don’t have anything dedicated for workloads yet, so for now, both of these servers are intended to also run some light workloads. I haven’t made my mind up yet on how to solve the workload issues. Now that I have a rack, I obviously want something rack-mountable, and I probably want it to run a Kubernetes cluster too.

In this regard, I could go for a set of Raspberry Pi units, there’s 3U mounts that can hold up to 12 Raspberry Pi machines, which would be a nice amount. However, I am not yet completely sold on full-ARM workloads, and I’m not entirely convinced of the power of Raspberry Pi units in general. I’d much rather standardize on another brand, Odroid, as they have more types of units available, and are not limited to just ARM. But since they’re not the popular kid in class, there’s very few off-the-shelf rack mounting equipment for it. I’ll be thinking about this for just a bit more before making a decision.

For now, though, I wanted to talk about the setup of the first server, Mieshu, who will be used as a storage server. Mieshu currently runs 8 HDDs, and 2 NVMe drives. One of the NVMe drives is used for the rootfs, and the other is used for caching in certain applications. The HDDs themselves offer the data storage capacity.

The HDDs are currently comprised of four 16TB drives, and four 8TB drives. The smaller disks come from my desktop, Edephas, which used to serve as data storage until Mieshu took over. All disks are configured into pairs, which themselves make mirrors. This means I have four sets of mirror pools, two times 16TB, and two times 8TB, for a total of 48TB of storage. I’m currently using about half of this, and it should give me plenty of time before needing to increase the size again.

I chose to use mirrors since it has a good chance of your data being recoverable on disk failure, and it allows me to buy disks per two, rather than in larger numbers. This hopefully keeps the cost of expansion within reasonable limits. The mirrors themselves are currently ZFS pools, but I hope to be able to use bcachefs very soon as well.

Just a bunch of mirrors is rather inconvenient, however, so I’m also leveraging MergerFS to combine all the mirrors into a single usable pool. This slightly odd setup was chosen over RAID-0 or RAID-Z* to lower the impact of disk failure. Even if two disks in the same mirror were the die at the same time, I wouldn’t lose all data, just the bits on that particular mirror. It would be very annoying, but it wouldn’t be disastrous.

Apart from generic mass storage, I also host S3 buckets for personal use. This is where I upload CI artifacts to, and my MissKey instance uses it for storing objects as well. Future services such as Mimir will probably leverage S3 for storage as well. This is achieved through Garage. I’ve also tried SeaweedFS, which is a very neat project on its own, but Garage is just simpler to configure, and allows a replicated setup with only two servers, whereas SeaweedFS demands an odd number of master servers.

And lastly, Mieshu runs K3s for its Kubernetes component. It is currently not serving anything yet, as the other server is supposed to become the database server, which is needed for most workloads. Once that is up and running, Mieshu will start hosting things such as Grafana and Loki, monitoring stuff basically. Perhaps I’ll move Laminar to this server as well, but I’m unsure if I will run that as a Kubernetes service.

The server itself runs on Gentoo, as it still is the most stable experience I can get out of any GNU+Linux distribution. I am, however, not using the default of OpenRC as the init system and service manager. For the first time, I’m running Gentoo with systemd. After several years, it appears to have become stable enough to trust with serious workloads. With its increased use, however, some things have become simpler by just using systemd. I hope to get a better understanding of it, and learn to bend it to my will as needed, by simply using it on my own systems.

I hope to have time to work on the other server sooner rather than later, so I can finish up the base of my new setup. Be on the lookout for the next post, where I’ll go into detail on Nouki, the database server.