About scherbelberg
Synopsis
scherbelberg provides both a command line interface (CLI) and a Python application programming interface (API) for deploying and managing small Dask-based HPC clusters in the Hetzner cloud.
Warning
Development status alpha, stability acceptable, security also acceptable but implementation needs a review.
Project’s Name
Next to impressive projects like Fugaku, which is named after Mount Fuji, the TOP500 are clearly missing an entry from the city of Leipzig. This project is named after one of the few significant “mountains” in the city, the “Scherbelberg”, also known as the “Rosentalhügel”: 20 meters above the surrounding landscape and 125 meters above sea level. Starting out as a late 19th century landfill, it has since become part of a park-like landscape. As of 1975, a famously shaky steel observation tower with a rather beautiful view is located at its top, overlooking not only the Leipziger Auenwald forest but also the city’s sewage treatment plant.
Motivation
While Dask is wonderful for automating large, parallel, distributed computations, it can not solve the problem of its own deployment onto computer clusters. Instead, Dask plays nicely with established tools in the arena such as slurm. Deploying Dask onto a custom cluster therefore requires a fair bit of time, background knowledge and technical skills in computer & network administration.
One of the really appealing features of Dask is that it enables users to exploit huge quantities of cloud compute resources really efficiently. Cloud compute instances can usually be rented on a per-hour basis, making them an interesting target for sizable, short-lived, on-demand clusters. For cloud deployments like this, there is the Dask-related cloud provider package, which surprisingly does not solve the entire problem of deployment. At the time of scherbelberg’s creation, it was both rather inflexible and lacking support for the Hetzner cloud. Companies like Coiled, which is also the primary developer of Dask, have filled this niche with polished, web-front-end services (and equally polished APIs) for creating clusters on clouds, which effectively makes them resellers of cloud resources. scherbelberg aims at eliminating the resellers from the equation while trying to provide a minimal, independent, self-contained, yet fully operational solution.
Implementation
scherbelberg creates clusters on the Hetzner cloud quickly and completely from scratch without any prerequisites on the cloud’s side. No pre-configured operating system, virtual machine or docker images are required. scherbelberg simply connects to the Hetzner cloud via its REST API, creates the required number and kind of compute instances based on the latest Ubuntu LTS release, networks as well as secures the compute instances and deploys mambaforge onto them. Depending on the size of the cluster, creating an entire cluster from scratch with a single command or single API call requires anywhere from two to ten minutes. Destroying a cluster is done in under ten seconds. In many ways, scherbelberg is a quick and dirty bare-bones solution. It heavily relies on ssh and the systems’ shell. It does not use any higher-end tools for mass-administration of computers such as Chef or Ansible.
Note
scherbelberg’s’ primary objective is to provide a stack of Dask, conda-forge and Ubuntu as simple and cleanly as possible.
Note
scherbelberg is designed as an asynchronous package using asyncio.
Use Cases
Anything that Dask can be used for: Parallel & distributed computations, distributed memory, all on demand. In many cases, high-performant computational resources are only needed for short periods of time. In those cases, it might not be worth spending a lot of money on matching hardware. Being able to quickly create and destroy custom-configured computer clusters enables a different, very interesting kind of thinking.