Cluster

scherbelberg.Cluster objects define Dask clusters and represent the core functionality of scherbelberg. The class can be used to create new clusters as well as to attach to existing ones and to destroy them. It generates matching Dask Client objects.

The Cluster Class

class scherbelberg.Cluster(client, scheduler, workers, network, firewall, dask_ipc=9753, dask_dash=9756, dask_nanny=9759, prefix='cluster', wait=1.0, log=None)

Defines a cluster of nodes. Mutable. Use the asynchronous from_* classmethods to instantiate objects of this class.

Parameters

  • client (Client) – A cloud-API client object.

  • scheduler (NodeABC) – A node running the Dask scheduler.

  • workers (List[NodeABC]) – A list of nodes running Dask workers.

  • network (BoundNetwork) – A cloud-API network object.

  • firewall (BoundFirewall) – A cloud-API firewall object.

  • dask_ipc (int) – Port used for Dask’s interprocess communication.

  • dask_dash (int) – Port used for Dask’s dashboard.

  • dask_nanny (int) – Port used for Dask’s nanny.

  • prefix (str) – Name of cluster, used as a prefix in names of every component.

  • wait (float) – Timeout in seconds before actions are repeated or exceptions are raised.

  • log (Optional[Logger]) – Allows to pass custom logger objects. Defaults to scherbelberg’s own default logger.

__repr__()

Interactive string representation

Return type

str

property alive: bool

Is cluster alive?

Return type

bool

property dask_dash: int

Port used for Dask’s dashboard

Return type

int

property dask_ipc: int

Port used for Dask’s interprocess communication

Return type

int

property dask_nanny: int

Port used for Dask’s nanny

Return type

int

async destroy()

Destroys a living cluster

async classmethod from_existing(prefix='cluster', tokenvar='HETZNER', wait=1.0, log=None)

Attaches to existing cluster. Raises scherbelberg.ClusterSchedulerNotFound if scheduler can not be found. This is also the most likely exception if a cluster for a given prefix simply does not exist. Raises scherbelberg.ClusterWorkerNotFound if a worker can not be found. Raises scherbelberg.ClusterFirewallNotFound if the firewall can not be found. Raises scherbelberg.ClusterNetworkNotFound if the private network can not be found.

Parameters

  • prefix (str) – Name of cluster, used as a prefix in names of every component.

  • tokenvar (str) – Name of the environment variable holding the cloud API login token.

  • wait (float) – Timeout in seconds before actions are repeated or exceptions are raised.

  • log (Optional[Logger]) – Allows to pass custom logger objects. Defaults to scherbelberg’s own default logger.

Return type

ClusterABC

Returns

Cluster object represeting an alive cluster.

async classmethod from_new(prefix='cluster', tokenvar='HETZNER', wait=1.0, dask_ipc=9753, dask_dash=9756, dask_nanny=9759, scheduler='cx11', worker='cx11', image='ubuntu-20.04', datacenter='fsn1-dc14', workers=1, log=None)

Creates a new cluster

Parameters

  • prefix (str) – Name of cluster, used as a prefix in names of every component.

  • tokenvar (str) – Name of the environment variable holding the cloud API login token.

  • wait (float) – Timeout in seconds before actions are repeated or exceptions are raised.

  • dask_ipc (int) – Port used for Dask’s interprocess communication.

  • dask_dash (int) – Port used for Dask’s dashboard.

  • dask_nanny (int) – Port used for Dask’s nanny.

  • scheduler (str) – Compute instance type used for Dask scheduler.

  • worker (str) – Compute instance type used for Dask workers.

  • image (str) – Operating system image.

  • datacenter (str) – Target data center.

  • workers (int) – Number of workers in cluster.

  • log (Optional[Logger]) – Allows to pass custom logger objects. Defaults to scherbelberg’s own default logger.

Return type

ClusterABC

Returns

Cluster object represeting an alive cluster.

async get_client(asynchronous=True)

Creates and returns a client.

Parameters

asynchronous (bool) – Specifies if the Client object runs in asynchronous mode.

Return type

Any

Returns

dask.distributed.Client object attached to the cluster.

async classmethod nuke(prefix='cluster', tokenvar='HETZNER', log=None)

Nukes a possibly incomplete or broken cluster without prior connecting to it

Parameters

  • prefix (str) – Name of cluster, used as a prefix in names of every component.

  • tokenvar (str) – Name of the environment variable holding the cloud API login token.

  • log (Optional[Logger]) – Allows to pass custom logger objects. Defaults to scherbelberg’s own default logger.

property prefix: str

Name of cluster, used as a prefix in names of every component

Return type

str

property scheduler: scherbelberg._core.abc.NodeABC

A node running the Dask scheduler

Return type

NodeABC

property workers: List[scherbelberg._core.abc.NodeABC]

A list of nodes running Dask workers

Return type

List[NodeABC]