Multi Datacenter guides on Cozystack

Tune etcd timeouts

Mon, 01 Jan 0001 00:00:00 +0000

Potential problems

etcd is a reliable key-value store for critical data, the place where the Kubernetes API server stores its data. Values are not considered written until a quorum of nodes reports that data has been written. Also, etcd constantly checks that there is a leader and every node is aware which one it is. Usually, an etcd cluster is run on nodes in the same datacenter (where latency is low) and default timeouts are tuned for the quickest error reporting possible. In a stretched cluster, where nodes are in different datacenters, the RTT is much higher and default timeouts can cause false alarms as if the network was partitioned. In this case, the data is still consistent, but etcd will enter readonly mode to prevent split-brain. This could happen so frequently that many other components of Kubernetes would start to fail with non-descriptive and unhelpful error messages.

Kube scheduler configuration

Mon, 01 Jan 0001 00:00:00 +0000

Label nodes

kubectl label node <nodename> topology.kubernetes.io/zone=A

Create global app topology spread constraints

apiVersion: v1
kind: ConfigMap
metadata:
 name: cozystack-scheduling
 namespace: cozy-system
data:
 globalAppTopologySpreadConstraints: |
 topologySpreadConstraints:
 - maxSkew: 1
 topologyKey: topology.kubernetes.io/zone
 whenUnsatisfiable: DoNotSchedule

Configure PodTopologySpread

See: https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#cluster-level-default-constraints

For Talm installation: Add to templates/_helpers.tpl.

cluster:
...
 scheduler:
 config:
 apiVersion: kubescheduler.config.k8s.io/v1beta3
 kind: KubeSchedulerConfiguration
 profiles:
 - schedulerName: default-scheduler
 pluginConfig:
 - name: PodTopologySpread
 args:
 defaultConstraints:
 - maxSkew: 1
 topologyKey: topology.kubernetes.io/zone
 whenUnsatisfiable: ScheduleAnyway
 defaultingType: List

Apply changes:

Node labels

Mon, 01 Jan 0001 00:00:00 +0000

How topology labels work

When running a Kubernetes cluster in multiple datacenters, it’s important to know when you want to schedule workloads close to each other (for example, a database and a backend application) or when you want to spread them out (for example, multiple replicas of frontend, or database replicas, or volume replicas). The first step to achieving this is to label Kubernetes nodes. Public clouds typically use the zone and region terms. In Kubernetes, the most common way to designate a geographical location is to use topology.kubernetes.io/zone and topology.kubernetes.io/region labels (or only the zone one).

LINSTOR DRBD Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

This guide explains the configuration needed to use LINSTOR storage in a stretched (distributed) Cozystack cluster.

DRBD (Distributed Replicated Block Device) is a kernel-level block device replication system that works over the network. LINSTOR server manages DRBD volumes, including their creation, deletion, and orchestration across nodes.

Challenges of using DRBD

DRBD only considers data as written once it reaches a quorum of nodes. But as it presents itself as a block device to the end user, it must return an error within a given timeout if there are not enough nodes to establish a quorum.

Configuring a Dedicated Network for Distributed Storage with LINSTOR

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

This guide explains how to improve storage reliability and performance in distributed Cozystack clusters.

In hyper-converged clusters, it’s common to dedicate a network to storage traffic. However, it’s not always possible to provision separate storage links between datacenters.

If you lack dedicated inter-datacenter links for storage, you have two options:

make storage nodes in each datacenter isolated,
make storage traffic share the existing uplinks with other workloads.

This guide shows how to configure LINSTOR to use a dedicated network for storage traffic within each datacenter, while falling back to shared links between datacenters when needed.

SeaweedFS Multi-DC Configuration

Mon, 01 Jan 0001 00:00:00 +0000

This guide explains how to deploy SeaweedFS over several data centers (“multi-DC”). Multi-zone configuration for SeaweedFS is available since Cozystack v0.34.0.

SeaweedFS Multi-DC Configuration

To span SeaweedFS over several DCs, create a new cluster in multi-DC mode.

By default, SeaweedFS runs in a single data centre (DC), and a running single-DC deployment cannot be switched to multi-DC mode. If you need to change the topology, delete the current SeaweedFS instance and create a new one with the desired mode.