Cozystack Troubleshooting Guide on Cozystack

Troubleshooting etcd

Mon, 01 Jan 0001 00:00:00 +0000

How to clean up etcd state

To flush the etcd state from a node, you can use talm or talosctl with the following commands:

Talm
talosctl

Replace nodeN with the name of the failed node, for instance, node0.yaml:

talm reset -f nodes/nodeN.yaml --system-labels-to-wipe=EPHEMERAL --graceful=false --reboot

talosctl reset --system-labels-to-wipe=EPHEMERAL --graceful=false --reboot

⚠️ This command will remove the state from the specified node. Use it with caution.

Troubleshooting Flux CD

Mon, 01 Jan 0001 00:00:00 +0000

Diagnosing `install retries exhausted` error

Sometimes you can face with the error:

# kubectl get hr -A -n cozy-dashboard dashboard
NAMESPACE NAME AGE READY STATUS
cozy-dashboard dashboard 15m False install retries exhausted

You can try to figure out by checking events:

kubectl describe hr -n cozy-dashboard dashboard

if Events: <none> then suspend and resume the release:

kubectl patch hr -n cozy-dashboard dashboard -p '{"spec": {"suspend": true}}' --type=merge
kubectl patch hr -n cozy-dashboard dashboard -p '{"spec": {"suspend": null}}' --type=merge

and check the events again:

Monitoring Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

This guide provides troubleshooting steps for common issues with monitoring components in Cozystack, including metrics collection, alerting, visualization, and log collection.

Diagnosing Missing Metrics

If metrics are not appearing in Grafana or VictoriaMetrics, follow these steps:

Check VMAgent Status

Ensure VMAgent is running and collecting metrics:

kubectl get pods -n cozy-monitoring -l app.kubernetes.io/name=vmagent
kubectl logs -n cozy-monitoring -l app.kubernetes.io/name=vmagent --tail=50

Verify Targets

Check if VMAgent can scrape targets:

kubectl exec -n cozy-monitoring -c vmagent deploy/vmagent -- curl -s http://localhost:8429/targets | jq .

Look for targets with health: "up". If targets are down, check network connectivity and RBAC permissions.

Troubleshooting Kube-OVN

Mon, 01 Jan 0001 00:00:00 +0000

Getting information about Kube-OVN database state

# Northbound DB
kubectl -n cozy-kubeovn exec deploy/ovn-central -c ovn-central -- ovn-appctl \
 -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
# Southbound DB
kubectl -n cozy-kubeovn exec deploy/ovn-central -c ovn-central -- ovn-appctl \
 -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound

Example output:

Name: OVN_Northbound
Cluster ID: abf6 (abf66f15-9382-4b2b-b14c-355d64ae1bda)
Server ID: 8d8a (8d8a2985-c444-43bb-99f6-21c82f05b58d)
Address: ssl:[10.200.1.22]:6643
Status: cluster member
Role: leader
Term: 3
Leader: self
Vote: self

Last Election started 146211 ms ago, reason: leadership_transfer
Last Election won: 146202 ms ago
Election timer: 5000
Log: [2, 1569]
Entries not yet committed: 0
Entries not yet applied: 0
Connections: ->2a66 ->c23f <-2a66 <-c23f
Disconnections: 30
Servers:
 2a66 (2a66 at ssl:[10.200.1.18]:6643) next_index=1569 match_index=1568 last msg 17 ms ago
 8d8a (8d8a at ssl:[10.200.1.22]:6643) (self) next_index=1471 match_index=1568
 c23f (c23f at ssl:[10.200.1.1]:6643) next_index=1569 match_index=1568 last msg 18 ms ago

To kick a node out of the cluster (for example, if it is down, and you want to remove it from the cluster), use:

Troubleshooting LINSTOR controller crash loops

Mon, 01 Jan 0001 00:00:00 +0000

Restarting the controller

If the controller is running but appears idle/unresponsive, try restarting it. This operation is idempotent and safe: pending (unfinished) work will resume after the restart.

LINSTOR controller crash loop

If linstor-controller can’t start, but logs do not contain any useful information, you can increase the log level (maximum level is TRACE).

Example of LINSTORCluster CR with increased log level:

apiVersion: piraeus.io/v1
kind: LINSTORCluster
spec:
 controller:
 podTemplate:
 spec:
 containers:
 - name: linstor-controller
 env:
 # both settings are used by linstor-controller
 - name: LS_LOG_LEVEL
 value: TRACE
 - name: LS_LOG_LEVEL_LINSTOR
 value: TRACE

Note: if linstor-controller is not in a crash loop, but you need to increase log level, you can do so temporarily in the runtime using the following command:

Troubleshooting LINSTOR CrashLoopBackOff related to a broken database

Mon, 01 Jan 0001 00:00:00 +0000

⚠️ Advanced Users Only

This guide is intended for experienced users who are comfortable with low-level troubleshooting and data recovery operations. Corrupted LINSTOR databases are rare and typically indicate a serious underlying issue.

If you encounter this situation in a production environment, we strongly recommend contacting qualified support rather than attempting to fix it yourself. Incorrect actions can lead to permanent data loss.

Introduction

When running outside of Kubernetes, LINSTOR controller uses some kind of SQL database (various kinds). In Kubernetes, linstor-controller does not require a persistent volume or external database. Instead, it stores all its information as custom resources (CRs) right in the Kubernetes control plane. Upon startup, LINSTOR controller reads all CRs and creates an in-memory database.

Troubleshooting Piraeus custom resources

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

Piraeus custom resources are the following:

LinstorCluster
LinstorSatellite
LinstorSatelliteConfiguration
LinstorNodeConnection

The Piraeus controller protects these resources from unintended changes and deletion. When you delete the resource, controller will first make sure that all downstream resources are deleted.

But the default setting of the webhook is to reject any changes if something goes wrong with the webhook itself. And, in a disturbed system, webhook may go down a lot, without useful stacktraces.

Cozystack Troubleshooting Guide on Cozystack

Troubleshooting etcd

How to clean up etcd state

Troubleshooting Flux CD

Diagnosing install retries exhausted error

Monitoring Troubleshooting

Diagnosing Missing Metrics

Check VMAgent Status

Verify Targets

Troubleshooting Kube-OVN

Getting information about Kube-OVN database state

Troubleshooting LINSTOR controller crash loops

Restarting the controller

LINSTOR controller crash loop

Troubleshooting LINSTOR CrashLoopBackOff related to a broken database

Introduction

Troubleshooting Piraeus custom resources

Introduction

Diagnosing `install retries exhausted` error