We have all been there. You get a page on Slack from Alertmanager telling you that a pod in your staging or production cluster has started crash-looping. You immediately log in, run kubectl logs --previous, and get met with a blank screen or a NotFound error.
By the time the alert reached you, the high-churn container had crashed multiple times, the replica set had recycled the pod, and the local Kubelet garbage collector had completely purged the terminated container. The logs, which contained the exact stack trace explaining why the service crashed, are gone. You are left debugging in the dark.
This was the exact loop of frustration that led me to design and build Kivert, a real-time, event-driven Kubernetes pod-restart alerting controller.
The Limits of the Prometheus Stack
Prometheus and Alertmanager form the gold standard for Kubernetes monitoring, but they were never designed for real-time log capture. They suffer from a few architectural bottlenecks when it comes to transient crash-loop detection:
- Scrape and Rule Delays: Prometheus is poll-based. It scrapes metrics on an interval (usually 30 or 60 seconds). Alertmanager then evaluates rules on another interval, and applies a grouping wait time. By the time an alert is generated and dispatched to your Slack channel, minutes have easily passed.
- Missing Short-Lived Events: In high-churn clusters, containers can crash, get deleted, and restart between scrape intervals. This churns the metric series, sometimes resetting counters and hiding the crash entirely from Prometheus.
- No Log Context: A metric is just a number. It cannot carry log lines. Stitching metrics and logs together using multiple systems (like Alertmanager and Loki) requires complex query setups, and it still does not solve the problem of logs disappearing before they can be collected.
Kivert is built to address this specific gap. It does not replace Prometheus. Instead, it acts as a rapid-response crash-log catcher.
How Kivert Works under the Hood
Kivert is written in Go and built on sigs.k8s.io/controller-runtime. It watches the core Pod type and dispatches alerts directly to channels (like webhooks) the instant a restart count increases.
Here are the key parts of its architecture that make it reliable and lightweight:
1. Watch, Don’t Poll
Unlike poll-based checkers, Kivert registers an informer cache with the Kubernetes API server. It receives real-time watch events. There are no heavy loops checking the API server on a timer. The detection runs on top of the local informer cache.
2. Preventing Startup Boot-Storms
When a controller starts, the informer cache replays the current state of every pod in the cluster, including pods with existing historical restarts. If the controller restarted, it would normally fire alerts for all past crashes.
To solve this, Kivert runs a seeding phase at startup. It lists all pods using a raw Clientset and seeds an in-memory, thread-safe BaselineStore with their current restart counts. Alerts are only fired when a container’s restart count increases beyond this initial baseline.
3. High-Performance Predicates
Kubernetes pods are incredibly noisy. They constantly update their statuses for IP changes, readiness gates, and condition updates. Enqueuing all of these events would choke the controller queue.
Kivert uses a custom event predicate filter. It compares the container status maps of the old and new pod objects during update events. If the restart count of standard, init, or ephemeral containers has not increased, the update event is cheaply discarded before it ever reaches the reconciler.
4. Best-Effort Log Enrichment
When a restart is detected, Kivert tries to fetch the logs of the previous (crashed) container instance using a raw Kubernetes client.
- Context Bounds: The API call is bound with a strict timeout (default 5 seconds) to prevent blocking.
- Safety Caps: Logs are read up to a configured limit (like 64KB) to avoid memory bloating.
- Regex Redaction: The retrieved logs pass through a line-by-line regex filter to strip out API keys, tokens, or PII before shipping.
- Strictly Best-Effort: If the container was already garbage collected or log fetching fails for any reason, Kivert logs the warning and sends the alert anyway. A log failure never silences the alert.
5. Pluggable Alert Channels
I wanted Kivert to be highly modular. The core controller has zero compile-time dependency on any specific alert channel. Alert destinations implement a simple Alerter interface:
type Alerter interface {
Name() string
Send(ctx context.Context, a Alert) error
}
Channels self-register in their package init() block using a factory registry. Adding a new channel (like Slack or PagerDuty) only requires creating a package and adding a single blank import line in main.go.
Wrapping Up
Kivert is small, lightweight, and easy to deploy with a single Helm install. It is currently configured to run with leader election, making it safe for high-availability setups.
By running Kivert alongside your Prometheus stack, you get the best of both worlds: Prometheus keeps track of long-term trends and capacity planning, while Kivert ensures that if a container crashes in the middle of the night, you get an immediate page containing the exact log snippet showing why it failed.
You can check out the project code, Helm templates, and roadmap on the Kivert GitHub repository.