Why ship Kubernetes Events to Loki?

Kubernetes events provide valuable insights into what’s happening in your cluster—from pod restarts and failed deployments to resource shortages. They’re often the first indicator of issues, but by default, they’re not stored long-term.

With Grafana Alloy (the successor to Grafana Agent), you can ship events centrally to Loki, making them searchable, visualizable, and alertable.

Alloy configuration: Shipping Kubernetes Events

The following Alloy configuration will send all Kubernetes events in JSON format directly to Loki:

loki.write "loki" {
  endpoint {
    url = "http://loki-gateway/loki/api/v1/push"
  }
}

loki.source.kubernetes_events "k8s_events" {
  forward_to = [loki.write.loki.receiver]
  log_format = "json"
}

Step by step: What’s happening here?

  1. loki.write defines the target for your logs (your Loki endpoint).
  2. loki.source.kubernetes_events collects all events in the cluster and ships them in JSON format to Loki.

Benefits

  • Central search: All events are searchable in Loki.
  • Alerting: You can set up alerts for specific event types or reasons.
  • Long-term archive: Events are no longer lost after a short retention period.

Tips & Notes

  • Alloy needs the correct RBAC permissions to read events (get, list, watch on events).
  • Events are stored as a log stream with their own labels—use these for targeted searches.
  • For production, consider a dedicated Loki gateway service with authentication.

Conclusion

With just a few lines of Alloy configuration, you can centrally ship Kubernetes events to Loki. This gives you full visibility into everything happening in your cluster—and lets you react faster to issues.

Happy Logging! 🚀