Logs (Loki)

Grafana Loki stores the logs from every node in the cluster. It runs on the control node and writes its chunks and index to the account bucket, so a cluster's logs outlive the cluster and can be read from any later cluster in the same tenant.

How logs get there

The OpenTelemetry Collector runs on every node and reads:

Fluent Bit reads the systemd journal on every node and forwards it to the collector with source="journald". OTLP logs (the OpenTelemetry Java agent, Spark jobs) arrive at the collector directly.

Cassandra's application logs (system.log, debug.log) reach Loki only over OTLP, from the OpenTelemetry Java agent in the Cassandra JVM. They carry service_name="cassandra" and no source label, and are stored once. The collector tails only the JVM GC log from disk, because the JVM writes it directly and the agent never sees it.

The collector sends everything to Loki's OTLP endpoint, http://loki.default.svc.cluster.local:3100/otlp, with the cluster's tenant in the X-Scope-OrgID header.

Labels

Four resource attributes become stream labels. Loki adds service_name, and for pod logs the Kubernetes labels (k8s_namespace_name, k8s_pod_name, k8s_container_name):

Every other attribute (the systemd unit, the log file, the trace and span ids, the severity) is stored as structured metadata. LogQL reads it with a label filter after the selector, such as | systemd_unit="docker.service".

Configuration

Loki 3.7.8 runs as one process (target: all) on the control node:

• Ports: 3100 (HTTP), 9098 (gRPC)
• Local data: /mnt/db1/loki on the control node (write-ahead log, open chunks and the index not yet shipped)
• Object storage: s3://<account-bucket>/loki/; chunks sit under the tenant, the TSDB index under index/
• Tenancy: native multi-tenancy; the tenant is the cluster's observability tenant

Loki flushes a chunk to S3 once it spans an hour, so a stream that keeps writing reaches S3 within about an hour. Its index uploads on Loki's own 15-minute rotation, and Loki re-reads the other clusters' index from S3 every minute. A line from another cluster in the tenant becomes readable here once its index file is in S3.

Out-of-order window. Loki rejects a line that is more than 30 minutes older than the newest line of its stream (half of the 1-hour chunk age). The window is wide so that a slow Loki restart does not drop logs. A line replayed or backdated by more than 30 minutes on a stream that is still writing is refused. The annotation mirror is not affected: each annotation is a stream of its own.

Nothing in the cluster deletes a chunk or an index file. The cluster's Loki compactor is idle, retention is off, and the delete API is not served. The account compactor merges each day's index files into one, with retention off, and removes the sources only after the merged file is written. Every log line stays in S3 until you delete it yourself.

Every cluster's Loki reads every tenant's chunks and index from the shared store; nothing about this changes with the account compactor.

Querying logs

The CLI

easy-db-lab logs query reads this cluster's logs:

# The last hour
easy-db-lab logs query

# By source, host or systemd unit (`cassandra` is the application log, `cassandra-gc` the JVM GC log)
easy-db-lab logs query --source cassandra --host db0
easy-db-lab logs query --source cassandra-gc --host db0
easy-db-lab logs query --source journald --unit docker.service

# Lines containing text
easy-db-lab logs query --grep "OutOfMemory"

# Time range and limit
easy-db-lab logs query --since 30m --limit 500

# A raw LogQL query, sent unchanged
easy-db-lab logs query -q '{service_name="cassandra"} |= "Exception"'

Every option but --query is scoped to this cluster. A raw query is sent as written, so it reads every cluster in the tenant unless it names a cluster.

Grafana

The Loki datasource (uid loki) sends the cluster's own tenant on every query. Grafana also has one logs datasource per tenant in the shared store, Loki (<tenant>) (uid loki-<tenant>), and one for all of them, Loki (all tenants) (uid loki--all). Each links a trace ID to the Tempo datasource of the same tenant view. See the Mimir page for how the tenants are found. Dashboards reach them through their Logs picker, which opens on Loki; the annotation markers follow it too.

The Log Investigation dashboard (Dashboards → Log Investigation) filters by:

For anything else, open Explore, choose the Loki datasource, and write LogQL:

{cluster="<name>-<clusterId>", service_name="cassandra"} |= "Exception"
{cluster=~".+", host_name="db0"} | systemd_unit="cassandra.service"
sum by (source) (count_over_time({cluster="<name>-<clusterId>"}[5m]))

A LogQL selector needs at least one matcher that cannot match an empty value, which is why "all clusters" is written cluster=~".+". See the LogQL documentation.

A trace in Tempo links to its logs through the trace_id.

HTTP API

A query that carries no tenant reads nothing, so always send X-Scope-OrgID:

source env.sh
with-proxy curl -G -H 'X-Scope-OrgID: <tenant>' \
  --data-urlencode 'query={service_name="cassandra"}' \
  "http://control0:3100/loki/api/v1/query_range"

Annotations

Every Grafana annotation is also written to Loki as its own stream (source="annotation", with the cluster label and an annotation_id), so the annotations stay readable from S3 after the cluster and its Grafana are gone. The core dashboards read them back from Loki. Loki accepts entries only from the last 8760 hours to 24 hours ahead, so an annotation outside that window is not mirrored; the tool warns with its id, and the annotation stays in Grafana's JSON backup.

Teardown

down flushes Loki before it removes anything, at the same time as it saves the other signals: it stops the ingester, which answers only once every open chunk is in S3, then scales Loki to 0 and waits for the pod to go, which builds and uploads its index. It checks nothing else. The logs are recorded as saved the moment this succeeds, so a later down skips them. If a step fails, down removes nothing and Loki stays as that step left it: it is never started again, and the report names the step, each backend's state, and down --force. See down.

Troubleshooting

No logs appearing

1. Check that Loki is running and ready:

   kubectl get pods -l app.kubernetes.io/name=loki
   kubectl logs -l app.kubernetes.io/name=loki
   

2. Check that the collector is sending. A failing exporter shows up in otelcol_exporter_send_failed_log_records_total in Mimir, and in the collector's log:

   kubectl logs -l app.kubernetes.io/name=otel-collector
   

3. Check that the cluster-config ConfigMap carries the tenant:

   kubectl get configmap cluster-config -o yaml
   

Connection errors from logs query

logs query reaches Loki through the cluster's SOCKS proxy, which it starts when needed. Check that the cluster is up (easy-db-lab status) and that the control node answers (ssh control0 hostname).