BrutlabsLLC Book a consult
Foundation layer

Telemetry an agent can reason over, not just dashboards to stare at

Telemetry at a resolution that shows the fault, logs in one queryable place, and alerting that earns the page it sends.

Most networks are not short of monitoring data. They are short of correlation. Metrics sit in one tool, syslog in another, flow data in a third, and the only thing that joins them is an engineer with three browser tabs at two in the morning.

Alert fatigue is the symptom. When everything pages, nothing does, and the signal that mattered is buried under nine hundred that did not.

We rebuild this layer around a different requirement: a machine has to be able to query it. That means telemetry at a resolution that shows the fault and consistent labels to join it on, logs with structured fields, baselines that make 'abnormal' meaningful, and alerts routed by service impact rather than by device.

Deliverables

What we build

01

High-fidelity telemetry

Resolution fine enough to see the fault, built from whatever your estate supports — gNMI and OpenConfig subscriptions where the platform offers them, optimised SNMP, CLI and API collection everywhere else.

02

Collection pipeline

Telegraf and Kafka moving high-volume telemetry reliably, with backpressure handling that does not fall over during the incident you needed it for.

03

Metrics and baselines

Prometheus with retention and recording rules tuned to your network, plus learned baselines so deviation is measurable rather than a matter of opinion.

04

Unified logging

Syslog, flow and event data centralised in OpenSearch or ELK with consistent device and site labels — the same labels the metrics use.

05

Dashboards that get used

Grafana built for the questions your team actually asks during an incident, not a wall of every available counter.

06

Alert routing

Alertmanager with deduplication, grouping and service-impact routing. Fewer alerts, each of which means something.

Agent

What the agent does with this layer

The agent reads this layer at the raw level. Giving it only pre-digested alerts would hide exactly the detail it is good at.

Raw signal access

Querying series, not summaries

The agent pulls the underlying time series and log lines, so it can see the 340 ms latency shoulder that never crossed an alert threshold.

Cross-layer joins

Metrics, logs and flow together

Consistent labelling is what makes a join possible. An interface counter, a syslog line and a flow record become one story about one interface.

Baseline reasoning

Knowing what normal looks like

Retention and baselines let the agent distinguish a genuine deviation from Tuesday-afternoon traffic that looks alarming to a static threshold.

Stack

Tools we use here

gNMI / OpenConfigstreaming telemetryPrometheusmetricsGrafanadashboardsTelegrafcollectionKafkatransportOpenSearch / ELKlog analyticsAlertmanagerroutingSuzieQstate over timeOpenTelemetrytraces
Questions

Questions about telemetry & observability

Do we have to rip out our existing monitoring?

No, and we usually advise against it. The common pattern is to add streaming telemetry and centralised logging alongside what you run today, prove the new path during real incidents, and decommission the old tooling only once nobody is reaching for it.

Is gNMI worth it if SNMP already works?

It depends on what you are trying to detect. SNMP polling at five-minute intervals will never show you a microburst or a sub-second flap. If your hardest incidents are transient, streaming telemetry is the difference between diagnosing them and guessing.

Won't streaming telemetry cost a fortune in storage?

Only if you keep everything at full resolution forever. We design retention in tiers — high resolution for a short window, downsampled recording rules for trend and baseline — which typically keeps costs close to existing monitoring spend.

How does this reduce alert fatigue rather than add to it?

By changing what generates a page. Alerts route on service impact with deduplication and grouping, and a large class of conditions that used to page a human instead becomes context the agent gathers on its own.

Book a 30-minute automation readiness consultation

In 30 minutes, we’ll evaluate your infrastructure maturity, identify operational risk areas, and highlight high-impact automation opportunities.

No scripts. No invasive discovery. Just clarity.