Bringing Geographic Context to Network Operations

At Grant County PUD Telecom, the data needed to understand network outages already existed. The problem was that it lived in separate systems. Zabbix showed hub status and monitoring health. ArcGIS showed hub locations, fiber patch locations, and service-area boundaries.

Each system was useful on its own, but during an outage, operators still had to connect the dots manually. A monitoring alert could show that a hub was down, but understanding where that hub was located and which areas or ISPs might be affected required jumping between tools.

That became the core problem:

How do we turn network monitoring data into geographic operational context?

The problem

A traditional monitoring alert is useful, but it is not always enough. If a hub goes down, Zabbix can tell you the device is unreachable or that a trigger has fired. But the alert alone does not immediately answer the next operational questions:

Where is the affected hub?
What service area does it support?
Are nearby hubs also impacted?
Is this isolated or part of a larger outage?
Which ISPs or customers may need to know?
Is this related to a storm, maintenance event, or broader network issue?

Most of those questions are about geography and operations. The network team had the monitoring data. The GIS team had the location and service-area data. The missing piece was a shared view that connected both.

The goal

The goal was to build a near-real-time operational map that joined hub monitoring status with GIS context. The map needed to help different groups answer the same basic question:

What is affected, and where?

Network admins wanted faster triage. Field technicians needed a clearer sense of location. ISPs needed better communication about possible service impact, and management wanted a quick read on how big an outage was.

Zabbix and ArcGIS would stay as they were. The map would connect them so outage information was easier to understand.

The initial concept

The first version was simple: take hub status from Zabbix and put it on a map using GIS data. That needed three things for each hub:

Its status from Zabbix
Its physical location
The service area it supports

With those connected, the map could show more than up or down. It could show where a hub was and what area might be affected.

The prototype

I started with a custom web app: React, TypeScript, and Vite, with MapLibre for the map and the Zabbix API for status. It handled search, clustered map features, and refreshed status on a timer. It showed the idea worked: hub status was much easier to read on a map.

It also showed the problem with a browser app. Zabbix and ArcGIS both need authenticated access, and putting those tokens in client-side code would have been a bad design. I needed a server-side layer that held the credentials, cached the data, and only exposed what the browser actually needed.

Moving to ArcGIS

The GIS data already lived in ArcGIS, and ArcGIS already handled mapping, authentication, layer rendering, and user interaction well. Instead of maintaining my own map app, it made more sense to display everything in ArcGIS Enterprise Map Viewer and keep my code focused on the data.

There was one hard constraint: I didn’t have write access to ArcGIS, so I couldn’t push features into GIS. The design had to work the other way around. My service publishes standard geospatial feeds, and ArcGIS reads them.

How it works now

The service is one Docker image that runs in two roles, a worker and an API, with Redis between them:

Zabbix (JSON-RPC)
      │
      ▼
Worker   holds the Zabbix token,
      │  sweeps every 5 minutes
      ▼
Redis    cache of the last good snapshot
      │
      ▼
API      read-only, no credentials
      │
      ▼
Caddy    TLS
      │
      ▼
ArcGIS Map Viewer

The split keeps upstream credentials out of the API. The worker is the only process that holds the Zabbix token and the only one that talks to Zabbix. The API refuses to start if it finds Zabbix or ArcGIS credentials in its environment, so the process that serves the feeds never has the token to leak.

It’s written in TypeScript (strict) on Node 22, with Fastify for the API, Redis for shared state, Zod to validate every response from Zabbix, pino for structured logs with secrets redacted, and Prometheus metrics. There are 148 tests, unit and mocked integration, using Node’s built-in test runner.

The sweep

Every five minutes the worker pulls the hub hosts from a dedicated Zabbix host group, then their latest metric values: ping, packet loss, and latency, uptime, and custom counters for how many customer ONTs on each hub are up and down. Then it pulls active problems. Disabled hosts are included on purpose, so a hub that gets disabled shows as inactive instead of disappearing and looking like it was deleted.

So “live” really means near-real-time: the feeds are refreshed once per sweep.

Zabbix 7 needed some workarounds. problem.get doesn’t return the related hosts, so problems are matched to hosts through trigger.get. item.get can’t sort by host ID either, so the code can’t rely on that order.

There are two kinds of failure, and they’re handled differently. If a query fails or a response fails validation, the whole sweep is thrown out and the map keeps showing the last good snapshot. I never want to publish a half-built picture. A problem with one host, like a hub ID that can’t be resolved, only drops that host, and the rest of the sweep goes ahead.

Matching hosts to hubs

The hardest part was the data matching. Zabbix and GIS weren’t designed around the same naming model, so every Zabbix host has to resolve to exactly one hub ID. The worker checks, in order: a hub_id tag on the host, an inventory field, a manual override list, and finally the hub number parsed from the hostname.

If a host can’t be resolved, or two hosts claim the same hub, that host is quarantined. It doesn’t get guessed. A hub linked to the wrong service area would give people wrong information in the middle of an outage, and that’s worse than a hub missing from the map.

Hub coordinates come from Zabbix inventory and get checked against a bounding box around Grant County, which catches 0,0 and other bad entries. A hub with bad coordinates stays in the data but isn’t plotted.

Service-area polygons come from a GeoJSON export of the GIS layer, loaded when the API starts. Each polygon lists its member hubs, and at request time the API joins current status onto that geometry. GIS is still where the shapes are maintained, but my copy doesn’t update itself. When the shapes change in GIS, the export has to be refreshed.

Deciding status

Each hub gets one status, and the order of precedence is strict:

Inactive
Maintenance
Down
Warning
Unknown
Up

The checks run in that order. A disabled host is inactive. A host in a Zabbix maintenance window is maintenance. A host is down if its SNMP interface is down, if a recent ping failed, or if it has an active problem that matches ping, unavailable, or down at high severity. Any other active problem makes it a warning. If the ping data is missing or stale, the status is unknown, and only then is a hub up.

Maintenance is checked before reachability on purpose. A hub that’s being worked on shouldn’t set off alarms, even if it’s unreachable. The tradeoff is that the map can’t show “unreachable during maintenance” today.

All of it is pure functions with the clock passed in, so every case can be unit tested.

Keeping the snapshot consistent

The worker writes each sweep to Redis as a staged generation, then switches a pointer to it in one atomic step. Readers always see a complete snapshot, never one in the middle of an update. A lease in Redis, renewed on a heartbeat, keeps two workers from sweeping at once. If the renewal fails, the sweep aborts.

There are a couple of safety checks before anything is published. If the host count drops by more than 20% from the last sweep, the sweep isn’t published. A hub that disappears from Zabbix is only removed after several clean sweeps in a row, so one bad sweep can’t wipe hubs off the map.

The feeds, and why there are several

The API serves the same data in a few formats:

JSON and GeoJSON, for general programmatic access
OGC API - Features, which is what ArcGIS Enterprise Map Viewer adds natively
CSV, as a fallback that ArcGIS can fetch and read attributes from
KML, which turned out to be a dead end for us

KML didn’t work because Map Viewer fetches it through Esri’s public utility service, and that service can’t reach a server on an internal network. The OGC and CSV feeds are fetched by the viewer’s own browser, so they work.

One thing I haven’t confirmed yet is how often our Map Viewer version re-reads these layers. The worker updates every five minutes, but that doesn’t guarantee the map refreshes every five minutes. That’s on my list to verify.

The bigger lesson was styling. Our ArcGIS Enterprise version couldn’t style a layer by attribute, so one layer colored by status wasn’t an option. Instead the API serves pre-filtered layers: hubs down, hubs warning, hubs up, and the same for service areas. Each one gets added to the map with a single color. The server does the filtering the client can’t.

Aggregate before you alarm

One down customer doesn’t mean an area is in trouble, so customer impact is added up before anything turns orange. There are three separate ideas here, and the feeds keep them separate: whether one hub is healthy, whether a group of related hubs has enough customers down, and whether a service area is affected.

A service area is red if any of its member hubs is down. If none are down but the loops down across its hubs reach the threshold (five by default), it’s orange. Down always wins, so an area never shows up as both.

The newest change, still in review as I write this, applies the same idea to individual hubs. Hubs that share a name suffix share an upstream path, so they’re grouped. A hub that’s down is always red. Otherwise it turns orange when its group’s total loops down reaches the threshold. A group at 1 + 1 + 4 = 6 down lights up all three hubs, while a group at 1 + 1 + 1 = 3 stays green. The red, orange, and green layers between them cover every hub, so none can fall through.

Security and deployment

The image is distroless and runs as a non-root user, with a read-only filesystem, all Linux capabilities dropped, and no privilege escalation.
The Zabbix token is mounted as a file at runtime, readable only by the container user. Host-specific config, certificates, and geometry live outside the repository.
Caddy terminates TLS with a certificate from an internal CA and keeps operational endpoints off the public side.
Configuration is validated at startup. A bad setting stops the service before it opens a port, and the error names the setting without printing its value.

When things break

Redis is a cache, not a system of record. If it’s lost, the next sweep rebuilds it, so there’s nothing in it to back up. The host-side config, certificates, secrets, and geometry export are different. They can’t be regenerated, so they need backups.

If Zabbix is unreachable, the last good snapshot keeps serving. The map can be out of date, but it never shows state that didn’t come from Zabbix. The GeoJSON and OGC feeds mark each hub as stale once its data is older than 15 minutes. The CSV feeds don’t have that flag yet, which is a gap, since CSV is what the map uses for hub points.

The places it can break are known and written down in the runbook: a renamed Zabbix host group, changed ONT counter keys, a change to hub naming, an expired certificate, or a rotated token. Each one has its symptom and fix.

User experience

During an outage people need answers fast, so the map has to make status obvious. Hubs show green, orange, or red, and service areas light up only when they’re actually affected. Hub names and site codes needed to be searchable, and clicking a hub needed to show useful context. The same view has to work for technical troubleshooting and for explaining an outage to someone non-technical.

I didn’t put every available field on the map, only what helps during an outage.

Operational value

The biggest value of the Hub Map was shared context. Before the map, an outage required people to mentally connect information across monitoring tools, GIS layers, and internal knowledge. After the map, the relationship between network health and geography became visible.

Finding the failed device is only the start of outage response. You also need to understand the impact, and a hub outage makes a lot more sense when you can see:

Where it is
What area it serves
Whether nearby hubs are affected
Whether the outage pattern matches a storm path or known maintenance event
Whether ISP communication may be needed

That’s where it stopped being a technical dashboard and became an operational tool.

What I’d improve

The engineering reference for this service lists its known limitations, and these are the ones I’d work on first:

Add a stale flag to the CSV feeds, and have the health check fail when the snapshot is too old instead of only when it’s missing
Check lease ownership when a generation is promoted. Today a worker that just lost its lease could still publish an older (but complete) snapshot before the next sweep replaces it
Reconcile the ONT counters. The map trusts the down count as reported, without checking that total equals up plus down or that the counters are fresh
Show “unreachable during maintenance” as its own state
Group hubs by their actual upstream instead of the name suffix, which only works as long as suffixes aren’t reused
Confirm the Map Viewer refresh rate in our ArcGIS version
A proper admin workflow for fixing bad hub mappings, instead of editing the override list
Outage history and timeline views
Separate views for internal teams and ISP-facing summaries
Better mobile views for field use
Links from Zabbix alerts straight to the hub on the map
Tracking which views people actually use during events

Tools used

Zabbix 7 (JSON-RPC API)
ArcGIS Enterprise Map Viewer
Node 22, TypeScript, and Fastify
Redis
Zod, pino, and Prometheus metrics
Docker, Docker Compose, and Caddy
React, Vite, and MapLibre for the original prototype

Final takeaway

More alerts wouldn’t have helped. What the team was missing was context: where the hub is, what it serves, and who’s affected. The service turns validated Zabbix data into standard GIS feeds, and the logic that matters (status, hub matching, and aggregation) lives in small, tested functions. The fragile parts are in two known places: the Zabbix data contract and the GIS geometry export.