Build a reliable MQTT client

What to handle in a client for the alerts and lightning feeds so it stays correct for months.

This is for a client you mean to leave running. A first client is in Subscribe to live alerts over MQTT; the topics and payloads are in the alerts feed and lightning feed references.

Give every client its own ID

Two clients sharing a client ID kick each other off in a loop. Derive it from something stable and per-installation rather than hardcoding a constant.

There is a 20-subscription cap per client. The 21st subscribe is refused with a failure code in the SUBACK while the connection stays up, so check the return codes.

On the alerts feed, a subscription per county-per-phenomenon will hit the cap. Subscribe to same/{code}/# and filter in your own code.

On the lightning feed, cover a large area with shorter geohash prefixes rather than many long ones.

Reconnect with exponential backoff

Every resubscribe replays the whole retained set. A hot reconnect loop is a bandwidth amplifier pointed at the broker, which is why it accepts at most 50 new WebSocket connections a second across all clients. Use exponential backoff, and note that holding an MQTT session open without reading the socket does not detect a dead connection.

Handle alert tombstones

A hazard ends with an empty retained payload on the topic it lived on. Test for a zero-length payload before you parse JSON, and drop the alert when you see one. Retained alerts, and tombstones explains why.

Decide an alert’s state from `status` and `ends`

expires is not the end of the hazard. It is CAP’s “expect an update by” deadline and routinely sits in the past on a live alert. Use ends, and trust status and tombstones over your own arithmetic.

action is the latest VTEC action, not the state. A hazard that has been cancelled for some of its counties stays active, on its remaining counties, with action set to CAN. Go by status.

Dedupe alerts on `id`

One hazard arrives more than once. A warning covering three counties is published on three same/ topics plus its office topic, with an identical payload each time. If your filter overlaps, dedupe on id.

Style alerts by `geometry_source`

geometry_source tells you how precise the shape is. polygon is a tight storm-based warning polygon; ugc is a coarser union of county boundaries. Worth styling differently. Geometry is simplified for the wire.

Read the lightning feed

This client subscribes to a geohash box, roughly the Dallas-Fort Worth area, and prints each flash.

import json
import paho.mqtt.client as mqtt

def on_connect(client, userdata, flags, reason_code, properties):
    # Roughly the Dallas-Fort Worth area.
    client.subscribe("wxalerts/glm/v1/9/v/g/#")

def on_message(client, userdata, msg):
    batch = json.loads(msg.payload)
    for flash in batch["flashes"]:
        print(flash["lat"], flash["lon"], flash["t"])

client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, transport="websockets")
client.username_pw_set("wxalerts", "wxalerts")
client.tls_set()
client.on_connect = on_connect
client.on_message = on_message
client.connect("mqtt.wxalerts.org", 443)
client.loop_forever()

Age lightning flashes out yourself

Flashes are not retained, so there is no tombstone when a flash is over. Age flashes out yourself, and ten minutes is a reasonable ceiling.

Time lightning by observation, not arrival

A flash is typically 20 to 40 seconds old when it reaches you. That is older than any sensible animation window. Fade on the observation timestamp, t; animate on arrival.

Keep your own history

The REST API keeps 24 hours of flashes. For anything finer over a longer period, keep the flashes yourself as they arrive.