Why polling regulatory data breaks
The default way to consume GET /updates is to poll. Every 15 minutes, every hour, every night — whatever the customer's tolerance allows. Polling has three problems that show up specifically with regulatory data:
Latency by design. The customer's tolerance is their maximum staleness. If you're an NDIS provider billing therapy sessions at $214.14 an hour, and the NDIA cuts that rate at 00:00 AEST, and your billing system polls at 06:00, you've just billed 6 hours of stale rates. Multiply by every customer and every rule change and the compound cost is real.
Cost linear in tolerance. Halve your polling interval, double your API bill. The economics push toward staleness, not freshness.
Easy to miss. A polling loop that dies at midnight and comes back at 03:00 has no idea it missed anything. There's no receipt, no X-Sequence-Number, no "you were offline for 3 hours". If a regulation changed and un-changed in that window — rare but real, especially with draft instruments — the poll never sees it.
Webhooks solve all three: latency is server-controlled, cost scales with actual change frequency (which is a lot lower than "twice a minute"), and every delivery carries an event_id you can dedup on.
What we built
Eight API endpoints under /api/webhooks/* — register, list, get, patch, delete, rotate-secret, test-send, list-deliveries. A delivery worker that fires every 30 seconds via systemd .timer, HMAC-SHA256-signs each payload, POSTs with a 10 s timeout, and updates delivery state per attempt. Two SQLite tables — one for registered endpoints, one for the per-attempt delivery log.
Total wire time: one focused day for the API side, an afternoon for the dashboard UI. Everything lives at regintelapi.com/docs.html#webhooks.
Three design decisions worth calling out because they're the ones you'll get wrong the first time.
Signing: HMAC-SHA256 over timestamp.body
Every delivery carries three headers:
X-RegIntel-Signature: sha256=<hex> X-RegIntel-Timestamp: 1721304191 X-RegIntel-Event-Id: evt_a1b2c3d4...
The signature is HMAC-SHA256(secret, "<timestamp>.<body>") — same shape Stripe uses. Timestamp goes into the signed content so an attacker who intercepts a delivery can't replay it 6 hours later against your endpoint. Event ID is idempotency: if you receive two deliveries with the same event_id (retries), you dedup.
Verification in Python is five lines:
import hmac, hashlib, time def verify(body: bytes, timestamp: str, signature: str, secret: str) -> bool: if abs(time.time() - int(timestamp)) > 300: # reject replays > 5 min old return False expected = "sha256=" + hmac.new( secret.encode(), f"{timestamp}.{body.decode()}".encode(), hashlib.sha256, ).hexdigest() return hmac.compare_digest(expected, signature)
Two things go wrong in this code the first time developers write it.
First, always use hmac.compare_digest, never ==. Timing-attack resistance matters even for internal endpoints — the number of bytes an attacker learns per failed request is small, but non-zero, and the attacks are cheap to run.
Second, sign the timestamp. A signature over just the body lets an attacker replay yesterday's payload today, undetectably. The timestamp check turns a valid-but-stale payload into a rejection.
The signing secret is returned exactly once — from POST /api/webhooks on register and POST /api/webhooks/{id}/rotate on rotate. RegIntel cannot re-serve it. If you lose the secret, rotate.
Retry semantics: 0 s, 30 s, 5 m, 30 m, 3 h
Any HTTP 2xx counts as delivered. Anything else — non-2xx, timeout past 10 s, network error — is a failed attempt. Attempts fire at 0 s, 30 s, 5 m, 30 m, 3 h off the initial schedule. After the fifth attempt the delivery is marked exhausted and the endpoint's consecutive_failures counter bumps.
The 3 h final wait is deliberately long. A customer whose endpoint is genuinely down for a two-hour maintenance window gets the retry after they come back up. A customer whose endpoint is broken forever fails within the first hour and the counter bumps — no harm in trying once more three hours later just in case.
Auto-pause after 24 hours of continuous failure. If the endpoint hasn't succeeded in a day and there's a pending un-delivered event that's also older than 24 hours, we flip active = 0 and email the account owner. This is the point where retrying stops being helpful and starts being noise — for us (worker cycles) and for the customer (their receiver logging errors for a dead endpoint).
Filtering: subscribe to what matters
Empty filter matches every regulation change. That's rarely what you want. The filter object supports three fields, AND'd together:
{
"url": "https://your-app.example.com/webhooks/regintel",
"filters": {
"jurisdiction": "Australia",
"industry": "Sustainability",
"tags": ["aasb-s2", "climate-disclosure"]
}
}
Within tags, ANY match qualifies. So the example above matches any Australian sustainability regulation tagged either aasb-s2 or climate-disclosure. jurisdiction and industry are exact-match against the canonical values from GET /jurisdictions. Tags are inclusive.
If we've picked wrong — if you want per-regulation-id subscriptions (only fire when regulation 351 changes), or Slack-native delivery, or grouped batches — those are on the Phase 2 list. Say so and we'll prioritise.
What we deliberately didn't ship
Three things a competitor might have and RegIntel doesn't, that we can defend:
No Slack, Discord, or MS Teams native integrations. You proxy via your own handler. Reason: every native integration is a support surface, a rate-limit surface, and an auth-scope surface. We'd rather do one thing (signed HTTP POST) well than five things badly.
No delivery batching. One regulation change → one delivery. Reason: batching creates uncomfortable timing questions ("how long do you hold?", "what happens mid-batch if a receiver dies?"), and the actual change frequency across a subscribed slice is nowhere near the volume that would justify it. If you're at 10 k deliveries per month, batching saves you nothing.
No per-regulation-id filter. Not yet. It's a defensible request and the SQL isn't hard; we've held it back so we don't ship a filter shape we'll have to break later.
How to try it
If you have a Starter or Pro key, the dashboard has a Webhooks tab — regintelapi.com/dashboard.html — where you can register an endpoint in a form and fire test deliveries. No API key yet? Get one here — 100 free credits, no card.
Full endpoint reference, payload shape, signature verification snippets in Python and Node, and the retry table are all at regintelapi.com/docs.html#webhooks.
If you build something with these and want a callout, reply to the weekly digest — happy to feature real integrations.
The other build log from this year — how RegIntel became a Model Context Protocol server, and why MCP matters for a REST API.