This is the deployment a senior SRE would build for jetway carrying real airline traffic: a carrier's reservations and departure-control host, or a message switch, with the availability the business expects from its Type B and EDIFACT plumbing. Not a web service. Long-lived TCP sessions pinned behind a passthrough load balancer, one writer per system, warm standbys that already hold the links, a regional database with a synchronous standby, and a release process that never restarts everything at once. Most of the availability comes from the topology, not from the code.
Every partner — a GDS, an interline carrier, a ground handler, SITA or ARINC — holds one or a few long-lived sessions to you. A link that is down is a partner queueing messages or, worse, routing around you. Link availability is the number they measure you by.
A sell over Type B is expected to be answered in seconds; the GDS's own timers run 30 to 120 seconds before it queues the booking for a human. EDIFACT interactive sessions are tighter. Availability is answered in hundreds of milliseconds or the shopping engine drops you.
The write-ahead spool and the message log exist for this. A message acknowledged and lost is a passenger booked at the GDS and unknown to the airline. A message applied twice is an oversell.
One record, one version, one writer at a time. Optimistic concurrency does the work inside one database; there must never be two databases that both think they are it.
The reference is Google Cloud because that is what the plan was written against; every piece has a direct equivalent elsewhere (an NLB is an NLB, Cloud SQL HA is RDS Multi-AZ, a MIG is an ASG). The shape is what matters.
Regional managed instance groups over three zones running the jetwayd container or the binary under systemd. Not serverless, not autopilot Kubernetes: the value is long-lived inbound TCP with source-IP identification and MATIP framing, stable addresses, no request-scoped scaling, and control over connection draining. Kubernetes Standard works if the organisation already runs it, with host networking or an internal passthrough LB per service and disruption budgets that keep one link instance per system up; it buys nothing over instance groups here.
A passthrough NLB with connection tracking by 5-tuple, persistence on unhealthy backends off so a failed instance's partners reconnect rather than hang, and client-IP affinity so a partner that opens several sessions lands on one instance (jetway's by_hello and source-IP resolvers assume a peer's sessions share a process). A TCP proxy is not usable: framing and identity must survive.
Link tier in a private subnet; NAT for the few outbound dials; partner-facing addresses reserved and static, because partners whitelist you by IP and change control on their side takes weeks. Private access to secrets, logging and registries without NAT. Regional residency: a European carrier's book does not replicate to another continent, even for DR, without the lawyers first.
jetway models many systems in one database (one node view per system). In production run one process per system per instance, not one for all: a panic in one carrier's decoder takes down one carrier. Each system gets its own container and unit, its own listener ports, its own metrics labels, its own spool directory, and the same database as its own node view.
wholesky runs every one of these as an embedded assembly in one process on a laptop and as separate machines on the deployed demo; the switch cannot tell the difference. In production the same seams are where you scale, isolate and delegate.
The rule for a split is that nothing crosses it but a message the industry already defines. That is what lets a departure-control system move to the airport, a switch move to a network provider, or a carrier's host move to a hosting company, without either side learning a private API.
Optimistic concurrency protects a single database from two writers racing on one record. It does not protect against two processes both believing they own a system's links, because a partner's messages would then be split between them and answered twice. The lease is the answer.
system_lease, in the book's own database, so the lease and the book share one failure domain ↓ON CONFLICT … WHERE expires_at < now() OR holder = EXCLUDED.holderThe other two instances hold the load; the lease moves in seconds; partners reconnect through the NLB. The database fails over to its synchronous standby in under a minute with no committed transaction lost. During the failover writes fail; the spool holds inbound messages and the gateway refuses acknowledgements rather than acknowledging what it cannot store. Partners see a pause, not a loss.
Autohealing recreates it; the lease has already moved. The spool directory is on a persistent disk that outlives the VM and is reattached, or a regional disk if the spool must survive a zone. Unflushed spool entries replay on start.
The failure mode that actually happens. statement_timeout of 5 seconds on the gateway's pool: the answer to a sell that takes longer is a NAK the partner retries, not a hung link. A queue-depth alert on the pooler. And the outbox: a slow database no longer stalls the read loop of every link; it becomes ErrCongested on the sends that cannot leave, recorded as undeliverable and retried.
Handled the same way, per link: the outbox fills, sends to that peer fail fast, congestion clears only when the writer has caught up, other links are unaffected. Alert on jetway_outbox_congested_total{peer}; it is a partner problem until it is not.
Rolling, one instance at a time, maxUnavailable=0, maxSurge=1, with a connection drain: mark not-ready, release the leases so standbys take the systems, wait for the outboxes to empty and the spool to flush, then stop. Partners reconnect once per deploy. Never deploy the link tier and a migration in the same window; run migrations from a one-off job first, because a long conversion on a populated book took eight minutes and doubled the table's disk while it ran.
A YAML config in the image or a pinned secret version, never mutable on the instance. Adding a partner is a config change and a SIGHUP: Node.ReloadPeers adds what is new; removing a peer is a restart. Rate limits per peer with a shared cap per ingress, so one flooding partner cannot take the others' share.
PostgreSQL 17, 16 vCPU and 128 GB primary with a synchronous standby, 500 GB SSD to start because IOPS scale with disk size. The GIN index on the record's state is the expensive one; it makes flight and locator lookups fast and is roughly the size of the data. Encryption with customer-managed keys for both the records and the raw message log, which holds the same personal data.
An auth proxy in front of pgbouncer in transaction mode per instance, pool_max_conns 24 per process. jetway sets default_query_exec_mode=cache_describe for transaction pooling and runs COPY inside explicit transactions, both of which work through the pooler; it uses transaction-level advisory locks only, because session-level ones would not.
The message log is partitioned by month and records by their retirement day: the last flight plus the carrier's purge grace, typically three days. A nightly job calls POST /api/admin/retire or jetwayctl retire --before: daily partitions drop, stragglers in the default partition are deleted, orphaned queue items go too. A partition that cannot be created never stops a write, and an occupied default partition is detected under a share lock rather than discovered by a catalogue scan that stops every reader.
Dense-numbered, never edited once applied anywhere. The runner takes an advisory lock and re-checks, so two instances racing to migrate on a rolling deploy is safe; still run them from a one-off job first so a long conversion does not happen inside an instance's startup timeout.
retire, and the store now survives the case anyway.A cross-region asynchronous replica (its lag is the RPO, normally under a second) and a cold link tier: the same instance template at size zero, the same load balancer configuration with its own reserved addresses. Region failover changes the addresses partners see and half of them will need a phone call, so it is a runbook: promote the replica, scale the tier up, repoint the partners' alternate address.
Most Type B contracts specify a primary and an alternate address. Give partners the DR region's address as the alternate on day one. Accept that in-flight messages at the failed region after the last replicated transaction are lost, which is why the spool is regional and the RPO for this case is replication lag, not zero.
Automated daily with 35-day point-in-time recovery, plus a weekly logical export of each system's records (GET /api/admin/export, newline-delimited JSON, oldest first) to object storage with a retention lock, because the regulator will ask for a booking from four years ago and a PITR window does not answer that. Restore quarterly into a scratch instance and run jetwayctl decode and the console against it.
| # | alert | condition | action |
|---|---|---|---|
| 1 | Links down | jetway_ingress_links below the contracted count for a partner for 60 s | page |
| 2 | Undeliverable rate | messages a switch or host could not deliver, above a floor and rising for 5 min | page |
| 3 | Reply latency | p99 inbound sell to reply above 5 s; page at 30 s | page |
| 4 | Dead letter queue | anything routed to the DLQ is a message a person must read | ticket · page above a rate |
| 5 | Spool depth and age | jetway_spool_oldest_seconds over 10: the database is not keeping up | page |
| 6 | Outbox congestion per peer | jetway_outbox_congested_total{peer} | ticket · escalate to the partner |
| 7 | Divergence queue growth | records the airline and a partner disagree about, growing faster than agents work them | ticket daily |
| 8 | Database | replication lag, connections, lock waits, disk, and the size of pnr_default, which should be near zero | standard alerting |
/metrics in Prometheus exposition without a client library; OpenTelemetry traces with a trace id in every log line that has one; slog JSON to stdout. Inventory gauges per carrier (sold, waitlisted, full cabins) and decisions by status; store partition failures by day; sequence gaps and repeats per peer.
The switch console's views are the operator's. The SRE dashboard is wholesky's stats page rebuilt from the same metrics: message rates by class, movements, bookings, sold per second, undeliverables, queue depths, and the number that should be zero, labelled as such.
A day of the world's schedule against a staging instance of the production topology, with the invariant suite as the pass criterion. Every number in this plan is one wholesky measured, and every production defect this year was found by it before a partner could.
Every shard reports its inventories, the core federates them, and skycheck exits non-zero on a cabin holding more than it has or a shard that did not answer, because a machine being down must not read as a pass. Conservation and convergence run in-process, because they need the wire quiet.
$ go run ./cmd/skycheck https://staging.example
6 shards, 27639 cabins, 101654 seats sold, 0 oversoldjetwayload runs the end-to-end scenario suite concurrently through the real assembly on real TCP and reports latency. A scenario exists for anything that crosses a link, because unit tests do not catch what only shows up with a partner on the other end.
In the order a team would build it. Eight of nine are done in full; the ninth runs in CI and lacks only a staging topology. The weekly archive export the DR section calls for exists too: jetwayctl export --out records.ndjson streams every record a node holds.
| # | item | state | what it is |
|---|---|---|---|
| 1 | A system lease | v0.1.48–51 | bind links only while holding the system's row; standbys poll and take over; a holder asked to stop drains first and releases after |
| 2 | Readiness | done | /readyz fails when the store cannot be reached, while standing by, and when the spool's oldest entry is older than 30 s; distinct from /healthz |
| 3 | Drain on SIGTERM | done | in-flight handlers, then every session's outbox, then HTTP; the lease is released after |
| 4 | Spool on by default | v0.1.52 | bounded by spool.max_entries, with depth and oldest-age metrics for the alert |
| 5 | Retire as an operation | done | POST /api/admin/retire and jetwayctl retire --before, so retention is a scheduled job |
| 6 | Hot peer reload | done | SIGHUP re-reads the config and adds new peers; removing one is a restart |
| 7 | Metrics for the outbox and the inventory | v0.1.53 | outbox depth and congestion per peer; inventory decisions by status and per-carrier gauges |
| 8 | Rate limiting per peer | v0.1.53 | per-peer pacing with a shared cap per ingress, and peers[].rate_limit for a peer's own share |
| 9 | Load test as a release gate | most | wholesky's gate workflow boots a world at warp 240 on every push, flies its banks and fails the build on an oversold cabin, a silent shard, or a sky that did not move or sell; a staging instance of the production topology is still open |
Per region at list prices: three 8-vCPU link instances about $600 a month; a 16-vCPU HA database with a read replica about $3,500; the cross-region DR replica another $1,700; load balancing, edge policy, NAT, logging and monitoring a few hundred. Call it $6,500 a month for one region with DR, which is what one Type B circuit from a network provider used to cost per year in the era the protocol was designed, and about a day of one GDS's segment fees for a mid-sized carrier.