This page describes the production deployment for jetway as a carrier's reservations and departure-control host, or as a message switch. The design is what a senior SRE would build, with the availability the business expects from its Type B and EDIFACT infrastructure. jetway is not a web service. The topology has long-lived TCP sessions pinned behind a passthrough load balancer, one writer per system, and warm standbys that already hold the links. It has a regional database with a synchronous standby, and a release process that never restarts everything at once. Most of the availability comes from the topology rather than from the code.
Every partner holds one or a few long-lived sessions to you. A partner is a GDS, an interline carrier, a ground handler, SITA or ARINC. When a link is down, the partner queues messages or routes around you. Partners measure you by link availability.
Partners expect an answer to a sell over Type B within seconds. The GDS's own timers run 30 to 120 s before it queues the booking for a person. EDIFACT interactive sessions allow less time. An availability request needs an answer within hundreds of milliseconds, or the shopping engine drops you.
The write-ahead spool and the message log exist for this requirement. A message that is acknowledged and then lost leaves a passenger booked at the GDS and unknown to the airline. A message applied twice is an oversell.
Each record has one version and one writer at a time. Optimistic concurrency enforces this inside one database. There must never be two databases that both hold the book of record.
The reference platform is Google Cloud, because the plan was written against it. Every piece has a direct equivalent elsewhere. An NLB is an NLB, Cloud SQL HA is RDS Multi-AZ, and a MIG is an ASG. The shape of the topology is the important part.
Regional managed instance groups over three zones run the jetwayd container or the binary under systemd. The plan does not use serverless or autopilot Kubernetes. The link tier needs long-lived inbound TCP with source-IP identification and MATIP framing, stable addresses, no request-scoped scaling, and control over connection draining. Kubernetes Standard works if the organisation already runs it. It needs host networking or an internal passthrough LB per service, and disruption budgets that keep one link instance per system up. It offers no advantage over instance groups here.
The load balancer is a passthrough NLB with connection tracking by 5-tuple. Persistence on unhealthy backends is off, and a failed instance's partners reconnect instead of hanging. Client-IP affinity puts a partner that opens several sessions on one instance. jetway's by_hello and source-IP resolvers assume a peer's sessions share a process. A TCP proxy is not usable, because framing and identity must survive.
The link tier sits in a private subnet, with NAT for the few outbound dials. Partner-facing addresses are reserved and static, because partners whitelist you by IP and change control on their side takes weeks. Secrets, logging and registries have private access without NAT. Data residency is regional. A European carrier's book does not replicate to another continent, even for DR, until the lawyers approve.
jetway models many systems in one database, with one node view per system. In production, run one process per system per instance. Do not run one process for all systems. A panic in one carrier's decoder then takes down only that carrier. Each system gets its own container and unit, its own listener ports, its own metrics labels and its own spool directory. It uses the same database through its own node view.
wholesky runs every one of these as an embedded assembly in one process on a laptop, and as separate machines on the deployed demo. The switch behaves the same in both cases. In production, the same seams are where you scale, isolate and delegate.
The rule for a split is that only a message the industry already defines crosses it. This rule lets a departure-control system move to the airport, a switch to a network provider, or a carrier's host to a hosting company. Neither side has to learn a private API.
Optimistic concurrency protects a single database from two writers racing on one record. It does not protect against two processes that both hold a system's links. In that case a partner's messages split between them, and both processes answer. The lease prevents this case.
system_lease in the book's own database. The lease and the book share one failure domain ↓ON CONFLICT … WHERE expires_at < now() OR holder = EXCLUDED.holderThe other two instances hold the load. The lease moves in seconds, and partners reconnect through the NLB. The database fails over to its synchronous standby in under a minute, and no committed transaction is lost. Writes fail during the failover. The spool holds inbound messages, and the gateway refuses acknowledgements for what it cannot store. Partners see a pause and no loss.
Autohealing recreates the instance, and the lease has already moved. The spool directory is on a persistent disk that outlives the VM and is reattached. Use a regional disk if the spool must survive a zone. Unflushed spool entries replay on start.
This is the common failure mode. The gateway's pool has a statement_timeout of 5 s. A sell that takes longer receives a NAK, which the partner retries, and the link does not hang. The pooler has a queue-depth alert. The outbox keeps a slow database from stalling the read loop of every link. The sends that cannot leave return ErrCongested, and the gateway records them as undeliverable and retries them.
A slow partner is handled the same way, per link. The outbox fills, and sends to that peer fail fast. Congestion clears only when the writer has caught up. Other links are unaffected. Alert on jetway_outbox_congested_total{peer}. Treat it as a partner problem until evidence shows otherwise.
Deploys are rolling, one instance at a time, with maxUnavailable=0 and maxSurge=1, and each instance drains its connections first. The drain marks the instance not-ready, releases the leases to the standbys, waits for empty outboxes and a flushed spool, and then stops. Partners reconnect once per deploy. Never deploy the link tier and a migration in the same window, and run migrations from a one-off job first. A long conversion on a populated book took 8 minutes and doubled the table's disk while it ran.
The YAML config lives in the image or in a pinned secret version. It is never mutable on the instance. Adding a partner is a config change and a SIGHUP, and Node.ReloadPeers adds the new peers. Removing a peer is a restart. Rate limits are per peer with a shared cap per ingress, and one flooding partner cannot take the others' share.
The primary is PostgreSQL 17 with 16 vCPU and 128 GB, and it has a synchronous standby. It starts with 500 GB of SSD, because IOPS scale with disk size. The GIN index on the record's state is the expensive index. It makes flight and locator lookups fast, and it is about the size of the data. Customer-managed keys encrypt both the records and the raw message log. The message log holds the same personal data.
Each instance runs an auth proxy in front of pgbouncer in transaction mode, with pool_max_conns 24 per process. jetway sets default_query_exec_mode=cache_describe for transaction pooling and runs COPY inside explicit transactions. Both work through the pooler. jetway uses transaction-level advisory locks only, because session-level locks do not work through the pooler.
The message log is partitioned by month. Records are partitioned by their retirement day, which is the last flight plus the carrier's purge grace, typically 3 days. A nightly job calls POST /api/admin/retire or jetwayctl retire --before. The job drops daily partitions, deletes stragglers in the default partition, and deletes orphaned queue items. A partition that cannot be created never stops a write. The store detects an occupied default partition under a share lock rather than by a catalogue scan that stops every reader.
Migrations are dense-numbered, and nobody edits one after it has been applied anywhere. The runner takes an advisory lock and re-checks. Two instances that race to migrate on a rolling deploy are therefore safe. Run migrations from a one-off job first anyway. A long conversion must not happen inside an instance's startup timeout.
retire, and the store now survives the case anyway.The cold region has a cross-region asynchronous replica and a cold link tier. The replica's lag is the RPO, and it is normally under 1 s. The cold link tier is the same instance template at size zero and the same load balancer configuration with its own reserved addresses. Region failover changes the addresses partners see, and half of them need a phone call. Failover is therefore a runbook. Promote the replica, scale the tier up, and repoint the partners' alternate address.
Most Type B contracts specify a primary and an alternate address. Give partners the DR region's address as the alternate on day one. In-flight messages at the failed region after the last replicated transaction are lost. The spool is regional for this reason, and the RPO for this case is the replication lag rather than zero.
Backups run daily, with 35-day point-in-time recovery. A weekly logical export writes each system's records to object storage with a retention lock. The export uses GET /api/admin/export and produces newline-delimited JSON, oldest first. The regulator will ask for a booking from 4 years ago, and a PITR window does not cover that. Restore quarterly into a scratch instance. Run jetwayctl decode and the console against it.
| # | alert | condition | action |
|---|---|---|---|
| 1 | Links down | jetway_ingress_links below the contracted count for a partner for 60 s | page |
| 2 | Undeliverable rate | messages a switch or host could not deliver, above a floor and rising for 5 min | page |
| 3 | Reply latency | p99 inbound sell to reply above 5 s, with a page at 30 s | page |
| 4 | Dead letter queue | anything routed to the DLQ is a message a person must read | ticket · page above a rate |
| 5 | Spool depth and age | jetway_spool_oldest_seconds over 10. The database is not keeping up | page |
| 6 | Outbox congestion per peer | jetway_outbox_congested_total{peer} | ticket · escalate to the partner |
| 7 | Divergence queue growth | records the airline and a partner disagree about, growing faster than agents work them | ticket daily |
| 8 | Database | replication lag, connections, lock waits, disk, and the size of pnr_default, which should be near zero | standard alerting |
/metrics uses Prometheus exposition without a client library. OpenTelemetry traces carry a trace id, and every log line that has one includes it. Logs are slog JSON to stdout. Inventory gauges per carrier cover sold, waitlisted and full cabins, and decisions are counted by status. The store reports partition failures by day. Each peer reports sequence gaps and repeats.
The operator uses the switch console's views. The SRE dashboard is wholesky's stats page rebuilt from the same metrics. It shows message rates by class, movements, bookings, sold per second, undeliverables, queue depths, and the numbers that should be zero, each labelled as such.
The load test runs a day of the world's schedule against a staging instance of the production topology. The invariant suite is the pass criterion. wholesky measured every number in this plan. wholesky found every production defect this year before a partner could.
Every shard reports its inventories, and the core federates them. skycheck exits non-zero on a cabin that holds more than its capacity or on a shard that did not answer. A machine that is down must not read as a pass. Conservation and convergence run in-process, because they need the wire quiet.
$ go run ./cmd/skycheck https://staging.example
6 shards, 27639 cabins, 101654 seats sold, 0 oversoldjetwayload runs the end-to-end scenario suite concurrently through the full assembly over TCP and reports latency. A scenario exists for anything that crosses a link. Unit tests do not catch a defect that only appears with a partner on the other end.
The items are in the order a team would build them. Items 1 to 8 are complete. Item 9 runs in CI and lacks only a staging topology. The weekly archive export that the DR section calls for also exists. jetwayctl export --out records.ndjson streams every record a node holds.
| # | item | state | what it is |
|---|---|---|---|
| 1 | A system lease | v0.1.48–51 | bind links only while holding the system's row. Standbys poll and take over. A holder asked to stop drains first and releases after |
| 2 | Readiness | done | /readyz fails when the store cannot be reached, while standing by, and when the spool's oldest entry is older than 30 s. It is distinct from /healthz |
| 3 | Drain on SIGTERM | done | in-flight handlers, then every session's outbox, then HTTP. The lease is released after |
| 4 | Spool on by default | v0.1.52 | bounded by spool.max_entries, with depth and oldest-age metrics for the alert |
| 5 | Retire as an operation | done | POST /api/admin/retire and jetwayctl retire --before, which make retention a scheduled job |
| 6 | Hot peer reload | done | SIGHUP re-reads the config and adds new peers. Removing one is a restart |
| 7 | Metrics for the outbox and the inventory | v0.1.53 | outbox depth and congestion per peer, inventory decisions by status, and per-carrier gauges |
| 8 | Rate limiting per peer | v0.1.53 | per-peer pacing with a shared cap per ingress, and peers[].rate_limit for a peer's own share |
| 9 | Load test as a release gate | most | wholesky's gate workflow boots a world at warp 240 on every push and flies its banks. It fails the build on an oversold cabin, a silent shard, or a sky that did not move or sell. A staging instance of the production topology is still open |
At list prices per region, three 8-vCPU link instances cost about $600 a month. A 16-vCPU HA database with a read replica costs about $3,500. The cross-region DR replica costs another $1,700, and load balancing, edge policy, NAT, logging and monitoring cost a few hundred. The total is about $6,500 a month for one region with DR. One Type B circuit from a network provider cost that per year in the era the protocol was designed. The total is also about a day of one GDS's segment fees for a mid-sized carrier.