Overload begins before the 429
By the time an upstream rejects traffic, your service may already be carrying too many large requests, retries, open streams, and waiting callers.
MoFlux makes capacity a workload-aware decision before execution. Tyr protects configured interactive capacity and sheds lower-priority work first when the provider gets tight. In eight counterbalanced live OpenAI overload runs, MoFlux preserved 99.4% of interactive requests versus 71.9% with an undifferentiated static cap.
Live OpenAI overload: 99.4% interactive success through Tyr versus 71.9% with the static cap; MoFlux led in all 8 counterbalanced seeds.MoFlux shed more batch work to preserve interactive capacity. No latency-win claim.
By the time an upstream rejects traffic, your service may already be carrying too many large requests, retries, open streams, and waiting callers.
A limit that is safe for one process can become unsafe when every replica independently assumes it owns the same shared capacity.
Waiting is not free. Under sustained load, queues can trade obvious rejection for timeouts, retry amplification, memory pressure, and a less predictable user experience.
Two different questions now have direct evidence. The live sweep tests whether protected concurrency changes who succeeds under real provider scarcity. The membership run tests whether Latchflo converges cleanly as the fleet changes.
Static concurrency completed 71.9%. MoFlux led in all 8 matched seeds, with a median paired advantage of +25.5 points. Both controlled arms kept provider 429s at zero.
Open reviewed OpenAI evidenceZero heartbeat revision churn, duplicate members, stale members after convergence, or revision regressions. Replacement joins propagated in 11.1ms p50; timeout removal was 1.40s p50.
This benchmark validates Latchflo membership and routing-topology convergence. It does not yet prove uninterrupted routed request traffic while peers churn.MoFlux is a specialized safety layer for the gap between “this request is allowed” and “this request is safe to start right now.” It is intended to integrate with an existing gateway and observability stack.
Pre-execution reservation, local admission, protected service-class capacity, fleet-level grants, and decision provenance—without a control-plane lookup on every request.
The goal is narrow: make capacity an explicit pre-execution decision, then leave ingress, provider governance, and broader gateway policy where they already live.
Admit, reject, or observe a request before it consumes provider-facing concurrency and expected in-flight token headroom.
Turn one capacity envelope into expiring local authority without placing the control plane on every provider request.
Map trusted identities to a bounded class table, apply hard ceilings and protected concurrency/token floors, and expose protected versus borrowed usage.
Surface rejection reason, policy revision, admission identity, grant ID, controller epoch, and bounded metrics for tuning and incident review.
moflux-bench v0.29.0 is public under Apache-2.0. The reviewed eight-seed OpenAI sweep now carries the protected-concurrency overload claim. The separately published August 26 synthetic ladder remains the evidence for token pressure, adaptive lending, and coordinator-distance behavior. Keeping those claims separate prevents one benchmark from pretending to prove the whole platform.
795/800 interactive requests completed through Tyr under live OpenAI overload.
575/800 completed with the same 36-request global concurrency ceiling and no workload protection.
Median MoFlux advantage across eight matched seeds; every seed was positive (17–58 points).
Both static and MoFlux kept provider 429s at zero; direct OpenAI produced 280 under the same offered workload.
pooled interactive success versus the static cap
counterbalanced seeds favored MoFlux on interactive success
median paired batch-success delta versus static
same global concurrency ceiling; Tyr protected 8 interactive / 4 batch
The synthetic ladder is still useful for token pressure, Redis coordination distance, and adaptive fleet behavior. It is no longer the primary proof that protected concurrency changes interactive outcomes under a real provider.
| Metric | No control | Static cap | Redis | MoFlux |
|---|---|---|---|---|
| Interactive successCompleted interactive logical requests | 81.1% · 75.8%–85.3% | 93.7% · 87.3%–98.7% | 92.7% · 88.3%–95.7% | 97.4% · 90.8%–100.0% |
| Interactive goodputSuccessful interactive requests per second | 4.79 · 4.38–5.13 | 5.52 · 5.07–5.91 | 5.50 · 4.89–6.04 | 5.73 · 5.13–6.24 |
| Interactive p50End-to-end latency among successful requests | 7.39s · 6.53s–8.51s | 5.77s · 5.07s–7.09s | 4.22s · 2.89s–5.64s | 5.25s · 3.08s–7.21s |
| Interactive p95End-to-end latency among successful requests | 14.87s · 12.70s–17.67s | 13.23s · 11.75s–15.14s | 10.30s · 7.67s–12.89s | 11.54s · 9.43s–13.12s |
| TTFT p50Time to first streamed output among successful requests | 2.96s · 1.27s–3.95s | 1.93s · 1.11s–2.53s | 0.79s · 0.56s–1.13s | 1.04s · 0.53s–1.84s |
| TTFT p95Time to first streamed output among successful requests | 7.23s · 6.44s–7.79s | 5.77s · 4.84s–5.98s | 2.92s · 2.14s–3.44s | 5.46s · 1.46s–7.27s |
| Retry amplificationAttempts per logical interactive request | 1.828 | 1.457 | 1.478 | 1.339 |
| Batch successCompleted batch requests; the adaptive policy guarantees liveness, not maximum batch throughput | 55.1% · 50.0%–69.8% | 23.0% · 17.2%–30.2% | 17.4% · 11.4%–23.3% | 9.2% · 8.6%–14.0% |
| Local rejectsAttempt-level work shed before provider execution | 0 | 295 · 196–443 | 306.5 · 231–420 | 264 · 160–401 |
| Token-budget rejectsRequest size was a binding admission constraint | 0 | 0 | 86 · 13–123 | 40.5 · 2–63 |
| Upstream 429sProvider-side rejection after capacity was spent | 375 · 241–471 | 0 | 0 | 0 |
| Peak provider occupancyActive requests against the 32-slot envelope | 32/32 | 32/32 | 32/32 · 30/32–32/32 | 32/32 |
Interactive request sizes varied by roughly 24.9×–32.0×. A median 50.6% of MoFlux capacity-bound rejects were token-bound, with 40.5 median token-budget rejects. No control and the static cap do not make request-size-aware token admission decisions. MoFlux and Redis exercised token-aware admission on all eight 0ms seeds.
Static caps and Redis both work. The benchmark is useful precisely because it shows where MoFlux adds value and where another control is currently better.
At the 0ms rung, MoFlux improved paired interactive completion by +4.0 pts and goodput by +4.2%. Across all four rungs, its median interactive-success advantage stayed between 3.42 and 4.72 percentage points.
At 0ms, median local rejects were 264 for MoFlux versus 306.5 for Redis — about 13.9% fewer attempt-level local rejects. Retry amplification was 1.339× versus 1.478×. Local rejects are attempt-level, so completion remains the stronger reliability outcome.
At the 0ms rung, Redis median TTFT p95 was 2.92s versus 5.46s for MoFlux, and Redis median end-to-end p95 was 10.30s versus 11.54s. These percentiles include successful logical requests only, so they should be read together with completion rate.
At 0ms, Redis median batch success was 17.4% versus 9.2% for MoFlux. MoFlux restored and served the protected batch floor on every seed, but this adaptive policy prioritizes interactive reliability over maximizing batch throughput.
The published ladder injects 0, 5, 20, and 50ms into the Redis coordinator path. Redis admitted-decision time rose 1.0141ms for each 1ms of injected distance, with all 8 seeds degrading. Tyr's admitted-decision slope was -0.0001ms/ms with a 95% interval from -0.0005 to +0.0002. The benchmark directly measures Tyr synchronous decisionDuration separately from queue wait. The near-zero MoFlux decision slope supports coordinator-distance independence for that measured local decision path; it does not mean end-to-end TTFT is zero-cost or invariant.
Median paired slope across eight seeds; 95% interval 1.0092–1.0183.
Measured Tyr decisionDuration excluding queue wait; 95% interval -0.0005 to +0.0002.
MoFlux median interactive-success advantage over Redis across the four tested coordinator-distance rungs.
Every rung recorded 8/8 passing seeds for the configured interactive target, batch-liveness floor, zero-upstream-429 requirement, and handoff safety checks.
All eight 0ms seeds observed and committed the reclaim handoff, restored the protected batch guarantee in controller and data-plane evidence, proved commit-before-admission with exact Tyr admission provenance, and recorded no applied over-allocation. Interactive traffic actually consumed borrowed slots on six of eight seeds. Controller-level lending was observed on all eight, so some seeds made capacity available without needing to occupy it above the protected floor. The median first batch admission window was 0.36–0.68s after commit. The roughly 14.28s commit-to-response-headers interval includes downstream simulated service for the large batch prompt and should not be read as capacity-control recovery latency.
Every seed committed in safe event order, before batch admission and fallback lease expiry, with no applied over-allocation.
Where demand timing was available, drain began immediately; acknowledgment dominated the handoff, while acknowledgment-to-commit was 0.01s at the median.
Interactive traffic actually consumed borrowed capacity on 6 of 8 seeds, reaching as many as 4 extra concurrency slots.
Median lower/upper bound for first batch admission after commit. The roughly 14.28s commit-to-response-headers interval includes downstream simulated service for the large batch prompt—queue/prefill/TTFT-equivalent work—not capacity-control recovery.
MoFlux recovered a median 60.6% of reserved tokens net of overruns, and 83.4% of refunded tokens returned while streams were still active. Median overrun was 133.5 tokens. This is earlier reuse of existing headroom, not newly created capacity.
Median across eight matched 0ms seeds.
Share of refunded tokens returned mid-stream.
This earlier run used Tyr 0.22.0 and Latchflo 0.8.0. Across five matched seeds, the protected policy was exercised successfully, both classes continued to receive service under contention, and every arm recorded zero upstream 429s. Against class ceilings alone, premium median success moved from 72.5% to 75.4% and contended goodput from 2.76 to 2.88 requests/s. Premium TTFT was effectively unchanged versus ceilings-only.
Same trace, both classes observed and served under contention, protected policy exercised, minimum noisy-class progress met, and zero upstream 429s in every arm.
Protected floors improved premium success and goodput in four of five seeds, but the effect is incremental rather than dramatic.
Median protected/ceilings premium p95 TTFT ratio and change. Protected TTFT was better in only 2/5 seeds, so the benchmark does not establish a TTFT advantage over ceilings-only.
Noisy median success edged up from 37.5% to 38.2%, but median contended completions fell. The floor guarantees progress; this run does not show a noisy-class throughput win.
The shared arm is an important counterweight: it reached 98.8% median premium completion and 4.52 premium contended goodput, both well above protected. Protected instead cut premium contended p95 TTFT to 0.535× shared at the median — about 46.5% lower — while reserving capacity for the noisy class. That is an isolation trade-off, not a universal performance win.
The newest evidence strengthens the safety claim substantially: across the published four-rung ladder, all 32 rung-seed combinations passed the recorded adaptive proof, and the 0ms evidence includes exact Tyr admission provenance for safe reclaim. Current Tyr/Latchflo releases also support opt-in demand-aware class-floor lending, but the class-performance numbers below are still from the earlier static protected-floor proof and should not be relabeled as current-stack performance evidence.
The provider is simulated and the policy intentionally prioritizes interactive traffic. The result supports claims about this tested overload regime, not a blanket statement that MoFlux is faster or more reliable in every workload.
At 0ms, static caps reached 93.7% median interactive success with zero upstream 429s. MoFlux reached 97.4% and adds token-aware admission, adaptive lending, fleet authority, provenance, and coordinator-independent local decisions — at the cost of more system complexity.
Redis had substantially lower median successful-request TTFT at 0ms and lower median end-to-end latency. MoFlux's value in this run is higher interactive completion, lower retry amplification, fewer attempt-level local rejects, and a request path that does not inherit coordinator RTT.
All 32 rung-seed combinations met the configured batch-liveness floor, but at 0ms MoFlux median batch success was 9.2% versus 17.4% for Redis. The site therefore describes the batch guarantee as non-starvation/liveness, not superior batch throughput.
The published Tyr 0.27.0 ladder exports synchronous admission decision time separately from queue wait. The coordinator-distance claim uses decisionDuration excluding queue wait; end-to-end TTFT still includes retry/backoff, queueing when exercised, provider prefill, and first-token service.
Protected class floors remain shipped, but the five-seed class proof cited here was generated with Tyr 0.22.0 and Latchflo 0.8.0. It is prior isolation evidence, not a current-runtime performance rerun.
MoFlux returned a median 60.6% of reserved tokens net of overruns, with 83.4% of refunded tokens released while streams were still active.
The benchmark harness is at v0.29.0. The August 26 adaptive coordinator ladder cited here is published under video-seed-sweep in the public results tree. Reproducing the MoFlux arms still requires licensed Tyr and Latchflo images. The older protected-class performance proof remains separately disclosed because it predates the current runtime.
Inspect the published reviewed result or clone mofluxhq/moflux-bench and run npm run verify. The current ladder driver is npm run demo:coordinator:adaptive; the protected-class driver is npm run demo:classes. Licensed Tyr and Latchflo images are required to reproduce the MoFlux arm.
The openai-live-overload-sweep artifact sends real requests to OpenAI under deliberate request-rate overload. Arm order is counterbalanced across eight seeds and each arm starts behind the same rate-limit headroom gate. The older August 26 Universal Scalability Law ladder remains separate synthetic evidence for coordinator distance, token pressure, and adaptive fleet behavior.
MoFlux completed 795/800 protected interactive requests versus 575/800 under the static cap, with a positive paired advantage in every seed. It shed more batch work to do so, and Single-node Tyr protected concurrency only. This sweep does not exercise Tyr in-flight token-budget admission or Latchflo fleet coordination.
5/5 membership runs passed with no revision churn, duplicates, stale members after convergence, or revision regressions. This benchmark validates Latchflo membership and routing-topology convergence. It does not yet prove uninterrupted routed request traffic while peers churn.
The August 26 ladder 20260826T204430Z was promoted under video-seed-sweep. The site links directly to the reviewed evidence in the public moflux-bench results tree rather than relying on an unpublished local corpus.
The published ladder is associated with moflux-bench 0.27.0, Tyr 0.27.0, Latchflo 0.12.4, async-bulkhead-llm 3.16.0, and async-bulkhead-ts 1.0.1.
Redis admitted decisions rose 1.0141ms per 1ms of injected coordinator distance. Tyr admitted decisions had a median slope of -0.0001ms/ms with a 95% interval of -0.0005 to +0.0002. This measures local decisionDuration excluding queue wait, not end-to-end response latency.
At 0ms, the safe handoff took a median 0.81s. First batch admission was bounded to 0.36s–0.68s after commit at the median, while response headers arrived 14.28s after commit because the large batch request still had downstream simulated service to do.
The reviewed npm run demo:classes run used Tyr 0.22.0 and Latchflo 0.8.0. It remains useful correctness/isolation evidence, but it is not presented as a current-stack performance measurement.
Peer routing ships in the current Tyr release. The published ladder measures the full MoFlux arm with routing enabled, but it does not isolate routing as a standalone causal effect.
Latchflo allocates bounded, expiring authority for physical pools and fixed admission classes, and now derives versioned routing topology from live agent membership. Tyr atomically applies complete desired state, reports bounded pool and per-class demand, acknowledges restrictive physical or class grants with bounded occupancy, and publishes fresh post-ACK evidence; Latchflo keeps staged authority unavailable until that evidence proves the drain is safe. Tyr still makes the final admission decision locally without a Latchflo round trip on every provider call.
Controller epoch, revision, grant ID, expiration, complete pool/class limits, apply acknowledgement, and immutable provenance make installed authority attributable.
Pool/class drain → applied ACK → fresh post-ACK occupancy proof → commit staged authority. A restrictive-grant ACK proves installation; fresh post-ACK pool or class occupancy proves the transferred authority is actually safe to reuse.
Live agent registration and timeout drive a versioned routing-topology snapshot without placing Latchflo on the provider request path. The current membership proof validates topology convergence; routed traffic continuity during churn remains a separate benchmark target.
Valid leases survive transient control-plane failures until expiration; when authority expires, Tyr applies the reserved zero-capacity revision rather than admitting indefinitely.
MoFlux can sit on a selected provider path while your existing platform continues to own ingress, broader authentication and authorization, secrets, routing, and AI governance. Tyr can optionally verify JWT/JWKS identity when admission classes or role-based policy need it.
The current releases are credible for technical evaluation, architecture review, and a tightly controlled design-partner pilot. Tyr and Latchflo are still pre-1.0 and should not be sold as a broadly hardened enterprise platform yet.
v0.28.0Makes local token-and-concurrency admission decisions, reports bounded pool and per-class demand, atomically applies managed desired state, acknowledges restrictive physical and class grants with bounded occupancy, publishes ordered post-ACK proof heartbeats, enforces protected class floors and hard ceilings, progressively reconciles streaming usage, and can route once toward a peer with better request-specific headroom.
v0.13.0Allocates versioned, expiring fleet grants, protected class floors and hard ceilings; tracks live agent membership and publishes versioned routing topology; coordinates acknowledged physical and class-only drain → applied ACK → fresh occupancy proof → commit; and retriggers pending reclaim when fresh member telemetry makes the transition provable — without entering the live provider request path.
v3.16.0Provides token estimation, exact reservation previews, protected admission-class floors, shared-capacity borrowing, progressive reconciliation, observe mode, and atomic versioned limit updates.
v1.0.2Provides the lower-level fail-fast and bounded-queue admission foundation used beneath the LLM-aware policy layer.
The project is moving quickly, but these constraints remain part of the architecture discussion from the start.
Latchflo currently persists to SQLite behind one controller. PostgreSQL-backed HA and hardened multi-region operation remain roadmap work.
There is no 24/7 support organization or enterprise SLA today. A pilot is direct, technical, and intentionally bounded.
Token budgets govern expected in-flight exposure. They are not monthly spend limits, invoices, or billing-grade accounting.
Tyr currently targets OpenAI Chat Completions and Anthropic Messages and supports direct JWT/JWKS identity. Broader provider protocols and dynamic authorization policy are not yet generalized.
Latchflo 0.13.0 now publishes versioned routing topology from live membership. The current 5-run proof validates clean remove/join convergence with no duplicate or stale members, but uninterrupted routed request traffic during peer churn remains a separate benchmark target.
Latchflo 0.13.0 with Tyr 0.28.0 can lend a fully observed idle class's protected floor while keeping hard ceilings fixed, then restore that floor before the old lease expires when ordered applied-ACK plus fresh class-occupancy evidence proves safe attrition. Older/incapable agents or missing evidence fall back to lease expiry; active work is never preempted.
At the 0ms rung, the adaptive policy restored the batch pool guarantee safely on all 8 seeds, and first batch admission was bounded to 0.36s–0.68s after commit at the median. Batch completion was still only 9.2% versus 17.4% for Redis, so the current policy proves liveness rather than superior batch throughput.
Across eight matched seeds at 0/5/20/50ms, Redis admitted-decision time rose 1.014ms for each 1ms of injected coordinator distance. Tyr's measured local admitted-decision slope was -0.0001ms/ms with a 95% interval spanning zero; queue wait and end-to-end response time remain separate metrics.
The strongest pilot has several AI-service replicas, a shared constrained provider or inference pool, and a workload where interactive traffic, batch jobs, large prompts, or retries compete for capacity. The goal is not a broad rollout; it is a defensible answer to one capacity question.
Pick a service, model route, environment, or batch flow where overload behavior is visible and reversible.
Exercise the real estimator and policy path without turning capacity-related decisions into hard rejection yet.
Review prospective rejects, latency, completion, token reservation accuracy, class behavior, and provider-throttling exposure.
Move to a narrow enforcement canary only when the observed trade-offs and operational model are acceptable to your team.
MoFlux is currently built and supported by one founder and is not yet an incorporated company.
Before the first paid client engagement, I will form an LLC and obtain any business licences required for the engagement.
I am looking for one technically engaged design partner with a real AI-capacity problem and a bounded pilot workload.
No sales team, contact form, or automated funnel. Your email goes directly to the person building the system.
Start the conversation