MOFLUX
Workload-aware LLM admission control

Keep batch LLM work from starving user-facing traffic.

99.4% interactive successunder live OpenAI overload · 71.9% with the same 36-slot static cap
  • Protect interactive capacity before lower-priority work can consume it.
  • Borrow idle headroom when protected demand is absent, then stop new borrowing when it returns.
  • Shed excess work before execution instead of waiting for provider throttling to decide.
Interactive completion · same 36-slot global ceiling
Static cap71.9%
MoFlux99.4%
MoFlux led in all 8/8 matched seeds. It shed more batch work to preserve interactive capacity; this is not a latency-win claim.
Open live OpenAI evidence Discuss a design-partner pilotSee all measured trade-offs
Built by Jan Balangue · backend and distributed-systems engineer · creator of async-bulkhead-llm.
Tyr is open source under Apache-2.0. Published evidence comes from the reproducible moflux-bench harness.
Admission under contention Local
InteractiveUser-facing requests
8 protected
Batch / backgroundDelay-tolerant work
4 protected
Workload-aware admissionTyr
Protect · borrow · shed
UpstreamLLM provider
Interactive capacity stays protected.When slots run out, lower-priority work is shed first.
Fleet authorityLatchflo stays off the request path
Control plane

Overload begins before the 429

By the time an upstream rejects traffic, your service may already be carrying too many large requests, retries, open streams, and waiting callers.

Replica-local limits multiply

A limit that is safe for one process can become unsafe when every replica independently assumes it owns the same shared capacity.

Queues can turn pressure into latency

Waiting is not free. Under sustained load, queues can trade obvious rejection for timeouts, retry amplification, memory pressure, and a less predictable user experience.

Measured progress

Live overload proof.
Changing fleet.

Two different questions now have direct evidence. The live sweep tests whether protected concurrency changes who succeeds under real provider scarcity. The membership run tests whether Latchflo converges cleanly as the fleet changes.

Counterbalanced live OpenAI overload
99.4%

Protected interactive requests completed under overload.

Static concurrency completed 71.9%. MoFlux led in all 8 matched seeds, with a median paired advantage of +25.5 points. Both controlled arms kept provider 429s at zero.

Open reviewed OpenAI evidence
Reviewed fleet proof · prior runtime
5/5

Reviewed membership runs converged cleanly.

Zero heartbeat revision churn, duplicate members, stale members after convergence, or revision regressions. Replacement joins propagated in 11.1ms p50; timeout removal was 1.40s p50.

This benchmark validates Latchflo membership and routing-topology convergence. It does not yet prove uninterrupted routed request traffic while peers churn.
Market position

Not another gateway.
Not just another limiter.

MoFlux is a specialized safety layer for the gap between “this request is allowed” and “this request is safe to start right now.” It is intended to integrate with an existing gateway and observability stack.

Adjacent category

AI gateway

Primary job
Routes requests and often owns provider access, authentication, observability, and broad API policy.
Boundary
That is a wider job than admission control, and many teams already have this layer.
Relationship
MoFlux is designed to integrate with it rather than replace it.
Adjacent category

RPM / TPM rate limit

Primary job
Controls request or token volume over a time window.
Boundary
A request can fit the current rate window and still be too large or too concurrent to start safely now.
Relationship
MoFlux adds an in-flight capacity decision before execution.
Adjacent category

Generic concurrency limiter

Primary job
Caps local in-flight work, sometimes with a queue.
Boundary
It protects the upstream, but an undifferentiated cap does not know whether a slot is being used by a waiting user or delay-tolerant background work.
Relationship
MoFlux adds protected workload classes, LLM-sized reservations, fleet authority, and progressive accounting.
Purpose-built layer

MoFlux

Primary job
Protects capacity for important workload classes before execution, then reserves expected LLM work and refuses excess demand while policy is still enforceable.
Boundary
It is not yet a turnkey HA control plane, universal gateway, billing system, or multi-region product.
Relationship
It is a focused admission-and-capacity layer for LLM services.
Defensible category

A focused capacity firewall for LLM traffic.

Pre-execution reservation, local admission, protected service-class capacity, fleet-level grants, and decision provenance—without a control-plane lookup on every request.

Operational value

Built around the failure modes platform teams actually see

The goal is narrow: make capacity an explicit pre-execution decision, then leave ingress, provider governance, and broader gateway policy where they already live.

Protect scarce execution capacity

Admit, reject, or observe a request before it consumes provider-facing concurrency and expected in-flight token headroom.

Coordinate a fleet

Turn one capacity envelope into expiring local authority without placing the control plane on every provider request.

Protect service classes

Map trusted identities to a bounded class table, apply hard ceilings and protected concurrency/token floors, and expose protected versus borrowed usage.

Explain the decision

Surface rejection reason, policy revision, admission identity, grant ID, controller epoch, and bounded metrics for tuning and incident review.

Latest evidence

Restoration is now measured twice.

The homepage keeps the product claim short. The evidence library carries the full tables, seeds, methodology, provenance, counter-results, and limitations.

Latest · vLLM Metal · September 23 · five counterbalanced seeds

Lending improved batch goodput over the static partition in every seed.

MoFlux matched native priority’s interactive SLO goodput in three seeds and trailed by one request in two. Batch borrow-phase goodput improved by +0.171 req/s at the paired median versus static.

Apple M1, 16 GB, Qwen2.5-1.5B. MoFlux completed 110/236 batch requests versus native priority’s 236/236, with retries disabled. Grant restoration and borrower occupancy are measured separately; GPU and KV-cache reclamation remain unproven.

Inspect the Metal comparison and restoration timings
Batch borrow-phase gain vs static+0.171req/s · median paired change across five seeds
Earlier CPU-only Ollama · 5 counterbalanced seeds

The controller restored the grant in 263ms median. 6 of 9 occupancy episodes still missed 15s.

All H1–H4 gates passed. Latchflo 0.16.0 enforced a native one-slot interactive reserve while lending the remaining idle capacity, but already-admitted work could still delay when restored capacity became usable.

Reviewed public result. CPU-only Ollama / qwen3:0.6b; no GPU, KV-cache, upstream reclamation, or production-scale claim.

Grant median → occupancy misses263ms → 6/9episodes exceeding the 15s objective
Reviewed live provider

99.4% vs 71.9%

Interactive completion through Tyr versus the same 36-slot static cap across eight counterbalanced live OpenAI overload seeds. MoFlux shed more batch work; no latency-win claim.

Open reviewed result
Reviewed local proof

5/5 seeds passed

Latchflo's native unlent-concurrency policy kept one interactive slot out of shared capacity while exact provenance verified transfer safety and borrowing order.

Inspect restoration evidence
Published distributed proof

32/32 ladder checks

The August 26 synthetic ladder remains the separate source for token-pressure, adaptive fleet behavior, and coordinator-distance claims.

Open full evidence library
Read the complete evidence library Open the reviewed local result
Current architecture

Fleet capacity authority.
Local request decisions.

Latchflo allocates bounded, expiring authority for physical pools and fixed admission classes, and now derives versioned routing topology from live agent membership. Tyr atomically applies complete desired state, reports bounded pool and per-class demand, acknowledges restrictive physical or class grants with bounded occupancy, and publishes fresh post-ACK evidence; Latchflo keeps staged authority unavailable until that evidence proves the drain is safe. Tyr still makes the final admission decision locally without a Latchflo round trip on every provider call.

MoFlux current-state architectureLatchflo control plane 0.19.0 owns versioned, expiring fleet authority for concurrency and in-flight token capacity. It distributes complete desired-state grants, protected admission-class floors and hard ceilings, supports opt-in per-class demand-aware floor lending, and can keep configured concurrency and token slices allocation-enforced and unlent. It coordinates acknowledged physical capacity-group reclaim and class-only acknowledged handoff for returning protected floors and restrictive class transitions. Authority expansion remains staged until drain installation plus fresh post-ack pool or class occupancy proves safe attrition; lease expiry is the conservative fallback and active work is not preempted. Tyr admission controller 0.33.0 atomically applies higher-revision desired-state batches, reports bounded pool and admission-class demand, includes bounded pool/class occupancy in applied acknowledgements, publishes immediate post-ack evidence heartbeats for restrictive transitions, fails managed pools closed when authority expires, and makes the final request admission decision locally. Latchflo derives a versioned routing-topology snapshot from live agent membership and includes that fleet state in managed desired state. Tyr can route a request once using fresh request-specific concurrency, token and class headroom; the destination Tyr performs authoritative admission. Current public membership evidence proves topology convergence, not uninterrupted routed traffic during churn. Latchflo does not receive prompts or caller identity. Client applications call Tyr on the Anthropic Messages or OpenAI Chat Completions-compatible routes; Latchflo remains off the live provider request path.Latchflo control planev0.19.0 · capacity + live membership · acknowledged handoffFleet + class policyPool guarantees + adaptive class floorsLive membership → versioned topologyHandoff coordinatorPool/class drain · applied ACKFresh occupancy proof → commitDesired state + topology ↓ACK + pool/class demand + occupancy ↑Tyr admission controllersv0.33.0 · local admission + managed topologyAtomic apply · fail closed on lease expiryPool/class demand + post-shrink proof heartbeatInstance Alocal admit / routeInstance Blocal admit / routeInstance Nlocal admit / routeClient appsagents, jobsAnthropicMessages APIOpenAIChat CompletionsLatchflo receives capacity, demand, and endpoint metadataPrompts and caller identity stay out of the control planePeer routing stays Tyr-local; no control-plane lookup per request
Current state: Latchflo 0.19.0 owns versioned pool/class authority, tracks live membership, publishes routing-topology snapshots, coordinates acknowledged handoff, and retries pending reclaim when fresh telemetry makes it actionable. Tyr 0.33.0 keeps the request admission decision local and destination-authoritative.

Versioned desired state

Controller epoch, revision, grant ID, expiration, complete pool/class limits, apply acknowledgement, and immutable provenance make installed authority attributable.

Proof before reuse

Pool/class drain → applied ACK → fresh post-ACK occupancy proof → commit staged authority. A restrictive-grant ACK proves installation; fresh post-ACK pool or class occupancy proves the transferred authority is actually safe to reuse.

Dynamic fleet membership

Live agent registration and timeout drive a versioned routing-topology snapshot without placing Latchflo on the provider request path. The current membership proof validates topology convergence; routed traffic continuity during churn remains a separate benchmark target.

Fail-closed managed mode

Valid leases survive transient control-plane failures until expiration; when authority expires, Tyr applies the reserved zero-capacity revision rather than admitting indefinitely.

Integration position

Keep the gateway you already have.

MoFlux can sit on a selected provider path while your existing platform continues to own ingress, broader authentication and authorization, secrets, routing, and AI governance. Tyr can optionally verify JWT/JWKS identity when admission classes or role-based policy need it.

Inside one Tyr admission controllerTyr admission controller 0.33.0 contains JWT and JWKS identity with bounded role and admission-class selection; an admission engine built on async-bulkhead-llm 3.17.0; first-class Latchflo managed mode; and optional one-hop peer routing. The admission engine enforces physical-pool and admission-class concurrency and in-flight token limits, protected floors and hard ceilings, shared-capacity borrowing, exact request reservations, progressive streaming reconciliation, observe or enforce mode, and bounded retry hints. Managed mode validates complete desired state, applies higher-revision batches atomically, reports pool and admission-class demand, acknowledges applied grants, includes bounded pool and admission-class occupancy in applied acknowledgements, publishes fresh post-ack evidence for physical and restrictive class transitions, preserves a valid lease through transient control-plane failures, and fails managed pools closed on expiry. After a physical or class drain Tyr emits an immediate evidence heartbeat and can temporarily use a bounded 500 millisecond cadence until an exact sent snapshot proves the transition safe. Managed state can carry versioned routing topology derived from Latchflo membership. Peer routing remains off path, forwards at most once using authenticated internal headers, and leaves final admission to the destination Tyr. Current public evidence validates topology convergence separately from routed traffic continuity during churn. /healthz and /readyz are public probes; /stats and /metrics require operator authority. Caller identity is not forwarded upstream, while the caller's provider credential is preserved.Tyr admission controller v0.33.0Local request decision; managed fleet authority arrives off pathIdentity + class selectionJWT / JWKS · configurable headerRoles + bounded admission classesIdentity stays Tyr-local for LatchfloDefault class when no rule matchesAdmission engineasync-bulkhead-llm v3.17.0Pool + class concurrency / token headroomProtected floors · hard ceilings · borrowingProgressive reconciliation · admit / rejectLatchflo managed modeComplete desired state + routing topologyApplied ACK + bounded pool/class occupancyPool + per-class demand heartbeatsPool/class post-shrink proof · fail closed on expiryActive work shrinks by attrition, never preemptedCapacity-aware peer routingVersioned membership topology + private secretFresh request-specific pool + class headroomForward at most once; no automatic retryDestination Tyr performs authoritative admissionPublic proof validates topology convergencePublic probes/healthz /readyzOperator authority/stats /metricsIdentity stripped before provider forwarding; caller provider credential preserved
Inside Tyr 0.33.0: identity-aware admission classes, protected floors and hard ceilings, bounded pool/per-class demand, atomic Latchflo desired-state application, applied ACK with bounded pool/class occupancy plus immediate post-ACK proof heartbeats for restrictive transitions, progressive reconciliation, and one-hop peer routing all stay in the data plane. Latchflo 0.19.0 now publishes versioned routing topology from live membership; routed churn remains a separate end-to-end proof target.
Current implementation

A substantial technical core,
still before production v1.

The current releases are credible for technical evaluation, architecture review, and a tightly controlled design-partner pilot. Tyr and Latchflo are still pre-1.0 and should not be sold as a broadly hardened enterprise platform yet. Tyr, the data plane in the request path, is open source under Apache-2.0. Latchflo, the control plane, is commercially licensed and stays off the provider request path.

Admission data planev0.33.0

Tyr

Makes local token-and-concurrency admission decisions, reports bounded pool and per-class demand, atomically applies managed desired state, acknowledges restrictive physical and class grants with bounded occupancy, publishes ordered post-ACK proof heartbeats, enforces protected class floors and hard ceilings, progressively reconciles streaming usage, and can route once toward a peer with better request-specific headroom.

Apache-2.0 · Source

Capacity control planev0.19.0

Latchflo

Allocates versioned, expiring fleet grants, protected class floors, hard ceilings, and native unlent concurrency/token slices; tracks live agent membership and publishes versioned routing topology; coordinates acknowledged physical and class-only drain → applied ACK → fresh occupancy proof → commit; and retriggers pending reclaim when fresh member telemetry makes the transition provable — without entering the live provider request path.

Commercial license

LLM-aware admission enginev3.17.0

async-bulkhead-llm

Provides token estimation, exact reservation previews, protected admission-class floors, shared-capacity borrowing, progressive reconciliation, observe mode, and atomic versioned limit updates.

Apache-2.0 · Source

Concurrency primitivev1.0.2

async-bulkhead-ts

Provides the lower-level fail-fast and bounded-queue admission foundation used beneath the LLM-aware policy layer.

Apache-2.0 · Source

Benchmark and proof harnessv0.44.0

moflux-bench

Runs deterministic and counterbalanced workload comparisons, validates safety gates and evidence provenance, and now includes the local-inference contention/restoration scenario used to separate grant restoration from usable-capacity restoration.

Apache-2.0 · Source

Current boundaries

Where the current boundary is

The project is moving quickly, but these constraints remain part of the architecture discussion from the start.

Single-controller control plane

Latchflo currently persists to SQLite behind one controller. PostgreSQL-backed HA and hardened multi-region operation remain roadmap work.

Founder-stage operations

There is no 24/7 support organization or enterprise SLA today. A pilot is direct, technical, and intentionally bounded.

Capacity policy, not billing

Token budgets govern expected in-flight exposure. They are not monthly spend limits, invoices, or billing-grade accounting.

Focused protocol and identity support

Tyr currently targets OpenAI Chat Completions and Anthropic Messages and supports direct JWT/JWKS identity. Broader provider protocols and dynamic authorization policy are not yet generalized.

Membership convergence is proven; routed churn is next

Latchflo 0.19.0 now publishes versioned routing topology from live membership. The current 5-run proof validates clean remove/join convergence with no duplicate or stale members, but uninterrupted routed request traffic during peer churn remains a separate benchmark target.

Restoration is resource-specific

The September 11 Ollama proof distinguishes grant restoration from usable occupancy restoration. Latchflo 0.16.0 and Tyr 0.30.0 restored the protected grant in 263ms median, but 6 of 9 occupancy-restoration episodes still missed 15s while already-admitted work drained. MoFlux does not relabel that delay as reclaimed GPU or provider capacity.

Batch fairness remains unfinished

At the 0ms rung, the adaptive policy restored the batch pool guarantee safely on all 8 seeds, and first batch admission was bounded to 0.36s–0.68s after commit at the median. Batch completion was still only 9.2% versus 17.4% for Redis, so the current policy proves liveness rather than superior batch throughput. In a published follow-up, lending interactive headroom to batch during active demand cost interactive a median 11.5% p95 without reliably adding batch completions.

Coordinator distance now has direct evidence

Across eight matched seeds at 0/5/20/50ms, Redis admitted-decision time rose 1.014ms for each 1ms of injected coordinator distance. Tyr's measured local admitted-decision slope was -0.0001ms/ms with a 95% interval spanning zero; queue wait and end-to-end response time remain separate metrics.

First design partner

Bring one real capacity problem. Measure it together.

The strongest pilot has several AI-service replicas, a shared constrained provider or inference pool, and a workload where interactive traffic, batch jobs, large prompts, or retries compete for capacity. The goal is not a broad rollout; it is a defensible answer to one capacity question.

A bounded non-critical or canary workload Access to latency, rejection, and usage telemetry An engineering owner who can review behavior directly
Email about a pilot Discuss the architecture
01

Choose one bounded workload

Pick a service, model route, environment, or batch flow where overload behavior is visible and reversible.

02

Start in observe mode

Exercise the real estimator and policy path without turning capacity-related decisions into hard rejection yet.

03

Compare policy with traffic

Review prospective rejects, latency, completion, token reservation accuracy, class behavior, and provider-throttling exposure.

04

Enforce only if the evidence supports it

Move to a narrow enforcement canary only when the observed trade-offs and operational model are acceptable to your team.

Commercial status

Founder-led, direct, and early.

MoFlux is currently built and supported by one founder and is not yet an incorporated company.

Before the first paid client engagement, I will form an LLC and obtain any business licences required for the engagement.

I am looking for one technically engaged design partner with a real AI-capacity problem and a bounded pilot workload.

Direct contactmofluxhq@gmail.com

No sales team, contact form, or automated funnel. Your email goes directly to the person building the system.

Start the conversation