Reverse Engineering Fireworks' Serving Stack from Black-Box API Calls

Aug 18, 2026
Research Security

One ordinary response from Meta’s newly released Muse Glimmer 30B leaked a vLLM Git revision. We used it to build a local twin, identify the precision and GPU configuration, reproduce the scheduler limit, map the serving fleet, and improve prompt-cache routing.

Each result enabled the next: fingerprint → local twin → FP8 / H100 / TP=1 → approximately 64-sequence limit → fleet topology → client optimization.

1. The API exposed its vLLM source revision

The first clue arrived without a special probe. An ordinary non-streaming chat-completions response contained a system_fingerprint value: vllm-0.1.dev19758+g075d645af-28e13d40.

{
  "object": "chat.completion",
  "system_fingerprint": "vllm-0.1.dev19758+g075d645af-28e13d40"
}

system_fingerprint is a standard OpenAI-compatible field intended to signal backend changes. Fireworks did not list it in its chat-completions response schema when we checked, but vLLM emits it from its OpenAI-compatible server. vLLM stamped the response and the Fireworks gateway passed the field through.

vllm-0.1.dev19758 identified a development build of vLLM.
g075d645af identified a real upstream Git revision.
28e13d40 identified a configuration hash.

The Git revision resolved to commit 075d645af, titled Remove MUSE_GLIMMER_README.md, at the tail of the Muse porting work. When we first observed the fingerprint on August 12, Muse support lived in an unmerged vLLM pull request, rather than a release. The PR merged into main on August 14.

First conclusion: A public API should not disclose that level of implementation detail.

2. The source revision let us build a local twin

We checked out the Muse vLLM branch and served the public Muse-Glimmer-30B weights. The local server reproduced the same vocabulary and token IDs, rendered prompt structure including the unusual =self special-token prefix, native vLLM context-limit error, max_model_len=131072, and successful generation from a prompt just below that limit.

The token-level match showed that we had the correct model code, tokenizer, and chat template, not merely a model with the same name. It also showed that the Fireworks gateway was not materially rewriting the prompt before vLLM saw it.

Second conclusion: the leaked revision converted a remote black box into a white-box comparison problem.

3. The local twin revealed FP8

Speed is not sufficient to identify weight precision, so we compared the model’s numerical output distribution. For high-entropy prompts, we requested alternative-token log probabilities and compared the same token positions across production and two local variants: FP8 on an H200 and NVFP4 on an RTX 5090.

top-5 token overlap with FP8 H200:   87%
top-5 token overlap with NVFP4 5090: 58%
Fireworks/H200 same-token gap:       0.15 nats
FP8/NVFP4 white-box gap:             2.32 nats

Across positions where FP8 and NVFP4 diverged, Fireworks was significantly more often closer to FP8 (p=0.036, sign test). High-confidence output tokens also matched.

Third conclusion: The serverless Muse route is using FP8 with very high probability.

4. FP8 timing identified the GPU and showed speculation was off

Decode matched one H100’s measured performance band

We measured decode with an output-length gradient: keep the prompt fixed, vary generated tokens, and fit processing time against output length. Production decode ranged from about 13.8 to 16.1 ms/token. The same weights decoded at 13.2 to 13.3 ms/token on our dedicated H100 and 11.2 ms/token on our H200.

A cold-prefill sweep of unique salted prompts gave a slope of 0.0765 ms per token, or about 13,100 prompt tokens per second. That ruled out A100 and pointed to Hopper-class hardware. The 30B FP8 weights fit on one 80 GB H100; the direct one-H100 TP=1 replay matched production without tensor-parallel communication.

The absence of a speculator

Muse supports a DFlash speculative-decoding drafter. Highly predictable prompts and adversarial high-temperature prompts both decoded at essentially the same ~62–65 tok/s, while a speculative-decoding control varied from 321 to 525 tok/s. Muse also lacked the speculative-match metric present on Fireworks routes using speculation.

Fourth conclusion: Muse was served on one H100-class GPU in FP8 at TP=1, with speculative decoding disabled.

5. A router leak gave us one replica at a time

An undocumented X-Session-Id response header Base64-decodes to values resembling private RFC 1918 addresses, such as 172.25.x.y:80. They cannot be geolocated or assigned to a public owner; they resemble internal Kubernetes Pod or service addresses.

Controlled requests showed that a fresh client-supplied prompt_cache_key emitted an address; repeated requests with that key stayed on the same replica and suppressed the header; a fresh key emitted the header even on a full prompt-cache hit; and one persistent TLS connection reached different addresses.

X-Session-Id marked a fresh router assignment and identified the chosen internal serving endpoint. Fireworks’ documented prompt_cache_key then kept later requests on that endpoint while the binding remained alive.

Fifth conclusion: by retaining an assigned affinity key, a client can remain pinned to a single Fireworks serving replica. This is expected behavior, which we verified experimentally.

6. Pinning one replica exposed a 64-sequence scheduler limit

We warmed one affinity key, then barrier-launched waves of 32, 36, 40, 48, 56, 64, 80, and 96 equal-length streams, all pinned to its replica. The knee was at 64: up to 64 all decode windows overlapped; above it, excess streams queued while admitted streams kept the same token spacing.

Wave	Excess over 64	Fastest completion	Slowest first token	Queue evidence
64	0	all windows overlap	all streams admitted	none
80	16	1.81 s	2.10 s	+0.29 s
96	32	1.83 s	2.66 s	+0.83 s

Our local H100 advanced all 96 streams together. Restarting the local twin with max_num_seqs=64 matched production: full overlap through 64, admission queueing above 64, and unchanged token spacing for admitted streams.

Sixth conclusion: the serving replica was configured with max_num_seqs=64 with high probability.

7. Repeating assignments revealed the fleet

Counting observed router assignments became a coupon-collection problem. Stable 200-call windows repeatedly converged on 30 distinct addresses. In one run, the last new address appeared at draw 89, followed by 111 draws with no new address or replica. Under approximately uniform routing, the maximum-likelihood pool size was 30.

The same 18 addresses appeared in every formal census.
Roughly 12 additional addresses changed between windows.
Transition windows briefly produced more than 30 addresses in the union because replacements occurred during the run.
In a separate keepalive experiment, all 18 stable replicas survived while two volatile replicas disappeared and their keys were reassigned.

The resulting fleet model was approximately 18 long-lived replicas plus approximately 12 rotating replicas, or approximately 30 active replicas in observed windows. We modeled this as an HPA-style configuration with minReplicas ≈ 18 and maxReplicas ≈ 30; the replacement mechanism needs further investigation.

Seventh conclusion: the route ran on approximately 30 active single-GPU serving replicas, with an 18-replica stable core and a 12-replica replaceable edge.

8. Turning the fleet map into a client-side optimization

Fireworks documents that prompt caching works within one replica, and documents prompt_cache_key as the preferred session-affinity field. With the same 29k-token prompt, twelve requests using fresh bindings spread across nine replicas and had a median TTFT of 2.08 seconds. Twelve requests on one warmed affinity key stayed on one replica and had a median TTFT of 0.77 seconds: 2.7 times faster, with much lower variance.

The stable cohort was also warmer. In formal censuses, 581 requests landed on the stable 18 without a cold result. When a flex replica changed address while an affinity key was active, the router reassigned it and its KV cache did not move: cached prompt tokens fell from a warm value to zero on a cold reassignment.

Census the current pool. Send fresh, salted requests and decode X-Session-Id until the active set is covered.
Classify fixed addresses. Repeat across windows; persistent addresses form the fixed set.
Mint affinity keys until one lands on a fixed address.
Retain the winning key for the real inference workload.
Keep several fixed-replica keys and distribute traffic between them.

The client never sends traffic directly to a private address. The address only screens router assignments; the retained prompt_cache_key performs routing.

Final conclusion: Periodically map stable replicas, mint affinity keys until they land on those replicas, and reuse those keys for production inference. This turns replica-local prompt caching from a routing lottery into a controlled client-side optimization.

In summary: one leaked build fingerprint enabled the local twin; the twin exposed GPU and engine settings; the router header exposed replicas; and the replica map produced a measurable latency optimization.