[vibecoded wip, do not merged yet] feat(inference): theoretical prefix tokens overlay + new input suffix throughput and TFLOP/s y-metrics / 理论 prefix token 悬浮框指标与新输入 suffix 吞吐量及 TFLOP/s Y 轴指标 by functionstackx · Pull Request #849 · SemiAnalysisAI/InferenceX-app

functionstackx · 2026-08-25T20:55:39Z

Summary

Adds two new rows to the point overlay (tooltip) for agentic-trace points:

Theoretical Prefix Tokens — the theoretical prefix for each point, computed as the sum of all the prefix it's seen. The harness already reports this sum as the infinite-cache theoretical_cache_hit_rate over served prompt tokens, so the token sum is recovered as round(total_prompt_tokens × theoretical_cache_hit_rate).
Input Tokens w/o Prefix Caching — total_prompt_tokens minus the theoretical prefix, i.e. the prompt tokens that must be prefilled even with an infinite prefix cache.

Also adds three new Throughput y-axis metrics derived from the same rate:

New Input Suffix + Output Token Throughput per Chip — total token throughput minus the theoretical prefix share of input throughput (tput_per_gpu − input_tput_per_gpu × rate)
New Input Suffix Token Throughput per Chip — input_tput_per_gpu × (1 − rate), plotted vs. P90 TTFT like the existing input-throughput metric
New Input Suffix + Output TFLOP/s per Chip — achieved model TFLOP/s on the theoretically necessary tokens: suffix_output_tput × (2 × N_active + attention FLOPs per computed token) / 1e12. The weight-GEMM term is 2 × N_active (Kaplan/PaLM convention, MoE-CAP S-MFU); the attention term is priced per architecture from each run's true (ISL, OSL) request distribution — see methodology below.

Named after what it measures: the new (theoretically uncachable) input suffix plus output tokens. Like MFU vs HFU, actual (uncached + output) throughput is always at least the theoretical (uncachable suffix + output) throughput.

This is deliberately the theoretical prefix (what could be cached given the trace), not the server-observed cache hits: a system with better real caching would otherwise show fewer "uncached" tokens and look worse on the derived metric. Follows the #ai-ml-system-research discussion of Horace's feedback (cached-input-counted throughput looks like 500% MFU); Alec's UI ask was a new row for input tokens without prefix caching in the point overlay.

Both tooltip rows render in all three point-overlay variants (official, unofficial-run overlay, and date-comparison GPU graph) via the shared agentic section, with /zh labels. Rows and metric fields are omitted when theoretical_cache_hit_rate (or total_prompt_tokens for the rows) is missing or out of range, so fixed-sequence points never render them and drop off the chart under the new metrics.

Attention-FLOPs methodology

Attention-score FLOPs (activation–activation matmuls: QKᵀ and scores·V; all weight matmuls live in the 2·N_active term; MAC = 2 FLOPs) depend on per-request context lengths, so aggregate token counts are not enough. The pipeline:

1. Exact request-length moments (stats v9, packages/db). For every profiling request we accumulate the exact joint moment sums (n, ΣP, ΣP², ΣO, ΣO², ΣPO) of (ISL P, OSL O) into the aggregate-stats bundle (with a live-derivation fallback from the stored profile_export.jsonl blob), served through /api/v1/derived-agentic-metrics alongside the normalized-interactivity fields. The STATS_VERSION 8→9 bump rolls the blob cache automatically; existing rows self-heal on read or can be backfilled with bun run --cwd packages/db db:backfill-aggregate-stats.

2. Closed-form integration under the theoretical prefix (attention-flops.ts). With theoretical hit rate r, request (P, O) computes its suffix at contexts rP+1 … P and decodes at P+1 … P+O (the cached prefix is not recomputed but is still attended). Summed over requests, total attended context and computed tokens are exactly:

Σctx    = (1−r²)/2·ΣP² + (1−r)/2·ΣP + ΣPO + (ΣO² + ΣO)/2
Σtokens = (1−r)·ΣP + ΣO

3. Per-architecture cost specs (model-architectures.ts). Every supported attention mechanism reduces to per-layer cost F(L) = lin·L + coeff·min(L, cap) + const at context L, which the moments integrate exactly (the min term uses coeff·min(cap·Σtokens, Σctx), exact when contexts sit on one side of the cap, an upper bound otherwise). Different requests contribute their own F(L) and are summed — heterogeneous request FLOPs are handled by construction. Per-model specs, each verified against configs + tech reports:

Model	Mechanism	Per-layer cost/token at context L (FLOPs)
DeepSeek-R1-0528	Absorbed MLA (MQA mode): 128 heads score vs shared 576-dim latent, aggregate in 512	61 layers × 278,528·L (config, V3.2 report on MQA-mode MLA)
DeepSeek-V4-Pro	Hybrid HCA + CSA, shared K=V 512-dim latent, 128-token sliding window, sink = pure logit (0 tokens)	31 HCA layers: 2048·L + 33.6M (window); 30 CSA layers: FP4 indexer 4096·L + top-1024 core 65,536·min(L, 4096) + 33.6M (V4 report, config `compress_ratios` ⇒ 31/30 split, HF docs: HCA has no indexer, release blog)
Llama-3.1/3.3-70B	GQA, 64 Q heads, d=128	80 layers × 65,536·L (config)
gpt-oss-120b	Alternating full / 128-token sliding window, 64 Q heads d=64, sinks are pure logits	18 full layers × 16,384·L + 18 sliding layers × 16,384·min(L, 128) (config, model card paper, sink implementation)
Kimi-K2.5	Absorbed MLA, 64 heads, 512+64 latent	61 layers × 139,264·L (config)
Kimi-K3	Hybrid KDA + gated MLA-NoPE (3:1)	69 KDA layers × 11.0M const (linear attention, recurrent form 7·H·d²; chunked prefill ≈ +11%, within noise) + 24 MLA layers × 196,608·L (Kimi Linear paper, K3 config, fla-org KDA kernel)
MiniMax-M2.5	GQA, 48 Q heads, d=128	62 layers × 24,576·L (config)
MiniMax-M3	Hybrid dense + MSA (top-k sparse over 8× compressed keys)	3 dense layers × 32,768·L; 57 MSA layers: compressed indexer 1024·L + top-2048 core 32,768·min(L, 2048) (MSA paper, M3 config)

Sanity checks reproduced from the specs: V4-Pro ≈ 27% of V3.2's per-token inference FLOPs at 1M context (matches the V4 report headline); MSA ≈ 28× attention-FLOP reduction at 1M (matches the MSA paper).

Scope/caveats: Qwen3.5 and GLM-5/5.2 have no attention spec yet, so they omit the TFLOP/s metric (as before). The metric requires request-length moments, which flow only through the persisted-benchmark chart path — unofficial-run overlays and AI-chart entries omit it. Σctx covers the suffix attending the cached prefix exactly via the (1−r²)/2·ΣP² term.

Test plan

bun run typecheck, bun run lint, bun run fmt — clean
New attention-flops unit suite: closed-form Σctx/Σtokens vs brute-force token enumeration (r = 0, 0.5, 1), capped saturation/fallback, additive group mixing, null guards
DB suites: exact moment accumulation from profile blobs (incl. cancelled-request exclusion and non-profiling-phase filtering), v9 bundle shape, derived-metrics pass-through — 597 tests pass
Chart suite: R1 fixture verified by hand (Σctx = 285,150, Σtokens = 300 ⇒ 37.86 TFLOP/s at 420 tok/s), metric omitted without moments or architecture entry

中文说明

为 agentic trace 数据点的 point overlay（悬浮框）新增两行指标：

理论 prefix token 数 — 每个点的理论 prefix，按该点在 trace 中已出现的全部 prefix 之和计算。测试框架已将该总和以无限 cache 理论命中率（theoretical_cache_hit_rate）的形式上报，因此用 total_prompt_tokens × theoretical_cache_hit_rate 取整即可还原 token 总数。
无 prefix cache 的输入 token 数 — prompt token 总数减去理论 prefix，即使拥有无限 prefix cache 也必须实际 prefill 的输入 token。

同时基于同一命中率新增三个吞吐量 Y 轴指标：

每芯片新输入 suffix + 输出 token 吞吐量 — 总吞吐量减去输入吞吐量中的理论 prefix 部分（tput_per_gpu − input_tput_per_gpu × rate）
每芯片新输入 suffix token 吞吐量 — input_tput_per_gpu × (1 − rate)，与现有输入吞吐量指标一样以 P90 TTFT 为 X 轴
每芯片新输入 suffix + 输出 TFLOP/s — 在理论必需 token 上实现的模型 TFLOP/s：suffix_output_tput × (2 × 激活参数量 + 每计算 token 的注意力 FLOPs) / 1e12。权重 GEMM 项为 2 × N_active（Kaplan/PaLM 约定、MoE-CAP S-MFU）；注意力项按各模型架构基于该 run 的真实 (ISL, OSL) 请求分布计算。

注意力 FLOPs 计算方法

注意力得分 FLOPs（QKᵀ 与 scores·V 两个激活-激活矩阵乘；所有权重矩阵乘计入 2·N_active 项；MAC = 2 FLOPs）依赖每个请求的上下文长度，仅有聚合 token 数不够。流程：

精确请求长度矩（stats v9）：对每个 profiling 请求累加 (ISL P, OSL O) 的精确联合矩 (n, ΣP, ΣP², ΣO, ΣO², ΣPO) 存入聚合统计（含从 profile_export.jsonl 实时推导的兜底路径），经 /api/v1/derived-agentic-metrics 下发。STATS_VERSION 8→9 自动滚动缓存；存量数据读取时自愈，也可用 bun run --cwd packages/db db:backfill-aggregate-stats 回填。
理论前缀下的闭式积分：命中率 r 时，请求 (P, O) 在上下文 rP+1…P 计算 suffix、在 P+1…P+O 解码（缓存前缀不重算但仍被 attend）。总上下文 Σctx = (1−r²)/2·ΣP² + (1−r)/2·ΣP + ΣPO + (ΣO²+ΣO)/2，总计算 token 数 Σtokens = (1−r)·ΣP + ΣO。
各架构成本规格：每种注意力机制归约为每层 F(L) = lin·L + coeff·min(L, cap) + const，由矩精确积分（min 项取 coeff·min(cap·Σtokens, Σctx)，上下文同侧时精确，否则为上界）；不同请求各自贡献 F(L) 后求和，天然处理异构请求 FLOPs。覆盖：DeepSeek-R1（absorbed MLA）、DeepSeek-V4-Pro（HCA + CSA + FP4 indexer + 滑窗，sink 为纯 logit 记 0 token）、Llama 3.1/3.3 70B（GQA）、gpt-oss-120b（全注意力/滑窗交替）、Kimi K2.5（MLA）、Kimi K3（KDA 线性注意力常数项 + gated MLA-NoPE）、MiniMax M2.5（GQA）、MiniMax M3（dense + MSA 稀疏）。各规格已对照 config 与技术报告核验（复现 V4-Pro 1M 上下文 ≈ V3.2 的 27%、MSA 1M ≈ 28× 注意力 FLOPs 缩减）。

范围与注意：Qwen3.5 与 GLM-5/5.2 暂无注意力规格，不显示该指标；指标依赖请求长度矩，仅在持久化 benchmark 图表路径可用，非官方 run overlay 与 AI 图表暂不显示。

测试：typecheck / lint / fmt 全部通过；新增 attention-flops 单测（闭式解对暴力枚举、capped 饱和、空值防护）、DB 矩累加与 v9 结构单测（597 个通过）、图表 R1 手算校验（Σctx = 285,150 ⇒ 420 tok/s 下 37.86 TFLOP/s）。

Fix: existing data showed "No data available" for the TFLOP/s metric

Root cause: production profile_export_jsonl_gz blobs reach 248 MB compressed, while Neon's serverless HTTP driver caps any single response at 64 MB (HTTP 507). The live fallbacks selected whole blobs inline (6/query in derived-agentic-metrics, 8/query in agentic-aggregates), and the v9 stats bump routes every pre-v9 row through those fallbacks — so one oversized batch 500'd the endpoint and every point lost its moments (and p75/p90 E2E-normalized interactivity).

Fix (commit 3513940): blobs are never selected whole anymore. A cheap metadata query maps ids to trace rows, then each blob streams through bounded, self-terminating substring chunks (8 MiB) into a streaming gunzip line parser — the same pattern backfill-aggregate-stats.ts already used. Per-row try/catch means one bad blob can never blank the whole response. Ingest, backfill, and both query fallbacks now share a single extractor (extractProfileSamples in agentic-shared.ts), so fast and slow paths cannot drift. server_metrics_json_gz (also >64 MB in the tail) is chunk-read too.

Verified against production data: the previously-failing 35 MB row returns full moments in <2 s; the worst-case 237 MB blob streams in 11 s at 93 MB RSS.

Deploy ordering: run db:backfill-aggregate-stats only after this PR is deployed. Backfilling v9 bundles first would push the current v8 production code onto its own (still-broken) inline-fetch fallback.

中文：TFLOP/s 指标在现有数据上显示 "No data available" 的修复

根因：生产环境 profile_export_jsonl_gz 压缩后最大达 248 MB，而 Neon serverless HTTP 驱动单次响应上限为 64 MB（HTTP 507）。回退路径整块内联拉取 blob（derived-agentic-metrics 每查询 6 个、agentic-aggregates 每查询 8 个），v9 版本升级使所有旧行都走回退——一个超大批次即令接口 500，所有点位丢失矩量（以及 p75/p90 端到端归一化交互性）。

修复（commit 3513940）：不再整块选取 blob。先用轻量元数据查询定位 trace 行，再通过有界、自终止的 substring 分块（8 MiB）流式送入 gunzip 逐行解析器——与 backfill-aggregate-stats.ts 已有模式一致。按行 try/catch，单个坏 blob 不会拖垮整个响应。摄取、回填和两个查询回退共用同一提取器（agentic-shared.ts 的 extractProfileSamples），快慢路径不会漂移。server_metrics_json_gz（尾部同样 >64 MB）也改为分块读取。

已用生产数据验证：此前失败的 35 MB 行 <2 秒返回完整矩量；最坏情况 237 MB blob 11 秒完成，内存峰值 93 MB。

部署顺序：db:backfill-aggregate-stats 必须在本 PR 部署之后再运行，否则 v9 bundle 会把仍在运行 v8 代码的生产环境推入其尚未修复的整块拉取回退。

Note

Medium Risk
Touches published derived-agentic-metrics contract, STATS_VERSION 9 aggregate bundles, and critical blob-read fallbacks for large agentic traces; deploy before running aggregate backfill per PR notes.

Overview
Adds agentic-trace analytics that treat prefix caching as theoretical (infinite-cache theoretical_cache_hit_rate), not observed server hits, so better real caches are not penalized on derived metrics.

UI: Agentic tooltips show theoretical prefix token count and input tokens without prefix caching. Three new throughput Y-axis options derive from the same rate—new input suffix throughput, suffix+output throughput, and suffix+output TFLOP/s per chip (GEMM 2 × active params plus architecture-specific attention FLOPs). Charts fetch request_length_moments via useDerivedAgenticMetrics and merge them in transformBenchmarkRows; points without moments or an attention spec omit TFLOP/s.

Backend (stats v9): Profile parsing now stores exact joint (ISL, OSL) moment sums in aggregate_stats and exposes them on /api/v1/derived-agentic-metrics. New attention-flops.ts integrates per-model specs from model-architectures.ts over those moments at the theoretical hit rate.

Reliability fix: Fallback paths no longer SELECT whole multi‑hundred‑MB gzip blobs (Neon’s 64 MB response cap). Blobs stream through 8 MiB substring chunks with shared extractProfileSamples; self-heal skips stamping when the server blob fails to parse so null KV/prefix fields are not cached forever.

^{Reviewed by Cursor Bugbot for commit 728011e. Bugbot is set up for automated code reviews on this repo. Configure here.}

… to point overlay For agentic points, derive the theoretical prefix tokens for each point as the sum of every prompt prefix the trace has already seen: the harness reports that sum as the infinite-cache theoretical_cache_hit_rate over served prompt tokens, so multiplying it back with total_prompt_tokens recovers the token sum. Add two new point-overlay rows: - Theoretical Prefix Tokens - Input Tokens w/o Prefix Caching (prompt total minus theoretical prefix) This is deliberately the theoretical prefix (what could be cached given the trace), not server-observed cache hits, so systems with better real caching are not penalized on the derived uncached-input view. 中文：为 agentic 数据点的 point overlay 新增两行指标：理论 prefix token 数（按该点在 trace 中已出现的全部 prefix 之和计算，即无限 cache 理论命中率乘以 prompt token 总数）以及无 prefix cache 的输入 token 数（prompt 总数减去理论 prefix）。刻意采用理论 prefix 而非服务端实际命中，避免 cache 能力更强的系统在该派生指标上被低估。

vercel · 2026-08-25T20:55:45Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
inferencemax-app	Ready	Preview	Aug 26, 2026 12:05am

Add two Throughput y-axis options derived from the trace-level theoretical (infinite-cache) prefix cache hit rate: - Uncached Token Throughput per Chip: total throughput minus the theoretical prefix share of input throughput - Uncached Input Token Throughput per Chip: input throughput x (1 - rate) Both deliberately use the theoretical rate rather than server-observed cache hits so systems with good cache storage aren't penalized. Only agentic trace points carry the rate; fixed-sequence points omit the fields and drop off the chart for these metrics. 中文：新增两个基于 trace 级理论（无限 cache）prefix cache 命中率的吞吐量 Y 轴指标： - 每芯片无 prefix cache token 吞吐量：总吞吐量减去输入吞吐量中的理论 prefix 部分 - 每芯片无 prefix cache 输入 token 吞吐量：输入吞吐量 x（1 - 命中率）两者刻意采用理论命中率而非服务端实际 cache 命中，避免惩罚 cache 存储能力强的系统。只有 agentic trace 点携带该命中率；固定序列点省略这些字段，在这两个指标下不显示。

The new Uncached Input Token Throughput per Chip metric also matches the "input token throughput" search, so the filtered option count is now three. 中文：新增的每芯片无 prefix cache 输入 token 吞吐量指标同样匹配 "input token throughput" 搜索，过滤后的选项数量变为三个。

…naming Per thread discussion, name the metric after what it measures: the prompt suffix (theoretically uncachable input) plus output tokens. Like MFU vs HFU, actual (uncached + output) throughput is always at least the theoretical (uncachable suffix + output) throughput. - Uncached Token Throughput per Chip -> Prompt Suffix + Output Token Throughput per Chip (key y_promptSuffixOutputTputPerGpu) - Uncached Input Token Throughput per Chip -> Prompt Suffix Token Throughput per Chip (key y_promptSuffixTputPerGpu) 中文：按讨论将指标以其实际度量对象命名：prompt suffix（理论上无法缓存的输入）加输出 token。类似 MFU 与 HFU 的关系，实际（未缓存 + 输出）吞吐量总是不低于理论（不可缓存 suffix + 输出）吞吐量。 - 每芯片无 prefix cache token 吞吐量 -> 每芯片 prompt suffix + 输出 token 吞吐量 - 每芯片无 prefix cache 输入 token 吞吐量 -> 每芯片 prompt suffix token 吞吐量

Per thread, call the metric "New Input Suffix + Output Token Throughput": the new (theoretically uncachable) input suffix plus output tokens. - Prompt Suffix + Output Token Throughput per Chip -> New Input Suffix + Output Token Throughput per Chip (key y_newInputSuffixOutputTputPerGpu) - Prompt Suffix Token Throughput per Chip -> New Input Suffix Token Throughput per Chip (key y_newInputSuffixTputPerGpu) 中文：按讨论将指标更名为 "New Input Suffix + Output Token Throughput"：新增（理论上无法缓存的）输入 suffix 加输出 token。 - 每芯片 prompt suffix + 输出 token 吞吐量 -> 每芯片新输入 suffix + 输出 token 吞吐量 - 每芯片 prompt suffix token 吞吐量 -> 每芯片新输入 suffix token 吞吐量

Bugbot: missing input throughput is normalized to 0 upstream, so the suffix+output metric would silently equal total throughput. Omit both fields when input throughput is absent. 中文：Bugbot 发现上游会将缺失的输入吞吐量归一化为 0，导致 suffix + 输出指标退化为总吞吐量。现在输入吞吐量缺失时两个字段均省略。

Achieved model TFLOP/s on the theoretically necessary tokens only: FLOPs/token = 2 x active params (GEMM-only Kaplan/PaLM convention), times the new-input-suffix + output token throughput, so cached prefix tokens contribute no compute credit. Attention-score FLOPs are excluded: they depend on per-request context lengths (only aggregate token counts reach the chart layer) and on the attention implementation (MHA/GQA/MLA/linear), so 2N_active is the comparable cross-model lower bound - same spirit as MFU counting only theoretically required work. Active params come from the model-architectures registry; points whose model lacks an entry omit the field. 中文：新增每芯片新输入 suffix + 输出 TFLOP/s 指标，仅统计理论上必需的 token：FLOPs/token = 2 x 激活参数量（GEMM-only Kaplan/PaLM 口径），乘以新输入 suffix + 输出 token 吞吐量，缓存的 prefix token 不计入算力。不含 attention score FLOPs：其依赖每个请求的上下文长度（图表层只有聚合 token 数）及 attention 实现（MHA/GQA/MLA/linear），2N_active 是可跨模型比较的下界，与 MFU 只统计理论必需计算量的思路一致。激活参数量取自 model-architectures 注册表；缺少架构条目的模型省略该字段。

The chip-untranslated rule only knew tok/s/chip; the new TFLOP/s/chip y-metric unit tripped it. Units stay English per AGENTS.md rule 6, so extend the exemption to [KMGT]FLOP/s/chip and add a fixture. 中文：chip-untranslated 规则原本只豁免 tok/s/chip，新增的 TFLOP/s/chip Y 轴指标单位触发了误报。按 AGENTS.md 第 6 条单位保留英文，故将豁免扩展到 [KMGT]FLOP/s/chip 并补充用例。

…OP/s Replace the GEMM-only FLOP estimate with GEMM + per-architecture attention FLOPs integrated in closed form over each run's true (ISL, OSL) request distribution. - db: stats v9 adds exact joint request-length moment sums (n, ΣP, ΣP², ΣO, ΣO², ΣPO) to the aggregate-stats bundle and to the derived-agentic-metrics fallback/self-heal path. - app: new attention-flops module prices any affine+capped per-context cost F(L) = lin·L + coeff·min(L, cap) + const exactly from the moments, under the theoretical infinite-cache prefix (cached prefix excluded from computed tokens but still attended by the suffix). - model-architectures: cited attention cost specs for DeepSeek-R1 (absorbed MLA), DeepSeek-V4-Pro (HCA + capped CSA + indexer), Llama 3.x 70B (GQA), gpt-oss (full + sliding-window), Kimi K2.5 (MLA), Kimi K3 (KDA + gated MLA NoPE), MiniMax M2.5 (GQA), MiniMax M3 (dense + capped MSA). - chart-utils: TFLOP/s/chip = suffix+output tput × (2·N_active + attention FLOPs/token) and is omitted when moments or an attention spec are missing. 中文：将 TFLOP/s/chip 指标从仅 GEMM 估算升级为 GEMM + 各模型注意力 FLOPs。数据库统计 v9 新增请求长度联合矩（n、ΣP、ΣP²、ΣO、ΣO²、ΣPO），前端按各架构的注意力成本公式 F(L)=lin·L+coeff·min(L,cap)+const 在理论无限缓存前缀假设下闭式积分。覆盖 DeepSeek-R1/V4-Pro、Llama 3.x 70B、gpt-oss、Kimi K2.5/K3、 MiniMax M2.5/M3 八个架构；缺少矩数据或注意力规格时该指标不显示。

Add the joint (ISL, OSL) request-length moment sums to the public API documentation for GET /api/v1/derived-agentic-metrics (schema + example) and refresh the route/shared-source SHA-256 digests in the catalog. 中文：在 GET /api/v1/derived-agentic-metrics 的公开 API 文档中补充 (ISL, OSL) 请求长度联合矩和字段（schema 与示例），并更新路由及共享源码的 SHA-256 摘要。

… MB response cap Production profile_export blobs reach 248 MB compressed while Neon's serverless HTTP driver rejects any response over 64 MB (HTTP 507). The derived-agentic-metrics fallback selected 6 whole blobs per query and the agentic-aggregates fallback 8 per query, so after the v9 stats bump routed every pre-v9 row through the fallback, one oversized batch failed the whole query, the endpoint 500'd, and the new TFLOP/s per chip metric (plus p75/p90 E2E-normalized interactivity) showed "No data available" on existing data. - agentic-shared: add streamTraceReplayBlob (bounded, self-terminating substring chunk reads) + a single shared streaming profile-sample extractor used by ingest, backfill, and both query fallbacks - derived-agentic-metrics + agentic-aggregates: metadata query first, then per-row streamed recompute with per-row error isolation so one bad blob can never blank the whole response; server_metrics blobs are also read via bounded chunks (they can exceed the cap too) - etl/compute-aggregate-stats: delegate to the shared extractor - verified against prod: the previously-507ing 35 MB row returns full moments in <2 s; the worst-case 237 MB blob streams in 11 s at 93 MB RSS 中文：生产环境的 profile_export blob 压缩后最大达 248 MB，而 Neon serverless HTTP 驱动拒绝超过 64 MB 的响应（HTTP 507）。derived-agentic-metrics 回退路径每次查询内联拉取 6 个完整 blob，agentic-aggregates 回退每次 8 个；v9 版本升级后所有旧行都走回退路径，一个超大批次即导致整个查询失败、接口 500，新的 TFLOP/s per chip 指标（以及 p75/p90 端到端归一化交互性）在现有数据上显示 "No data available"。 - agentic-shared：新增 streamTraceReplayBlob（有界、自终止的 substring 分块读取）及统一的流式 profile 样本提取器，供摄取、回填和两个查询回退共用 - derived-agentic-metrics 与 agentic-aggregates：先做元数据查询，再逐行流式重算并按行隔离错误，单个坏 blob 不再拖垮整个响应；server_metrics blob 同样改为分块读取（也可能超过上限） - etl/compute-aggregate-stats：委托给共享提取器 - 已用生产数据验证：此前 507 的 35 MB 行 <2 秒返回完整矩量；最坏情况 237 MB blob 11 秒流式完成，内存峰值 93 MB

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 3513940. Configure here.}

中文：提交钩子二次格式化后刷新 agentic-aggregates 的共享源码 SHA-256 摘要。

…r fields Bugbot findings on #849: - Llama 3.1/3.3 70B linPerCtx was 65536 but 4·H·d with 64 heads × dim 128 is 32768 — attention FLOPs were doubled for both Llama models. - agentic-aggregates pass 2: when a server blob exists but fails to read or parse, drop the id from the self-heal set instead of stamping a current-version bundle with null kvCacheUtil/prefixCacheHitRate, which would permanently cache the miss (fast path would never retry the blob). 中文： - Llama 3.1/3.3 70B 的 linPerCtx 误写为 65536；按 4·H·d（64 头 × 128 维）应为 32768，此前两款 Llama 的注意力 FLOPs 被高估一倍。 - agentic-aggregates 第二遍：server blob 存在但读取/解析失败时，从自愈集合中剔除该 id，而不是把 kvCacheUtil/prefixCacheHitRate 为 null 的当前版本 bundle 写回——否则缺失会被永久缓存，快路径不再重试。

functionstackx requested a review from adibarra as a code owner August 25, 2026 20:55

vercel Bot deployed to Preview August 25, 2026 20:56 View deployment

vercel Bot deployed to Preview August 25, 2026 21:07 View deployment

This comment has been minimized.

Sign in to view

cursor Bot reviewed Aug 25, 2026

View reviewed changes

Comment thread packages/app/src/lib/chart-utils.ts

vercel Bot deployed to Preview August 25, 2026 21:13 View deployment

vercel Bot deployed to Preview August 25, 2026 21:20 View deployment

vercel Bot deployed to Preview August 25, 2026 21:29 View deployment

vercel Bot deployed to Preview August 25, 2026 21:42 View deployment

vercel Bot deployed to Preview August 25, 2026 22:03 View deployment

This comment has been minimized.

Sign in to view

vercel Bot deployed to Preview August 25, 2026 22:21 View deployment

vercel Bot deployed to Preview August 25, 2026 23:19 View deployment

cursor Bot reviewed Aug 25, 2026

View reviewed changes

Comment thread packages/app/src/lib/model-architectures.ts

Comment thread packages/db/src/queries/derived-agentic-metrics.ts Outdated

vercel Bot deployed to Preview August 25, 2026 23:27 View deployment

vercel Bot deployed to Preview August 25, 2026 23:53 View deployment

cursor Bot reviewed Aug 25, 2026

View reviewed changes

Comment thread packages/db/src/queries/agentic-aggregates.ts

chore: refresh agentic-aggregates shared-source digest after format hook

f042265

中文：提交钩子二次格式化后刷新 agentic-aggregates 的共享源码 SHA-256 摘要。

vercel Bot deployed to Preview August 25, 2026 23:59 View deployment

This comment has been minimized.

Sign in to view

vercel Bot deployed to Preview August 26, 2026 00:05 View deployment

Navigation Menu

Uh oh!

[vibecoded wip, do not merged yet] feat(inference): theoretical prefix tokens overlay + new input suffix throughput and TFLOP/s y-metrics / 理论 prefix token 悬浮框指标与新输入 suffix 吞吐量及 TFLOP/s Y 轴指标 - #849

functionstackx commented Aug 25, 2026 •

edited by cursor Bot

Loading

Uh oh!

vercel Bot commented Aug 25, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

This comment has been minimized.

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

functionstackx commented Aug 25, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Attention-FLOPs methodology

Test plan

中文说明

注意力 FLOPs 计算方法

Fix: existing data showed "No data available" for the TFLOP/s metric

中文：TFLOP/s 指标在现有数据上显示 "No data available" 的修复

Uh oh!

vercel Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

functionstackx commented Aug 25, 2026 •

edited by cursor Bot

Loading

vercel Bot commented Aug 25, 2026 •

edited

Loading