Compare commits
13
Commits
63509dba20
...
d376bd0df7
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d376bd0df7 | ||
|
|
b2e06a2933 | ||
|
|
adbe9d129d | ||
|
|
cd661b99c7 | ||
|
|
8a14ce8722 | ||
|
|
8e50ecef77 | ||
|
|
90de3a698c | ||
|
|
fe1e3065d6 | ||
|
|
871d0ac5cb | ||
|
|
602c0b99da | ||
|
|
a22844d2dd | ||
|
|
001f0402b2 | ||
|
|
2bf3c62dbf |
@@ -1,4 +1,5 @@
|
||||
.superpowers/
|
||||
.worktrees/
|
||||
AGENTS.md
|
||||
agents.md
|
||||
node_modules/
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,472 @@
|
||||
# LLM 后台架构观测标签页设计
|
||||
|
||||
## 1. 背景
|
||||
|
||||
当前首页提供 `GEO 文章优化` 和 `普通文案优化` 两个功能标签。GEO 主流程通过 `POST /api/jobs/optimize-stream` 返回 NDJSON 事件,前端可以看到事实卡、优化草稿、质量检查、定向修复和终稿等阶段性结果。
|
||||
|
||||
所有真实模型调用集中在 `src/lib/llm/client.ts`。该客户端目前会向服务端控制台打印 `[llm:start]`、`[llm:response]`、`[llm:validated]` 和 `[llm:error]`,并通过审计回调保存 Provider、模型、任务名、耗时、Schema 状态和输入输出哈希。现有持久化审计不保存 SDK 实际发送的完整请求对象,也不保存 SDK 收到的完整响应对象,因此前端不能准确还原一次 LLM 调用。
|
||||
|
||||
本功能增加第三个首页标签 `后台架构`,用于只读展示文章优化后台的真实执行过程,重点呈现每次 LLM 调用的完整请求、完整响应和校验结果。架构图必须由后端真实任务事件驱动,而不是前端按预估时间播放动画。
|
||||
|
||||
## 2. 目标
|
||||
|
||||
本功能需要同时服务两类受众:
|
||||
|
||||
- 客户或非技术人员通过默认摘要视图理解文章经历了哪些处理阶段。
|
||||
- 已授权的研发或运营人员查看完整的 LLM 请求、响应、耗时、Token 用量、错误和 Schema 校验结果。
|
||||
|
||||
完成后,用户应当能够:
|
||||
|
||||
1. 在首页切换到 `后台架构` 标签。
|
||||
2. 观察当前任务的架构节点随真实后台状态变化。
|
||||
3. 在没有运行中任务时查看最近一次终态任务。
|
||||
4. 按真实发生顺序选择任意一次 LLM 调用。
|
||||
5. 在授权后查看 SDK 实际发送的完整请求对象。
|
||||
6. 在授权后查看 SDK 收到的完整响应 JSON。
|
||||
7. 区分 Provider 失败、响应解析失败、Schema 校验失败和业务质检未通过。
|
||||
|
||||
## 3. 非目标
|
||||
|
||||
本次不包含:
|
||||
|
||||
- 浏览、筛选或搜索全部历史任务的完整 LLM 日志。
|
||||
- 在架构页发起、重试、取消或管理任务。
|
||||
- 接入 OpenTelemetry、Sentry、Grafana 或其他外部可观测平台。
|
||||
- 将现有优化流程改造成持久化队列、Durable Workflow 或后台任务系统。
|
||||
- 为已有历史案例补造过去没有保存的请求或响应。
|
||||
- 展示 API Key、Authorization、HTTP 请求头或底层网络数据包。
|
||||
- 改变现有文章优化、质量检查、定向修复和导出业务规则。
|
||||
|
||||
## 4. 已选方案
|
||||
|
||||
采用一等公民的 LLM 调用事件方案。
|
||||
|
||||
`src/lib/llm/client.ts` 在模型请求前后产生包含真实对象的内部追踪事件。任务追踪收集器先持久化完整正文,再通过现有 NDJSON 响应流向首页发送不含正文的状态事件。架构标签页与 GEO 标签共享当前页面的任务状态,并在用户选择某次调用后通过授权接口按需读取完整正文。页面刷新后,通过只读追踪接口恢复运行中任务已经保存的事件,或读取最近一次终态任务。
|
||||
|
||||
没有采用以下方案:
|
||||
|
||||
- 前端根据阶段事件和控制台文本日志重建调用过程。该方案无法保证完整请求和响应与真实 SDK 数据一致。
|
||||
- 外部可观测平台。该方案适合更大规模的运维场景,但超出当前产品范围。
|
||||
|
||||
## 5. 总体架构
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph UI["浏览器界面"]
|
||||
GEO["GEO 文章优化标签"]
|
||||
ARCH["后台架构标签<br/>只读观察器"]
|
||||
end
|
||||
|
||||
subgraph API["Next.js / Cloudflare Worker"]
|
||||
STREAM["POST /api/jobs/optimize-stream<br/>NDJSON 实时事件"]
|
||||
TRACEAPI["只读 LLM Trace API<br/>刷新后恢复"]
|
||||
COLLECTOR["任务追踪收集器"]
|
||||
end
|
||||
|
||||
subgraph WORKFLOW["真实文章优化工作流"]
|
||||
FACT["fact_extractor<br/>事实提取"]
|
||||
DRAFT["article_optimizer<br/>生成优化稿"]
|
||||
QA["quality_inspector<br/>质量检查"]
|
||||
REWRITE["targeted_rewriter<br/>定向修复"]
|
||||
FACT --> DRAFT --> QA
|
||||
QA -->|"未通过,最多两轮"| REWRITE
|
||||
REWRITE -->|"重新检查"| QA
|
||||
end
|
||||
|
||||
CLIENT["src/lib/llm/client.ts<br/>唯一 LLM 调用边界"]
|
||||
LLM["DeepSeek / OpenAI-compatible API"]
|
||||
D1[("D1<br/>任务、调用索引、状态")]
|
||||
R2[("R2<br/>完整请求与响应 JSON")]
|
||||
AUDIT[("案例结果版本<br/>摘要与内容哈希")]
|
||||
|
||||
GEO -->|"携带访问密钥发起任务"| STREAM
|
||||
STREAM --> FACT
|
||||
FACT & DRAFT & QA & REWRITE --> CLIENT
|
||||
CLIENT -->|"SDK 实际请求对象"| LLM
|
||||
LLM -->|"SDK 完整响应对象"| CLIENT
|
||||
CLIENT -->|"started / responded / validated / failed"| COLLECTOR
|
||||
COLLECTOR -->|"实时事件"| STREAM
|
||||
STREAM -->|"同一任务流"| GEO
|
||||
GEO -.->|"切换标签,共享当前任务状态"| ARCH
|
||||
COLLECTOR --> D1
|
||||
COLLECTOR --> R2
|
||||
COLLECTOR -->|"任务结束后写入"| AUDIT
|
||||
ARCH -->|"携带访问密钥读取"| TRACEAPI
|
||||
TRACEAPI --> D1
|
||||
TRACEAPI --> R2
|
||||
```
|
||||
|
||||
### 5.1 真实请求的定义
|
||||
|
||||
“完整请求”指应用完成默认值合并后、立即传给 OpenAI-compatible SDK 的请求对象,包括:
|
||||
|
||||
- `model`
|
||||
- `temperature`
|
||||
- `response_format`
|
||||
- 按实际顺序排列的完整 `messages`
|
||||
- 后续真实加入 SDK 请求的其他非敏感参数
|
||||
|
||||
追踪对象不包含 SDK 客户端构造参数、API Key、Authorization 或 HTTP 请求头。
|
||||
|
||||
### 5.2 真实响应的定义
|
||||
|
||||
“完整响应”指 SDK 返回、但尚未提取 `choices[0].message.content`、执行 JSON 解析或 Zod 校验之前的完整可序列化响应对象。它应保留 Provider 实际返回的字段,包括存在时的:
|
||||
|
||||
- `id`
|
||||
- `object`
|
||||
- `created`
|
||||
- `model`
|
||||
- `choices`
|
||||
- `finish_reason`
|
||||
- `usage`
|
||||
- Provider 返回的其他可序列化非敏感字段
|
||||
|
||||
若 Provider 调用没有返回正常响应,则保存请求和安全化错误对象,不伪造响应。
|
||||
|
||||
## 6. 模块边界
|
||||
|
||||
### 6.1 LLM 客户端
|
||||
|
||||
`src/lib/llm/client.ts` 继续作为唯一模型调用边界,负责:
|
||||
|
||||
- 构造最终 SDK 请求对象。
|
||||
- 在调用前把最终 SDK 请求对象交给追踪收集器。
|
||||
- 保存 SDK 返回的完整响应对象。
|
||||
- 在 JSON 解析和 Schema 校验后把对应结果交给追踪收集器。
|
||||
- 将现有审计摘要从同一份追踪事实派生出来。
|
||||
|
||||
工作流节点只传递任务名、修复轮次和追踪回调,不自行拼装日志。
|
||||
|
||||
### 6.2 任务追踪收集器
|
||||
|
||||
新增独立的任务追踪收集器,负责:
|
||||
|
||||
- 为每次调用分配稳定 `call_id` 和递增 `sequence`。
|
||||
- 将 LLM 事件映射到任务、案例、工作流阶段和修复轮次。
|
||||
- 先保存请求或响应正文,再将不含完整正文的实时状态事件发送给现有优化流。
|
||||
- 增量写入 D1 索引和 R2 正文。
|
||||
- 在任务结束后执行完整日志保留清理。
|
||||
- 生成现有结果版本需要的审计摘要和哈希。
|
||||
|
||||
追踪收集器不参与 Prompt 构造、文章优化或业务质检判断。
|
||||
|
||||
### 6.3 追踪存储
|
||||
|
||||
D1 使用 `llm_trace_runs` 和 `llm_trace_calls` 两张表保存轻量、可查询的结构化数据:
|
||||
|
||||
- 追踪任务标识、任务状态和当前阶段。
|
||||
- 调用顺序、任务名、Provider、模型和耗时。
|
||||
- 请求及响应的 R2 对象键。
|
||||
- Schema 名称、校验状态和问题摘要。
|
||||
- 错误分类、错误摘要和追踪完整性状态。
|
||||
- 创建、响应、校验和完成时间。
|
||||
|
||||
R2 保存完整正文:
|
||||
|
||||
- `llm-traces/<jobId>/<callId>/request.json`
|
||||
- `llm-traces/<jobId>/<callId>/response.json`
|
||||
|
||||
R2 对象保持私有,只能通过应用 API 读取。应用不返回公开对象 URL。
|
||||
|
||||
### 6.4 架构观察组件
|
||||
|
||||
前端组件只负责:
|
||||
|
||||
- 消费当前页面已经收到的追踪事件。
|
||||
- 在刷新后读取追踪清单。
|
||||
- 按调用选择并按需读取请求或响应正文。
|
||||
- 根据后端事件映射固定架构图的节点状态。
|
||||
- 以纯文本方式渲染 JSON。
|
||||
|
||||
组件不推断不存在的阶段,不解析服务器控制台日志,也不发起业务操作。
|
||||
|
||||
## 7. 追踪数据模型
|
||||
|
||||
### 7.1 Trace Run
|
||||
|
||||
每个文章优化任务对应一个追踪运行记录,至少包含:
|
||||
|
||||
- `job_id`
|
||||
- `case_id`
|
||||
- `status`: `running | completed | failed | interrupted`
|
||||
- `current_stage`
|
||||
- `trace_completeness`: `complete | incomplete`
|
||||
- `started_at`
|
||||
- `finished_at`
|
||||
- `error_stage`
|
||||
- `error_summary`
|
||||
|
||||
### 7.2 Trace Call
|
||||
|
||||
每次 LLM 调用对应一个记录,至少包含:
|
||||
|
||||
- `call_id`
|
||||
- `job_id`
|
||||
- `sequence`
|
||||
- `task`
|
||||
- `workflow_stage`
|
||||
- `rewrite_round`
|
||||
- `provider`
|
||||
- `model`
|
||||
- `status`: `started | responded | validated | failed`
|
||||
- `request_object_key`
|
||||
- `response_object_key`
|
||||
- `schema_name`
|
||||
- `schema_valid`
|
||||
- `validation_issues`
|
||||
- `duration_ms`
|
||||
- `started_at`
|
||||
- `responded_at`
|
||||
- `validated_at`
|
||||
- `error_type`
|
||||
- `error_summary`
|
||||
|
||||
完整请求和响应不重复写入结果版本的 D1 JSON。结果版本继续保存紧凑审计摘要和哈希。
|
||||
|
||||
## 8. 事件契约
|
||||
|
||||
现有 `OptimizationStreamEvent` 联合类型增加以下事件:
|
||||
|
||||
### 8.1 `llm_call_started`
|
||||
|
||||
SDK 调用前产生。内部追踪事件包含最终请求对象;写入 R2 后,NDJSON 事件只包含调用标识、任务信息、Provider、模型、最终参数摘要和 `request_available: true`。架构页收到后立即将对应节点标记为运行中,用户打开“请求”视图时再调用授权接口读取完整对象。
|
||||
|
||||
### 8.2 `llm_call_responded`
|
||||
|
||||
SDK 正常返回后、解析前产生。内部追踪事件包含完整响应对象;写入 R2 后,NDJSON 事件只包含调用标识、耗时、Token 用量和 `response_available: true`。架构页允许立即切换到响应视图并按需读取完整对象。
|
||||
|
||||
### 8.3 `llm_call_validated`
|
||||
|
||||
JSON 解析和 Zod 校验后产生。包含 Schema 名称、校验结果和完整问题列表。业务质检结果 `pass` 或 `fail` 与 Schema 是否有效分开表达。
|
||||
|
||||
### 8.4 `llm_call_failed`
|
||||
|
||||
Provider、JSON 解析或 Schema 校验造成调用失败时产生。包含错误分类和安全化错误详情。已经保存的请求或响应继续可见。
|
||||
|
||||
### 8.5 `trace_warning`
|
||||
|
||||
请求或响应正文持久化、追踪索引更新或清理失败时产生。包含安全化错误摘要和 `trace_completeness: incomplete`,但不改变文章优化业务状态。
|
||||
|
||||
事件必须按单次调用顺序发送。不同调用通过 `call_id` 区分,通过 `sequence` 排列。顺序固定为:
|
||||
|
||||
- 成功:`started -> responded -> validated(success=true)`。
|
||||
- Provider 失败:`started -> failed(provider)`。
|
||||
- JSON 解析失败:`started -> responded -> failed(json_parse)`。
|
||||
- Schema 失败:`started -> responded -> validated(success=false) -> failed(schema_validation)`。
|
||||
|
||||
## 9. API 设计
|
||||
|
||||
### 9.1 实时事件
|
||||
|
||||
`POST /api/jobs/optimize-stream` 保持当前请求方式和 NDJSON 格式,并增加 LLM 追踪事件。现有文章结果事件保持兼容。
|
||||
|
||||
### 9.2 最近追踪
|
||||
|
||||
`GET /api/llm-traces/latest`
|
||||
|
||||
- 有运行中任务时返回运行中任务的追踪清单。
|
||||
- 没有运行中任务时返回最近一次终态任务。
|
||||
- 没有任何追踪记录时返回明确的空状态,不返回错误页面。
|
||||
|
||||
### 9.3 任务追踪清单
|
||||
|
||||
`GET /api/jobs/:jobId/llm-trace`
|
||||
|
||||
返回任务状态、当前阶段、架构节点状态和按顺序排列的调用元数据,不内嵌大体积请求或响应正文。
|
||||
|
||||
### 9.4 完整请求
|
||||
|
||||
`GET /api/jobs/:jobId/llm-trace/:callId/request`
|
||||
|
||||
返回该调用保存的完整请求 JSON。
|
||||
|
||||
### 9.5 完整响应
|
||||
|
||||
`GET /api/jobs/:jobId/llm-trace/:callId/response`
|
||||
|
||||
返回该调用保存的完整响应 JSON。调用仍在进行时返回明确的等待状态;调用失败且无响应时返回明确的无响应状态。
|
||||
|
||||
所有追踪读取接口复用现有访问密钥校验,并设置 `Cache-Control: no-store`。
|
||||
|
||||
## 10. 保留策略
|
||||
|
||||
完整日志只保留:
|
||||
|
||||
- 当前仍在运行的任务。
|
||||
- 最近一次进入终态的任务。
|
||||
|
||||
当新任务进入终态后:
|
||||
|
||||
1. 它成为最近一次终态任务。
|
||||
2. 删除更旧终态任务的请求和响应 R2 对象。
|
||||
3. 删除更旧终态任务的 `llm_trace_runs` 和 `llm_trace_calls` 记录。
|
||||
4. 保留案例结果版本中已有的审计摘要、错误摘要和输入输出哈希。
|
||||
|
||||
若存在多个并发运行任务,不删除任何运行中任务的完整记录。最近一次终态任务按完成时间确定。
|
||||
|
||||
## 11. 页面与交互设计
|
||||
|
||||
### 11.1 应用标签
|
||||
|
||||
首页导航增加第三项:
|
||||
|
||||
- `GEO 文章优化`
|
||||
- `普通文案优化`
|
||||
- `后台架构`
|
||||
|
||||
`后台架构` 是只读观察器。切换标签不取消正在读取的优化流,也不清空首页中的当前任务状态。
|
||||
|
||||
### 11.2 页面结构
|
||||
|
||||
页面按以下顺序组成:
|
||||
|
||||
1. 任务状态条:显示当前任务或最近一次任务、任务 ID、Provider、模型、状态和技术详情授权状态。
|
||||
2. 固定执行图:显示输入归一化、事实提取、生成草稿、质量检查、定向修复、保存与导出。
|
||||
3. LLM 调用轨迹:按 `sequence` 显示每次实际调用;复检和多轮修复均为独立记录。
|
||||
4. 调用详情:提供 `请求`、`响应`、`校验` 三个子视图。
|
||||
|
||||
### 11.3 架构图状态
|
||||
|
||||
固定拓扑不随调用次数改变,节点状态由真实事件更新:
|
||||
|
||||
- `waiting`: 尚未开始。
|
||||
- `running`: 当前正在处理。
|
||||
- `completed`: 已成功完成。
|
||||
- `failed`: Provider、解析或 Schema 失败。
|
||||
|
||||
业务质检未通过不是技术失败。质量检查节点应显示“检查完成,需要修复”,然后高亮定向修复节点,并在修复后回到质量检查节点。
|
||||
|
||||
### 11.4 分级展示
|
||||
|
||||
追踪页面整体复用现有访问密钥校验。访问密钥缺失或无效时,不返回任务清单、摘要或正文。授权成功后,页面默认进入客户友好视图,只展示:
|
||||
|
||||
- 阶段名称和中文说明。
|
||||
- Provider、模型、耗时和 Token 用量。
|
||||
- 请求与响应的脱敏摘要。
|
||||
- Schema 和业务结果。
|
||||
|
||||
用户切换到技术详情后,前端通过同一访问密钥按需加载:
|
||||
|
||||
- 完整 SDK 请求对象。
|
||||
- 完整 SDK 响应对象。
|
||||
- 完整 Schema 校验问题。
|
||||
- 安全化错误对象。
|
||||
|
||||
前端不把完整请求或响应内嵌在任务清单和 NDJSON 状态事件中,也不依靠 CSS 隐藏已经下载的敏感内容。
|
||||
|
||||
### 11.5 空状态和恢复
|
||||
|
||||
- 无当前或历史追踪:解释需要先在 GEO 标签运行一次优化。
|
||||
- 当前任务:实时跟随 NDJSON 事件。
|
||||
- 最近一次任务:明确标记为历史快照,避免误认为仍在运行。
|
||||
- 追踪不完整:显示已保存数据,并明确指出缺失阶段或正文。
|
||||
|
||||
## 12. 安全与隐私
|
||||
|
||||
- API Key、Authorization 和请求头永不进入追踪事件、D1、R2 或前端状态。
|
||||
- Provider 错误采用字段白名单序列化,只保留安全的状态码、错误类型、错误码和消息。
|
||||
- R2 追踪对象保持私有。
|
||||
- 所有追踪读取接口都要求访问密钥,并禁用缓存。
|
||||
- 前端通过文本节点或 JSON 文本组件展示内容,不使用 `dangerouslySetInnerHTML`。
|
||||
- 模型响应中的 HTML、Markdown 或脚本不会作为可执行页面内容渲染。
|
||||
- 完整请求和响应不进入 Git、导出文章文件或客户端持久化存储。
|
||||
|
||||
## 13. 错误处理
|
||||
|
||||
### 13.1 Provider 失败
|
||||
|
||||
保存完整请求和安全化错误,产生 `llm_call_failed`,将对应架构节点标记为失败,并保持现有优化错误上抛行为。
|
||||
|
||||
### 13.2 JSON 解析失败
|
||||
|
||||
保存完整原始响应和解析错误。响应视图仍可读取完整响应,校验视图标记为未进入 Schema 校验。
|
||||
|
||||
### 13.3 Schema 校验失败
|
||||
|
||||
保存完整响应、Schema 名称和所有 Zod 问题。架构页区分 Schema 失败与 QA 业务检查未通过。
|
||||
|
||||
### 13.4 追踪持久化失败
|
||||
|
||||
追踪系统不得使原本可以完成的文章优化失败。持久化异常产生 `trace_warning`,将 `trace_completeness` 设为 `incomplete`,主流程继续执行,架构页显示“追踪记录不完整”。
|
||||
|
||||
### 13.5 连接中断
|
||||
|
||||
切换首页标签不会中断同一页面中的流式请求。页面刷新或网络中断后,架构页可以恢复已经增量保存的事件,但本次功能不承诺现有非持久化工作流在浏览器请求断开后继续执行。
|
||||
|
||||
## 14. 测试策略
|
||||
|
||||
### 14.1 LLM 客户端单元测试
|
||||
|
||||
- 保存的请求对象与传给 SDK mock 的对象深度相等。
|
||||
- 保存的响应对象与 SDK mock 返回对象深度相等。
|
||||
- 请求事件先于 Provider 调用产生。
|
||||
- 响应事件先于正文解析和 Schema 校验产生。
|
||||
- API Key、Authorization 和请求头不出现在任何追踪对象中。
|
||||
|
||||
### 14.2 追踪存储测试
|
||||
|
||||
- D1 元数据和 R2 请求、响应对象键正确关联。
|
||||
- 调用顺序、任务名、阶段和修复轮次正确。
|
||||
- 每个事件增量持久化。
|
||||
- 并发运行任务不会被清理。
|
||||
- 新终态任务产生后,旧完整正文被删除,审计摘要和哈希仍保留。
|
||||
|
||||
### 14.3 API 测试
|
||||
|
||||
- NDJSON 事件顺序为 `started -> responded -> validated`。
|
||||
- Provider、解析和 Schema 失败产生正确的失败事件。
|
||||
- 未授权读取完整追踪返回 401。
|
||||
- 请求或响应尚不存在时返回明确状态。
|
||||
- 所有读取响应包含 `Cache-Control: no-store`。
|
||||
|
||||
### 14.4 UI 组件测试
|
||||
|
||||
- 覆盖无任务、当前任务、最近任务、成功、失败和追踪不完整。
|
||||
- 选择不同调用会同步更新架构节点和详情。
|
||||
- 请求、响应、校验子视图显示对应数据。
|
||||
- 业务质检未通过不会显示为 Provider 或 Schema 技术失败。
|
||||
- 未授权状态不下载完整正文。
|
||||
|
||||
### 14.5 浏览器端到端测试
|
||||
|
||||
- 从 GEO 标签发起任务后切换到后台架构标签,优化流继续。
|
||||
- 每次调用按真实顺序出现。
|
||||
- 修复循环在固定架构图中正确高亮。
|
||||
- 刷新后可读取已经保存的当前状态或最近终态任务。
|
||||
- 移动端和窄屏下调用清单与详情改为纵向排列,无横向溢出。
|
||||
|
||||
### 14.6 真实 Provider 冒烟测试
|
||||
|
||||
使用安全样例调用真实 DeepSeek,核对:
|
||||
|
||||
- 模型和最终参数。
|
||||
- 完整 `messages` 顺序与正文。
|
||||
- 完整响应 `choices`、`finish_reason` 和 `usage`。
|
||||
- Schema 校验结果。
|
||||
- 架构节点最终状态。
|
||||
|
||||
真实 Provider 冒烟测试不进入普通 CI,避免消耗额度并避免依赖生产密钥。
|
||||
|
||||
实现完成后运行:
|
||||
|
||||
```bash
|
||||
npm run lint
|
||||
npm test
|
||||
npm run build
|
||||
```
|
||||
|
||||
## 15. 验收标准
|
||||
|
||||
功能完成必须满足:
|
||||
|
||||
1. 首页出现 `后台架构` 标签,且为只读观察器。
|
||||
2. 架构节点由真实后端事件驱动,不使用模拟计时。
|
||||
3. 技术详情中的请求与传给 SDK 的对象一致。
|
||||
4. 技术详情中的响应与 SDK 返回对象一致。
|
||||
5. 完整请求、完整响应和 Schema 校验可以按调用查看。
|
||||
6. 当前任务和最近一次终态任务可被恢复。
|
||||
7. 更旧任务不保留完整正文,但现有审计摘要和哈希继续存在。
|
||||
8. API Key、Authorization 和请求头不会被记录。
|
||||
9. 追踪失败不会使文章优化失败,但会明确显示记录不完整。
|
||||
10. Provider 失败、解析失败、Schema 失败和业务质检未通过在界面中可以区分。
|
||||
11. 单元、集成、组件、端到端测试以及项目 lint、test、build 全部通过。
|
||||
@@ -0,0 +1,48 @@
|
||||
CREATE TABLE IF NOT EXISTS llm_trace_runs (
|
||||
job_id TEXT PRIMARY KEY,
|
||||
case_id TEXT,
|
||||
status TEXT NOT NULL,
|
||||
current_stage TEXT NOT NULL,
|
||||
trace_completeness TEXT NOT NULL,
|
||||
error_stage TEXT,
|
||||
error_summary TEXT,
|
||||
started_at TEXT NOT NULL,
|
||||
finished_at TEXT,
|
||||
updated_at TEXT NOT NULL,
|
||||
FOREIGN KEY (job_id) REFERENCES article_jobs(id) ON DELETE CASCADE,
|
||||
FOREIGN KEY (case_id) REFERENCES optimization_cases(id) ON DELETE SET NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS llm_trace_calls (
|
||||
call_id TEXT PRIMARY KEY,
|
||||
job_id TEXT NOT NULL,
|
||||
sequence INTEGER NOT NULL,
|
||||
task TEXT NOT NULL,
|
||||
workflow_stage TEXT NOT NULL,
|
||||
rewrite_round INTEGER,
|
||||
provider TEXT NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
request_object_key TEXT,
|
||||
response_object_key TEXT,
|
||||
token_usage TEXT,
|
||||
schema_name TEXT,
|
||||
schema_valid INTEGER,
|
||||
validation_issues TEXT NOT NULL DEFAULT '[]',
|
||||
business_status TEXT,
|
||||
duration_ms INTEGER,
|
||||
started_at TEXT NOT NULL,
|
||||
responded_at TEXT,
|
||||
validated_at TEXT,
|
||||
failed_at TEXT,
|
||||
error_type TEXT,
|
||||
error_summary TEXT,
|
||||
UNIQUE (job_id, sequence),
|
||||
FOREIGN KEY (job_id) REFERENCES llm_trace_runs(job_id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_llm_trace_runs_status_updated
|
||||
ON llm_trace_runs(status, updated_at DESC);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_llm_trace_calls_job_sequence
|
||||
ON llm_trace_calls(job_id, sequence);
|
||||
@@ -205,18 +205,17 @@ describe("job API routes", () => {
|
||||
});
|
||||
|
||||
it("streams a one-click optimization from body-only input", async () => {
|
||||
llmMocks.generateValidatedJson
|
||||
.mockResolvedValueOnce(validCandidateFactCard)
|
||||
.mockResolvedValueOnce({
|
||||
title: "流式优化标题",
|
||||
summary: "流式优化摘要。",
|
||||
body_markdown:
|
||||
"## 服务能力\nExample Technology Co., Ltd. 提供 GEO optimization 服务。",
|
||||
image_suggestions: [],
|
||||
changed_sections: ["title", "body"],
|
||||
requires_user_confirmation: [],
|
||||
})
|
||||
.mockResolvedValueOnce({ checks: [] });
|
||||
mockTrackedLlmResult(validCandidateFactCard, "llmcall_fact");
|
||||
mockTrackedLlmResult({
|
||||
title: "流式优化标题",
|
||||
summary: "流式优化摘要。",
|
||||
body_markdown:
|
||||
"## 服务能力\nExample Technology Co., Ltd. 提供 GEO optimization 服务。",
|
||||
image_suggestions: [],
|
||||
changed_sections: ["title", "body"],
|
||||
requires_user_confirmation: [],
|
||||
}, "llmcall_draft");
|
||||
mockTrackedLlmResult({ checks: [] }, "llmcall_qa");
|
||||
|
||||
const response = await optimizeStream(
|
||||
request({
|
||||
@@ -231,13 +230,25 @@ describe("job API routes", () => {
|
||||
expect(response.status).toBe(200);
|
||||
expect(events.map((event) => event.type)).toEqual([
|
||||
"job_created",
|
||||
"llm_call_started",
|
||||
"llm_call_responded",
|
||||
"llm_call_validated",
|
||||
"fact_card_ready",
|
||||
"draft_started",
|
||||
"llm_call_started",
|
||||
"llm_call_responded",
|
||||
"llm_call_validated",
|
||||
"draft_ready",
|
||||
"qa_started",
|
||||
"llm_call_started",
|
||||
"llm_call_responded",
|
||||
"llm_call_validated",
|
||||
"qa_ready",
|
||||
"final_ready",
|
||||
]);
|
||||
expect(JSON.stringify(events)).not.toContain("messages");
|
||||
expect(JSON.stringify(events)).not.toContain("choices");
|
||||
expect(JSON.stringify(events)).not.toContain("test-key");
|
||||
expect(
|
||||
events.find((event) => event.type === "fact_card_ready")?.fact_card,
|
||||
).toEqual(expect.objectContaining({ confirmed_by_user: false }));
|
||||
@@ -784,6 +795,47 @@ async function createJobFixture() {
|
||||
return response.json() as Promise<{ job: { id: string } }>;
|
||||
}
|
||||
|
||||
function mockTrackedLlmResult(value: unknown, callId: string) {
|
||||
llmMocks.generateValidatedJson.mockImplementationOnce(async (input) => {
|
||||
const onTraceEvent = input.onTraceEvent as
|
||||
| ((event: Record<string, unknown>) => void | Promise<void>)
|
||||
| undefined;
|
||||
await onTraceEvent?.({
|
||||
type: "started",
|
||||
call_id: callId,
|
||||
task: input.task ?? "unknown",
|
||||
context: {
|
||||
workflow_stage: input.traceStage ?? "unknown",
|
||||
rewrite_round: input.rewriteRound,
|
||||
schema_name: input.schemaName,
|
||||
},
|
||||
provider: "deepseek",
|
||||
model: "deepseek-v4-pro",
|
||||
request: { model: "deepseek-v4-pro", messages: [] },
|
||||
started_at: "2026-07-16T00:00:00.000Z",
|
||||
});
|
||||
await onTraceEvent?.({
|
||||
type: "responded",
|
||||
call_id: callId,
|
||||
response: {
|
||||
choices: [{ message: { content: JSON.stringify(value) } }],
|
||||
usage: { prompt_tokens: 10, completion_tokens: 4, total_tokens: 14 },
|
||||
},
|
||||
duration_ms: 1200,
|
||||
responded_at: "2026-07-16T00:00:01.200Z",
|
||||
});
|
||||
await onTraceEvent?.({
|
||||
type: "validated",
|
||||
call_id: callId,
|
||||
schema_name: input.schemaName ?? "anonymousSchema",
|
||||
schema_valid: true,
|
||||
validation_issues: [],
|
||||
validated_at: "2026-07-16T00:00:01.300Z",
|
||||
});
|
||||
return value;
|
||||
});
|
||||
}
|
||||
|
||||
function request(body: unknown, options: { apiKey?: string | null } = {}) {
|
||||
const headers: Record<string, string> = { "content-type": "application/json" };
|
||||
const apiKey = options.apiKey === undefined ? "test-key" : options.apiKey;
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
|
||||
import { createSqliteRepository } from "../../../lib/db/sqlite-repository";
|
||||
import { createLocalTracePayloadStore } from "../../../lib/llm/trace-payload-store";
|
||||
import { createSqliteTraceRepository } from "../../../lib/llm/sqlite-trace-repository";
|
||||
import type {
|
||||
LlmTraceCall,
|
||||
LlmTraceManifest,
|
||||
LlmTraceRun,
|
||||
} from "../../../lib/llm/trace-types";
|
||||
import { GET as getLatestTrace } from "../llm-traces/latest/route";
|
||||
import { GET as getJobTrace } from "../jobs/[jobId]/llm-trace/route";
|
||||
import { GET as getTraceRequest } from "../jobs/[jobId]/llm-trace/[callId]/request/route";
|
||||
import { GET as getTraceResponse } from "../jobs/[jobId]/llm-trace/[callId]/response/route";
|
||||
|
||||
const exactRequestFixture = {
|
||||
model: "deepseek-v4-pro",
|
||||
temperature: 0.1,
|
||||
response_format: { type: "json_object" },
|
||||
messages: [{ role: "user", content: "完整文章正文" }],
|
||||
};
|
||||
|
||||
const exactResponseFixture = {
|
||||
id: "chatcmpl_1",
|
||||
choices: [{ message: { content: '{"ok":true}' }, finish_reason: "stop" }],
|
||||
usage: { prompt_tokens: 10, completion_tokens: 4, total_tokens: 14 },
|
||||
};
|
||||
|
||||
describe("LLM trace read APIs", () => {
|
||||
let tempDir: string;
|
||||
let jobId: string;
|
||||
const originalDataDir = process.env.APP_DATA_DIR;
|
||||
const originalApiKey = process.env.API_ACCESS_KEY;
|
||||
const originalAuthDisabled = process.env.API_AUTH_DISABLED;
|
||||
const originalRuntime = process.env.APP_RUNTIME;
|
||||
|
||||
beforeEach(async () => {
|
||||
tempDir = mkdtempSync(join(tmpdir(), "geo-llm-trace-api-"));
|
||||
process.env.APP_DATA_DIR = tempDir;
|
||||
process.env.API_ACCESS_KEY = "test-key";
|
||||
process.env.API_AUTH_DISABLED = "false";
|
||||
delete process.env.APP_RUNTIME;
|
||||
|
||||
const appRepository = createSqliteRepository();
|
||||
const job = await appRepository.createArticleJob({
|
||||
source_title: "",
|
||||
source_body: "完整文章正文",
|
||||
image_inputs: [],
|
||||
publish_platform: "official_site",
|
||||
user_instructions: "",
|
||||
});
|
||||
jobId = job.id;
|
||||
|
||||
const traceRepository = createSqliteTraceRepository();
|
||||
const run: LlmTraceRun = {
|
||||
job_id: jobId,
|
||||
case_id: null,
|
||||
status: "running",
|
||||
current_stage: "draft",
|
||||
trace_completeness: "complete",
|
||||
error_stage: null,
|
||||
error_summary: null,
|
||||
started_at: "2026-07-16T00:00:00.000Z",
|
||||
finished_at: null,
|
||||
updated_at: "2026-07-16T00:00:01.000Z",
|
||||
};
|
||||
const call: LlmTraceCall = {
|
||||
call_id: "llmcall_1",
|
||||
job_id: jobId,
|
||||
sequence: 1,
|
||||
task: "article_optimizer",
|
||||
workflow_stage: "draft",
|
||||
rewrite_round: null,
|
||||
provider: "deepseek",
|
||||
model: "deepseek-v4-pro",
|
||||
status: "validated",
|
||||
request_object_key: `llm-traces/${jobId}/llmcall_1/request.json`,
|
||||
response_object_key: `llm-traces/${jobId}/llmcall_1/response.json`,
|
||||
token_usage: { prompt_tokens: 10, completion_tokens: 4, total_tokens: 14 },
|
||||
schema_name: "optimizedArticleSchema",
|
||||
schema_valid: true,
|
||||
validation_issues: [],
|
||||
business_status: null,
|
||||
duration_ms: 1200,
|
||||
started_at: "2026-07-16T00:00:00.000Z",
|
||||
responded_at: "2026-07-16T00:00:01.000Z",
|
||||
validated_at: "2026-07-16T00:00:01.200Z",
|
||||
failed_at: null,
|
||||
error_type: null,
|
||||
error_summary: null,
|
||||
};
|
||||
await traceRepository.putRun(run);
|
||||
await traceRepository.putCall(call);
|
||||
|
||||
const payloadStore = createLocalTracePayloadStore();
|
||||
await payloadStore.putJson(call.request_object_key!, exactRequestFixture);
|
||||
await payloadStore.putJson(call.response_object_key!, exactResponseFixture);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
process.env.APP_DATA_DIR = originalDataDir;
|
||||
process.env.API_ACCESS_KEY = originalApiKey;
|
||||
process.env.API_AUTH_DISABLED = originalAuthDisabled;
|
||||
process.env.APP_RUNTIME = originalRuntime;
|
||||
rmSync(tempDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("rejects trace reads without API access", async () => {
|
||||
const response = await getLatestTrace(request(null));
|
||||
expect(response.status).toBe(401);
|
||||
expect(response.headers.get("cache-control")).toBe("no-store");
|
||||
});
|
||||
|
||||
it("returns a no-store latest manifest without raw bodies or object keys", async () => {
|
||||
const response = await getLatestTrace(request("test-key"));
|
||||
const body = await response.json() as LlmTraceManifest;
|
||||
|
||||
expect(response.headers.get("cache-control")).toBe("no-store");
|
||||
expect(body.run.job_id).toBe(jobId);
|
||||
expect(body.calls[0]).toMatchObject({
|
||||
call_id: "llmcall_1",
|
||||
request_available: true,
|
||||
response_available: true,
|
||||
});
|
||||
expect(body.calls[0]).not.toHaveProperty("request_object_key");
|
||||
expect(body.calls[0]).not.toHaveProperty("response_object_key");
|
||||
expect(JSON.stringify(body)).not.toContain("messages");
|
||||
expect(JSON.stringify(body)).not.toContain("choices");
|
||||
});
|
||||
|
||||
it("returns the same protected manifest for a specific job", async () => {
|
||||
const response = await getJobTrace(
|
||||
request("test-key"),
|
||||
params({ jobId }),
|
||||
);
|
||||
const body = await response.json() as LlmTraceManifest;
|
||||
|
||||
expect(response.status).toBe(200);
|
||||
expect(response.headers.get("cache-control")).toBe("no-store");
|
||||
expect(body.run.job_id).toBe(jobId);
|
||||
expect(body.calls).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("returns the exact stored request and response through protected routes", async () => {
|
||||
const requestResponse = await getTraceRequest(
|
||||
request("test-key"),
|
||||
params({ jobId, callId: "llmcall_1" }),
|
||||
);
|
||||
const responseResponse = await getTraceResponse(
|
||||
request("test-key"),
|
||||
params({ jobId, callId: "llmcall_1" }),
|
||||
);
|
||||
|
||||
expect(requestResponse.headers.get("cache-control")).toBe("no-store");
|
||||
await expect(requestResponse.json()).resolves.toEqual(exactRequestFixture);
|
||||
await expect(responseResponse.json()).resolves.toEqual(exactResponseFixture);
|
||||
});
|
||||
|
||||
it("returns clear waiting and unavailable states when a response is absent", async () => {
|
||||
const repository = createSqliteTraceRepository();
|
||||
const [existing] = await repository.listCalls(jobId);
|
||||
await repository.putCall({
|
||||
...existing,
|
||||
call_id: "llmcall_waiting",
|
||||
sequence: 2,
|
||||
status: "started",
|
||||
response_object_key: null,
|
||||
});
|
||||
await repository.putCall({
|
||||
...existing,
|
||||
call_id: "llmcall_failed",
|
||||
sequence: 3,
|
||||
status: "failed",
|
||||
response_object_key: null,
|
||||
failed_at: "2026-07-16T00:00:02.000Z",
|
||||
error_type: "provider",
|
||||
error_summary: "Error: timeout",
|
||||
});
|
||||
|
||||
const waiting = await getTraceResponse(
|
||||
request("test-key"),
|
||||
params({ jobId, callId: "llmcall_waiting" }),
|
||||
);
|
||||
const unavailable = await getTraceResponse(
|
||||
request("test-key"),
|
||||
params({ jobId, callId: "llmcall_failed" }),
|
||||
);
|
||||
|
||||
expect(waiting.status).toBe(202);
|
||||
await expect(waiting.json()).resolves.toEqual({ state: "waiting" });
|
||||
expect(unavailable.status).toBe(404);
|
||||
await expect(unavailable.json()).resolves.toEqual({
|
||||
error: "该调用未产生响应",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
function request(apiKey: string | null) {
|
||||
return new Request("http://localhost/api/llm-traces/latest", {
|
||||
headers: apiKey ? { "x-api-key": apiKey } : undefined,
|
||||
});
|
||||
}
|
||||
|
||||
function params<T extends Record<string, string>>(values: T) {
|
||||
return { params: Promise.resolve(values) };
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
import { requireApiAccess } from "../../../../../../../lib/api/auth";
|
||||
import {
|
||||
noStoreResponse,
|
||||
readTracePayload,
|
||||
} from "../../../../../../../lib/llm/trace-http";
|
||||
|
||||
interface RouteContext {
|
||||
params: Promise<{ jobId: string; callId: string }>;
|
||||
}
|
||||
|
||||
export async function GET(request: Request, context: RouteContext) {
|
||||
const access = requireApiAccess(request);
|
||||
if (!access.ok) return noStoreResponse(access.response);
|
||||
|
||||
const { jobId, callId } = await context.params;
|
||||
return readTracePayload({ jobId, callId, kind: "request" });
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
import { requireApiAccess } from "../../../../../../../lib/api/auth";
|
||||
import {
|
||||
noStoreResponse,
|
||||
readTracePayload,
|
||||
} from "../../../../../../../lib/llm/trace-http";
|
||||
|
||||
interface RouteContext {
|
||||
params: Promise<{ jobId: string; callId: string }>;
|
||||
}
|
||||
|
||||
export async function GET(request: Request, context: RouteContext) {
|
||||
const access = requireApiAccess(request);
|
||||
if (!access.ok) return noStoreResponse(access.response);
|
||||
|
||||
const { jobId, callId } = await context.params;
|
||||
return readTracePayload({ jobId, callId, kind: "response" });
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
import { requireApiAccess } from "../../../../../lib/api/auth";
|
||||
import {
|
||||
getTraceManifest,
|
||||
noStoreJson,
|
||||
noStoreResponse,
|
||||
} from "../../../../../lib/llm/trace-http";
|
||||
|
||||
interface RouteContext {
|
||||
params: Promise<{ jobId: string }>;
|
||||
}
|
||||
|
||||
export async function GET(request: Request, context: RouteContext) {
|
||||
const access = requireApiAccess(request);
|
||||
if (!access.ok) return noStoreResponse(access.response);
|
||||
|
||||
const { jobId } = await context.params;
|
||||
const manifest = await getTraceManifest(jobId);
|
||||
return manifest
|
||||
? noStoreJson(manifest)
|
||||
: noStoreJson({ error: "追踪任务不存在" }, 404);
|
||||
}
|
||||
@@ -7,6 +7,13 @@ import { getRepositoryFromRuntime } from "../../../../lib/db/repository";
|
||||
import { optimizationFactCardSchema } from "../../../../lib/domain/validation";
|
||||
import type { LlmAuditSummary } from "../../../../lib/llm/audit";
|
||||
import { LlmValidationError } from "../../../../lib/llm/client";
|
||||
import { getLlmTracePayloadStoreFromRuntime } from "../../../../lib/llm/trace-payload-store";
|
||||
import {
|
||||
createLlmTraceRecorder,
|
||||
createNoopLlmTraceRecorder,
|
||||
safeTraceError,
|
||||
} from "../../../../lib/llm/trace-recorder";
|
||||
import { getLlmTraceRepositoryFromRuntime } from "../../../../lib/llm/trace-repository";
|
||||
import { getExportStoreFromRuntime } from "../../../../lib/workflow/export-store";
|
||||
import { extractCandidateFactCard } from "../../../../lib/workflow/fact-extractor";
|
||||
import {
|
||||
@@ -61,6 +68,7 @@ export async function POST(request: Request) {
|
||||
let jobId: string | undefined;
|
||||
let caseId: string | undefined;
|
||||
let stage: OptimizationStreamStage = "job";
|
||||
let traceRecorder = createNoopLlmTraceRecorder();
|
||||
const llmAuditSummary: LlmAuditSummary[] = [];
|
||||
const processSummary: ProcessSummaryStep[] = [];
|
||||
const requestStartedAt = Date.now();
|
||||
@@ -98,11 +106,31 @@ export async function POST(request: Request) {
|
||||
user_instructions: normalized.articleInput.user_instructions,
|
||||
},
|
||||
});
|
||||
let traceSetupError: unknown;
|
||||
try {
|
||||
traceRecorder = await createLlmTraceRecorder({
|
||||
jobId: job.id,
|
||||
caseId: optimizationCase.id,
|
||||
repository: getLlmTraceRepositoryFromRuntime(),
|
||||
payloadStore: getLlmTracePayloadStoreFromRuntime(),
|
||||
publish: (event) => send(event),
|
||||
});
|
||||
} catch (error) {
|
||||
traceSetupError = error;
|
||||
}
|
||||
send({
|
||||
type: "job_created",
|
||||
job: { id: job.id },
|
||||
case: { id: optimizationCase.id, case_type: "article" },
|
||||
});
|
||||
if (traceSetupError) {
|
||||
send({
|
||||
type: "trace_warning",
|
||||
job_id: job.id,
|
||||
trace_completeness: "incomplete",
|
||||
error_summary: safeTraceError(traceSetupError),
|
||||
});
|
||||
}
|
||||
|
||||
stage = "fact_card";
|
||||
const factCardStartedAt = Date.now();
|
||||
@@ -112,6 +140,7 @@ export async function POST(request: Request) {
|
||||
onAuditSummary: (summary) => {
|
||||
llmAuditSummary.push(summary);
|
||||
},
|
||||
onTraceEvent: traceRecorder.onLlmEvent,
|
||||
})),
|
||||
);
|
||||
processSummary.push(
|
||||
@@ -124,20 +153,24 @@ export async function POST(request: Request) {
|
||||
}),
|
||||
);
|
||||
const savedFactCard = await repository.saveFactCard(job.id, factCard);
|
||||
send({
|
||||
const factCardReadyEvent: OptimizationStreamEvent = {
|
||||
type: "fact_card_ready",
|
||||
job_id: job.id,
|
||||
fact_card: savedFactCard,
|
||||
});
|
||||
};
|
||||
await traceRecorder.onWorkflowEvent(factCardReadyEvent);
|
||||
send(factCardReadyEvent);
|
||||
|
||||
const result = await runStreamingOptimizationWorkflow({
|
||||
jobId: job.id,
|
||||
input: normalized.articleInput,
|
||||
factCard: savedFactCard,
|
||||
onEvent: (event) => {
|
||||
onEvent: async (event) => {
|
||||
stage = stageForEvent(event, stage);
|
||||
await traceRecorder.onWorkflowEvent(event);
|
||||
send(event);
|
||||
},
|
||||
onTraceEvent: traceRecorder.onLlmEvent,
|
||||
onAuditSummary: (summary) => {
|
||||
llmAuditSummary.push(summary);
|
||||
},
|
||||
@@ -180,7 +213,7 @@ export async function POST(request: Request) {
|
||||
error_stage: null,
|
||||
error_summary: null,
|
||||
});
|
||||
send({
|
||||
const finalReadyEvent: OptimizationStreamEvent = {
|
||||
type: "final_ready",
|
||||
job_id: job.id,
|
||||
case: { id: optimizationCase.id, case_type: "article" },
|
||||
@@ -188,7 +221,10 @@ export async function POST(request: Request) {
|
||||
optimized_article: optimizedArticle,
|
||||
qa_report: qaReport,
|
||||
export_paths: exportPaths,
|
||||
});
|
||||
};
|
||||
await traceRecorder.onWorkflowEvent(finalReadyEvent);
|
||||
await traceRecorder.finish({ status: "completed" });
|
||||
send(finalReadyEvent);
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : "优化失败";
|
||||
let failedVersion:
|
||||
@@ -225,14 +261,21 @@ export async function POST(request: Request) {
|
||||
failedVersion = undefined;
|
||||
}
|
||||
}
|
||||
send({
|
||||
const failedEvent: OptimizationStreamEvent = {
|
||||
type: "failed",
|
||||
job_id: jobId,
|
||||
case: caseId ? { id: caseId, case_type: "article" } : undefined,
|
||||
result_version: failedVersion,
|
||||
stage,
|
||||
error: message,
|
||||
};
|
||||
await traceRecorder.onWorkflowEvent(failedEvent);
|
||||
await traceRecorder.finish({
|
||||
status: "failed",
|
||||
errorStage: stage,
|
||||
errorSummary: message,
|
||||
});
|
||||
send(failedEvent);
|
||||
} finally {
|
||||
controller.close();
|
||||
}
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
import { requireApiAccess } from "../../../../lib/api/auth";
|
||||
import {
|
||||
noStoreJson,
|
||||
noStoreResponse,
|
||||
toPublicTraceCall,
|
||||
} from "../../../../lib/llm/trace-http";
|
||||
import { getLlmTraceRepositoryFromRuntime } from "../../../../lib/llm/trace-repository";
|
||||
|
||||
export async function GET(request: Request) {
|
||||
const access = requireApiAccess(request);
|
||||
if (!access.ok) return noStoreResponse(access.response);
|
||||
|
||||
const repository = getLlmTraceRepositoryFromRuntime();
|
||||
const run = await repository.getLatestRun();
|
||||
if (!run) return noStoreJson({ run: null, calls: [] });
|
||||
const calls = await repository.listCalls(run.job_id);
|
||||
return noStoreJson({ run, calls: calls.map(toPublicTraceCall) });
|
||||
}
|
||||
+283
-1
@@ -222,6 +222,278 @@ h3 {
|
||||
font-weight: 800;
|
||||
}
|
||||
|
||||
.architecture-observer {
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.architecture-task-strip {
|
||||
align-items: center;
|
||||
background: #172033;
|
||||
border-radius: 8px;
|
||||
color: #ffffff;
|
||||
display: flex;
|
||||
gap: 1rem;
|
||||
justify-content: space-between;
|
||||
padding: 0.85rem 1rem;
|
||||
}
|
||||
|
||||
.architecture-task-strip > div:first-child {
|
||||
display: grid;
|
||||
gap: 0.2rem;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.architecture-task-strip strong,
|
||||
.architecture-task-strip small {
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
}
|
||||
|
||||
.architecture-task-strip small,
|
||||
.architecture-eyebrow {
|
||||
color: #cbd3df;
|
||||
}
|
||||
|
||||
.architecture-eyebrow {
|
||||
font-size: 0.72rem;
|
||||
font-weight: 800;
|
||||
letter-spacing: 0.08em;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.architecture-task-meta {
|
||||
align-items: center;
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 0.5rem;
|
||||
justify-content: flex-end;
|
||||
}
|
||||
|
||||
.architecture-task-meta > span {
|
||||
background: rgba(255, 255, 255, 0.12);
|
||||
border-radius: 999px;
|
||||
font-size: 0.78rem;
|
||||
padding: 0.35rem 0.6rem;
|
||||
}
|
||||
|
||||
.technical-detail-toggle {
|
||||
align-items: center;
|
||||
background: #ffffff;
|
||||
border-radius: 6px;
|
||||
color: #172033;
|
||||
cursor: pointer;
|
||||
display: flex;
|
||||
gap: 0.45rem;
|
||||
padding: 0.4rem 0.6rem;
|
||||
}
|
||||
|
||||
.technical-detail-toggle input {
|
||||
accent-color: #2f6fdd;
|
||||
height: 1rem;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
width: 1rem;
|
||||
}
|
||||
|
||||
.technical-detail-toggle span {
|
||||
color: inherit;
|
||||
font-size: 0.76rem;
|
||||
}
|
||||
|
||||
.architecture-warning {
|
||||
background: #fff7e6;
|
||||
border: 1px solid #f2c879;
|
||||
border-radius: 8px;
|
||||
color: #78520a;
|
||||
margin: 0;
|
||||
padding: 0.75rem 0.9rem;
|
||||
}
|
||||
|
||||
.architecture-flow-panel {
|
||||
background: #ffffff;
|
||||
border: 1px solid #dce2eb;
|
||||
border-radius: 8px;
|
||||
overflow-x: auto;
|
||||
padding: 1rem;
|
||||
}
|
||||
|
||||
.architecture-flow {
|
||||
align-items: stretch;
|
||||
display: flex;
|
||||
min-width: 62rem;
|
||||
}
|
||||
|
||||
.architecture-flow-step {
|
||||
align-items: center;
|
||||
display: flex;
|
||||
flex: 1 0 0;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.architecture-node {
|
||||
border: 1px solid #cbd3df;
|
||||
border-radius: 8px;
|
||||
display: grid;
|
||||
flex: 1 0 0;
|
||||
gap: 0.3rem;
|
||||
min-height: 8.5rem;
|
||||
min-width: 8.5rem;
|
||||
padding: 0.75rem;
|
||||
}
|
||||
|
||||
.architecture-node > span:not(.architecture-node-index),
|
||||
.architecture-node small {
|
||||
color: #586174;
|
||||
font-size: 0.76rem;
|
||||
}
|
||||
|
||||
.architecture-node-index {
|
||||
align-items: center;
|
||||
background: #eef1f6;
|
||||
border-radius: 999px;
|
||||
display: flex;
|
||||
font-size: 0.72rem;
|
||||
font-weight: 800;
|
||||
height: 1.5rem;
|
||||
justify-content: center;
|
||||
width: 1.5rem;
|
||||
}
|
||||
|
||||
.architecture-node-running {
|
||||
background: #eef5ff;
|
||||
border-color: #2f6fdd;
|
||||
box-shadow: 0 0 0 2px rgba(47, 111, 221, 0.1);
|
||||
}
|
||||
|
||||
.architecture-node-completed {
|
||||
background: #effaf4;
|
||||
border-color: #55a47a;
|
||||
}
|
||||
|
||||
.architecture-node-failed {
|
||||
background: #fff1f1;
|
||||
border-color: #d05858;
|
||||
}
|
||||
|
||||
.architecture-node-skipped {
|
||||
background: #f6f7f9;
|
||||
border-style: dashed;
|
||||
opacity: 0.72;
|
||||
}
|
||||
|
||||
.architecture-arrow {
|
||||
color: #8992a3;
|
||||
flex: 0 0 1.4rem;
|
||||
font-size: 1.1rem;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.architecture-loop-label {
|
||||
color: #586174;
|
||||
font-size: 0.78rem;
|
||||
margin: 0.75rem 0 0;
|
||||
text-align: right;
|
||||
}
|
||||
|
||||
.architecture-workspace {
|
||||
display: grid;
|
||||
gap: 1rem;
|
||||
grid-template-columns: minmax(16rem, 0.7fr) minmax(0, 1.3fr);
|
||||
}
|
||||
|
||||
.llm-call-list,
|
||||
.llm-call-detail {
|
||||
min-height: 26rem;
|
||||
}
|
||||
|
||||
.llm-call-list {
|
||||
align-content: start;
|
||||
display: grid;
|
||||
gap: 0.5rem;
|
||||
}
|
||||
|
||||
.llm-call-list .panel-heading {
|
||||
padding-bottom: 0.5rem;
|
||||
}
|
||||
|
||||
.llm-call-list button {
|
||||
background: #ffffff;
|
||||
border-color: #dce2eb;
|
||||
color: #172033;
|
||||
display: grid;
|
||||
gap: 0.25rem;
|
||||
min-width: 0;
|
||||
padding: 0.7rem;
|
||||
text-align: left;
|
||||
white-space: normal;
|
||||
}
|
||||
|
||||
.llm-call-list button.selected {
|
||||
border-color: #2f6fdd;
|
||||
box-shadow: 0 0 0 2px rgba(47, 111, 221, 0.1);
|
||||
}
|
||||
|
||||
.llm-call-list code,
|
||||
.llm-call-list small {
|
||||
color: #586174;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
.llm-call-detail {
|
||||
display: grid;
|
||||
grid-template-rows: auto auto minmax(18rem, 1fr);
|
||||
}
|
||||
|
||||
.llm-detail-heading,
|
||||
.llm-detail-tabs {
|
||||
align-items: center;
|
||||
display: flex;
|
||||
gap: 0.5rem;
|
||||
}
|
||||
|
||||
.llm-detail-heading {
|
||||
justify-content: space-between;
|
||||
}
|
||||
|
||||
.llm-detail-heading > div {
|
||||
display: grid;
|
||||
gap: 0.25rem;
|
||||
}
|
||||
|
||||
.llm-detail-heading small {
|
||||
color: #586174;
|
||||
}
|
||||
|
||||
.llm-detail-tabs {
|
||||
border-bottom: 1px solid #dce2eb;
|
||||
margin-top: 0.8rem;
|
||||
}
|
||||
|
||||
.llm-detail-tabs button {
|
||||
background: transparent;
|
||||
border: 0;
|
||||
border-bottom: 2px solid transparent;
|
||||
border-radius: 0;
|
||||
color: #586174;
|
||||
}
|
||||
|
||||
.llm-detail-tabs button.active-tab {
|
||||
border-bottom-color: #2f6fdd;
|
||||
color: #172033;
|
||||
}
|
||||
|
||||
.llm-json-view {
|
||||
background: #111827;
|
||||
border-radius: 6px;
|
||||
color: #dbeafe;
|
||||
font-size: 0.78rem;
|
||||
margin: 0.8rem 0 0;
|
||||
overflow: auto;
|
||||
padding: 1rem;
|
||||
white-space: pre-wrap;
|
||||
word-break: break-word;
|
||||
}
|
||||
|
||||
.panel-actions {
|
||||
align-items: center;
|
||||
display: flex;
|
||||
@@ -769,10 +1041,20 @@ h3 {
|
||||
.calibration-metrics,
|
||||
.case-toolbar,
|
||||
.case-detail-grid,
|
||||
.case-meta-grid {
|
||||
.case-meta-grid,
|
||||
.architecture-workspace {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.architecture-task-strip {
|
||||
align-items: stretch;
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
.architecture-task-meta {
|
||||
justify-content: flex-start;
|
||||
}
|
||||
|
||||
.case-row,
|
||||
.case-row-heading {
|
||||
grid-template-columns: 1fr;
|
||||
|
||||
+27
-2
@@ -7,6 +7,7 @@ import {
|
||||
ArticleInputForm,
|
||||
type ArticleInputPayload,
|
||||
} from "../components/article-input-form";
|
||||
import { ArchitectureObserverPanel } from "../components/architecture/architecture-observer-panel";
|
||||
import { FactCardEditor } from "../components/fact-card-editor";
|
||||
import { OptimizedPreview } from "../components/optimized-preview";
|
||||
import { PerformanceCalibrationPanel } from "../components/performance-calibration-panel";
|
||||
@@ -47,7 +48,7 @@ interface TimingSummary {
|
||||
}
|
||||
|
||||
export default function Home() {
|
||||
const [activeTab, setActiveTab] = useState<"geo" | "copy">("geo");
|
||||
const [activeTab, setActiveTab] = useState<"geo" | "copy" | "architecture">("geo");
|
||||
const [input, setInput] = useState(initialInput);
|
||||
const [jobId, setJobId] = useState<string | null>(null);
|
||||
const [factCard, setFactCard] = useState<OptimizationFactCard | null>(null);
|
||||
@@ -61,6 +62,7 @@ export default function Home() {
|
||||
const [elapsedSeconds, setElapsedSeconds] = useState(0);
|
||||
const [lastTiming, setLastTiming] = useState<TimingSummary | null>(null);
|
||||
const [streamActivity, setStreamActivity] = useState("");
|
||||
const [architectureEvents, setArchitectureEvents] = useState<OptimizationStreamEvent[]>([]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!busyAction) return;
|
||||
@@ -84,6 +86,7 @@ export default function Home() {
|
||||
setOptimizedArticle(null);
|
||||
setQaReport(null);
|
||||
setStreamActivity("正在创建任务");
|
||||
setArchitectureEvents([]);
|
||||
|
||||
try {
|
||||
const payload: ArticleInputPayload & {
|
||||
@@ -116,6 +119,9 @@ export default function Home() {
|
||||
}
|
||||
|
||||
function handleStreamEvent(event: OptimizationStreamEvent) {
|
||||
setArchitectureEvents((current) => event.type === "job_created"
|
||||
? [event]
|
||||
: [...current, event]);
|
||||
switch (event.type) {
|
||||
case "job_created":
|
||||
setJobId(event.job.id);
|
||||
@@ -157,6 +163,12 @@ export default function Home() {
|
||||
: "优化完成。",
|
||||
);
|
||||
break;
|
||||
case "llm_call_started":
|
||||
case "llm_call_responded":
|
||||
case "llm_call_validated":
|
||||
case "llm_call_failed":
|
||||
case "trace_warning":
|
||||
break;
|
||||
case "failed":
|
||||
setStreamActivity("优化失败");
|
||||
throw new Error(event.error);
|
||||
@@ -200,6 +212,13 @@ export default function Home() {
|
||||
>
|
||||
普通文案优化
|
||||
</button>
|
||||
<button
|
||||
className={activeTab === "architecture" ? "active-tab" : undefined}
|
||||
onClick={() => setActiveTab("architecture")}
|
||||
type="button"
|
||||
>
|
||||
后台架构
|
||||
</button>
|
||||
</nav>
|
||||
{activeTab === "geo" ? (
|
||||
<>
|
||||
@@ -237,10 +256,16 @@ export default function Home() {
|
||||
/>
|
||||
</div>
|
||||
</>
|
||||
) : (
|
||||
) : activeTab === "copy" ? (
|
||||
<RenweiCopyOptimizerPanel
|
||||
apiAccessKey={apiAccessKey}
|
||||
/>
|
||||
) : (
|
||||
<ArchitectureObserverPanel
|
||||
apiAccessKey={apiAccessKey}
|
||||
currentJobId={jobId}
|
||||
liveEvents={architectureEvents}
|
||||
/>
|
||||
)}
|
||||
</main>
|
||||
);
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { getTracePayload } from "../api-client";
|
||||
|
||||
describe("architecture trace API client", () => {
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
it("loads request bodies only when explicitly requested", async () => {
|
||||
const exactRequest = {
|
||||
model: "deepseek-v4-pro",
|
||||
messages: [{ role: "user", content: "完整原文" }],
|
||||
};
|
||||
const fetchMock = vi.fn(async () => Response.json(exactRequest));
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
|
||||
await expect(
|
||||
getTracePayload("job_1", "llmcall_1", "request", "test-key"),
|
||||
).resolves.toEqual(exactRequest);
|
||||
expect(fetchMock).toHaveBeenCalledWith(
|
||||
"/api/jobs/job_1/llm-trace/llmcall_1/request",
|
||||
expect.objectContaining({
|
||||
credentials: "same-origin",
|
||||
headers: { "x-api-key": "test-key" },
|
||||
cache: "no-store",
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it("surfaces protected API errors without returning partial data", async () => {
|
||||
vi.stubGlobal("fetch", vi.fn(async () => Response.json(
|
||||
{ error: "Unauthorized" },
|
||||
{ status: 401 },
|
||||
)));
|
||||
|
||||
await expect(
|
||||
getTracePayload("job_1", "llmcall_1", "response", "wrong"),
|
||||
).rejects.toThrow("Unauthorized");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,193 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import type {
|
||||
LlmTraceCallPublic,
|
||||
LlmTraceManifest,
|
||||
LlmTraceRun,
|
||||
} from "../../../lib/llm/trace-types";
|
||||
import {
|
||||
applyLiveOptimizationEvent,
|
||||
applyLiveTraceEvent,
|
||||
deriveArchitectureNodes,
|
||||
formatCallLabel,
|
||||
formatNodeStatus,
|
||||
} from "../trace-state";
|
||||
|
||||
const run: LlmTraceRun = {
|
||||
job_id: "job_1",
|
||||
case_id: "case_1",
|
||||
status: "running",
|
||||
current_stage: "qa",
|
||||
trace_completeness: "complete",
|
||||
error_stage: null,
|
||||
error_summary: null,
|
||||
started_at: "2026-07-16T00:00:00.000Z",
|
||||
finished_at: null,
|
||||
updated_at: "2026-07-16T00:00:01.000Z",
|
||||
};
|
||||
|
||||
function call(
|
||||
overrides: Partial<LlmTraceCallPublic> = {},
|
||||
): LlmTraceCallPublic {
|
||||
return {
|
||||
call_id: "llmcall_1",
|
||||
job_id: "job_1",
|
||||
sequence: 1,
|
||||
task: "quality_inspector",
|
||||
workflow_stage: "qa",
|
||||
rewrite_round: 0,
|
||||
provider: "deepseek",
|
||||
model: "deepseek-v4-pro",
|
||||
status: "validated",
|
||||
token_usage: { total_tokens: 14 },
|
||||
schema_name: "llmQaPatchSchema",
|
||||
schema_valid: true,
|
||||
validation_issues: [],
|
||||
business_status: "pass",
|
||||
duration_ms: 1200,
|
||||
started_at: "2026-07-16T00:00:00.000Z",
|
||||
responded_at: "2026-07-16T00:00:01.000Z",
|
||||
validated_at: "2026-07-16T00:00:01.200Z",
|
||||
failed_at: null,
|
||||
error_type: null,
|
||||
error_summary: null,
|
||||
request_available: true,
|
||||
response_available: true,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe("architecture trace state", () => {
|
||||
it("formats every architecture node and call label in Chinese", () => {
|
||||
expect(formatNodeStatus("running")).toBe("运行中");
|
||||
expect(formatNodeStatus("failed")).toBe("失败");
|
||||
expect(formatCallLabel(call({
|
||||
sequence: 4,
|
||||
task: "targeted_rewriter",
|
||||
rewrite_round: 1,
|
||||
duration_ms: 22600,
|
||||
}))).toBe("4 · 定向修复第 1 轮 · 22.6 秒");
|
||||
});
|
||||
|
||||
it("distinguishes QA business failure from schema failure", () => {
|
||||
const nodes = deriveArchitectureNodes(run, [
|
||||
call({ schema_valid: true, business_status: "fail" }),
|
||||
]);
|
||||
expect(nodes.qa).toMatchObject({
|
||||
status: "completed",
|
||||
detail: "检查完成,需要修复",
|
||||
});
|
||||
|
||||
const failedNodes = deriveArchitectureNodes(run, [
|
||||
call({ schema_valid: false, status: "failed" }),
|
||||
]);
|
||||
expect(failedNodes.qa.status).toBe("failed");
|
||||
});
|
||||
|
||||
it("merges repeated live events by call id and preserves sequence order", () => {
|
||||
const initial: LlmTraceManifest = { run, calls: [] };
|
||||
const started = applyLiveTraceEvent(initial, {
|
||||
type: "llm_call_started",
|
||||
job_id: "job_1",
|
||||
call_id: "llmcall_2",
|
||||
sequence: 2,
|
||||
task: "targeted_rewriter",
|
||||
workflow_stage: "rewrite",
|
||||
rewrite_round: 1,
|
||||
provider: "deepseek",
|
||||
model: "deepseek-v4-pro",
|
||||
started_at: "2026-07-16T00:00:02.000Z",
|
||||
request_available: true,
|
||||
});
|
||||
const responded = applyLiveTraceEvent(started, {
|
||||
type: "llm_call_responded",
|
||||
job_id: "job_1",
|
||||
call_id: "llmcall_2",
|
||||
duration_ms: 1500,
|
||||
token_usage: { total_tokens: 20 },
|
||||
responded_at: "2026-07-16T00:00:03.500Z",
|
||||
response_available: true,
|
||||
});
|
||||
const duplicate = applyLiveTraceEvent(responded, {
|
||||
type: "llm_call_responded",
|
||||
job_id: "job_1",
|
||||
call_id: "llmcall_2",
|
||||
duration_ms: 1500,
|
||||
token_usage: { total_tokens: 20 },
|
||||
responded_at: "2026-07-16T00:00:03.500Z",
|
||||
response_available: true,
|
||||
});
|
||||
|
||||
expect(duplicate.calls).toHaveLength(1);
|
||||
expect(duplicate.calls[0]).toMatchObject({
|
||||
call_id: "llmcall_2",
|
||||
status: "responded",
|
||||
response_available: true,
|
||||
token_usage: { total_tokens: 20 },
|
||||
});
|
||||
expect(duplicate.run.current_stage).toBe("rewrite");
|
||||
});
|
||||
|
||||
it("marks only trace completeness when a trace warning arrives", () => {
|
||||
const manifest = applyLiveTraceEvent(
|
||||
{ run, calls: [call()] },
|
||||
{
|
||||
type: "trace_warning",
|
||||
job_id: "job_1",
|
||||
trace_completeness: "incomplete",
|
||||
error_summary: "Error: payload unavailable",
|
||||
},
|
||||
);
|
||||
|
||||
expect(manifest.run).toMatchObject({
|
||||
status: "running",
|
||||
trace_completeness: "incomplete",
|
||||
});
|
||||
expect(manifest.calls[0].status).toBe("validated");
|
||||
});
|
||||
|
||||
it("uses workflow events to finish the run and retain QA business status", () => {
|
||||
const withQa = applyLiveOptimizationEvent(
|
||||
{ run, calls: [call({ business_status: null })] },
|
||||
{
|
||||
type: "qa_ready",
|
||||
job_id: "job_1",
|
||||
qa_report: { overall_status: "fail", checks: [] } as never,
|
||||
},
|
||||
);
|
||||
const completed = applyLiveOptimizationEvent(withQa, {
|
||||
type: "final_ready",
|
||||
job_id: "job_1",
|
||||
optimized_article: {} as never,
|
||||
qa_report: { overall_status: "fail", checks: [] } as never,
|
||||
export_paths: {},
|
||||
});
|
||||
|
||||
expect(withQa.calls[0].business_status).toBe("fail");
|
||||
expect(completed.run).toMatchObject({
|
||||
status: "completed",
|
||||
current_stage: "final",
|
||||
});
|
||||
expect(deriveArchitectureNodes(completed.run, completed.calls).final.status)
|
||||
.toBe("completed");
|
||||
});
|
||||
|
||||
it("uses workflow failure events to mark the matching architecture stage", () => {
|
||||
const failed = applyLiveOptimizationEvent(
|
||||
{ run: { ...run, current_stage: "draft" }, calls: [] },
|
||||
{
|
||||
type: "failed",
|
||||
job_id: "job_1",
|
||||
stage: "qa",
|
||||
error: "质量检查失败",
|
||||
},
|
||||
);
|
||||
|
||||
expect(failed.run).toMatchObject({
|
||||
status: "failed",
|
||||
current_stage: "qa",
|
||||
error_stage: "qa",
|
||||
error_summary: "质量检查失败",
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,44 @@
|
||||
import type { LlmTraceManifest } from "../../lib/llm/trace-types";
|
||||
|
||||
function traceHeaders(apiAccessKey: string): Record<string, string> {
|
||||
return apiAccessKey ? { "x-api-key": apiAccessKey } : {};
|
||||
}
|
||||
|
||||
async function traceFetch<T>(path: string, apiAccessKey: string): Promise<T> {
|
||||
const response = await fetch(path, {
|
||||
credentials: "same-origin",
|
||||
headers: traceHeaders(apiAccessKey),
|
||||
cache: "no-store",
|
||||
});
|
||||
const body = await response.json().catch(() => ({})) as T & {
|
||||
error?: string;
|
||||
};
|
||||
if (!response.ok) throw new Error(body.error ?? "读取后台追踪失败");
|
||||
return body;
|
||||
}
|
||||
|
||||
export function getLatestTrace(apiAccessKey: string) {
|
||||
return traceFetch<LlmTraceManifest | { run: null; calls: [] }>(
|
||||
"/api/llm-traces/latest",
|
||||
apiAccessKey,
|
||||
);
|
||||
}
|
||||
|
||||
export function getJobTrace(jobId: string, apiAccessKey: string) {
|
||||
return traceFetch<LlmTraceManifest>(
|
||||
`/api/jobs/${encodeURIComponent(jobId)}/llm-trace`,
|
||||
apiAccessKey,
|
||||
);
|
||||
}
|
||||
|
||||
export function getTracePayload(
|
||||
jobId: string,
|
||||
callId: string,
|
||||
kind: "request" | "response",
|
||||
apiAccessKey: string,
|
||||
) {
|
||||
return traceFetch<unknown>(
|
||||
`/api/jobs/${encodeURIComponent(jobId)}/llm-trace/${encodeURIComponent(callId)}/${kind}`,
|
||||
apiAccessKey,
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
import {
|
||||
formatNodeStatus,
|
||||
type ArchitectureNodeView,
|
||||
} from "./trace-state";
|
||||
|
||||
export function ArchitectureFlow({
|
||||
nodes,
|
||||
}: {
|
||||
nodes: Record<ArchitectureNodeView["id"], ArchitectureNodeView>;
|
||||
}) {
|
||||
const order: ArchitectureNodeView["id"][] = [
|
||||
"input",
|
||||
"fact_card",
|
||||
"draft",
|
||||
"qa",
|
||||
"rewrite",
|
||||
"final",
|
||||
];
|
||||
|
||||
return (
|
||||
<section className="architecture-flow-panel" aria-label="文章优化后台架构">
|
||||
<div className="architecture-flow">
|
||||
{order.map((id, index) => {
|
||||
const node = nodes[id];
|
||||
return (
|
||||
<div className="architecture-flow-step" key={node.id}>
|
||||
<article
|
||||
className={`architecture-node architecture-node-${node.status}`}
|
||||
data-node={node.id}
|
||||
>
|
||||
<span className="architecture-node-index">{index + 1}</span>
|
||||
<strong>{node.label}</strong>
|
||||
<span>{formatNodeStatus(node.status)}</span>
|
||||
<small>{node.detail}</small>
|
||||
</article>
|
||||
{index < order.length - 1 && (
|
||||
<span aria-hidden="true" className="architecture-arrow">→</span>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
<p className="architecture-loop-label">
|
||||
质量检查未通过时,定向修复后返回复检,最多两轮。
|
||||
</p>
|
||||
</section>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,191 @@
|
||||
"use client";
|
||||
|
||||
import { useEffect, useMemo, useState } from "react";
|
||||
|
||||
import type {
|
||||
LlmTraceManifest,
|
||||
} from "../../lib/llm/trace-types";
|
||||
import type { OptimizationStreamEvent } from "../../lib/workflow/stream-events";
|
||||
import { getJobTrace, getLatestTrace } from "./api-client";
|
||||
import { ArchitectureFlow } from "./architecture-flow";
|
||||
import { LlmCallDetail } from "./llm-call-detail";
|
||||
import {
|
||||
applyLiveOptimizationEvent,
|
||||
deriveArchitectureNodes,
|
||||
formatCallLabel,
|
||||
formatNodeStatus,
|
||||
} from "./trace-state";
|
||||
|
||||
interface ArchitectureObserverPanelProps {
|
||||
apiAccessKey: string;
|
||||
currentJobId: string | null;
|
||||
liveEvents: OptimizationStreamEvent[];
|
||||
}
|
||||
|
||||
interface ManifestLoadState {
|
||||
key: string;
|
||||
status: "loaded" | "empty" | "error";
|
||||
manifest: LlmTraceManifest | null;
|
||||
error: string;
|
||||
}
|
||||
|
||||
function loadError(error: unknown) {
|
||||
return error instanceof Error ? error.message : "后台追踪加载失败";
|
||||
}
|
||||
|
||||
function runStatusLabel(status: LlmTraceManifest["run"]["status"]) {
|
||||
if (status === "running") return "运行中";
|
||||
if (status === "completed") return "已完成";
|
||||
if (status === "interrupted") return "已中断";
|
||||
return "失败";
|
||||
}
|
||||
|
||||
export function ArchitectureObserverPanel({
|
||||
apiAccessKey,
|
||||
currentJobId,
|
||||
liveEvents,
|
||||
}: ArchitectureObserverPanelProps) {
|
||||
const [loadState, setLoadState] = useState<ManifestLoadState | null>(null);
|
||||
const [selectedCallId, setSelectedCallId] = useState<string | null>(null);
|
||||
const [technicalDetailsEnabled, setTechnicalDetailsEnabled] = useState(false);
|
||||
const loadKey = `${currentJobId ?? "latest"}:${apiAccessKey}`;
|
||||
|
||||
useEffect(() => {
|
||||
let current = true;
|
||||
const request = currentJobId
|
||||
? getJobTrace(currentJobId, apiAccessKey)
|
||||
: getLatestTrace(apiAccessKey);
|
||||
request
|
||||
.then((result) => {
|
||||
if (!current) return;
|
||||
if (result.run === null) {
|
||||
setLoadState({
|
||||
key: loadKey,
|
||||
status: "empty",
|
||||
manifest: null,
|
||||
error: "",
|
||||
});
|
||||
return;
|
||||
}
|
||||
setLoadState({
|
||||
key: loadKey,
|
||||
status: "loaded",
|
||||
manifest: result,
|
||||
error: "",
|
||||
});
|
||||
})
|
||||
.catch((error) => {
|
||||
if (!current) return;
|
||||
setLoadState({
|
||||
key: loadKey,
|
||||
status: "error",
|
||||
manifest: null,
|
||||
error: loadError(error),
|
||||
});
|
||||
});
|
||||
return () => {
|
||||
current = false;
|
||||
};
|
||||
}, [apiAccessKey, currentJobId, loadKey]);
|
||||
|
||||
const manifest = useMemo(() => {
|
||||
if (loadState?.key !== loadKey || !loadState.manifest) return null;
|
||||
return liveEvents.reduce(applyLiveOptimizationEvent, loadState.manifest);
|
||||
}, [liveEvents, loadKey, loadState]);
|
||||
|
||||
if (loadState?.key !== loadKey) {
|
||||
return <section className="panel empty-panel">正在读取后台架构…</section>;
|
||||
}
|
||||
if (loadState.status === "error") {
|
||||
return (
|
||||
<section className="panel empty-panel">
|
||||
<strong>无法读取后台追踪</strong>
|
||||
<p>{loadState.error}</p>
|
||||
</section>
|
||||
);
|
||||
}
|
||||
if (loadState.status === "empty" || !manifest) {
|
||||
return (
|
||||
<section className="panel empty-panel">
|
||||
暂无后台追踪。请先在“GEO 文章优化”标签运行一次优化。
|
||||
</section>
|
||||
);
|
||||
}
|
||||
|
||||
const nodes = deriveArchitectureNodes(manifest.run, manifest.calls);
|
||||
const selectedCall = manifest.calls.find(
|
||||
(call) => call.call_id === selectedCallId,
|
||||
) ?? manifest.calls[0] ?? null;
|
||||
const latestCall = manifest.calls.at(-1) ?? null;
|
||||
const isCurrentTask = currentJobId === manifest.run.job_id;
|
||||
|
||||
return (
|
||||
<section className="architecture-observer stack">
|
||||
<header className="architecture-task-strip">
|
||||
<div>
|
||||
<span className="architecture-eyebrow">
|
||||
{isCurrentTask ? "当前任务" : "最近一次任务"}
|
||||
</span>
|
||||
<strong>{manifest.run.job_id}</strong>
|
||||
<small>
|
||||
{latestCall
|
||||
? `${latestCall.provider} · ${latestCall.model}`
|
||||
: "尚未发起 LLM 调用"}
|
||||
</small>
|
||||
</div>
|
||||
<div className="architecture-task-meta">
|
||||
<span>{runStatusLabel(manifest.run.status)}</span>
|
||||
<span>{formatNodeStatus(nodes[manifest.run.current_stage === "unknown" ? "input" : manifest.run.current_stage].status)}</span>
|
||||
<label className="technical-detail-toggle">
|
||||
<input
|
||||
checked={technicalDetailsEnabled}
|
||||
onChange={(event) => setTechnicalDetailsEnabled(event.target.checked)}
|
||||
type="checkbox"
|
||||
/>
|
||||
<span>技术详情</span>
|
||||
</label>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
{manifest.run.trace_completeness === "incomplete" && (
|
||||
<p className="architecture-warning" role="status">
|
||||
追踪记录不完整:已保存的数据仍可查看,但部分阶段或正文可能缺失。
|
||||
</p>
|
||||
)}
|
||||
|
||||
<ArchitectureFlow nodes={nodes} />
|
||||
|
||||
<div className="architecture-workspace">
|
||||
<aside className="panel llm-call-list">
|
||||
<div className="panel-heading">LLM 调用轨迹</div>
|
||||
{manifest.calls.length === 0 ? (
|
||||
<p className="empty-panel">等待第一次 LLM 调用。</p>
|
||||
) : manifest.calls.map((call) => (
|
||||
<button
|
||||
className={selectedCall?.call_id === call.call_id ? "selected" : undefined}
|
||||
key={call.call_id}
|
||||
onClick={() => setSelectedCallId(call.call_id)}
|
||||
type="button"
|
||||
>
|
||||
<strong>{formatCallLabel(call)}</strong>
|
||||
<code>{call.task}</code>
|
||||
<small>{call.schema_valid === false ? "Schema 失败" : call.status}</small>
|
||||
</button>
|
||||
))}
|
||||
</aside>
|
||||
|
||||
{selectedCall ? (
|
||||
<LlmCallDetail
|
||||
apiAccessKey={apiAccessKey}
|
||||
call={selectedCall}
|
||||
jobId={manifest.run.job_id}
|
||||
key={selectedCall.call_id}
|
||||
technicalDetailsEnabled={technicalDetailsEnabled}
|
||||
/>
|
||||
) : (
|
||||
<section className="panel empty-panel">请选择一次 LLM 调用。</section>
|
||||
)}
|
||||
</div>
|
||||
</section>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
"use client";
|
||||
|
||||
import { useEffect, useState } from "react";
|
||||
|
||||
import type { LlmTraceCallPublic } from "../../lib/llm/trace-types";
|
||||
import { getTracePayload } from "./api-client";
|
||||
|
||||
type DetailTab = "request" | "response" | "validation";
|
||||
|
||||
interface LoadedPayload {
|
||||
key: string;
|
||||
value: unknown;
|
||||
}
|
||||
|
||||
function detailTabLabel(tab: DetailTab) {
|
||||
if (tab === "request") return "请求";
|
||||
if (tab === "response") return "响应";
|
||||
return "校验";
|
||||
}
|
||||
|
||||
function errorMessage(error: unknown) {
|
||||
return error instanceof Error ? error.message : "追踪正文加载失败";
|
||||
}
|
||||
|
||||
function validationView(call: LlmTraceCallPublic) {
|
||||
return {
|
||||
schema_name: call.schema_name,
|
||||
schema_valid: call.schema_valid,
|
||||
validation_issues: call.validation_issues,
|
||||
business_status: call.business_status,
|
||||
error_type: call.error_type,
|
||||
error_summary: call.error_summary,
|
||||
duration_ms: call.duration_ms,
|
||||
token_usage: call.token_usage,
|
||||
};
|
||||
}
|
||||
|
||||
export function LlmCallDetail({
|
||||
jobId,
|
||||
call,
|
||||
apiAccessKey,
|
||||
technicalDetailsEnabled,
|
||||
}: {
|
||||
jobId: string;
|
||||
call: LlmTraceCallPublic;
|
||||
apiAccessKey: string;
|
||||
technicalDetailsEnabled: boolean;
|
||||
}) {
|
||||
const [tab, setTab] = useState<DetailTab>("validation");
|
||||
const [loadedPayload, setLoadedPayload] = useState<LoadedPayload | null>(null);
|
||||
const payloadKey = `${call.call_id}:${tab}`;
|
||||
|
||||
useEffect(() => {
|
||||
if (!technicalDetailsEnabled || tab === "validation") return;
|
||||
let current = true;
|
||||
getTracePayload(jobId, call.call_id, tab, apiAccessKey)
|
||||
.then((value) => {
|
||||
if (current) setLoadedPayload({ key: payloadKey, value });
|
||||
})
|
||||
.catch((error) => {
|
||||
if (current) {
|
||||
setLoadedPayload({
|
||||
key: payloadKey,
|
||||
value: { error: errorMessage(error) },
|
||||
});
|
||||
}
|
||||
});
|
||||
return () => {
|
||||
current = false;
|
||||
};
|
||||
}, [apiAccessKey, call.call_id, jobId, payloadKey, tab, technicalDetailsEnabled]);
|
||||
|
||||
const displayValue = tab === "validation"
|
||||
? validationView(call)
|
||||
: !technicalDetailsEnabled
|
||||
? { state: "locked", message: "开启技术详情后按需读取完整正文。" }
|
||||
: loadedPayload?.key === payloadKey
|
||||
? loadedPayload.value
|
||||
: { state: "loading", message: "正在读取完整 JSON…" };
|
||||
|
||||
return (
|
||||
<section className="panel llm-call-detail" aria-live="polite">
|
||||
<div className="llm-detail-heading">
|
||||
<div>
|
||||
<strong>{call.task}</strong>
|
||||
<small>{call.provider} · {call.model}</small>
|
||||
</div>
|
||||
<span className={`status-pill ${call.status === "failed" ? "fail" : "pass"}`}>
|
||||
{call.status}
|
||||
</span>
|
||||
</div>
|
||||
<div className="llm-detail-tabs" role="tablist" aria-label="LLM 调用详情">
|
||||
{(["request", "response", "validation"] as const).map((value) => (
|
||||
<button
|
||||
aria-selected={tab === value}
|
||||
className={tab === value ? "active-tab" : undefined}
|
||||
key={value}
|
||||
onClick={() => setTab(value)}
|
||||
role="tab"
|
||||
type="button"
|
||||
>
|
||||
{detailTabLabel(value)}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
<pre className="llm-json-view">{JSON.stringify(displayValue, null, 2)}</pre>
|
||||
</section>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,368 @@
|
||||
import type {
|
||||
LlmTraceCallPublic,
|
||||
LlmTraceManifest,
|
||||
LlmTraceRun,
|
||||
LlmTraceStreamEvent,
|
||||
LlmTraceWorkflowStage,
|
||||
} from "../../lib/llm/trace-types";
|
||||
import type { OptimizationStreamEvent } from "../../lib/workflow/stream-events";
|
||||
|
||||
export type ArchitectureNodeStatus =
|
||||
| "waiting"
|
||||
| "running"
|
||||
| "completed"
|
||||
| "failed";
|
||||
|
||||
export interface ArchitectureNodeView {
|
||||
id: "input" | "fact_card" | "draft" | "qa" | "rewrite" | "final";
|
||||
label: string;
|
||||
status: ArchitectureNodeStatus;
|
||||
detail: string;
|
||||
}
|
||||
|
||||
export function formatNodeStatus(status: ArchitectureNodeStatus) {
|
||||
return {
|
||||
waiting: "等待中",
|
||||
running: "运行中",
|
||||
completed: "已完成",
|
||||
failed: "失败",
|
||||
}[status];
|
||||
}
|
||||
|
||||
export function formatTaskName(call: LlmTraceCallPublic) {
|
||||
if (call.task === "fact_extractor") return "事实提取";
|
||||
if (call.task === "article_optimizer") return "生成优化稿";
|
||||
if (call.task === "quality_inspector") {
|
||||
return call.rewrite_round && call.rewrite_round > 0
|
||||
? `质量复检第 ${call.rewrite_round} 轮`
|
||||
: "质量检查";
|
||||
}
|
||||
if (call.task === "targeted_rewriter") {
|
||||
return `定向修复第 ${call.rewrite_round ?? 1} 轮`;
|
||||
}
|
||||
if (call.task === "renwei_copy_optimizer") return "普通文案优化";
|
||||
return "未知调用";
|
||||
}
|
||||
|
||||
export function formatCallLabel(call: LlmTraceCallPublic) {
|
||||
const duration = call.duration_ms == null
|
||||
? "进行中"
|
||||
: `${(call.duration_ms / 1000).toFixed(1)} 秒`;
|
||||
return `${call.sequence} · ${formatTaskName(call)} · ${duration}`;
|
||||
}
|
||||
|
||||
const nodeDefinitions: Array<Pick<ArchitectureNodeView, "id" | "label">> = [
|
||||
{ id: "input", label: "输入归一化" },
|
||||
{ id: "fact_card", label: "事实提取" },
|
||||
{ id: "draft", label: "生成草稿" },
|
||||
{ id: "qa", label: "质量检查" },
|
||||
{ id: "rewrite", label: "定向修复" },
|
||||
{ id: "final", label: "保存与导出" },
|
||||
];
|
||||
|
||||
function waitingNodes() {
|
||||
return Object.fromEntries(nodeDefinitions.map((node) => [
|
||||
node.id,
|
||||
{ ...node, status: "waiting", detail: "等待执行" },
|
||||
])) as Record<ArchitectureNodeView["id"], ArchitectureNodeView>;
|
||||
}
|
||||
|
||||
function updateCall(
|
||||
calls: LlmTraceCallPublic[],
|
||||
callId: string,
|
||||
update: (call: LlmTraceCallPublic) => LlmTraceCallPublic,
|
||||
) {
|
||||
return calls
|
||||
.map((call) => call.call_id === callId ? update(call) : call)
|
||||
.sort((left, right) => left.sequence - right.sequence);
|
||||
}
|
||||
|
||||
export function applyLiveTraceEvent(
|
||||
state: LlmTraceManifest,
|
||||
event: LlmTraceStreamEvent,
|
||||
): LlmTraceManifest {
|
||||
if (event.job_id !== state.run.job_id) return state;
|
||||
|
||||
if (event.type === "trace_warning") {
|
||||
return {
|
||||
...state,
|
||||
run: {
|
||||
...state.run,
|
||||
trace_completeness: "incomplete",
|
||||
updated_at: new Date().toISOString(),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
if (event.type === "llm_call_started") {
|
||||
const call: LlmTraceCallPublic = {
|
||||
call_id: event.call_id,
|
||||
job_id: event.job_id,
|
||||
sequence: event.sequence,
|
||||
task: event.task,
|
||||
workflow_stage: event.workflow_stage,
|
||||
rewrite_round: event.rewrite_round,
|
||||
provider: event.provider,
|
||||
model: event.model,
|
||||
status: "started",
|
||||
token_usage: null,
|
||||
schema_name: null,
|
||||
schema_valid: null,
|
||||
validation_issues: [],
|
||||
business_status: null,
|
||||
duration_ms: null,
|
||||
started_at: event.started_at,
|
||||
responded_at: null,
|
||||
validated_at: null,
|
||||
failed_at: null,
|
||||
error_type: null,
|
||||
error_summary: null,
|
||||
request_available: event.request_available,
|
||||
response_available: false,
|
||||
};
|
||||
const withoutExisting = state.calls.filter(
|
||||
(existing) => existing.call_id !== event.call_id,
|
||||
);
|
||||
return {
|
||||
run: {
|
||||
...state.run,
|
||||
current_stage: event.workflow_stage,
|
||||
updated_at: event.started_at,
|
||||
},
|
||||
calls: [...withoutExisting, call].sort(
|
||||
(left, right) => left.sequence - right.sequence,
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
if (event.type === "llm_call_responded") {
|
||||
return {
|
||||
...state,
|
||||
run: { ...state.run, updated_at: event.responded_at },
|
||||
calls: updateCall(state.calls, event.call_id, (call) => ({
|
||||
...call,
|
||||
status: "responded",
|
||||
duration_ms: event.duration_ms,
|
||||
token_usage: event.token_usage,
|
||||
responded_at: event.responded_at,
|
||||
response_available: event.response_available,
|
||||
})),
|
||||
};
|
||||
}
|
||||
|
||||
if (event.type === "llm_call_validated") {
|
||||
return {
|
||||
...state,
|
||||
run: { ...state.run, updated_at: event.validated_at },
|
||||
calls: updateCall(state.calls, event.call_id, (call) => ({
|
||||
...call,
|
||||
status: "validated",
|
||||
schema_name: event.schema_name,
|
||||
schema_valid: event.schema_valid,
|
||||
validation_issues: event.validation_issues,
|
||||
validated_at: event.validated_at,
|
||||
})),
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
...state,
|
||||
run: {
|
||||
...state.run,
|
||||
status: "failed",
|
||||
error_stage: state.run.current_stage,
|
||||
error_summary: event.error_summary,
|
||||
updated_at: event.failed_at,
|
||||
},
|
||||
calls: updateCall(state.calls, event.call_id, (call) => ({
|
||||
...call,
|
||||
status: "failed",
|
||||
failed_at: event.failed_at,
|
||||
error_type: event.error_type,
|
||||
error_summary: event.error_summary,
|
||||
})),
|
||||
};
|
||||
}
|
||||
|
||||
function isTraceEvent(
|
||||
event: OptimizationStreamEvent,
|
||||
): event is LlmTraceStreamEvent {
|
||||
return event.type === "llm_call_started"
|
||||
|| event.type === "llm_call_responded"
|
||||
|| event.type === "llm_call_validated"
|
||||
|| event.type === "llm_call_failed"
|
||||
|| event.type === "trace_warning";
|
||||
}
|
||||
|
||||
function workflowStageForEvent(
|
||||
event: OptimizationStreamEvent,
|
||||
): LlmTraceWorkflowStage | null {
|
||||
if (event.type === "fact_card_ready") return "fact_card";
|
||||
if (event.type === "draft_started" || event.type === "draft_ready") return "draft";
|
||||
if (event.type === "qa_started" || event.type === "qa_ready") return "qa";
|
||||
if (event.type === "rewrite_started" || event.type === "rewrite_ready") return "rewrite";
|
||||
if (event.type === "final_ready") return "final";
|
||||
if (event.type === "failed") {
|
||||
return event.stage === "job" ? "input" : event.stage;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export function applyLiveOptimizationEvent(
|
||||
state: LlmTraceManifest,
|
||||
event: OptimizationStreamEvent,
|
||||
): LlmTraceManifest {
|
||||
if (isTraceEvent(event)) return applyLiveTraceEvent(state, event);
|
||||
if (event.type === "job_created") return state;
|
||||
if (event.job_id !== state.run.job_id) return state;
|
||||
|
||||
const now = new Date().toISOString();
|
||||
if (event.type === "qa_ready") {
|
||||
const qaCalls = state.calls.filter((call) => call.task === "quality_inspector");
|
||||
const latestQaCall = qaCalls.at(-1);
|
||||
return {
|
||||
run: { ...state.run, current_stage: "qa", updated_at: now },
|
||||
calls: latestQaCall
|
||||
? updateCall(state.calls, latestQaCall.call_id, (call) => ({
|
||||
...call,
|
||||
business_status: event.qa_report.overall_status,
|
||||
}))
|
||||
: state.calls,
|
||||
};
|
||||
}
|
||||
|
||||
const stage = workflowStageForEvent(event);
|
||||
if (!stage) return state;
|
||||
if (event.type === "final_ready") {
|
||||
return {
|
||||
...state,
|
||||
run: {
|
||||
...state.run,
|
||||
status: "completed",
|
||||
current_stage: "final",
|
||||
finished_at: now,
|
||||
updated_at: now,
|
||||
},
|
||||
};
|
||||
}
|
||||
if (event.type === "failed") {
|
||||
return {
|
||||
...state,
|
||||
run: {
|
||||
...state.run,
|
||||
status: "failed",
|
||||
current_stage: stage,
|
||||
error_stage: event.stage,
|
||||
error_summary: event.error,
|
||||
finished_at: now,
|
||||
updated_at: now,
|
||||
},
|
||||
};
|
||||
}
|
||||
return {
|
||||
...state,
|
||||
run: { ...state.run, current_stage: stage, updated_at: now },
|
||||
};
|
||||
}
|
||||
|
||||
function callDetail(call: LlmTraceCallPublic) {
|
||||
if (call.status === "failed" || call.schema_valid === false) {
|
||||
if (call.error_type === "provider") return "模型服务调用失败";
|
||||
if (call.error_type === "json_parse") return "响应 JSON 解析失败";
|
||||
return "Schema 校验失败";
|
||||
}
|
||||
if (call.status === "started" || call.status === "responded") {
|
||||
return "LLM 调用进行中";
|
||||
}
|
||||
if (call.task === "quality_inspector" && call.business_status === "fail") {
|
||||
return "检查完成,需要修复";
|
||||
}
|
||||
if (call.task === "quality_inspector" && call.business_status === "warn") {
|
||||
return "检查完成,存在警告";
|
||||
}
|
||||
return "LLM 调用完成";
|
||||
}
|
||||
|
||||
function statusForCall(call: LlmTraceCallPublic): ArchitectureNodeStatus {
|
||||
if (call.status === "failed" || call.schema_valid === false) return "failed";
|
||||
if (call.status === "started" || call.status === "responded") return "running";
|
||||
return "completed";
|
||||
}
|
||||
|
||||
function isNodeStage(
|
||||
stage: LlmTraceWorkflowStage,
|
||||
): stage is ArchitectureNodeView["id"] {
|
||||
return nodeDefinitions.some((node) => node.id === stage);
|
||||
}
|
||||
|
||||
export function deriveArchitectureNodes(
|
||||
run: LlmTraceRun | null,
|
||||
calls: LlmTraceCallPublic[],
|
||||
): Record<ArchitectureNodeView["id"], ArchitectureNodeView> {
|
||||
const nodes = waitingNodes();
|
||||
if (!run) return nodes;
|
||||
|
||||
nodes.input = {
|
||||
...nodes.input,
|
||||
status: "completed",
|
||||
detail: "输入已规范化",
|
||||
};
|
||||
|
||||
const latestByStage = new Map<LlmTraceWorkflowStage, LlmTraceCallPublic>();
|
||||
for (const call of [...calls].sort((left, right) => left.sequence - right.sequence)) {
|
||||
latestByStage.set(call.workflow_stage, call);
|
||||
}
|
||||
for (const [stage, call] of latestByStage) {
|
||||
if (!isNodeStage(stage)) continue;
|
||||
nodes[stage] = {
|
||||
...nodes[stage],
|
||||
status: statusForCall(call),
|
||||
detail: callDetail(call),
|
||||
};
|
||||
}
|
||||
|
||||
if (isNodeStage(run.current_stage) && nodes[run.current_stage].status === "waiting") {
|
||||
nodes[run.current_stage] = {
|
||||
...nodes[run.current_stage],
|
||||
status: run.status === "failed" ? "failed" : "running",
|
||||
detail: run.status === "failed" ? "任务在此阶段失败" : "后台正在执行",
|
||||
};
|
||||
}
|
||||
|
||||
const progression: ArchitectureNodeView["id"][] = [
|
||||
"input",
|
||||
"fact_card",
|
||||
"draft",
|
||||
"qa",
|
||||
"rewrite",
|
||||
"final",
|
||||
];
|
||||
const currentIndex = progression.indexOf(
|
||||
run.current_stage as ArchitectureNodeView["id"],
|
||||
);
|
||||
if (currentIndex > 0) {
|
||||
for (const stage of progression.slice(0, currentIndex)) {
|
||||
if (stage === "rewrite" && !latestByStage.has("rewrite")) continue;
|
||||
if (nodes[stage].status === "waiting") {
|
||||
nodes[stage] = {
|
||||
...nodes[stage],
|
||||
status: "completed",
|
||||
detail: "阶段已完成",
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (run.status === "completed") {
|
||||
nodes.final = {
|
||||
...nodes.final,
|
||||
status: "completed",
|
||||
detail: "终稿与导出已保存",
|
||||
};
|
||||
if (nodes.rewrite.status === "waiting") {
|
||||
nodes.rewrite.detail = "本次未触发修复";
|
||||
}
|
||||
}
|
||||
|
||||
return nodes;
|
||||
}
|
||||
@@ -52,6 +52,8 @@ describe("sqlite repositories", () => {
|
||||
"calibration_events",
|
||||
"case_inputs",
|
||||
"fact_cards",
|
||||
"llm_trace_calls",
|
||||
"llm_trace_runs",
|
||||
"optimization_cases",
|
||||
"optimization_result_versions",
|
||||
"optimized_articles",
|
||||
|
||||
@@ -115,6 +115,49 @@ export function initializeSchema(db: Database.Database) {
|
||||
foreign key (article_job_id) references article_jobs(id) on delete set null
|
||||
);
|
||||
|
||||
create table if not exists llm_trace_runs (
|
||||
job_id text primary key,
|
||||
case_id text,
|
||||
status text not null,
|
||||
current_stage text not null,
|
||||
trace_completeness text not null,
|
||||
error_stage text,
|
||||
error_summary text,
|
||||
started_at text not null,
|
||||
finished_at text,
|
||||
updated_at text not null,
|
||||
foreign key (job_id) references article_jobs(id) on delete cascade,
|
||||
foreign key (case_id) references optimization_cases(id) on delete set null
|
||||
);
|
||||
|
||||
create table if not exists llm_trace_calls (
|
||||
call_id text primary key,
|
||||
job_id text not null,
|
||||
sequence integer not null,
|
||||
task text not null,
|
||||
workflow_stage text not null,
|
||||
rewrite_round integer,
|
||||
provider text not null,
|
||||
model text not null,
|
||||
status text not null,
|
||||
request_object_key text,
|
||||
response_object_key text,
|
||||
token_usage text,
|
||||
schema_name text,
|
||||
schema_valid integer,
|
||||
validation_issues text not null default '[]',
|
||||
business_status text,
|
||||
duration_ms integer,
|
||||
started_at text not null,
|
||||
responded_at text,
|
||||
validated_at text,
|
||||
failed_at text,
|
||||
error_type text,
|
||||
error_summary text,
|
||||
unique (job_id, sequence),
|
||||
foreign key (job_id) references llm_trace_runs(job_id) on delete cascade
|
||||
);
|
||||
|
||||
create table if not exists rubric_versions (
|
||||
id text primary key,
|
||||
version text not null,
|
||||
@@ -204,6 +247,12 @@ export function initializeSchema(db: Database.Database) {
|
||||
|
||||
create index if not exists idx_performance_snapshots_publication
|
||||
on performance_snapshots(publication_id);
|
||||
|
||||
create index if not exists idx_llm_trace_runs_status_updated
|
||||
on llm_trace_runs(status, updated_at desc);
|
||||
|
||||
create index if not exists idx_llm_trace_calls_job_sequence
|
||||
on llm_trace_calls(job_id, sequence);
|
||||
`);
|
||||
|
||||
ensureColumn(db, "article_jobs", "case_id", "text");
|
||||
|
||||
@@ -2,6 +2,7 @@ import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { z } from "zod";
|
||||
|
||||
import * as client from "../client";
|
||||
import type { LlmClientTraceEvent } from "../trace-types";
|
||||
|
||||
describe("generateValidatedJson", () => {
|
||||
const originalProvider = process.env.LLM_PROVIDER;
|
||||
@@ -13,9 +14,133 @@ describe("generateValidatedJson", () => {
|
||||
delete process.env.LLM_LOG_RAW_LIMIT;
|
||||
delete process.env.DEEPSEEK_MODEL;
|
||||
client.setGenerateJsonForValidation(client.generateJson);
|
||||
client.setChatCompletionForTesting(null);
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it("traces the exact SDK request and full SDK response", async () => {
|
||||
process.env.LLM_PROVIDER = "deepseek";
|
||||
process.env.DEEPSEEK_API_KEY = "test-key";
|
||||
const traced: LlmClientTraceEvent[] = [];
|
||||
let sdkRequest: unknown;
|
||||
const sdkResponse = {
|
||||
id: "chatcmpl_1",
|
||||
object: "chat.completion",
|
||||
created: 1784188800,
|
||||
model: "deepseek-v4-pro",
|
||||
choices: [{
|
||||
index: 0,
|
||||
message: { role: "assistant", content: '{"value":"ok"}' },
|
||||
finish_reason: "stop",
|
||||
}],
|
||||
usage: { prompt_tokens: 10, completion_tokens: 4, total_tokens: 14 },
|
||||
};
|
||||
client.setChatCompletionForTesting(async (request) => {
|
||||
sdkRequest = request;
|
||||
return sdkResponse;
|
||||
});
|
||||
|
||||
await client.generateValidatedJson({
|
||||
schema: z.object({ value: z.string() }),
|
||||
schemaName: "valueSchema",
|
||||
prompt: "Return JSON.",
|
||||
task: "article_optimizer",
|
||||
traceStage: "draft",
|
||||
onTraceEvent: (event) => {
|
||||
traced.push(event);
|
||||
},
|
||||
});
|
||||
|
||||
expect(traced.find((event) => event.type === "started")).toMatchObject({
|
||||
type: "started",
|
||||
request: sdkRequest,
|
||||
});
|
||||
expect(traced.find((event) => event.type === "responded")).toMatchObject({
|
||||
type: "responded",
|
||||
response: sdkResponse,
|
||||
});
|
||||
expect(traced.map((event) => event.type)).toEqual([
|
||||
"started",
|
||||
"responded",
|
||||
"validated",
|
||||
]);
|
||||
expect(JSON.stringify(traced)).not.toContain("test-key");
|
||||
});
|
||||
|
||||
it("traces JSON parse and schema failures in their actual order", async () => {
|
||||
process.env.LLM_PROVIDER = "deepseek";
|
||||
process.env.DEEPSEEK_API_KEY = "test-key";
|
||||
const jsonParseEvents: LlmClientTraceEvent[] = [];
|
||||
client.setChatCompletionForTesting(async () => ({
|
||||
choices: [{ message: { content: "not json" } }],
|
||||
}));
|
||||
|
||||
await expect(client.generateValidatedJson({
|
||||
schema: z.object({ value: z.string() }),
|
||||
schemaName: "valueSchema",
|
||||
prompt: "Return JSON.",
|
||||
onTraceEvent: (event) => {
|
||||
jsonParseEvents.push(event);
|
||||
},
|
||||
})).rejects.toThrow("not valid JSON");
|
||||
expect(jsonParseEvents.map((event) => event.type)).toEqual([
|
||||
"started",
|
||||
"responded",
|
||||
"failed",
|
||||
]);
|
||||
expect(jsonParseEvents.at(-1)).toMatchObject({
|
||||
type: "failed",
|
||||
error_type: "json_parse",
|
||||
});
|
||||
|
||||
const schemaEvents: LlmClientTraceEvent[] = [];
|
||||
client.setChatCompletionForTesting(async () => ({
|
||||
choices: [{ message: { content: '{"value":42}' } }],
|
||||
}));
|
||||
await expect(client.generateValidatedJson({
|
||||
schema: z.object({ value: z.string() }),
|
||||
schemaName: "valueSchema",
|
||||
prompt: "Return JSON.",
|
||||
onTraceEvent: (event) => {
|
||||
schemaEvents.push(event);
|
||||
},
|
||||
})).rejects.toThrow("schema validation");
|
||||
expect(schemaEvents.map((event) => event.type)).toEqual([
|
||||
"started",
|
||||
"responded",
|
||||
"validated",
|
||||
"failed",
|
||||
]);
|
||||
expect(schemaEvents.at(-1)).toMatchObject({
|
||||
type: "failed",
|
||||
error_type: "schema_validation",
|
||||
});
|
||||
});
|
||||
|
||||
it("redacts configured secrets from traced provider errors", async () => {
|
||||
process.env.LLM_PROVIDER = "deepseek";
|
||||
process.env.DEEPSEEK_API_KEY = "super-secret-key";
|
||||
const traced: LlmClientTraceEvent[] = [];
|
||||
client.setChatCompletionForTesting(async () => {
|
||||
throw new Error("request failed for super-secret-key");
|
||||
});
|
||||
|
||||
await expect(client.generateValidatedJson({
|
||||
schema: z.object({ value: z.string() }),
|
||||
prompt: "Return JSON.",
|
||||
onTraceEvent: (event) => {
|
||||
traced.push(event);
|
||||
},
|
||||
})).rejects.toThrow("request failed");
|
||||
|
||||
expect(traced.map((event) => event.type)).toEqual(["started", "failed"]);
|
||||
expect(JSON.stringify(traced)).not.toContain("super-secret-key");
|
||||
expect(traced.at(-1)).toMatchObject({
|
||||
type: "failed",
|
||||
error_type: "provider",
|
||||
});
|
||||
});
|
||||
|
||||
it("throws clearly when no provider key is configured", async () => {
|
||||
process.env.LLM_PROVIDER = "deepseek";
|
||||
delete process.env.DEEPSEEK_API_KEY;
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { createD1TraceRepository } from "../d1-trace-repository";
|
||||
import type { LlmTraceCall, LlmTraceRun } from "../trace-types";
|
||||
|
||||
const run: LlmTraceRun = {
|
||||
job_id: "job_1",
|
||||
case_id: "case_1",
|
||||
status: "running",
|
||||
current_stage: "draft",
|
||||
trace_completeness: "complete",
|
||||
error_stage: null,
|
||||
error_summary: null,
|
||||
started_at: "2026-07-16T00:00:00.000Z",
|
||||
finished_at: null,
|
||||
updated_at: "2026-07-16T00:00:01.000Z",
|
||||
};
|
||||
|
||||
const call: LlmTraceCall = {
|
||||
call_id: "llmcall_1",
|
||||
job_id: "job_1",
|
||||
sequence: 1,
|
||||
task: "article_optimizer",
|
||||
workflow_stage: "draft",
|
||||
rewrite_round: null,
|
||||
provider: "deepseek",
|
||||
model: "deepseek-v4-pro",
|
||||
status: "validated",
|
||||
request_object_key: "llm-traces/job_1/llmcall_1/request.json",
|
||||
response_object_key: "llm-traces/job_1/llmcall_1/response.json",
|
||||
token_usage: { prompt_tokens: 10, completion_tokens: 4, total_tokens: 14 },
|
||||
schema_name: "optimizedArticleSchema",
|
||||
schema_valid: true,
|
||||
validation_issues: ["title: required"],
|
||||
business_status: null,
|
||||
duration_ms: 1200,
|
||||
started_at: "2026-07-16T00:00:00.000Z",
|
||||
responded_at: "2026-07-16T00:00:01.000Z",
|
||||
validated_at: "2026-07-16T00:00:01.200Z",
|
||||
failed_at: null,
|
||||
error_type: null,
|
||||
error_summary: null,
|
||||
};
|
||||
|
||||
describe("createD1TraceRepository", () => {
|
||||
it("binds run and call snapshots with JSON fields serialized", async () => {
|
||||
const runStatement = vi.fn().mockResolvedValue({ success: true });
|
||||
const bind = vi.fn().mockReturnValue({ run: runStatement });
|
||||
const prepare = vi.fn().mockReturnValue({ bind });
|
||||
const repository = createD1TraceRepository(
|
||||
{ prepare } as unknown as D1Database,
|
||||
);
|
||||
|
||||
await repository.putRun(run);
|
||||
await repository.putCall(call);
|
||||
|
||||
expect(prepare).toHaveBeenNthCalledWith(
|
||||
1,
|
||||
expect.stringContaining("insert into llm_trace_runs"),
|
||||
);
|
||||
expect(prepare).toHaveBeenNthCalledWith(
|
||||
2,
|
||||
expect.stringContaining("insert into llm_trace_calls"),
|
||||
);
|
||||
expect(bind).toHaveBeenNthCalledWith(
|
||||
2,
|
||||
call.call_id,
|
||||
call.job_id,
|
||||
call.sequence,
|
||||
call.task,
|
||||
call.workflow_stage,
|
||||
null,
|
||||
call.provider,
|
||||
call.model,
|
||||
call.status,
|
||||
call.request_object_key,
|
||||
call.response_object_key,
|
||||
JSON.stringify(call.token_usage),
|
||||
call.schema_name,
|
||||
1,
|
||||
JSON.stringify(call.validation_issues),
|
||||
null,
|
||||
call.duration_ms,
|
||||
call.started_at,
|
||||
call.responded_at,
|
||||
call.validated_at,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
);
|
||||
});
|
||||
|
||||
it("reads calls in sequence order and parses JSON fields", async () => {
|
||||
const all = vi.fn().mockResolvedValue({
|
||||
results: [{
|
||||
...call,
|
||||
token_usage: JSON.stringify(call.token_usage),
|
||||
schema_valid: 1,
|
||||
validation_issues: JSON.stringify(call.validation_issues),
|
||||
}],
|
||||
});
|
||||
const bind = vi.fn().mockReturnValue({ all });
|
||||
const prepare = vi.fn().mockReturnValue({ bind });
|
||||
const repository = createD1TraceRepository(
|
||||
{ prepare } as unknown as D1Database,
|
||||
);
|
||||
|
||||
await expect(repository.listCalls("job_1")).resolves.toEqual([call]);
|
||||
expect(prepare).toHaveBeenCalledWith(
|
||||
expect.stringContaining("order by sequence"),
|
||||
);
|
||||
expect(bind).toHaveBeenCalledWith("job_1");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,98 @@
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import {
|
||||
createLocalTracePayloadStore,
|
||||
createR2TracePayloadStore,
|
||||
} from "../trace-payload-store";
|
||||
|
||||
describe("LLM trace payload stores", () => {
|
||||
let tempDir: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tempDir = mkdtempSync(join(tmpdir(), "geo-llm-payloads-"));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(tempDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("round-trips exact JSON locally and deletes one job prefix", async () => {
|
||||
const store = createLocalTracePayloadStore(tempDir);
|
||||
const payload = {
|
||||
model: "deepseek-v4-pro",
|
||||
messages: [{ role: "user", content: "原文" }],
|
||||
};
|
||||
const key = "llm-traces/job_1/llmcall_1/request.json";
|
||||
|
||||
await store.putJson(key, payload);
|
||||
await expect(store.getJson(key)).resolves.toEqual(payload);
|
||||
await store.deleteJob("job_1");
|
||||
await expect(store.getJson(key)).resolves.toBeNull();
|
||||
});
|
||||
|
||||
it("stores private JSON in R2 without a public URL", async () => {
|
||||
const put = vi.fn().mockResolvedValue(undefined);
|
||||
const get = vi.fn().mockResolvedValue({
|
||||
json: async () => ({ ok: true }),
|
||||
});
|
||||
const list = vi.fn().mockResolvedValue({ objects: [], truncated: false });
|
||||
const deleteObjects = vi.fn().mockResolvedValue(undefined);
|
||||
const bucket = {
|
||||
put,
|
||||
get,
|
||||
list,
|
||||
delete: deleteObjects,
|
||||
} as unknown as R2Bucket;
|
||||
const store = createR2TracePayloadStore(bucket);
|
||||
|
||||
await store.putJson(
|
||||
"llm-traces/job_1/llmcall_1/request.json",
|
||||
{ ok: true },
|
||||
);
|
||||
await expect(
|
||||
store.getJson("llm-traces/job_1/llmcall_1/request.json"),
|
||||
).resolves.toEqual({ ok: true });
|
||||
expect(put).toHaveBeenCalledWith(
|
||||
"llm-traces/job_1/llmcall_1/request.json",
|
||||
JSON.stringify({ ok: true }),
|
||||
{ httpMetadata: { contentType: "application/json; charset=utf-8" } },
|
||||
);
|
||||
});
|
||||
|
||||
it("deletes every paginated R2 object under one job prefix", async () => {
|
||||
const list = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce({
|
||||
objects: [{ key: "llm-traces/job_1/call_1/request.json" }],
|
||||
truncated: true,
|
||||
cursor: "next-page",
|
||||
})
|
||||
.mockResolvedValueOnce({
|
||||
objects: [{ key: "llm-traces/job_1/call_1/response.json" }],
|
||||
truncated: false,
|
||||
});
|
||||
const deleteObjects = vi.fn().mockResolvedValue(undefined);
|
||||
const bucket = { list, delete: deleteObjects } as unknown as R2Bucket;
|
||||
|
||||
await createR2TracePayloadStore(bucket).deleteJob("job_1");
|
||||
|
||||
expect(list).toHaveBeenNthCalledWith(1, {
|
||||
prefix: "llm-traces/job_1/",
|
||||
cursor: undefined,
|
||||
});
|
||||
expect(list).toHaveBeenNthCalledWith(2, {
|
||||
prefix: "llm-traces/job_1/",
|
||||
cursor: "next-page",
|
||||
});
|
||||
expect(deleteObjects).toHaveBeenNthCalledWith(1, [
|
||||
"llm-traces/job_1/call_1/request.json",
|
||||
]);
|
||||
expect(deleteObjects).toHaveBeenNthCalledWith(2, [
|
||||
"llm-traces/job_1/call_1/response.json",
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,226 @@
|
||||
import { beforeEach, describe, expect, it } from "vitest";
|
||||
|
||||
import type { LlmTracePayloadStore } from "../trace-payload-store";
|
||||
import {
|
||||
createLlmTraceRecorder,
|
||||
type LlmTraceRecorder,
|
||||
} from "../trace-recorder";
|
||||
import type { LlmTraceRepository } from "../trace-repository";
|
||||
import type {
|
||||
LlmTraceCall,
|
||||
LlmTraceRun,
|
||||
LlmTraceStreamEvent,
|
||||
} from "../trace-types";
|
||||
|
||||
class MemoryTraceRepository implements LlmTraceRepository {
|
||||
runs = new Map<string, LlmTraceRun>();
|
||||
calls = new Map<string, LlmTraceCall>();
|
||||
deletedRuns: string[] = [];
|
||||
failWrites = false;
|
||||
|
||||
async putRun(run: LlmTraceRun) {
|
||||
if (this.failWrites) throw new Error("index unavailable");
|
||||
this.runs.set(run.job_id, structuredClone(run));
|
||||
}
|
||||
|
||||
async putCall(call: LlmTraceCall) {
|
||||
if (this.failWrites) throw new Error("index unavailable");
|
||||
this.calls.set(call.call_id, structuredClone(call));
|
||||
}
|
||||
|
||||
async getRun(jobId: string) {
|
||||
return this.runs.get(jobId) ?? null;
|
||||
}
|
||||
|
||||
async getLatestRun() {
|
||||
return [...this.runs.values()][0] ?? null;
|
||||
}
|
||||
|
||||
async listCalls(jobId: string) {
|
||||
return [...this.calls.values()]
|
||||
.filter((call) => call.job_id === jobId)
|
||||
.sort((left, right) => left.sequence - right.sequence);
|
||||
}
|
||||
|
||||
async listTerminalRunsExcept(jobId: string) {
|
||||
return [...this.runs.values()].filter(
|
||||
(run) => run.job_id !== jobId && run.status !== "running",
|
||||
);
|
||||
}
|
||||
|
||||
async deleteRun(jobId: string) {
|
||||
this.deletedRuns.push(jobId);
|
||||
this.runs.delete(jobId);
|
||||
for (const call of this.calls.values()) {
|
||||
if (call.job_id === jobId) this.calls.delete(call.call_id);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
class MemoryPayloadStore implements LlmTracePayloadStore {
|
||||
values = new Map<string, unknown>();
|
||||
deletedJobs: string[] = [];
|
||||
failPuts = false;
|
||||
|
||||
async putJson(key: string, value: unknown) {
|
||||
if (this.failPuts) throw new Error("payload unavailable");
|
||||
this.values.set(key, structuredClone(value));
|
||||
}
|
||||
|
||||
async getJson(key: string) {
|
||||
return this.values.get(key) ?? null;
|
||||
}
|
||||
|
||||
async deleteJob(jobId: string) {
|
||||
this.deletedJobs.push(jobId);
|
||||
}
|
||||
}
|
||||
|
||||
describe("createLlmTraceRecorder", () => {
|
||||
let repository: MemoryTraceRepository;
|
||||
let payloadStore: MemoryPayloadStore;
|
||||
let published: LlmTraceStreamEvent[];
|
||||
let recorder: LlmTraceRecorder;
|
||||
|
||||
beforeEach(async () => {
|
||||
repository = new MemoryTraceRepository();
|
||||
payloadStore = new MemoryPayloadStore();
|
||||
published = [];
|
||||
recorder = await createLlmTraceRecorder({
|
||||
jobId: "job_1",
|
||||
caseId: "case_1",
|
||||
repository,
|
||||
payloadStore,
|
||||
publish: (event) => {
|
||||
published.push(event);
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("persists exact bodies before publishing public metadata", async () => {
|
||||
await recorder.onLlmEvent({
|
||||
type: "started",
|
||||
call_id: "llmcall_1",
|
||||
task: "quality_inspector",
|
||||
context: { workflow_stage: "qa", schema_name: "llmQaPatchSchema" },
|
||||
provider: "deepseek",
|
||||
model: "deepseek-v4-pro",
|
||||
request: { model: "deepseek-v4-pro", messages: [] },
|
||||
started_at: "2026-07-16T00:00:00.000Z",
|
||||
});
|
||||
await recorder.onLlmEvent({
|
||||
type: "responded",
|
||||
call_id: "llmcall_1",
|
||||
response: {
|
||||
choices: [{ message: { content: "{}" } }],
|
||||
usage: { prompt_tokens: 10, completion_tokens: 4, total_tokens: 14 },
|
||||
},
|
||||
duration_ms: 1200,
|
||||
responded_at: "2026-07-16T00:00:01.200Z",
|
||||
});
|
||||
|
||||
expect(payloadStore.values.get(
|
||||
"llm-traces/job_1/llmcall_1/request.json",
|
||||
)).toEqual({ model: "deepseek-v4-pro", messages: [] });
|
||||
expect(payloadStore.values.get(
|
||||
"llm-traces/job_1/llmcall_1/response.json",
|
||||
)).toEqual({
|
||||
choices: [{ message: { content: "{}" } }],
|
||||
usage: { prompt_tokens: 10, completion_tokens: 4, total_tokens: 14 },
|
||||
});
|
||||
expect(published[0]).toMatchObject({
|
||||
type: "llm_call_started",
|
||||
request_available: true,
|
||||
});
|
||||
expect(published[1]).toMatchObject({
|
||||
type: "llm_call_responded",
|
||||
token_usage: { prompt_tokens: 10, completion_tokens: 4, total_tokens: 14 },
|
||||
response_available: true,
|
||||
});
|
||||
expect(JSON.stringify(published)).not.toContain("messages");
|
||||
expect(JSON.stringify(published)).not.toContain("choices");
|
||||
});
|
||||
|
||||
it("stores QA business failure separately from schema success", async () => {
|
||||
await recorder.onLlmEvent({
|
||||
type: "started",
|
||||
call_id: "llmcall_qa",
|
||||
task: "quality_inspector",
|
||||
context: { workflow_stage: "qa", schema_name: "llmQaPatchSchema" },
|
||||
provider: "deepseek",
|
||||
model: "deepseek-v4-pro",
|
||||
request: { messages: [] },
|
||||
started_at: "2026-07-16T00:00:00.000Z",
|
||||
});
|
||||
await recorder.onLlmEvent({
|
||||
type: "validated",
|
||||
call_id: "llmcall_qa",
|
||||
schema_name: "llmQaPatchSchema",
|
||||
schema_valid: true,
|
||||
validation_issues: [],
|
||||
validated_at: "2026-07-16T00:00:01.000Z",
|
||||
});
|
||||
|
||||
await recorder.onWorkflowEvent({
|
||||
type: "qa_ready",
|
||||
job_id: "job_1",
|
||||
qa_report: { overall_status: "fail", checks: [] },
|
||||
});
|
||||
|
||||
await expect(repository.listCalls("job_1")).resolves.toEqual([
|
||||
expect.objectContaining({
|
||||
task: "quality_inspector",
|
||||
schema_valid: true,
|
||||
business_status: "fail",
|
||||
}),
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps running runs and only the newest terminal full trace", async () => {
|
||||
repository.runs.set("job_old_completed", {
|
||||
...repository.runs.get("job_1")!,
|
||||
job_id: "job_old_completed",
|
||||
status: "completed",
|
||||
});
|
||||
repository.runs.set("job_other_running", {
|
||||
...repository.runs.get("job_1")!,
|
||||
job_id: "job_other_running",
|
||||
status: "running",
|
||||
});
|
||||
|
||||
await recorder.finish({ status: "completed" });
|
||||
|
||||
expect(payloadStore.deletedJobs).toEqual(["job_old_completed"]);
|
||||
expect(repository.deletedRuns).toEqual(["job_old_completed"]);
|
||||
expect(repository.deletedRuns).not.toContain("job_other_running");
|
||||
});
|
||||
|
||||
it("marks the trace incomplete without throwing when payload storage fails", async () => {
|
||||
payloadStore.failPuts = true;
|
||||
|
||||
await expect(recorder.onLlmEvent({
|
||||
type: "started",
|
||||
call_id: "llmcall_failed_storage",
|
||||
task: "fact_extractor",
|
||||
context: { workflow_stage: "fact_card" },
|
||||
provider: "deepseek",
|
||||
model: "deepseek-v4-pro",
|
||||
request: { messages: [{ role: "user", content: "原文" }] },
|
||||
started_at: "2026-07-16T00:00:00.000Z",
|
||||
})).resolves.toBeUndefined();
|
||||
|
||||
expect(published).toEqual([
|
||||
expect.objectContaining({
|
||||
type: "llm_call_started",
|
||||
request_available: false,
|
||||
}),
|
||||
expect.objectContaining({
|
||||
type: "trace_warning",
|
||||
trace_completeness: "incomplete",
|
||||
}),
|
||||
]);
|
||||
await expect(repository.getRun("job_1")).resolves.toMatchObject({
|
||||
trace_completeness: "incomplete",
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,99 @@
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
|
||||
import { createSqliteRepository } from "../../db/sqlite-repository";
|
||||
import type { LlmTraceCall, LlmTraceRun } from "../trace-types";
|
||||
import { createSqliteTraceRepository } from "../sqlite-trace-repository";
|
||||
|
||||
function runFixture(jobId: string): LlmTraceRun {
|
||||
return {
|
||||
job_id: jobId,
|
||||
case_id: null,
|
||||
status: "running",
|
||||
current_stage: "fact_card",
|
||||
trace_completeness: "complete",
|
||||
error_stage: null,
|
||||
error_summary: null,
|
||||
started_at: "2026-07-16T00:00:00.000Z",
|
||||
finished_at: null,
|
||||
updated_at: "2026-07-16T00:00:00.000Z",
|
||||
};
|
||||
}
|
||||
|
||||
function callFixture(
|
||||
jobId: string,
|
||||
callId: string,
|
||||
sequence: number,
|
||||
): LlmTraceCall {
|
||||
return {
|
||||
call_id: callId,
|
||||
job_id: jobId,
|
||||
sequence,
|
||||
task: sequence === 1 ? "fact_extractor" : "article_optimizer",
|
||||
workflow_stage: sequence === 1 ? "fact_card" : "draft",
|
||||
rewrite_round: null,
|
||||
provider: "deepseek",
|
||||
model: "deepseek-v4-pro",
|
||||
status: "started",
|
||||
request_object_key: `llm-traces/${jobId}/${callId}/request.json`,
|
||||
response_object_key: null,
|
||||
token_usage: null,
|
||||
schema_name: null,
|
||||
schema_valid: null,
|
||||
validation_issues: [],
|
||||
business_status: null,
|
||||
duration_ms: null,
|
||||
started_at: `2026-07-16T00:00:0${sequence}.000Z`,
|
||||
responded_at: null,
|
||||
validated_at: null,
|
||||
failed_at: null,
|
||||
error_type: null,
|
||||
error_summary: null,
|
||||
};
|
||||
}
|
||||
|
||||
describe("createSqliteTraceRepository", () => {
|
||||
let tempDir: string;
|
||||
let dbPath: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tempDir = mkdtempSync(join(tmpdir(), "geo-llm-traces-"));
|
||||
dbPath = join(tempDir, "app.db");
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(tempDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("stores a run and ordered calls, then deletes only trace rows", async () => {
|
||||
const appRepository = createSqliteRepository(dbPath);
|
||||
const job = await appRepository.createArticleJob({
|
||||
source_title: "Title",
|
||||
source_body: "Body",
|
||||
image_inputs: [],
|
||||
publish_platform: "official_site",
|
||||
user_instructions: "",
|
||||
});
|
||||
const repository = createSqliteTraceRepository(dbPath);
|
||||
|
||||
await repository.putRun(runFixture(job.id));
|
||||
await repository.putCall(callFixture(job.id, "llmcall_1", 1));
|
||||
await repository.putCall(callFixture(job.id, "llmcall_2", 2));
|
||||
|
||||
await expect(repository.getLatestRun()).resolves.toMatchObject({
|
||||
job_id: job.id,
|
||||
status: "running",
|
||||
});
|
||||
await expect(repository.listCalls(job.id)).resolves.toEqual([
|
||||
expect.objectContaining({ call_id: "llmcall_1", sequence: 1 }),
|
||||
expect.objectContaining({ call_id: "llmcall_2", sequence: 2 }),
|
||||
]);
|
||||
|
||||
await repository.deleteRun(job.id);
|
||||
await expect(appRepository.getArticleJob(job.id)).resolves.not.toBeNull();
|
||||
await expect(repository.getRun(job.id)).resolves.toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -1,4 +1,5 @@
|
||||
import type { LlmProviderStatus, LlmTaskName } from "./client";
|
||||
import type { LlmProviderStatus } from "./client";
|
||||
import type { LlmTaskName } from "./trace-types";
|
||||
|
||||
export interface LlmAuditSummary {
|
||||
provider: LlmProviderStatus["provider"];
|
||||
|
||||
+177
-38
@@ -1,15 +1,17 @@
|
||||
import OpenAI from "openai";
|
||||
import { nanoid } from "nanoid";
|
||||
import type { z } from "zod";
|
||||
|
||||
import { createLlmAuditSummary, type LlmAuditSummary } from "./audit";
|
||||
import type {
|
||||
LlmClientTraceEvent,
|
||||
LlmClientTraceHandler,
|
||||
LlmTaskName,
|
||||
LlmTraceErrorType,
|
||||
LlmTraceWorkflowStage,
|
||||
} from "./trace-types";
|
||||
|
||||
export type LlmTaskName =
|
||||
| "unknown"
|
||||
| "fact_extractor"
|
||||
| "article_optimizer"
|
||||
| "quality_inspector"
|
||||
| "targeted_rewriter"
|
||||
| "renwei_copy_optimizer";
|
||||
export type { LlmTaskName } from "./trace-types";
|
||||
|
||||
export interface GenerateInput {
|
||||
system?: string;
|
||||
@@ -17,7 +19,12 @@ export interface GenerateInput {
|
||||
model?: string;
|
||||
temperature?: number;
|
||||
task?: LlmTaskName;
|
||||
schemaName?: string;
|
||||
traceStage?: LlmTraceWorkflowStage;
|
||||
rewriteRound?: number;
|
||||
onTraceEvent?: LlmClientTraceHandler;
|
||||
onAuditSummary?: (summary: LlmAuditSummary) => void | Promise<void>;
|
||||
traceCallId?: string;
|
||||
}
|
||||
|
||||
export interface GenerateValidatedJsonInput<T> extends GenerateInput {
|
||||
@@ -98,6 +105,64 @@ function getTask(input: GenerateInput): LlmTaskName {
|
||||
return input.task || "unknown";
|
||||
}
|
||||
|
||||
function callIdFor(input: GenerateInput) {
|
||||
return input.traceCallId ?? `llmcall_${nanoid(12)}`;
|
||||
}
|
||||
|
||||
function redactConfiguredSecrets(value: string) {
|
||||
return [process.env.DEEPSEEK_API_KEY, process.env.OPENAI_API_KEY]
|
||||
.filter((secret): secret is string => Boolean(secret))
|
||||
.reduce(
|
||||
(redacted, secret) => redacted.split(secret).join("[redacted]"),
|
||||
value,
|
||||
);
|
||||
}
|
||||
|
||||
function safeErrorSummary(error: unknown) {
|
||||
if (error instanceof Error) {
|
||||
return `${error.name}: ${redactConfiguredSecrets(error.message)}`;
|
||||
}
|
||||
return `Error: ${redactConfiguredSecrets(String(error))}`;
|
||||
}
|
||||
|
||||
async function emitTrace(input: GenerateInput, event: LlmClientTraceEvent) {
|
||||
try {
|
||||
await input.onTraceEvent?.(event);
|
||||
} catch (error) {
|
||||
console.warn(`[llm:trace-warning] ${safeErrorSummary(error)}`);
|
||||
}
|
||||
}
|
||||
|
||||
function failedEvent(
|
||||
callId: string,
|
||||
errorType: LlmTraceErrorType,
|
||||
error: unknown,
|
||||
startedAt: number,
|
||||
): Extract<LlmClientTraceEvent, { type: "failed" }> {
|
||||
return {
|
||||
type: "failed",
|
||||
call_id: callId,
|
||||
error_type: errorType,
|
||||
error_summary: safeErrorSummary(error),
|
||||
duration_ms: Date.now() - startedAt,
|
||||
failed_at: new Date().toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
function buildChatCompletionRequest(input: GenerateInput, model: string) {
|
||||
return {
|
||||
model,
|
||||
temperature: input.temperature ?? 0.1,
|
||||
response_format: { type: "json_object" as const },
|
||||
messages: [
|
||||
...(input.system
|
||||
? [{ role: "system" as const, content: input.system }]
|
||||
: []),
|
||||
{ role: "user" as const, content: input.prompt },
|
||||
],
|
||||
} satisfies ChatCompletionRequest;
|
||||
}
|
||||
|
||||
function getRawLogLimit() {
|
||||
const parsed = Number(process.env.LLM_LOG_RAW_LIMIT ?? "4000");
|
||||
return Number.isFinite(parsed) && parsed >= 0 ? parsed : 4000;
|
||||
@@ -171,39 +236,75 @@ export async function generateText(input: GenerateInput) {
|
||||
export async function generateJson<T>(input: GenerateInput): Promise<T> {
|
||||
const task = getTask(input);
|
||||
const startedAt = Date.now();
|
||||
try {
|
||||
const status = getLlmProviderStatus();
|
||||
if (!status.configured) {
|
||||
throw new Error(status.reason ?? "LLM provider is not configured");
|
||||
}
|
||||
const effectiveModel = input.model ?? status.model;
|
||||
console.info(
|
||||
`[llm:start] provider=${status.provider} model=${effectiveModel} task=${task}`,
|
||||
);
|
||||
const request = {
|
||||
model: effectiveModel,
|
||||
temperature: input.temperature ?? 0.1,
|
||||
response_format: { type: "json_object" },
|
||||
messages: [
|
||||
...(input.system ? [{ role: "system" as const, content: input.system }] : []),
|
||||
{ role: "user" as const, content: input.prompt },
|
||||
],
|
||||
} satisfies ChatCompletionRequest;
|
||||
const response = chatCompletionForTesting
|
||||
? await chatCompletionForTesting(request)
|
||||
: await createClient().client.chat.completions.create(request);
|
||||
const content = response.choices[0]?.message.content ?? "{}";
|
||||
console.info(
|
||||
`[llm:response] task=${task} duration_ms=${Date.now() - startedAt} raw=${truncateRaw(content)}`,
|
||||
);
|
||||
return JSON.parse(content) as T;
|
||||
} catch (error) {
|
||||
const callId = callIdFor(input);
|
||||
const status = getLlmProviderStatus();
|
||||
if (!status.configured) {
|
||||
const error = new Error(status.reason ?? "LLM provider is not configured");
|
||||
const normalized = normalizeLlmError(error);
|
||||
console.error(
|
||||
`[llm:error] task=${task} duration_ms=${Date.now() - startedAt} message=${quoteLogValue(normalized.message)}`,
|
||||
);
|
||||
throw normalized;
|
||||
}
|
||||
|
||||
const effectiveModel = input.model ?? status.model;
|
||||
console.info(
|
||||
`[llm:start] provider=${status.provider} model=${effectiveModel} task=${task}`,
|
||||
);
|
||||
const request = buildChatCompletionRequest(input, effectiveModel);
|
||||
await emitTrace(input, {
|
||||
type: "started",
|
||||
call_id: callId,
|
||||
task,
|
||||
context: {
|
||||
workflow_stage: input.traceStage ?? "unknown",
|
||||
rewrite_round: input.rewriteRound,
|
||||
schema_name: input.schemaName,
|
||||
},
|
||||
provider: status.provider,
|
||||
model: effectiveModel,
|
||||
request,
|
||||
started_at: new Date(startedAt).toISOString(),
|
||||
});
|
||||
|
||||
let response: ChatCompletionResult;
|
||||
try {
|
||||
response = chatCompletionForTesting
|
||||
? await chatCompletionForTesting(request)
|
||||
: await createClient().client.chat.completions.create(request);
|
||||
} catch (error) {
|
||||
await emitTrace(input, failedEvent(callId, "provider", error, startedAt));
|
||||
const normalized = normalizeLlmError(error);
|
||||
console.error(
|
||||
`[llm:error] task=${task} duration_ms=${Date.now() - startedAt} message=${quoteLogValue(normalized.message)}`,
|
||||
);
|
||||
throw normalized;
|
||||
}
|
||||
|
||||
await emitTrace(input, {
|
||||
type: "responded",
|
||||
call_id: callId,
|
||||
response,
|
||||
duration_ms: Date.now() - startedAt,
|
||||
responded_at: new Date().toISOString(),
|
||||
});
|
||||
|
||||
const content = response.choices[0]?.message.content ?? "{}";
|
||||
console.info(
|
||||
`[llm:response] task=${task} duration_ms=${Date.now() - startedAt} raw=${truncateRaw(content)}`,
|
||||
);
|
||||
try {
|
||||
return JSON.parse(content) as T;
|
||||
} catch (error) {
|
||||
await emitTrace(input, failedEvent(callId, "json_parse", error, startedAt));
|
||||
const message = `LLM response is not valid JSON: ${redactConfiguredSecrets(
|
||||
error instanceof Error ? error.message : String(error),
|
||||
)}`;
|
||||
console.error(
|
||||
`[llm:error] task=${task} duration_ms=${Date.now() - startedAt} message=${quoteLogValue(message)}`,
|
||||
);
|
||||
throw new Error(message);
|
||||
}
|
||||
}
|
||||
|
||||
export let generateJsonForValidation: <T>(input: GenerateInput) => Promise<T> =
|
||||
@@ -232,6 +333,9 @@ export async function generateValidatedJson<T>({
|
||||
const status = getLlmProviderStatus();
|
||||
const startedAt = Date.now();
|
||||
const effectiveModel = input.model ?? status.model;
|
||||
const traceCallId = callIdFor(input);
|
||||
const tracedInput: GenerateInput = { ...input, traceCallId };
|
||||
const schemaName = input.schemaName ?? "anonymousSchema";
|
||||
const emitAudit = async ({
|
||||
schemaValid,
|
||||
output,
|
||||
@@ -262,7 +366,7 @@ export async function generateValidatedJson<T>({
|
||||
}
|
||||
|
||||
try {
|
||||
const generated = await generateJsonForValidation<unknown>(input);
|
||||
const generated = await generateJsonForValidation<unknown>(tracedInput);
|
||||
if (!usesDefaultGenerator) {
|
||||
console.info(
|
||||
`[llm:response] task=${task} duration_ms=${Date.now() - startedAt} raw=${truncateRaw(stringifyForLog(generated))}`,
|
||||
@@ -270,6 +374,14 @@ export async function generateValidatedJson<T>({
|
||||
}
|
||||
const parsed = schema.safeParse(generated);
|
||||
if (parsed.success) {
|
||||
await emitTrace(tracedInput, {
|
||||
type: "validated",
|
||||
call_id: traceCallId,
|
||||
schema_name: schemaName,
|
||||
schema_valid: true,
|
||||
validation_issues: [],
|
||||
validated_at: new Date().toISOString(),
|
||||
});
|
||||
await emitAudit({
|
||||
schemaValid: true,
|
||||
output: parsed.data,
|
||||
@@ -279,6 +391,18 @@ export async function generateValidatedJson<T>({
|
||||
return parsed.data;
|
||||
}
|
||||
const zodSummary = summarizeZodError(parsed.error);
|
||||
const validationIssues = parsed.error.issues.map((issue) => {
|
||||
const path = issue.path.length > 0 ? issue.path.join(".") : "<root>";
|
||||
return `${path}: ${issue.message}`;
|
||||
});
|
||||
await emitTrace(tracedInput, {
|
||||
type: "validated",
|
||||
call_id: traceCallId,
|
||||
schema_name: schemaName,
|
||||
schema_valid: false,
|
||||
validation_issues: validationIssues,
|
||||
validated_at: new Date().toISOString(),
|
||||
});
|
||||
console.warn(
|
||||
`[llm:validated] task=${task} ok=false zod_error=${quoteLogValue(zodSummary)}`,
|
||||
);
|
||||
@@ -287,16 +411,29 @@ export async function generateValidatedJson<T>({
|
||||
output: generated,
|
||||
errorSummary: zodSummary,
|
||||
});
|
||||
throw new LlmValidationError(
|
||||
const validationError = new LlmValidationError(
|
||||
`LLM response failed schema validation: ${zodSummary}`,
|
||||
task,
|
||||
);
|
||||
await emitTrace(
|
||||
tracedInput,
|
||||
failedEvent(traceCallId, "schema_validation", validationError, startedAt),
|
||||
);
|
||||
throw validationError;
|
||||
} catch (error) {
|
||||
if (error instanceof LlmValidationError) {
|
||||
throw error;
|
||||
}
|
||||
console.info(`[llm:validated] task=${task} ok=false reason=provider_error`);
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
const message = redactConfiguredSecrets(
|
||||
error instanceof Error ? error.message : String(error),
|
||||
);
|
||||
if (!usesDefaultGenerator) {
|
||||
await emitTrace(
|
||||
tracedInput,
|
||||
failedEvent(traceCallId, "provider", error, startedAt),
|
||||
);
|
||||
}
|
||||
await emitAudit({
|
||||
schemaValid: false,
|
||||
output: null,
|
||||
@@ -312,6 +449,8 @@ export async function generateValidatedJson<T>({
|
||||
}
|
||||
|
||||
function normalizeLlmError(error: unknown) {
|
||||
const message = error instanceof Error ? error.message : "Unknown LLM error";
|
||||
const message = redactConfiguredSecrets(
|
||||
error instanceof Error ? error.message : "Unknown LLM error",
|
||||
);
|
||||
return new Error(`LLM provider error: ${message}`);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,191 @@
|
||||
import type { LlmTraceRepository } from "./trace-repository";
|
||||
import type {
|
||||
LlmBusinessStatus,
|
||||
LlmProviderName,
|
||||
LlmTaskName,
|
||||
LlmTraceCall,
|
||||
LlmTraceCallStatus,
|
||||
LlmTraceCompleteness,
|
||||
LlmTraceErrorType,
|
||||
LlmTraceRunStatus,
|
||||
LlmTraceWorkflowStage,
|
||||
} from "./trace-types";
|
||||
|
||||
interface TraceRunRow {
|
||||
job_id: string;
|
||||
case_id: string | null;
|
||||
status: LlmTraceRunStatus;
|
||||
current_stage: LlmTraceWorkflowStage;
|
||||
trace_completeness: LlmTraceCompleteness;
|
||||
error_stage: string | null;
|
||||
error_summary: string | null;
|
||||
started_at: string;
|
||||
finished_at: string | null;
|
||||
updated_at: string;
|
||||
}
|
||||
|
||||
interface TraceCallRow {
|
||||
call_id: string;
|
||||
job_id: string;
|
||||
sequence: number;
|
||||
task: LlmTaskName;
|
||||
workflow_stage: LlmTraceWorkflowStage;
|
||||
rewrite_round: number | null;
|
||||
provider: LlmProviderName;
|
||||
model: string;
|
||||
status: LlmTraceCallStatus;
|
||||
request_object_key: string | null;
|
||||
response_object_key: string | null;
|
||||
token_usage: string | null;
|
||||
schema_name: string | null;
|
||||
schema_valid: 0 | 1 | null;
|
||||
validation_issues: string;
|
||||
business_status: LlmBusinessStatus | null;
|
||||
duration_ms: number | null;
|
||||
started_at: string;
|
||||
responded_at: string | null;
|
||||
validated_at: string | null;
|
||||
failed_at: string | null;
|
||||
error_type: LlmTraceErrorType | null;
|
||||
error_summary: string | null;
|
||||
}
|
||||
|
||||
const putRunSql = `
|
||||
insert into llm_trace_runs (
|
||||
job_id, case_id, status, current_stage, trace_completeness,
|
||||
error_stage, error_summary, started_at, finished_at, updated_at
|
||||
) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
on conflict(job_id) do update set
|
||||
case_id = excluded.case_id,
|
||||
status = excluded.status,
|
||||
current_stage = excluded.current_stage,
|
||||
trace_completeness = excluded.trace_completeness,
|
||||
error_stage = excluded.error_stage,
|
||||
error_summary = excluded.error_summary,
|
||||
started_at = excluded.started_at,
|
||||
finished_at = excluded.finished_at,
|
||||
updated_at = excluded.updated_at
|
||||
`;
|
||||
|
||||
const putCallSql = `
|
||||
insert into llm_trace_calls (
|
||||
call_id, job_id, sequence, task, workflow_stage, rewrite_round,
|
||||
provider, model, status, request_object_key, response_object_key,
|
||||
token_usage, schema_name, schema_valid, validation_issues,
|
||||
business_status, duration_ms, started_at, responded_at, validated_at,
|
||||
failed_at, error_type, error_summary
|
||||
) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
on conflict(call_id) do update set
|
||||
sequence = excluded.sequence,
|
||||
task = excluded.task,
|
||||
workflow_stage = excluded.workflow_stage,
|
||||
rewrite_round = excluded.rewrite_round,
|
||||
provider = excluded.provider,
|
||||
model = excluded.model,
|
||||
status = excluded.status,
|
||||
request_object_key = excluded.request_object_key,
|
||||
response_object_key = excluded.response_object_key,
|
||||
token_usage = excluded.token_usage,
|
||||
schema_name = excluded.schema_name,
|
||||
schema_valid = excluded.schema_valid,
|
||||
validation_issues = excluded.validation_issues,
|
||||
business_status = excluded.business_status,
|
||||
duration_ms = excluded.duration_ms,
|
||||
started_at = excluded.started_at,
|
||||
responded_at = excluded.responded_at,
|
||||
validated_at = excluded.validated_at,
|
||||
failed_at = excluded.failed_at,
|
||||
error_type = excluded.error_type,
|
||||
error_summary = excluded.error_summary
|
||||
`;
|
||||
|
||||
function toTraceCall(row: TraceCallRow): LlmTraceCall {
|
||||
return {
|
||||
...row,
|
||||
token_usage: row.token_usage == null
|
||||
? null
|
||||
: JSON.parse(row.token_usage) as Record<string, number>,
|
||||
schema_valid: row.schema_valid == null ? null : row.schema_valid === 1,
|
||||
validation_issues: JSON.parse(row.validation_issues) as string[],
|
||||
};
|
||||
}
|
||||
|
||||
export function createD1TraceRepository(db: D1Database): LlmTraceRepository {
|
||||
return {
|
||||
async putRun(run) {
|
||||
await db.prepare(putRunSql).bind(
|
||||
run.job_id,
|
||||
run.case_id,
|
||||
run.status,
|
||||
run.current_stage,
|
||||
run.trace_completeness,
|
||||
run.error_stage,
|
||||
run.error_summary,
|
||||
run.started_at,
|
||||
run.finished_at,
|
||||
run.updated_at,
|
||||
).run();
|
||||
},
|
||||
async putCall(call) {
|
||||
await db.prepare(putCallSql).bind(
|
||||
call.call_id,
|
||||
call.job_id,
|
||||
call.sequence,
|
||||
call.task,
|
||||
call.workflow_stage,
|
||||
call.rewrite_round,
|
||||
call.provider,
|
||||
call.model,
|
||||
call.status,
|
||||
call.request_object_key,
|
||||
call.response_object_key,
|
||||
call.token_usage == null ? null : JSON.stringify(call.token_usage),
|
||||
call.schema_name,
|
||||
call.schema_valid == null ? null : Number(call.schema_valid),
|
||||
JSON.stringify(call.validation_issues),
|
||||
call.business_status,
|
||||
call.duration_ms,
|
||||
call.started_at,
|
||||
call.responded_at,
|
||||
call.validated_at,
|
||||
call.failed_at,
|
||||
call.error_type,
|
||||
call.error_summary,
|
||||
).run();
|
||||
},
|
||||
async getRun(jobId) {
|
||||
return await db
|
||||
.prepare("select * from llm_trace_runs where job_id = ?")
|
||||
.bind(jobId)
|
||||
.first<TraceRunRow>() ?? null;
|
||||
},
|
||||
async getLatestRun() {
|
||||
return await db.prepare(`
|
||||
select * from llm_trace_runs
|
||||
order by case when status = 'running' then 0 else 1 end, updated_at desc
|
||||
limit 1
|
||||
`).first<TraceRunRow>() ?? null;
|
||||
},
|
||||
async listCalls(jobId) {
|
||||
const result = await db
|
||||
.prepare("select * from llm_trace_calls where job_id = ? order by sequence")
|
||||
.bind(jobId)
|
||||
.all<TraceCallRow>();
|
||||
return result.results.map(toTraceCall);
|
||||
},
|
||||
async listTerminalRunsExcept(jobId) {
|
||||
const result = await db.prepare(`
|
||||
select * from llm_trace_runs
|
||||
where status <> 'running' and job_id <> ?
|
||||
order by finished_at desc, updated_at desc
|
||||
`).bind(jobId).all<TraceRunRow>();
|
||||
return result.results;
|
||||
},
|
||||
async deleteRun(jobId) {
|
||||
await db
|
||||
.prepare("delete from llm_trace_runs where job_id = ?")
|
||||
.bind(jobId)
|
||||
.run();
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,194 @@
|
||||
import { createDatabase, getDefaultDatabasePath } from "../db/connection";
|
||||
import { initializeSchema } from "../db/schema";
|
||||
import type { LlmTraceRepository } from "./trace-repository";
|
||||
import type {
|
||||
LlmBusinessStatus,
|
||||
LlmProviderName,
|
||||
LlmTaskName,
|
||||
LlmTraceCall,
|
||||
LlmTraceCallStatus,
|
||||
LlmTraceCompleteness,
|
||||
LlmTraceErrorType,
|
||||
LlmTraceRunStatus,
|
||||
LlmTraceWorkflowStage,
|
||||
} from "./trace-types";
|
||||
|
||||
interface TraceRunRow {
|
||||
job_id: string;
|
||||
case_id: string | null;
|
||||
status: LlmTraceRunStatus;
|
||||
current_stage: LlmTraceWorkflowStage;
|
||||
trace_completeness: LlmTraceCompleteness;
|
||||
error_stage: string | null;
|
||||
error_summary: string | null;
|
||||
started_at: string;
|
||||
finished_at: string | null;
|
||||
updated_at: string;
|
||||
}
|
||||
|
||||
interface TraceCallRow {
|
||||
call_id: string;
|
||||
job_id: string;
|
||||
sequence: number;
|
||||
task: LlmTaskName;
|
||||
workflow_stage: LlmTraceWorkflowStage;
|
||||
rewrite_round: number | null;
|
||||
provider: LlmProviderName;
|
||||
model: string;
|
||||
status: LlmTraceCallStatus;
|
||||
request_object_key: string | null;
|
||||
response_object_key: string | null;
|
||||
token_usage: string | null;
|
||||
schema_name: string | null;
|
||||
schema_valid: 0 | 1 | null;
|
||||
validation_issues: string;
|
||||
business_status: LlmBusinessStatus | null;
|
||||
duration_ms: number | null;
|
||||
started_at: string;
|
||||
responded_at: string | null;
|
||||
validated_at: string | null;
|
||||
failed_at: string | null;
|
||||
error_type: LlmTraceErrorType | null;
|
||||
error_summary: string | null;
|
||||
}
|
||||
|
||||
const putRunSql = `
|
||||
insert into llm_trace_runs (
|
||||
job_id, case_id, status, current_stage, trace_completeness,
|
||||
error_stage, error_summary, started_at, finished_at, updated_at
|
||||
) values (
|
||||
@job_id, @case_id, @status, @current_stage, @trace_completeness,
|
||||
@error_stage, @error_summary, @started_at, @finished_at, @updated_at
|
||||
)
|
||||
on conflict(job_id) do update set
|
||||
case_id = excluded.case_id,
|
||||
status = excluded.status,
|
||||
current_stage = excluded.current_stage,
|
||||
trace_completeness = excluded.trace_completeness,
|
||||
error_stage = excluded.error_stage,
|
||||
error_summary = excluded.error_summary,
|
||||
started_at = excluded.started_at,
|
||||
finished_at = excluded.finished_at,
|
||||
updated_at = excluded.updated_at
|
||||
`;
|
||||
|
||||
const putCallSql = `
|
||||
insert into llm_trace_calls (
|
||||
call_id, job_id, sequence, task, workflow_stage, rewrite_round,
|
||||
provider, model, status, request_object_key, response_object_key,
|
||||
token_usage, schema_name, schema_valid, validation_issues,
|
||||
business_status, duration_ms, started_at, responded_at, validated_at,
|
||||
failed_at, error_type, error_summary
|
||||
) values (
|
||||
@call_id, @job_id, @sequence, @task, @workflow_stage, @rewrite_round,
|
||||
@provider, @model, @status, @request_object_key, @response_object_key,
|
||||
@token_usage, @schema_name, @schema_valid, @validation_issues,
|
||||
@business_status, @duration_ms, @started_at, @responded_at, @validated_at,
|
||||
@failed_at, @error_type, @error_summary
|
||||
)
|
||||
on conflict(call_id) do update set
|
||||
sequence = excluded.sequence,
|
||||
task = excluded.task,
|
||||
workflow_stage = excluded.workflow_stage,
|
||||
rewrite_round = excluded.rewrite_round,
|
||||
provider = excluded.provider,
|
||||
model = excluded.model,
|
||||
status = excluded.status,
|
||||
request_object_key = excluded.request_object_key,
|
||||
response_object_key = excluded.response_object_key,
|
||||
token_usage = excluded.token_usage,
|
||||
schema_name = excluded.schema_name,
|
||||
schema_valid = excluded.schema_valid,
|
||||
validation_issues = excluded.validation_issues,
|
||||
business_status = excluded.business_status,
|
||||
duration_ms = excluded.duration_ms,
|
||||
started_at = excluded.started_at,
|
||||
responded_at = excluded.responded_at,
|
||||
validated_at = excluded.validated_at,
|
||||
failed_at = excluded.failed_at,
|
||||
error_type = excluded.error_type,
|
||||
error_summary = excluded.error_summary
|
||||
`;
|
||||
|
||||
function toCallParams(call: LlmTraceCall) {
|
||||
return {
|
||||
...call,
|
||||
token_usage: call.token_usage == null ? null : JSON.stringify(call.token_usage),
|
||||
schema_valid: call.schema_valid == null ? null : Number(call.schema_valid),
|
||||
validation_issues: JSON.stringify(call.validation_issues),
|
||||
};
|
||||
}
|
||||
|
||||
function toTraceCall(row: TraceCallRow): LlmTraceCall {
|
||||
return {
|
||||
...row,
|
||||
token_usage: row.token_usage == null
|
||||
? null
|
||||
: JSON.parse(row.token_usage) as Record<string, number>,
|
||||
schema_valid: row.schema_valid == null ? null : row.schema_valid === 1,
|
||||
validation_issues: JSON.parse(row.validation_issues) as string[],
|
||||
};
|
||||
}
|
||||
|
||||
export function createSqliteTraceRepository(
|
||||
dbPath = getDefaultDatabasePath(),
|
||||
): LlmTraceRepository {
|
||||
function withDb<T>(action: (db: ReturnType<typeof createDatabase>) => T) {
|
||||
const db = createDatabase(dbPath);
|
||||
initializeSchema(db);
|
||||
try {
|
||||
return action(db);
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
async putRun(run) {
|
||||
withDb((db) => db.prepare(putRunSql).run(run));
|
||||
},
|
||||
async putCall(call) {
|
||||
withDb((db) => db.prepare(putCallSql).run(toCallParams(call)));
|
||||
},
|
||||
async getRun(jobId) {
|
||||
return withDb((db) => {
|
||||
const row = db
|
||||
.prepare("select * from llm_trace_runs where job_id = ?")
|
||||
.get(jobId) as TraceRunRow | undefined;
|
||||
return row ?? null;
|
||||
});
|
||||
},
|
||||
async getLatestRun() {
|
||||
return withDb((db) => {
|
||||
const row = db.prepare(`
|
||||
select * from llm_trace_runs
|
||||
order by case when status = 'running' then 0 else 1 end, updated_at desc
|
||||
limit 1
|
||||
`).get() as TraceRunRow | undefined;
|
||||
return row ?? null;
|
||||
});
|
||||
},
|
||||
async listCalls(jobId) {
|
||||
return withDb((db) => {
|
||||
const rows = db
|
||||
.prepare("select * from llm_trace_calls where job_id = ? order by sequence")
|
||||
.all(jobId) as TraceCallRow[];
|
||||
return rows.map(toTraceCall);
|
||||
});
|
||||
},
|
||||
async listTerminalRunsExcept(jobId) {
|
||||
return withDb((db) => db
|
||||
.prepare(`
|
||||
select * from llm_trace_runs
|
||||
where status <> 'running' and job_id <> ?
|
||||
order by finished_at desc, updated_at desc
|
||||
`)
|
||||
.all(jobId) as TraceRunRow[]);
|
||||
},
|
||||
async deleteRun(jobId) {
|
||||
withDb((db) => db
|
||||
.prepare("delete from llm_trace_runs where job_id = ?")
|
||||
.run(jobId));
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
import { NextResponse } from "next/server";
|
||||
|
||||
import { getLlmTracePayloadStoreFromRuntime } from "./trace-payload-store";
|
||||
import { getLlmTraceRepositoryFromRuntime } from "./trace-repository";
|
||||
import type {
|
||||
LlmTraceCall,
|
||||
LlmTraceCallPublic,
|
||||
LlmTraceManifest,
|
||||
} from "./trace-types";
|
||||
|
||||
export function noStoreJson(body: unknown, status = 200) {
|
||||
return NextResponse.json(body, {
|
||||
status,
|
||||
headers: { "cache-control": "no-store" },
|
||||
});
|
||||
}
|
||||
|
||||
export function noStoreResponse<T extends Response>(response: T): T {
|
||||
response.headers.set("cache-control", "no-store");
|
||||
return response;
|
||||
}
|
||||
|
||||
export function toPublicTraceCall(call: LlmTraceCall): LlmTraceCallPublic {
|
||||
const {
|
||||
request_object_key: requestObjectKey,
|
||||
response_object_key: responseObjectKey,
|
||||
...publicFields
|
||||
} = call;
|
||||
return {
|
||||
...publicFields,
|
||||
request_available: Boolean(requestObjectKey),
|
||||
response_available: Boolean(responseObjectKey),
|
||||
};
|
||||
}
|
||||
|
||||
export async function getTraceManifest(
|
||||
jobId: string,
|
||||
): Promise<LlmTraceManifest | null> {
|
||||
const repository = getLlmTraceRepositoryFromRuntime();
|
||||
const run = await repository.getRun(jobId);
|
||||
if (!run) return null;
|
||||
const calls = await repository.listCalls(jobId);
|
||||
return { run, calls: calls.map(toPublicTraceCall) };
|
||||
}
|
||||
|
||||
export async function readTracePayload({
|
||||
jobId,
|
||||
callId,
|
||||
kind,
|
||||
}: {
|
||||
jobId: string;
|
||||
callId: string;
|
||||
kind: "request" | "response";
|
||||
}) {
|
||||
const repository = getLlmTraceRepositoryFromRuntime();
|
||||
const call = (await repository.listCalls(jobId)).find(
|
||||
(candidate) => candidate.call_id === callId,
|
||||
);
|
||||
if (!call) return noStoreJson({ error: "追踪调用不存在" }, 404);
|
||||
|
||||
const key = kind === "request"
|
||||
? call.request_object_key
|
||||
: call.response_object_key;
|
||||
if (!key) {
|
||||
if (kind === "response" && call.status === "started") {
|
||||
return noStoreJson({ state: "waiting" }, 202);
|
||||
}
|
||||
return noStoreJson({
|
||||
error: kind === "response" ? "该调用未产生响应" : "追踪请求正文不存在",
|
||||
}, 404);
|
||||
}
|
||||
|
||||
const payload = await getLlmTracePayloadStoreFromRuntime().getJson(key);
|
||||
return payload == null
|
||||
? noStoreJson({ error: "追踪正文不存在" }, 404)
|
||||
: noStoreJson(payload);
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
import {
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
readFileSync,
|
||||
rmSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { dirname, join, relative, resolve } from "node:path";
|
||||
|
||||
import { getAppDataDir } from "../db/connection";
|
||||
import { getAppCloudflareEnv } from "../runtime/cloudflare";
|
||||
|
||||
const JSON_CONTENT_TYPE = "application/json; charset=utf-8";
|
||||
|
||||
export interface LlmTracePayloadStore {
|
||||
putJson(key: string, value: unknown): Promise<void>;
|
||||
getJson(key: string): Promise<unknown | null>;
|
||||
deleteJob(jobId: string): Promise<void>;
|
||||
}
|
||||
|
||||
function assertSafeSegment(value: string, label: string) {
|
||||
if (!/^[A-Za-z0-9_-]+$/.test(value)) {
|
||||
throw new Error(`Invalid LLM trace ${label}`);
|
||||
}
|
||||
}
|
||||
|
||||
export function tracePayloadKey(
|
||||
jobId: string,
|
||||
callId: string,
|
||||
kind: "request" | "response",
|
||||
) {
|
||||
assertSafeSegment(jobId, "job id");
|
||||
assertSafeSegment(callId, "call id");
|
||||
return `llm-traces/${jobId}/${callId}/${kind}.json`;
|
||||
}
|
||||
|
||||
function localPathForKey(dataDir: string, key: string) {
|
||||
const traceRoot = resolve(dataDir, "llm-traces");
|
||||
const path = resolve(dataDir, key);
|
||||
const pathFromTraceRoot = relative(traceRoot, path);
|
||||
if (
|
||||
pathFromTraceRoot.startsWith("..") ||
|
||||
pathFromTraceRoot === "" ||
|
||||
key.startsWith("/")
|
||||
) {
|
||||
throw new Error("Invalid LLM trace payload key");
|
||||
}
|
||||
return path;
|
||||
}
|
||||
|
||||
export function createLocalTracePayloadStore(
|
||||
dataDir = getAppDataDir(),
|
||||
): LlmTracePayloadStore {
|
||||
return {
|
||||
async putJson(key, value) {
|
||||
const path = localPathForKey(dataDir, key);
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
writeFileSync(path, JSON.stringify(value), "utf8");
|
||||
},
|
||||
async getJson(key) {
|
||||
const path = localPathForKey(dataDir, key);
|
||||
if (!existsSync(path)) return null;
|
||||
return JSON.parse(readFileSync(path, "utf8")) as unknown;
|
||||
},
|
||||
async deleteJob(jobId) {
|
||||
assertSafeSegment(jobId, "job id");
|
||||
rmSync(join(dataDir, "llm-traces", jobId), {
|
||||
recursive: true,
|
||||
force: true,
|
||||
});
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function createR2TracePayloadStore(
|
||||
bucket: R2Bucket,
|
||||
): LlmTracePayloadStore {
|
||||
return {
|
||||
async putJson(key, value) {
|
||||
await bucket.put(key, JSON.stringify(value), {
|
||||
httpMetadata: { contentType: JSON_CONTENT_TYPE },
|
||||
});
|
||||
},
|
||||
async getJson(key) {
|
||||
const object = await bucket.get(key);
|
||||
return object ? await object.json() : null;
|
||||
},
|
||||
async deleteJob(jobId) {
|
||||
assertSafeSegment(jobId, "job id");
|
||||
const prefix = `llm-traces/${jobId}/`;
|
||||
let cursor: string | undefined;
|
||||
do {
|
||||
const page = await bucket.list({ prefix, cursor });
|
||||
const keys = page.objects.map((object) => object.key);
|
||||
if (keys.length > 0) await bucket.delete(keys);
|
||||
cursor = page.truncated ? page.cursor : undefined;
|
||||
} while (cursor);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function getLlmTracePayloadStoreFromRuntime(): LlmTracePayloadStore {
|
||||
if (process.env.APP_RUNTIME === "cloudflare") {
|
||||
const bucket = getAppCloudflareEnv()?.EXPORT_BUCKET;
|
||||
if (!bucket) {
|
||||
throw new Error("Cloudflare R2 binding EXPORT_BUCKET is required");
|
||||
}
|
||||
return createR2TracePayloadStore(bucket);
|
||||
}
|
||||
return createLocalTracePayloadStore(getAppDataDir());
|
||||
}
|
||||
@@ -0,0 +1,404 @@
|
||||
import type { OptimizationStreamEvent } from "../workflow/stream-events";
|
||||
import {
|
||||
tracePayloadKey,
|
||||
type LlmTracePayloadStore,
|
||||
} from "./trace-payload-store";
|
||||
import type { LlmTraceRepository } from "./trace-repository";
|
||||
import type {
|
||||
LlmClientTraceEvent,
|
||||
LlmClientTraceHandler,
|
||||
LlmTraceCall,
|
||||
LlmTraceRun,
|
||||
LlmTraceRunStatus,
|
||||
LlmTraceStreamEvent,
|
||||
LlmTraceWorkflowStage,
|
||||
} from "./trace-types";
|
||||
|
||||
export interface LlmTraceRecorder {
|
||||
onLlmEvent: LlmClientTraceHandler;
|
||||
onWorkflowEvent(event: OptimizationStreamEvent): Promise<void>;
|
||||
finish(input: {
|
||||
status: Exclude<LlmTraceRunStatus, "running">;
|
||||
errorStage?: string;
|
||||
errorSummary?: string;
|
||||
}): Promise<void>;
|
||||
}
|
||||
|
||||
interface CreateLlmTraceRecorderInput {
|
||||
jobId: string;
|
||||
caseId: string | null;
|
||||
repository: LlmTraceRepository;
|
||||
payloadStore: LlmTracePayloadStore;
|
||||
publish: (event: LlmTraceStreamEvent) => void | Promise<void>;
|
||||
}
|
||||
|
||||
function nowIso() {
|
||||
return new Date().toISOString();
|
||||
}
|
||||
|
||||
function createRunningRun(jobId: string, caseId: string | null): LlmTraceRun {
|
||||
const timestamp = nowIso();
|
||||
return {
|
||||
job_id: jobId,
|
||||
case_id: caseId,
|
||||
status: "running",
|
||||
current_stage: "input",
|
||||
trace_completeness: "complete",
|
||||
error_stage: null,
|
||||
error_summary: null,
|
||||
started_at: timestamp,
|
||||
finished_at: null,
|
||||
updated_at: timestamp,
|
||||
};
|
||||
}
|
||||
|
||||
export function safeTraceError(error: unknown) {
|
||||
if (error instanceof Error) return `${error.name}: ${error.message}`;
|
||||
return String(error);
|
||||
}
|
||||
|
||||
export function createNoopLlmTraceRecorder(): LlmTraceRecorder {
|
||||
return {
|
||||
onLlmEvent: async () => undefined,
|
||||
onWorkflowEvent: async () => undefined,
|
||||
finish: async () => undefined,
|
||||
};
|
||||
}
|
||||
|
||||
function tokenUsageFromResponse(response: unknown) {
|
||||
if (!response || typeof response !== "object") return null;
|
||||
const usage = (response as { usage?: unknown }).usage;
|
||||
if (!usage || typeof usage !== "object") return null;
|
||||
return Object.fromEntries(
|
||||
Object.entries(usage).filter(
|
||||
(entry): entry is [string, number] => typeof entry[1] === "number",
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
function stageForWorkflowEvent(
|
||||
event: OptimizationStreamEvent,
|
||||
): LlmTraceWorkflowStage | null {
|
||||
switch (event.type) {
|
||||
case "job_created":
|
||||
return "input";
|
||||
case "fact_card_ready":
|
||||
return "fact_card";
|
||||
case "draft_started":
|
||||
case "draft_ready":
|
||||
return "draft";
|
||||
case "qa_started":
|
||||
case "qa_ready":
|
||||
return "qa";
|
||||
case "rewrite_started":
|
||||
case "rewrite_ready":
|
||||
return "rewrite";
|
||||
case "final_ready":
|
||||
return "final";
|
||||
case "failed":
|
||||
return event.stage === "job" ? "input" : event.stage;
|
||||
default:
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function isTraceStage(value: string): value is LlmTraceWorkflowStage {
|
||||
return [
|
||||
"unknown",
|
||||
"input",
|
||||
"fact_card",
|
||||
"draft",
|
||||
"qa",
|
||||
"rewrite",
|
||||
"final",
|
||||
].includes(value);
|
||||
}
|
||||
|
||||
export async function createLlmTraceRecorder({
|
||||
jobId,
|
||||
caseId,
|
||||
repository,
|
||||
payloadStore,
|
||||
publish,
|
||||
}: CreateLlmTraceRecorderInput): Promise<LlmTraceRecorder> {
|
||||
const calls = new Map<string, LlmTraceCall>();
|
||||
let sequence = 0;
|
||||
let run = createRunningRun(jobId, caseId);
|
||||
await repository.putRun(run);
|
||||
|
||||
async function publishSafely(event: LlmTraceStreamEvent) {
|
||||
try {
|
||||
await publish(event);
|
||||
} catch {
|
||||
// A disconnected observer must never fail the article workflow.
|
||||
}
|
||||
}
|
||||
|
||||
async function warn(error: unknown) {
|
||||
run = {
|
||||
...run,
|
||||
trace_completeness: "incomplete",
|
||||
updated_at: nowIso(),
|
||||
};
|
||||
try {
|
||||
await repository.putRun(run);
|
||||
} catch {
|
||||
// Keep the in-memory incomplete state even when the index is unavailable.
|
||||
}
|
||||
await publishSafely({
|
||||
type: "trace_warning",
|
||||
job_id: jobId,
|
||||
trace_completeness: "incomplete",
|
||||
error_summary: safeTraceError(error),
|
||||
});
|
||||
}
|
||||
|
||||
async function saveCall(call: LlmTraceCall) {
|
||||
calls.set(call.call_id, call);
|
||||
await repository.putCall(call);
|
||||
}
|
||||
|
||||
async function applyStartedEvent(
|
||||
event: Extract<LlmClientTraceEvent, { type: "started" }>,
|
||||
) {
|
||||
const requestKey = tracePayloadKey(jobId, event.call_id, "request");
|
||||
let storedRequestKey: string | null = null;
|
||||
let traceError: unknown;
|
||||
try {
|
||||
await payloadStore.putJson(requestKey, event.request);
|
||||
storedRequestKey = requestKey;
|
||||
} catch (error) {
|
||||
traceError = error;
|
||||
}
|
||||
|
||||
const call: LlmTraceCall = {
|
||||
call_id: event.call_id,
|
||||
job_id: jobId,
|
||||
sequence: ++sequence,
|
||||
task: event.task,
|
||||
workflow_stage: event.context.workflow_stage,
|
||||
rewrite_round: event.context.rewrite_round ?? null,
|
||||
provider: event.provider,
|
||||
model: event.model,
|
||||
status: "started",
|
||||
request_object_key: storedRequestKey,
|
||||
response_object_key: null,
|
||||
token_usage: null,
|
||||
schema_name: event.context.schema_name ?? null,
|
||||
schema_valid: null,
|
||||
validation_issues: [],
|
||||
business_status: null,
|
||||
duration_ms: null,
|
||||
started_at: event.started_at,
|
||||
responded_at: null,
|
||||
validated_at: null,
|
||||
failed_at: null,
|
||||
error_type: null,
|
||||
error_summary: null,
|
||||
};
|
||||
|
||||
try {
|
||||
await saveCall(call);
|
||||
} catch (error) {
|
||||
traceError ??= error;
|
||||
calls.set(call.call_id, call);
|
||||
}
|
||||
await publishSafely({
|
||||
type: "llm_call_started",
|
||||
job_id: jobId,
|
||||
call_id: call.call_id,
|
||||
sequence: call.sequence,
|
||||
task: call.task,
|
||||
workflow_stage: call.workflow_stage,
|
||||
rewrite_round: call.rewrite_round,
|
||||
provider: call.provider,
|
||||
model: call.model,
|
||||
started_at: call.started_at,
|
||||
request_available: storedRequestKey !== null,
|
||||
});
|
||||
if (traceError) await warn(traceError);
|
||||
}
|
||||
|
||||
async function applyRespondedEvent(
|
||||
event: Extract<LlmClientTraceEvent, { type: "responded" }>,
|
||||
) {
|
||||
const existing = calls.get(event.call_id);
|
||||
if (!existing) throw new Error(`Unknown LLM trace call: ${event.call_id}`);
|
||||
|
||||
const responseKey = tracePayloadKey(jobId, event.call_id, "response");
|
||||
let storedResponseKey: string | null = null;
|
||||
let traceError: unknown;
|
||||
try {
|
||||
await payloadStore.putJson(responseKey, event.response);
|
||||
storedResponseKey = responseKey;
|
||||
} catch (error) {
|
||||
traceError = error;
|
||||
}
|
||||
|
||||
const call: LlmTraceCall = {
|
||||
...existing,
|
||||
status: "responded",
|
||||
response_object_key: storedResponseKey,
|
||||
token_usage: tokenUsageFromResponse(event.response),
|
||||
duration_ms: event.duration_ms,
|
||||
responded_at: event.responded_at,
|
||||
};
|
||||
try {
|
||||
await saveCall(call);
|
||||
} catch (error) {
|
||||
traceError ??= error;
|
||||
calls.set(call.call_id, call);
|
||||
}
|
||||
await publishSafely({
|
||||
type: "llm_call_responded",
|
||||
job_id: jobId,
|
||||
call_id: call.call_id,
|
||||
duration_ms: event.duration_ms,
|
||||
token_usage: call.token_usage,
|
||||
responded_at: event.responded_at,
|
||||
response_available: storedResponseKey !== null,
|
||||
});
|
||||
if (traceError) await warn(traceError);
|
||||
}
|
||||
|
||||
async function applyValidatedEvent(
|
||||
event: Extract<LlmClientTraceEvent, { type: "validated" }>,
|
||||
) {
|
||||
const existing = calls.get(event.call_id);
|
||||
if (!existing) throw new Error(`Unknown LLM trace call: ${event.call_id}`);
|
||||
const call: LlmTraceCall = {
|
||||
...existing,
|
||||
status: "validated",
|
||||
schema_name: event.schema_name,
|
||||
schema_valid: event.schema_valid,
|
||||
validation_issues: event.validation_issues,
|
||||
validated_at: event.validated_at,
|
||||
};
|
||||
await saveCall(call);
|
||||
await publishSafely({
|
||||
type: "llm_call_validated",
|
||||
job_id: jobId,
|
||||
call_id: call.call_id,
|
||||
schema_name: event.schema_name,
|
||||
schema_valid: event.schema_valid,
|
||||
validation_issues: event.validation_issues,
|
||||
validated_at: event.validated_at,
|
||||
});
|
||||
}
|
||||
|
||||
async function applyFailedEvent(
|
||||
event: Extract<LlmClientTraceEvent, { type: "failed" }>,
|
||||
) {
|
||||
const existing = calls.get(event.call_id);
|
||||
if (!existing) throw new Error(`Unknown LLM trace call: ${event.call_id}`);
|
||||
const call: LlmTraceCall = {
|
||||
...existing,
|
||||
status: "failed",
|
||||
duration_ms: event.duration_ms,
|
||||
failed_at: event.failed_at,
|
||||
error_type: event.error_type,
|
||||
error_summary: event.error_summary,
|
||||
};
|
||||
await saveCall(call);
|
||||
await publishSafely({
|
||||
type: "llm_call_failed",
|
||||
job_id: jobId,
|
||||
call_id: call.call_id,
|
||||
error_type: event.error_type,
|
||||
error_summary: event.error_summary,
|
||||
failed_at: event.failed_at,
|
||||
});
|
||||
}
|
||||
|
||||
async function applyLlmEvent(event: LlmClientTraceEvent) {
|
||||
switch (event.type) {
|
||||
case "started":
|
||||
await applyStartedEvent(event);
|
||||
return;
|
||||
case "responded":
|
||||
await applyRespondedEvent(event);
|
||||
return;
|
||||
case "validated":
|
||||
await applyValidatedEvent(event);
|
||||
return;
|
||||
case "failed":
|
||||
await applyFailedEvent(event);
|
||||
}
|
||||
}
|
||||
|
||||
async function applyWorkflowEvent(event: OptimizationStreamEvent) {
|
||||
const stage = stageForWorkflowEvent(event);
|
||||
if (stage) {
|
||||
run = { ...run, current_stage: stage, updated_at: nowIso() };
|
||||
await repository.putRun(run);
|
||||
}
|
||||
|
||||
if (event.type === "qa_ready") {
|
||||
const qualityCall = [...calls.values()]
|
||||
.filter((call) => call.task === "quality_inspector")
|
||||
.sort((left, right) => right.sequence - left.sequence)[0];
|
||||
if (qualityCall) {
|
||||
await saveCall({
|
||||
...qualityCall,
|
||||
business_status: event.qa_report.overall_status,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function finishRun(input: {
|
||||
status: Exclude<LlmTraceRunStatus, "running">;
|
||||
errorStage?: string;
|
||||
errorSummary?: string;
|
||||
}) {
|
||||
const finishedAt = nowIso();
|
||||
run = {
|
||||
...run,
|
||||
status: input.status,
|
||||
current_stage: input.status === "completed"
|
||||
? "final"
|
||||
: input.errorStage && isTraceStage(input.errorStage)
|
||||
? input.errorStage
|
||||
: run.current_stage,
|
||||
error_stage: input.errorStage ?? null,
|
||||
error_summary: input.errorSummary ?? null,
|
||||
finished_at: finishedAt,
|
||||
updated_at: finishedAt,
|
||||
};
|
||||
await repository.putRun(run);
|
||||
|
||||
const expired = await repository.listTerminalRunsExcept(jobId);
|
||||
for (const oldRun of expired) {
|
||||
try {
|
||||
await payloadStore.deleteJob(oldRun.job_id);
|
||||
await repository.deleteRun(oldRun.job_id);
|
||||
} catch (error) {
|
||||
await warn(error);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
onLlmEvent: async (event) => {
|
||||
try {
|
||||
await applyLlmEvent(event);
|
||||
} catch (error) {
|
||||
await warn(error);
|
||||
}
|
||||
},
|
||||
onWorkflowEvent: async (event) => {
|
||||
try {
|
||||
await applyWorkflowEvent(event);
|
||||
} catch (error) {
|
||||
await warn(error);
|
||||
}
|
||||
},
|
||||
finish: async (input) => {
|
||||
try {
|
||||
await finishRun(input);
|
||||
} catch (error) {
|
||||
await warn(error);
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
import { getDefaultDatabasePath } from "../db/connection";
|
||||
import { getAppCloudflareEnv } from "../runtime/cloudflare";
|
||||
import { createD1TraceRepository } from "./d1-trace-repository";
|
||||
import { createSqliteTraceRepository } from "./sqlite-trace-repository";
|
||||
import type { LlmTraceCall, LlmTraceRun } from "./trace-types";
|
||||
|
||||
export interface LlmTraceRepository {
|
||||
putRun(run: LlmTraceRun): Promise<void>;
|
||||
putCall(call: LlmTraceCall): Promise<void>;
|
||||
getRun(jobId: string): Promise<LlmTraceRun | null>;
|
||||
getLatestRun(): Promise<LlmTraceRun | null>;
|
||||
listCalls(jobId: string): Promise<LlmTraceCall[]>;
|
||||
listTerminalRunsExcept(jobId: string): Promise<LlmTraceRun[]>;
|
||||
deleteRun(jobId: string): Promise<void>;
|
||||
}
|
||||
|
||||
export function getLlmTraceRepositoryFromRuntime(): LlmTraceRepository {
|
||||
if (process.env.APP_RUNTIME === "cloudflare") {
|
||||
const env = getAppCloudflareEnv();
|
||||
if (!env?.DB) throw new Error("Cloudflare D1 binding DB is required");
|
||||
return createD1TraceRepository(env.DB);
|
||||
}
|
||||
return createSqliteTraceRepository(getDefaultDatabasePath());
|
||||
}
|
||||
@@ -0,0 +1,181 @@
|
||||
export type LlmProviderName = "deepseek" | "openai";
|
||||
|
||||
export type LlmTaskName =
|
||||
| "unknown"
|
||||
| "fact_extractor"
|
||||
| "article_optimizer"
|
||||
| "quality_inspector"
|
||||
| "targeted_rewriter"
|
||||
| "renwei_copy_optimizer";
|
||||
|
||||
export type LlmTraceWorkflowStage =
|
||||
| "unknown"
|
||||
| "input"
|
||||
| "fact_card"
|
||||
| "draft"
|
||||
| "qa"
|
||||
| "rewrite"
|
||||
| "final";
|
||||
|
||||
export type LlmTraceRunStatus =
|
||||
| "running"
|
||||
| "completed"
|
||||
| "failed"
|
||||
| "interrupted";
|
||||
|
||||
export type LlmTraceCallStatus =
|
||||
| "started"
|
||||
| "responded"
|
||||
| "validated"
|
||||
| "failed";
|
||||
|
||||
export type LlmTraceCompleteness = "complete" | "incomplete";
|
||||
export type LlmBusinessStatus = "pass" | "warn" | "fail";
|
||||
export type LlmTraceErrorType =
|
||||
| "provider"
|
||||
| "json_parse"
|
||||
| "schema_validation";
|
||||
|
||||
export interface LlmTraceContext {
|
||||
workflow_stage: LlmTraceWorkflowStage;
|
||||
rewrite_round?: number;
|
||||
schema_name?: string;
|
||||
}
|
||||
|
||||
export interface LlmTraceRun {
|
||||
job_id: string;
|
||||
case_id: string | null;
|
||||
status: LlmTraceRunStatus;
|
||||
current_stage: LlmTraceWorkflowStage;
|
||||
trace_completeness: LlmTraceCompleteness;
|
||||
error_stage: string | null;
|
||||
error_summary: string | null;
|
||||
started_at: string;
|
||||
finished_at: string | null;
|
||||
updated_at: string;
|
||||
}
|
||||
|
||||
export interface LlmTraceCall {
|
||||
call_id: string;
|
||||
job_id: string;
|
||||
sequence: number;
|
||||
task: LlmTaskName;
|
||||
workflow_stage: LlmTraceWorkflowStage;
|
||||
rewrite_round: number | null;
|
||||
provider: LlmProviderName;
|
||||
model: string;
|
||||
status: LlmTraceCallStatus;
|
||||
request_object_key: string | null;
|
||||
response_object_key: string | null;
|
||||
token_usage: Record<string, number> | null;
|
||||
schema_name: string | null;
|
||||
schema_valid: boolean | null;
|
||||
validation_issues: string[];
|
||||
business_status: LlmBusinessStatus | null;
|
||||
duration_ms: number | null;
|
||||
started_at: string;
|
||||
responded_at: string | null;
|
||||
validated_at: string | null;
|
||||
failed_at: string | null;
|
||||
error_type: LlmTraceErrorType | null;
|
||||
error_summary: string | null;
|
||||
}
|
||||
|
||||
export type LlmTraceCallPublic = Omit<
|
||||
LlmTraceCall,
|
||||
"request_object_key" | "response_object_key"
|
||||
> & {
|
||||
request_available: boolean;
|
||||
response_available: boolean;
|
||||
};
|
||||
|
||||
export interface LlmTraceManifest {
|
||||
run: LlmTraceRun;
|
||||
calls: LlmTraceCallPublic[];
|
||||
}
|
||||
|
||||
export type LlmClientTraceEvent =
|
||||
| {
|
||||
type: "started";
|
||||
call_id: string;
|
||||
task: LlmTaskName;
|
||||
context: LlmTraceContext;
|
||||
provider: LlmProviderName;
|
||||
model: string;
|
||||
request: unknown;
|
||||
started_at: string;
|
||||
}
|
||||
| {
|
||||
type: "responded";
|
||||
call_id: string;
|
||||
response: unknown;
|
||||
duration_ms: number;
|
||||
responded_at: string;
|
||||
}
|
||||
| {
|
||||
type: "validated";
|
||||
call_id: string;
|
||||
schema_name: string;
|
||||
schema_valid: boolean;
|
||||
validation_issues: string[];
|
||||
validated_at: string;
|
||||
}
|
||||
| {
|
||||
type: "failed";
|
||||
call_id: string;
|
||||
error_type: LlmTraceErrorType;
|
||||
error_summary: string;
|
||||
duration_ms: number;
|
||||
failed_at: string;
|
||||
};
|
||||
|
||||
export type LlmClientTraceHandler = (
|
||||
event: LlmClientTraceEvent,
|
||||
) => void | Promise<void>;
|
||||
|
||||
export type LlmTraceStreamEvent =
|
||||
| {
|
||||
type: "llm_call_started";
|
||||
job_id: string;
|
||||
call_id: string;
|
||||
sequence: number;
|
||||
task: LlmTaskName;
|
||||
workflow_stage: LlmTraceWorkflowStage;
|
||||
rewrite_round: number | null;
|
||||
provider: LlmProviderName;
|
||||
model: string;
|
||||
started_at: string;
|
||||
request_available: boolean;
|
||||
}
|
||||
| {
|
||||
type: "llm_call_responded";
|
||||
job_id: string;
|
||||
call_id: string;
|
||||
duration_ms: number;
|
||||
token_usage: Record<string, number> | null;
|
||||
responded_at: string;
|
||||
response_available: boolean;
|
||||
}
|
||||
| {
|
||||
type: "llm_call_validated";
|
||||
job_id: string;
|
||||
call_id: string;
|
||||
schema_name: string;
|
||||
schema_valid: boolean;
|
||||
validation_issues: string[];
|
||||
validated_at: string;
|
||||
}
|
||||
| {
|
||||
type: "llm_call_failed";
|
||||
job_id: string;
|
||||
call_id: string;
|
||||
error_type: LlmTraceErrorType;
|
||||
error_summary: string;
|
||||
failed_at: string;
|
||||
}
|
||||
| {
|
||||
type: "trace_warning";
|
||||
job_id: string;
|
||||
trace_completeness: "incomplete";
|
||||
error_summary: string;
|
||||
};
|
||||
@@ -19,6 +19,28 @@ describe("optimization stream events", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("encodes LLM call metadata without raw request or response bodies", () => {
|
||||
const event: OptimizationStreamEvent = {
|
||||
type: "llm_call_started",
|
||||
job_id: "job_123",
|
||||
call_id: "llmcall_1",
|
||||
sequence: 1,
|
||||
task: "fact_extractor",
|
||||
workflow_stage: "fact_card",
|
||||
rewrite_round: null,
|
||||
provider: "deepseek",
|
||||
model: "deepseek-v4-pro",
|
||||
started_at: "2026-07-16T00:00:00.000Z",
|
||||
request_available: true,
|
||||
};
|
||||
|
||||
const encoded = encodeOptimizationStreamEvent(event);
|
||||
|
||||
expect(JSON.parse(encoded)).toEqual(event);
|
||||
expect(encoded).not.toContain("messages");
|
||||
expect(encoded).not.toContain("Authorization");
|
||||
});
|
||||
|
||||
it("parses chunked NDJSON while preserving incomplete lines", () => {
|
||||
const first = parseOptimizationStreamChunk(
|
||||
"",
|
||||
|
||||
@@ -13,12 +13,14 @@ import {
|
||||
export interface OptimizeArticleInput {
|
||||
input: ArticleInput;
|
||||
factCard: OptimizationFactCard;
|
||||
onTraceEvent?: GenerateInput["onTraceEvent"];
|
||||
onAuditSummary?: GenerateInput["onAuditSummary"];
|
||||
}
|
||||
|
||||
export async function optimizeArticle({
|
||||
input,
|
||||
factCard,
|
||||
onTraceEvent,
|
||||
onAuditSummary,
|
||||
}: OptimizeArticleInput): Promise<OptimizedArticle> {
|
||||
const llmArticle = await generateValidatedJson({
|
||||
@@ -27,6 +29,9 @@ export async function optimizeArticle({
|
||||
prompt: buildArticleOptimizerPrompt(input, factCard),
|
||||
temperature: 0.2,
|
||||
task: "article_optimizer",
|
||||
schemaName: "optimizedArticleSchema",
|
||||
traceStage: "draft",
|
||||
onTraceEvent,
|
||||
onAuditSummary,
|
||||
});
|
||||
|
||||
|
||||
@@ -8,7 +8,10 @@ import {
|
||||
|
||||
export async function extractCandidateFactCard(
|
||||
input: ArticleInput,
|
||||
options: { onAuditSummary?: GenerateInput["onAuditSummary"] } = {},
|
||||
options: {
|
||||
onAuditSummary?: GenerateInput["onAuditSummary"];
|
||||
onTraceEvent?: GenerateInput["onTraceEvent"];
|
||||
} = {},
|
||||
): Promise<CandidateFactCard> {
|
||||
return generateValidatedJson({
|
||||
schema: candidateFactCardSchema,
|
||||
@@ -16,6 +19,9 @@ export async function extractCandidateFactCard(
|
||||
prompt: buildFactExtractorPrompt(input),
|
||||
temperature: 0.1,
|
||||
task: "fact_extractor",
|
||||
schemaName: "candidateFactCardSchema",
|
||||
traceStage: "fact_card",
|
||||
onTraceEvent: options.onTraceEvent,
|
||||
onAuditSummary: options.onAuditSummary,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -43,6 +43,8 @@ export interface InspectQualityInput {
|
||||
factCard: OptimizationFactCard;
|
||||
platform: PublishPlatform;
|
||||
sourceImages: ImageInput[];
|
||||
rewriteRound?: number;
|
||||
onTraceEvent?: GenerateInput["onTraceEvent"];
|
||||
onAuditSummary?: GenerateInput["onAuditSummary"];
|
||||
}
|
||||
|
||||
@@ -72,6 +74,10 @@ export async function inspectQualityWithLlm(
|
||||
}),
|
||||
temperature: 0.1,
|
||||
task: "quality_inspector",
|
||||
schemaName: "llmQaPatchSchema",
|
||||
traceStage: "qa",
|
||||
rewriteRound: input.rewriteRound,
|
||||
onTraceEvent: input.onTraceEvent,
|
||||
onAuditSummary: input.onAuditSummary,
|
||||
});
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ import type {
|
||||
OptimizedArticle,
|
||||
QaReport,
|
||||
} from "../domain/types";
|
||||
import type { LlmTraceStreamEvent } from "../llm/trace-types";
|
||||
|
||||
export type OptimizationStreamStage =
|
||||
| "input"
|
||||
@@ -13,7 +14,7 @@ export type OptimizationStreamStage =
|
||||
| "rewrite"
|
||||
| "final";
|
||||
|
||||
export type OptimizationStreamEvent =
|
||||
type ExistingOptimizationStreamEvents =
|
||||
| {
|
||||
type: "job_created";
|
||||
job: { id: string };
|
||||
@@ -53,6 +54,10 @@ export type OptimizationStreamEvent =
|
||||
error: string;
|
||||
};
|
||||
|
||||
export type OptimizationStreamEvent =
|
||||
| LlmTraceStreamEvent
|
||||
| ExistingOptimizationStreamEvents;
|
||||
|
||||
export function encodeOptimizationStreamEvent(
|
||||
event: OptimizationStreamEvent,
|
||||
) {
|
||||
|
||||
@@ -13,6 +13,7 @@ export interface RunStreamingOptimizationWorkflowInput {
|
||||
input: ArticleInput;
|
||||
factCard: OptimizationFactCard;
|
||||
onEvent: (event: OptimizationStreamEvent) => void | Promise<void>;
|
||||
onTraceEvent?: GenerateInput["onTraceEvent"];
|
||||
onAuditSummary?: GenerateInput["onAuditSummary"];
|
||||
}
|
||||
|
||||
@@ -21,6 +22,7 @@ export async function runStreamingOptimizationWorkflow({
|
||||
input,
|
||||
factCard,
|
||||
onEvent,
|
||||
onTraceEvent,
|
||||
onAuditSummary,
|
||||
}: RunStreamingOptimizationWorkflowInput) {
|
||||
const processSummary: ProcessSummaryStep[] = [];
|
||||
@@ -31,7 +33,12 @@ export async function runStreamingOptimizationWorkflow({
|
||||
message: "正在生成优化草稿",
|
||||
});
|
||||
let stageStartedAt = Date.now();
|
||||
let article = await optimizeArticle({ input, factCard, onAuditSummary });
|
||||
let article = await optimizeArticle({
|
||||
input,
|
||||
factCard,
|
||||
onTraceEvent,
|
||||
onAuditSummary,
|
||||
});
|
||||
processSummary.push(
|
||||
createProcessStep({
|
||||
stage: "draft",
|
||||
@@ -54,6 +61,8 @@ export async function runStreamingOptimizationWorkflow({
|
||||
factCard,
|
||||
platform: input.platform,
|
||||
sourceImages: input.images,
|
||||
rewriteRound: 0,
|
||||
onTraceEvent,
|
||||
onAuditSummary,
|
||||
});
|
||||
processSummary.push(
|
||||
@@ -81,6 +90,8 @@ export async function runStreamingOptimizationWorkflow({
|
||||
article,
|
||||
factCard,
|
||||
failedChecks,
|
||||
rewriteRound: nextRound,
|
||||
onTraceEvent,
|
||||
onAuditSummary,
|
||||
});
|
||||
rewriteRounds = nextRound;
|
||||
@@ -112,6 +123,8 @@ export async function runStreamingOptimizationWorkflow({
|
||||
factCard,
|
||||
platform: input.platform,
|
||||
sourceImages: input.images,
|
||||
rewriteRound: nextRound,
|
||||
onTraceEvent,
|
||||
onAuditSummary,
|
||||
});
|
||||
processSummary.push(
|
||||
|
||||
@@ -10,6 +10,8 @@ export interface RewriteFailedSectionsInput {
|
||||
article: OptimizedArticle;
|
||||
factCard: OptimizationFactCard;
|
||||
failedChecks: QaCheck[];
|
||||
rewriteRound?: number;
|
||||
onTraceEvent?: GenerateInput["onTraceEvent"];
|
||||
onAuditSummary?: GenerateInput["onAuditSummary"];
|
||||
}
|
||||
|
||||
@@ -17,6 +19,8 @@ export async function rewriteFailedSections({
|
||||
article,
|
||||
factCard,
|
||||
failedChecks,
|
||||
rewriteRound,
|
||||
onTraceEvent,
|
||||
onAuditSummary,
|
||||
}: RewriteFailedSectionsInput): Promise<OptimizedArticle> {
|
||||
const llmArticle = await generateValidatedJson({
|
||||
@@ -25,6 +29,10 @@ export async function rewriteFailedSections({
|
||||
prompt: buildTargetedRewritePrompt({ article, factCard, failedChecks }),
|
||||
temperature: 0.15,
|
||||
task: "targeted_rewriter",
|
||||
schemaName: "optimizedArticleSchema",
|
||||
traceStage: "rewrite",
|
||||
rewriteRound,
|
||||
onTraceEvent,
|
||||
onAuditSummary,
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,179 @@
|
||||
import { expect, test } from "@playwright/test";
|
||||
|
||||
const jobId = "job_architecture";
|
||||
const callId = "llmcall_draft";
|
||||
|
||||
const run = {
|
||||
job_id: jobId,
|
||||
case_id: "case_architecture",
|
||||
status: "completed",
|
||||
current_stage: "final",
|
||||
trace_completeness: "complete",
|
||||
error_stage: null,
|
||||
error_summary: null,
|
||||
started_at: "2026-07-16T00:00:00.000Z",
|
||||
finished_at: "2026-07-16T00:00:03.000Z",
|
||||
updated_at: "2026-07-16T00:00:03.000Z",
|
||||
};
|
||||
|
||||
const call = {
|
||||
call_id: callId,
|
||||
job_id: jobId,
|
||||
sequence: 2,
|
||||
task: "article_optimizer",
|
||||
workflow_stage: "draft",
|
||||
rewrite_round: null,
|
||||
provider: "deepseek",
|
||||
model: "deepseek-chat",
|
||||
status: "validated",
|
||||
token_usage: { prompt_tokens: 120, completion_tokens: 80, total_tokens: 200 },
|
||||
schema_name: "optimizedArticleSchema",
|
||||
schema_valid: true,
|
||||
validation_issues: [],
|
||||
business_status: null,
|
||||
duration_ms: 1600,
|
||||
started_at: "2026-07-16T00:00:01.000Z",
|
||||
responded_at: "2026-07-16T00:00:02.500Z",
|
||||
validated_at: "2026-07-16T00:00:02.600Z",
|
||||
failed_at: null,
|
||||
error_type: null,
|
||||
error_summary: null,
|
||||
request_available: true,
|
||||
response_available: true,
|
||||
};
|
||||
|
||||
const article = {
|
||||
job_id: jobId,
|
||||
revision: 1,
|
||||
title: "可观测的 GEO 优化稿",
|
||||
summary: "展示真实后台调用。",
|
||||
body_markdown: "## 优化结果\n正文内容。",
|
||||
image_suggestions: [],
|
||||
changed_sections: ["title", "body"],
|
||||
requires_user_confirmation: [],
|
||||
};
|
||||
|
||||
const qaReport = {
|
||||
job_id: jobId,
|
||||
revision: 1,
|
||||
overall_status: "pass",
|
||||
checks: [],
|
||||
};
|
||||
|
||||
test("后台架构标签按需展示完整 LLM 请求与响应", async ({ page }) => {
|
||||
const payloadReads = { request: 0, response: 0 };
|
||||
const streamEvents = [
|
||||
{ type: "job_created", job: { id: jobId } },
|
||||
{
|
||||
type: "llm_call_started",
|
||||
job_id: jobId,
|
||||
call_id: callId,
|
||||
sequence: 2,
|
||||
task: "article_optimizer",
|
||||
workflow_stage: "draft",
|
||||
rewrite_round: null,
|
||||
provider: "deepseek",
|
||||
model: "deepseek-chat",
|
||||
started_at: call.started_at,
|
||||
request_available: true,
|
||||
},
|
||||
{
|
||||
type: "llm_call_responded",
|
||||
job_id: jobId,
|
||||
call_id: callId,
|
||||
duration_ms: call.duration_ms,
|
||||
token_usage: call.token_usage,
|
||||
responded_at: call.responded_at,
|
||||
response_available: true,
|
||||
},
|
||||
{
|
||||
type: "llm_call_validated",
|
||||
job_id: jobId,
|
||||
call_id: callId,
|
||||
schema_name: call.schema_name,
|
||||
schema_valid: true,
|
||||
validation_issues: [],
|
||||
validated_at: call.validated_at,
|
||||
},
|
||||
{
|
||||
type: "final_ready",
|
||||
job_id: jobId,
|
||||
optimized_article: article,
|
||||
qa_report: qaReport,
|
||||
export_paths: {
|
||||
markdown: `/api/jobs/${jobId}/exports/optimized.md`,
|
||||
docx: `/api/jobs/${jobId}/exports/optimized.docx`,
|
||||
qa_report: `/api/jobs/${jobId}/exports/qa_report.json`,
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
await page.route("**/api/jobs/optimize-stream", async (route) => {
|
||||
await route.fulfill({
|
||||
status: 200,
|
||||
contentType: "application/x-ndjson; charset=utf-8",
|
||||
body: `${streamEvents.map((event) => JSON.stringify(event)).join("\n")}\n`,
|
||||
});
|
||||
});
|
||||
await page.route(`**/api/jobs/${jobId}/llm-trace`, async (route) => {
|
||||
await route.fulfill({ json: { run, calls: [call] } });
|
||||
});
|
||||
await page.route("**/api/llm-traces/latest", async (route) => {
|
||||
await route.fulfill({ json: { run, calls: [call] } });
|
||||
});
|
||||
await page.route(`**/api/jobs/${jobId}/llm-trace/${callId}/request`, async (route) => {
|
||||
payloadReads.request += 1;
|
||||
await route.fulfill({
|
||||
json: {
|
||||
model: "deepseek-chat",
|
||||
messages: [
|
||||
{ role: "system", content: "你是 GEO 文章优化器。" },
|
||||
{ role: "user", content: "优化这篇原始文章。" },
|
||||
],
|
||||
response_format: { type: "json_object" },
|
||||
temperature: 0.2,
|
||||
},
|
||||
});
|
||||
});
|
||||
await page.route(`**/api/jobs/${jobId}/llm-trace/${callId}/response`, async (route) => {
|
||||
payloadReads.response += 1;
|
||||
await route.fulfill({
|
||||
json: {
|
||||
id: "chatcmpl_observer",
|
||||
choices: [{ message: { role: "assistant", content: JSON.stringify(article) } }],
|
||||
usage: call.token_usage,
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
await page.goto("/");
|
||||
await page.getByLabel("访问密钥").fill("local-dev-key");
|
||||
await page.getByLabel("文章内容").fill("这是一篇需要优化的原始文章。");
|
||||
await page.getByRole("button", { name: "开始优化" }).click();
|
||||
await page.getByRole("button", { name: "后台架构" }).click();
|
||||
|
||||
await expect(page.getByLabel("文章优化后台架构")).toBeVisible();
|
||||
await expect(page.locator(".llm-call-list code", { hasText: "article_optimizer" }))
|
||||
.toBeVisible();
|
||||
await expect(page.locator('[data-node="final"]')).toContainText("已完成");
|
||||
|
||||
await page.getByRole("tab", { name: "请求" }).click();
|
||||
await expect(page.locator(".llm-json-view"))
|
||||
.toContainText("开启技术详情后按需读取完整正文。");
|
||||
expect(payloadReads.request).toBe(0);
|
||||
expect(payloadReads.response).toBe(0);
|
||||
|
||||
await page.getByLabel("技术详情").check();
|
||||
await expect(page.locator(".llm-json-view")).toContainText("messages");
|
||||
await expect(page.locator(".llm-json-view")).toContainText("你是 GEO 文章优化器。");
|
||||
expect(payloadReads.request).toBe(1);
|
||||
|
||||
await page.getByRole("tab", { name: "响应" }).click();
|
||||
await expect(page.locator(".llm-json-view")).toContainText("choices");
|
||||
await expect(page.locator(".llm-json-view")).toContainText("chatcmpl_observer");
|
||||
expect(payloadReads.response).toBe(1);
|
||||
|
||||
await page.getByRole("button", { name: "GEO 文章优化" }).click();
|
||||
await expect(page.getByText("优化完成。", { exact: true })).toBeVisible();
|
||||
await expect(page.getByRole("link", { name: "optimized.md" })).toBeVisible();
|
||||
});
|
||||
+55
-21
@@ -1,38 +1,72 @@
|
||||
import { expect, test } from "@playwright/test";
|
||||
|
||||
test("中文界面可以生成优化文章和导出链接", async ({ page }) => {
|
||||
test.setTimeout(180_000);
|
||||
const jobId = "job_mvp";
|
||||
const factCard = {
|
||||
company_full_name: "Example Technology Co., Ltd.",
|
||||
company_short_names: ["Example Tech"],
|
||||
brand_names: ["Example"],
|
||||
product_names: ["Example GEO"],
|
||||
target_industry: "GEO optimization",
|
||||
target_audience: "市场团队",
|
||||
experience_years: 8,
|
||||
core_claims: ["拥有 8 年 GEO 优化经验"],
|
||||
forbidden_claims: [],
|
||||
image_topics: ["产品仪表盘"],
|
||||
uncertain_items: [],
|
||||
is_ready_for_optimization: true,
|
||||
confirmed_by_user: false,
|
||||
};
|
||||
const article = {
|
||||
job_id: jobId,
|
||||
revision: 1,
|
||||
title: "Example Technology Co., Ltd. GEO 指南",
|
||||
summary: "面向市场团队的 GEO 优化说明。",
|
||||
body_markdown: "## 服务能力\nExample GEO 帮助团队优化内容结构。",
|
||||
image_suggestions: [],
|
||||
changed_sections: ["title", "body"],
|
||||
requires_user_confirmation: [],
|
||||
};
|
||||
const qaReport = {
|
||||
job_id: jobId,
|
||||
revision: 1,
|
||||
overall_status: "pass",
|
||||
checks: [],
|
||||
};
|
||||
|
||||
await page.route("**/api/jobs/optimize-stream", async (route) => {
|
||||
const events = [
|
||||
{ type: "job_created", job: { id: jobId } },
|
||||
{ type: "fact_card_ready", job_id: jobId, fact_card: factCard },
|
||||
{ type: "draft_ready", job_id: jobId, article },
|
||||
{ type: "qa_ready", job_id: jobId, qa_report: qaReport },
|
||||
{
|
||||
type: "final_ready",
|
||||
job_id: jobId,
|
||||
optimized_article: article,
|
||||
qa_report: qaReport,
|
||||
export_paths: {},
|
||||
},
|
||||
];
|
||||
await route.fulfill({
|
||||
status: 200,
|
||||
contentType: "application/x-ndjson; charset=utf-8",
|
||||
body: `${events.map((event) => JSON.stringify(event)).join("\n")}\n`,
|
||||
});
|
||||
});
|
||||
|
||||
await page.goto("/");
|
||||
|
||||
await page.getByLabel("访问密钥").fill("local-dev-key");
|
||||
await page.getByLabel("标题").fill("Example Technology Co., Ltd. GEO 指南");
|
||||
await page
|
||||
.getByLabel("正文")
|
||||
.getByLabel("文章内容")
|
||||
.fill(
|
||||
"Example Technology Co., Ltd. has 8 years of GEO optimization experience. Example GEO 帮助市场团队优化内容结构。",
|
||||
);
|
||||
await page.getByLabel("图片描述或图片链接").fill("产品仪表盘截图");
|
||||
await page.getByLabel("用户要求").fill("保持事实准确,语气自然。");
|
||||
|
||||
await page.getByRole("button", { name: "分析文章" }).click();
|
||||
await expect(page.getByText("候选事实卡已生成")).toBeVisible({
|
||||
timeout: 70_000,
|
||||
});
|
||||
|
||||
const confirmUncertainItemButtons = page.getByRole("button", {
|
||||
name: "采纳为核心事实",
|
||||
});
|
||||
while ((await confirmUncertainItemButtons.count()) > 0) {
|
||||
await confirmUncertainItemButtons.first().click();
|
||||
}
|
||||
await page.getByRole("button", { name: "确认事实卡" }).click();
|
||||
await expect(page.getByText("事实卡已确认。")).toBeVisible();
|
||||
|
||||
await page.getByRole("button", { name: "开始优化" }).click();
|
||||
await expect(page.getByRole("link", { name: "optimized.md" })).toBeVisible({
|
||||
timeout: 120_000,
|
||||
});
|
||||
await expect(page.getByRole("link", { name: "optimized.md" })).toBeVisible();
|
||||
await expect(page.getByText("质量报告")).toBeVisible();
|
||||
await expect(page.getByRole("link", { name: "optimized.docx" })).toBeVisible();
|
||||
await expect(page.getByRole("link", { name: "qa_report.json" })).toBeVisible();
|
||||
|
||||
@@ -18,7 +18,7 @@ const samplesDir =
|
||||
const reportDir =
|
||||
process.env.GEO_SAMPLE_REPORT_DIR ??
|
||||
join(repoRoot, "test-results", "geo-sample-flow", "manual");
|
||||
const baseURL = process.env.GEO_SAMPLE_BASE_URL ?? "http://127.0.0.1:3000";
|
||||
const baseURL = process.env.GEO_SAMPLE_BASE_URL ?? "http://localhost:3000";
|
||||
const apiAccessKey = process.env.API_ACCESS_KEY ?? "local-dev-key";
|
||||
const timeoutMs = Number(process.env.GEO_SAMPLE_TIMEOUT_MS ?? "600000");
|
||||
const filter = process.env.GEO_SAMPLE_FILTER;
|
||||
@@ -26,8 +26,13 @@ const limit = Number(process.env.GEO_SAMPLE_LIMIT ?? "0");
|
||||
|
||||
const loaded = loadArticleSamples(samplesDir);
|
||||
const selectedSamples = selectSamples(loaded.valid, filter, limit);
|
||||
const provider = process.env.LLM_PROVIDER ?? "deepseek";
|
||||
const hasLiveLlmCredentials = provider === "openai"
|
||||
? Boolean(process.env.OPENAI_API_KEY)
|
||||
: Boolean(process.env.DEEPSEEK_API_KEY);
|
||||
|
||||
test.describe("GEO sample article live E2E flow", () => {
|
||||
test.skip(!hasLiveLlmCredentials, "需要真实 LLM 提供商密钥");
|
||||
test.beforeAll(() => {
|
||||
mkdirSync(reportDir, { recursive: true });
|
||||
writeFileSync(
|
||||
|
||||
@@ -37,6 +37,16 @@ interface QaReportJson {
|
||||
checks?: Array<{ rule_id?: string; status?: "pass" | "warn" | "fail" }>;
|
||||
}
|
||||
|
||||
interface TraceManifestJson {
|
||||
run?: { job_id?: string; status?: string; trace_completeness?: string };
|
||||
calls?: Array<{
|
||||
task?: string;
|
||||
status?: string;
|
||||
request_available?: boolean;
|
||||
response_available?: boolean;
|
||||
}>;
|
||||
}
|
||||
|
||||
export async function runSamplePageFlow({
|
||||
page,
|
||||
request,
|
||||
@@ -87,17 +97,29 @@ export async function runSamplePageFlow({
|
||||
jobId,
|
||||
exportsDir,
|
||||
});
|
||||
const traceResult = await validateLlmTrace({
|
||||
request,
|
||||
baseURL,
|
||||
apiAccessKey,
|
||||
jobId,
|
||||
});
|
||||
const qa = readQaReport(exportsDir);
|
||||
const finalScreenshot = join(sampleDir, "final.png");
|
||||
await page.screenshot({ path: finalScreenshot, fullPage: true });
|
||||
|
||||
const failedExports = exportResults.filter((result) => result.status === "failed");
|
||||
const failureMessage = [
|
||||
...failedExports.map(
|
||||
(result) => `${result.fileName}: ${result.error ?? result.statusCode}`,
|
||||
),
|
||||
...(traceResult.error ? [`LLM trace: ${traceResult.error}`] : []),
|
||||
].join("; ");
|
||||
|
||||
return {
|
||||
file: sample.filePath,
|
||||
name: sample.name,
|
||||
slug: sample.slug,
|
||||
status: failedExports.length === 0 ? "passed" : "failed",
|
||||
status: failedExports.length === 0 && !traceResult.error ? "passed" : "failed",
|
||||
duration_ms: Date.now() - startedAt,
|
||||
job_id: jobId,
|
||||
qa_status: qa.overall_status,
|
||||
@@ -108,18 +130,61 @@ export async function runSamplePageFlow({
|
||||
exports: Object.fromEntries(
|
||||
exportResults.map((result) => [result.fileName, result.status]),
|
||||
),
|
||||
llm_tasks: [],
|
||||
failure_category: failedExports.length > 0 ? "export_failed" : undefined,
|
||||
failure_message:
|
||||
failedExports
|
||||
.map((result) => `${result.fileName}: ${result.error ?? result.statusCode}`)
|
||||
.join("; ") || undefined,
|
||||
llm_tasks: traceResult.tasks,
|
||||
failure_category: failedExports.length > 0
|
||||
? "export_failed"
|
||||
: traceResult.error
|
||||
? "llm_failed"
|
||||
: undefined,
|
||||
failure_message: failureMessage || undefined,
|
||||
artifacts: {
|
||||
final_screenshot: finalScreenshot,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function validateLlmTrace({
|
||||
request,
|
||||
baseURL,
|
||||
apiAccessKey,
|
||||
jobId,
|
||||
}: {
|
||||
request: APIRequestContext;
|
||||
baseURL: string;
|
||||
apiAccessKey: string;
|
||||
jobId: string;
|
||||
}) {
|
||||
const headers: Record<string, string> | undefined = apiAccessKey
|
||||
? { "x-api-key": apiAccessKey }
|
||||
: undefined;
|
||||
const response = await request.get(
|
||||
`${baseURL}/api/jobs/${jobId}/llm-trace`,
|
||||
{ headers },
|
||||
);
|
||||
if (!response.ok()) {
|
||||
return { tasks: [], error: `metadata endpoint returned ${response.status()}` };
|
||||
}
|
||||
|
||||
const manifest = await response.json() as TraceManifestJson;
|
||||
const calls = manifest.calls ?? [];
|
||||
const tasks = calls
|
||||
.map((call) => call.task)
|
||||
.filter((task): task is string => Boolean(task));
|
||||
if (manifest.run?.job_id !== jobId) {
|
||||
return { tasks, error: "run job_id does not match completed job" };
|
||||
}
|
||||
if (manifest.run.status !== "completed") {
|
||||
return { tasks, error: `run status is ${manifest.run.status ?? "missing"}` };
|
||||
}
|
||||
if (calls.length === 0) {
|
||||
return { tasks, error: "no LLM calls were recorded" };
|
||||
}
|
||||
if (calls.some((call) => !call.status || call.request_available !== true)) {
|
||||
return { tasks, error: "call metadata is incomplete" };
|
||||
}
|
||||
return { tasks, error: undefined };
|
||||
}
|
||||
|
||||
async function fillFirstAvailable(page: Page, labels: string[], value: string) {
|
||||
for (const label of labels) {
|
||||
const locator = page.getByLabel(label);
|
||||
|
||||
Reference in New Issue
Block a user