48 KiB
GEO 样例文章 E2E Skill 工作流 Implementation Plan
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: Build a live-LLM sample article E2E runner that opens the current web UI, runs samples/articles/*.json through the one-click optimization flow, validates exports, and writes reviewable reports for future skill wrapping.
Architecture: Keep project-specific behavior inside the repo, not inside the future Codex skill. Add focused TypeScript helpers for loading samples, reporting, page automation, and export validation; add a thin Node runner that starts Playwright with isolated environment and aggregates per-sample results.
Tech Stack: Next.js 16, React 19, Playwright, Vitest, Node.js fs/child_process, existing NDJSON/LLM logs, local .env.local.
File Structure
- Create
tests/e2e/sample-flow/types.ts: shared sample, result, failure, and report interfaces. - Create
tests/e2e/sample-flow/sample-loader.ts: read and validatesamples/articles/*.json. - Create
tests/e2e/sample-flow/reporting.ts: create report directories, redact secrets, write per-sampleresult.json, and aggregatesummary.json/summary.md. - Create
tests/e2e/sample-flow/page-flow.ts: Playwright UI workflow and export validation helpers. - Create
tests/e2e/sample-flow.spec.ts: dynamic live-LLM Playwright tests, one test per selected sample. - Create
tests/e2e/sample-flow/__tests__/sample-loader.test.ts: Vitest coverage for sample loading and validation. - Create
tests/e2e/sample-flow/__tests__/reporting.test.ts: Vitest coverage for summary aggregation and redaction. - Create
playwright.samples.config.ts: dedicated Playwright config for live sample flow, independent from the old MVP config. - Create
scripts/run-geo-sample-e2e.mjs: CLI runner that loads.env.local, prepares an isolated report directory, finds a port, launches Playwright, and aggregates output. - Modify
package.json: addtest:e2e:samplesscript. - Modify
.gitignoreonly if needed. Current.gitignorealready ignorestest-results/, so no edit is expected. - Modify
README.md: document the sample E2E command and report path. - Keep
tests/e2e/mvp.spec.tsunchanged in the first pass. The new runner will point only atsample-flow.spec.ts; removing or rewriting the old test can be a separate cleanup after the new workflow is proven.
Task 1: Sample Loader and Types
Files:
-
Create:
tests/e2e/sample-flow/types.ts -
Create:
tests/e2e/sample-flow/sample-loader.ts -
Create:
tests/e2e/sample-flow/__tests__/sample-loader.test.ts -
Step 1: Write the failing sample loader tests
Create tests/e2e/sample-flow/__tests__/sample-loader.test.ts:
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { loadArticleSamples } from "../sample-loader";
describe("loadArticleSamples", () => {
let tempDir: string;
beforeEach(() => {
tempDir = mkdtempSync(join(tmpdir(), "geo-samples-"));
});
afterEach(() => {
rmSync(tempDir, { recursive: true, force: true });
});
it("loads valid samples in filename order and normalizes optional fields", () => {
writeFileSync(
join(tempDir, "b-sample.json"),
JSON.stringify({
name: "B sample",
input: {
body: "第二篇文章正文",
platform: "media_article",
},
}),
);
writeFileSync(
join(tempDir, "a-sample.json"),
JSON.stringify({
name: "A sample",
input: {
title: "标题",
body: "第一篇文章正文",
image_lines: "图一",
platform: "official_site",
user_instructions: "保持事实准确",
},
expectedHardFailures: ["body_quality"],
expectedWarnings: ["context_sensitive_terms"],
}),
);
const result = loadArticleSamples(tempDir);
expect(result.valid.map((sample) => sample.fileName)).toEqual([
"a-sample.json",
"b-sample.json",
]);
expect(result.invalid).toEqual([]);
expect(result.valid[1]).toEqual(
expect.objectContaining({
name: "B sample",
input: expect.objectContaining({
title: "",
image_lines: "",
user_instructions: "",
}),
expectedHardFailures: [],
expectedWarnings: [],
}),
);
});
it("returns invalid entries for malformed samples without throwing", () => {
writeFileSync(
join(tempDir, "empty-body.json"),
JSON.stringify({
name: "Empty body",
input: {
body: " ",
platform: "official_site",
},
}),
);
writeFileSync(
join(tempDir, "bad-platform.json"),
JSON.stringify({
name: "Bad platform",
input: {
body: "正文",
platform: "unknown",
},
}),
);
const result = loadArticleSamples(tempDir);
expect(result.valid).toEqual([]);
expect(result.invalid).toEqual([
expect.objectContaining({
fileName: "bad-platform.json",
reason: "input.platform must be one of official_site, media_article, comparison_review, recommendation_list",
}),
expect.objectContaining({
fileName: "empty-body.json",
reason: "input.body must be a non-empty string",
}),
]);
});
});
- Step 2: Run tests and verify they fail
Run:
npm test -- tests/e2e/sample-flow/__tests__/sample-loader.test.ts
Expected: FAIL because tests/e2e/sample-flow/sample-loader.ts does not exist.
- Step 3: Add shared types
Create tests/e2e/sample-flow/types.ts:
export const supportedPlatforms = [
"official_site",
"media_article",
"comparison_review",
"recommendation_list",
] as const;
export type SamplePlatform = (typeof supportedPlatforms)[number];
export interface ArticleSampleInput {
title: string;
body: string;
image_lines: string;
platform: SamplePlatform;
user_instructions: string;
}
export interface ArticleSample {
filePath: string;
fileName: string;
slug: string;
name: string;
input: ArticleSampleInput;
expectedHardFailures: string[];
expectedWarnings: string[];
}
export interface InvalidArticleSample {
filePath: string;
fileName: string;
reason: string;
}
export interface LoadedArticleSamples {
valid: ArticleSample[];
invalid: InvalidArticleSample[];
}
export type SampleStatus = "passed" | "failed" | "skipped";
export type FailureCategory =
| "preflight_failed"
| "sample_invalid"
| "page_flow_failed"
| "stream_timeout"
| "stream_failed"
| "llm_failed"
| "export_failed"
| "console_error"
| "unknown_failed";
export interface ExportValidationResult {
fileName: "optimized.md" | "optimized.docx" | "qa_report.json";
status: "passed" | "failed";
statusCode?: number;
error?: string;
}
export interface SampleResult {
file: string;
name: string;
slug: string;
status: SampleStatus;
duration_ms: number;
job_id?: string;
qa_status?: "pass" | "warn" | "fail";
qa_fail_rules: string[];
qa_warn_rules: string[];
expected_hard_failures: string[];
expected_warnings: string[];
exports: Record<string, "passed" | "failed">;
llm_tasks: string[];
failure_category?: FailureCategory;
failure_message?: string;
artifacts: {
final_screenshot?: string;
failure_screenshot?: string;
trace?: string;
};
}
export interface RunSummary {
started_at: string;
finished_at: string;
mode: "live";
provider: string;
model: string;
base_url: string;
report_dir: string;
totals: {
passed: number;
failed: number;
skipped: number;
};
samples: SampleResult[];
invalid_samples: InvalidArticleSample[];
}
- Step 4: Implement the sample loader
Create tests/e2e/sample-flow/sample-loader.ts:
import { existsSync, readdirSync, readFileSync } from "node:fs";
import { basename, join } from "node:path";
import {
type ArticleSample,
type LoadedArticleSamples,
supportedPlatforms,
type SamplePlatform,
} from "./types";
interface RawSample {
name?: unknown;
input?: {
title?: unknown;
body?: unknown;
image_lines?: unknown;
platform?: unknown;
user_instructions?: unknown;
};
expectedHardFailures?: unknown;
expectedWarnings?: unknown;
}
export function loadArticleSamples(samplesDir: string): LoadedArticleSamples {
if (!existsSync(samplesDir)) {
return { valid: [], invalid: [] };
}
const files = readdirSync(samplesDir)
.filter((fileName) => fileName.endsWith(".json"))
.sort((left, right) => left.localeCompare(right));
const valid: ArticleSample[] = [];
const invalid: LoadedArticleSamples["invalid"] = [];
for (const fileName of files) {
const filePath = join(samplesDir, fileName);
try {
const raw = JSON.parse(readFileSync(filePath, "utf8")) as RawSample;
const sample = normalizeSample(raw, filePath, fileName);
valid.push(sample);
} catch (error) {
invalid.push({
filePath,
fileName,
reason: error instanceof Error ? error.message : "sample could not be parsed",
});
}
}
return { valid, invalid };
}
function normalizeSample(
raw: RawSample,
filePath: string,
fileName: string,
): ArticleSample {
if (!raw || typeof raw !== "object") {
throw new Error("sample must be a JSON object");
}
if (!raw.input || typeof raw.input !== "object") {
throw new Error("input must be an object");
}
if (typeof raw.input.body !== "string" || raw.input.body.trim().length === 0) {
throw new Error("input.body must be a non-empty string");
}
if (!isSupportedPlatform(raw.input.platform)) {
throw new Error(
`input.platform must be one of ${supportedPlatforms.join(", ")}`,
);
}
return {
filePath,
fileName,
slug: slugFromFileName(fileName),
name:
typeof raw.name === "string" && raw.name.trim().length > 0
? raw.name.trim()
: basename(fileName, ".json"),
input: {
title: stringOrEmpty(raw.input.title),
body: raw.input.body,
image_lines: stringOrEmpty(raw.input.image_lines),
platform: raw.input.platform,
user_instructions: stringOrEmpty(raw.input.user_instructions),
},
expectedHardFailures: stringArrayOrEmpty(raw.expectedHardFailures),
expectedWarnings: stringArrayOrEmpty(raw.expectedWarnings),
};
}
function isSupportedPlatform(value: unknown): value is SamplePlatform {
return (
typeof value === "string" &&
supportedPlatforms.includes(value as SamplePlatform)
);
}
function stringOrEmpty(value: unknown) {
return typeof value === "string" ? value : "";
}
function stringArrayOrEmpty(value: unknown) {
return Array.isArray(value)
? value.filter((item): item is string => typeof item === "string")
: [];
}
function slugFromFileName(fileName: string) {
return basename(fileName, ".json")
.toLowerCase()
.replace(/[^a-z0-9-]+/g, "-")
.replace(/^-+|-+$/g, "");
}
- Step 5: Run tests and verify they pass
Run:
npm test -- tests/e2e/sample-flow/__tests__/sample-loader.test.ts
Expected: PASS with both loader tests passing.
- Step 6: Commit
Run:
git add tests/e2e/sample-flow/types.ts tests/e2e/sample-flow/sample-loader.ts tests/e2e/sample-flow/__tests__/sample-loader.test.ts
git commit -m "新增样例文章加载器"
Task 2: Report Writer and Redaction
Files:
-
Create:
tests/e2e/sample-flow/reporting.ts -
Create:
tests/e2e/sample-flow/__tests__/reporting.test.ts -
Step 1: Write the failing reporting tests
Create tests/e2e/sample-flow/__tests__/reporting.test.ts:
import { existsSync, mkdtempSync, readFileSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import {
aggregateSummary,
redactSensitiveText,
writeSampleResult,
writeSummaryFiles,
} from "../reporting";
import type { InvalidArticleSample, SampleResult } from "../types";
describe("reporting helpers", () => {
let reportDir: string;
beforeEach(() => {
reportDir = mkdtempSync(join(tmpdir(), "geo-report-"));
});
afterEach(() => {
rmSync(reportDir, { recursive: true, force: true });
});
it("redacts API keys and authorization tokens", () => {
const redacted = redactSensitiveText(
'x-api-key: local-dev-key Authorization: Bearer secret DEEPSEEK_API_KEY="abc"',
["local-dev-key", "abc"],
);
expect(redacted).toBe(
'x-api-key: [REDACTED] Authorization: Bearer [REDACTED] DEEPSEEK_API_KEY="[REDACTED]"',
);
});
it("writes per-sample results and aggregate summary files", () => {
const sample: SampleResult = {
file: "samples/articles/title-quality.json",
name: "Title quality",
slug: "title-quality",
status: "passed",
duration_ms: 1234,
job_id: "job_123",
qa_status: "warn",
qa_fail_rules: [],
qa_warn_rules: ["title_quality"],
expected_hard_failures: [],
expected_warnings: ["title_quality"],
exports: {
"optimized.md": "passed",
"optimized.docx": "passed",
"qa_report.json": "passed",
},
llm_tasks: ["fact_extractor", "article_optimizer", "quality_inspector"],
artifacts: {
final_screenshot: "samples/title-quality/final.png",
},
};
const invalid: InvalidArticleSample[] = [
{
filePath: "/repo/samples/articles/bad.json",
fileName: "bad.json",
reason: "input.body must be a non-empty string",
},
];
writeSampleResult(reportDir, sample);
const summary = aggregateSummary({
startedAt: "2026-07-01T00:00:00.000Z",
finishedAt: "2026-07-01T00:01:00.000Z",
provider: "deepseek",
model: "deepseek-v4-pro",
baseURL: "http://127.0.0.1:3000",
reportDir,
samples: [sample],
invalidSamples: invalid,
});
writeSummaryFiles(reportDir, summary);
expect(
existsSync(join(reportDir, "samples", "title-quality", "result.json")),
).toBe(true);
expect(JSON.parse(readFileSync(join(reportDir, "summary.json"), "utf8"))).toEqual(
expect.objectContaining({
provider: "deepseek",
totals: { passed: 1, failed: 0, skipped: 1 },
}),
);
expect(readFileSync(join(reportDir, "summary.md"), "utf8")).toContain(
"| Title quality | passed | warn |",
);
});
});
- Step 2: Run tests and verify they fail
Run:
npm test -- tests/e2e/sample-flow/__tests__/reporting.test.ts
Expected: FAIL because tests/e2e/sample-flow/reporting.ts does not exist.
- Step 3: Implement reporting helpers
Create tests/e2e/sample-flow/reporting.ts:
import { mkdirSync, readFileSync, readdirSync, writeFileSync } from "node:fs";
import { join, relative } from "node:path";
import type {
InvalidArticleSample,
RunSummary,
SampleResult,
} from "./types";
interface AggregateInput {
startedAt: string;
finishedAt: string;
provider: string;
model: string;
baseURL: string;
reportDir: string;
samples: SampleResult[];
invalidSamples: InvalidArticleSample[];
}
export function writeSampleResult(reportDir: string, result: SampleResult) {
const sampleDir = join(reportDir, "samples", result.slug);
mkdirSync(sampleDir, { recursive: true });
writeFileSync(join(sampleDir, "result.json"), `${JSON.stringify(result, null, 2)}\n`);
}
export function readSampleResults(reportDir: string): SampleResult[] {
const samplesDir = join(reportDir, "samples");
try {
return readdirSync(samplesDir, { withFileTypes: true })
.filter((entry) => entry.isDirectory())
.map((entry) => join(samplesDir, entry.name, "result.json"))
.map((filePath) => JSON.parse(readFileSync(filePath, "utf8")) as SampleResult)
.sort((left, right) => left.file.localeCompare(right.file));
} catch {
return [];
}
}
export function aggregateSummary(input: AggregateInput): RunSummary {
const passed = input.samples.filter((sample) => sample.status === "passed").length;
const failed = input.samples.filter((sample) => sample.status === "failed").length;
const skipped =
input.samples.filter((sample) => sample.status === "skipped").length +
input.invalidSamples.length;
return {
started_at: input.startedAt,
finished_at: input.finishedAt,
mode: "live",
provider: input.provider,
model: input.model,
base_url: input.baseURL,
report_dir: input.reportDir,
totals: { passed, failed, skipped },
samples: input.samples,
invalid_samples: input.invalidSamples,
};
}
export function writeSummaryFiles(reportDir: string, summary: RunSummary) {
mkdirSync(reportDir, { recursive: true });
writeFileSync(join(reportDir, "summary.json"), `${JSON.stringify(summary, null, 2)}\n`);
writeFileSync(join(reportDir, "summary.md"), renderSummaryMarkdown(summary));
}
export function redactSensitiveText(text: string, secrets: Array<string | undefined>) {
let redacted = text
.replace(/(x-api-key:\s*)([^\s]+)/gi, "$1[REDACTED]")
.replace(/(authorization:\s*bearer\s+)([^\s]+)/gi, "$1[REDACTED]")
.replace(/((?:DEEPSEEK|OPENAI|API)_?[A-Z_]*KEY=)"?([^"\s]+)"?/g, '$1"[REDACTED]"');
for (const secret of secrets) {
if (!secret) continue;
redacted = redacted.split(secret).join("[REDACTED]");
}
return redacted;
}
function renderSummaryMarkdown(summary: RunSummary) {
const lines = [
"# GEO 样例文章 E2E 测试报告",
"",
`- 模式:${summary.mode}`,
`- Provider:${summary.provider}`,
`- Model:${summary.model}`,
`- Base URL:${summary.base_url}`,
`- 报告目录:${summary.report_dir}`,
`- 结果:通过 ${summary.totals.passed},失败 ${summary.totals.failed},跳过 ${summary.totals.skipped}`,
"",
"| 样例 | 状态 | QA | 耗时 | 失败分类 |",
"| --- | --- | --- | ---: | --- |",
...summary.samples.map((sample) =>
[
sample.name,
sample.status,
sample.qa_status ?? "",
`${Math.round(sample.duration_ms / 1000)}s`,
sample.failure_category ?? "",
].join(" | "),
).map((row) => `| ${row} |`),
];
if (summary.invalid_samples.length > 0) {
lines.push("", "## 无效样例", "");
for (const sample of summary.invalid_samples) {
lines.push(`- ${sample.fileName}: ${sample.reason}`);
}
}
lines.push("");
return lines.join("\n");
}
export function relativeArtifactPath(reportDir: string, artifactPath: string) {
return relative(reportDir, artifactPath);
}
- Step 4: Run tests and verify they pass
Run:
npm test -- tests/e2e/sample-flow/__tests__/reporting.test.ts
Expected: PASS with both reporting tests passing.
- Step 5: Commit
Run:
git add tests/e2e/sample-flow/reporting.ts tests/e2e/sample-flow/__tests__/reporting.test.ts
git commit -m "新增样例验收报告工具"
Task 3: Page Flow and Export Validation Helpers
Files:
-
Create:
tests/e2e/sample-flow/page-flow.ts -
Step 1: Add page automation helpers
Create tests/e2e/sample-flow/page-flow.ts:
import { mkdirSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { expect, type APIRequestContext, type Page } from "@playwright/test";
import type {
ArticleSample,
ExportValidationResult,
SampleResult,
} from "./types";
const exportFileNames = [
"optimized.md",
"optimized.docx",
"qa_report.json",
] as const;
interface RunSamplePageFlowOptions {
page: Page;
request: APIRequestContext;
sample: ArticleSample;
baseURL: string;
apiAccessKey: string;
reportDir: string;
timeoutMs: number;
}
interface QaReportJson {
overall_status?: "pass" | "warn" | "fail";
checks?: Array<{ rule_id?: string; status?: "pass" | "warn" | "fail" }>;
}
export async function runSamplePageFlow({
page,
request,
sample,
baseURL,
apiAccessKey,
reportDir,
timeoutMs,
}: RunSamplePageFlowOptions): Promise<SampleResult> {
const startedAt = Date.now();
const sampleDir = join(reportDir, "samples", sample.slug);
const exportsDir = join(sampleDir, "exports");
mkdirSync(exportsDir, { recursive: true });
await page.goto(baseURL);
await fillIfVisible(page, "访问密钥", apiAccessKey);
await fillFirstAvailable(page, ["文章内容", "粘贴文章", "正文"], sample.input.body);
await fillIfVisible(page, "图片描述或图片链接", sample.input.image_lines);
await page.getByLabel("目标平台").selectOption(sample.input.platform);
await fillIfVisible(page, "用户要求", sample.input.user_instructions);
await page.getByRole("button", { name: "开始优化" }).click();
await expect(page.getByText("事实卡")).toBeVisible({ timeout: 30_000 });
await expect(page.getByText("优化结果")).toBeVisible({ timeout: 30_000 });
await expect(page.getByText(/优化完成|质检发现需要复核的问题/)).toBeVisible({
timeout: timeoutMs,
});
await expect(page.getByText("质量报告")).toBeVisible();
for (const fileName of exportFileNames) {
await expect(page.getByRole("link", { name: fileName })).toBeVisible();
}
const jobId = await extractJobIdFromExportLink(page);
const exportResults = await validateExports({
request,
baseURL,
apiAccessKey,
jobId,
exportsDir,
});
const qa = await readQaReport(exportsDir);
const finalScreenshot = join(sampleDir, "final.png");
await page.screenshot({ path: finalScreenshot, fullPage: true });
return {
file: sample.filePath,
name: sample.name,
slug: sample.slug,
status: exportResults.every((result) => result.status === "passed")
? "passed"
: "failed",
duration_ms: Date.now() - startedAt,
job_id: jobId,
qa_status: qa.overall_status,
qa_fail_rules: rulesWithStatus(qa, "fail"),
qa_warn_rules: rulesWithStatus(qa, "warn"),
expected_hard_failures: sample.expectedHardFailures,
expected_warnings: sample.expectedWarnings,
exports: Object.fromEntries(
exportResults.map((result) => [result.fileName, result.status]),
),
llm_tasks: [],
failure_category: exportResults.some((result) => result.status === "failed")
? "export_failed"
: undefined,
failure_message: exportResults
.filter((result) => result.status === "failed")
.map((result) => `${result.fileName}: ${result.error ?? result.statusCode}`)
.join("; ") || undefined,
artifacts: {
final_screenshot: finalScreenshot,
},
};
}
async function fillFirstAvailable(
page: Page,
labels: string[],
value: string,
) {
for (const label of labels) {
const locator = page.getByLabel(label);
if ((await locator.count()) > 0) {
await locator.fill(value);
return;
}
}
throw new Error(`none of these labels were found: ${labels.join(", ")}`);
}
async function fillIfVisible(page: Page, label: string, value: string) {
const locator = page.getByLabel(label);
if ((await locator.count()) > 0) {
await locator.fill(value);
}
}
async function extractJobIdFromExportLink(page: Page) {
const href = await page
.getByRole("link", { name: "optimized.md" })
.getAttribute("href");
const match = href?.match(/\/api\/jobs\/([^/]+)\/exports\//);
if (!match?.[1]) {
throw new Error(`could not extract job id from export link: ${href ?? ""}`);
}
return match[1];
}
async function validateExports({
request,
baseURL,
apiAccessKey,
jobId,
exportsDir,
}: {
request: APIRequestContext;
baseURL: string;
apiAccessKey: string;
jobId: string;
exportsDir: string;
}): Promise<ExportValidationResult[]> {
const results: ExportValidationResult[] = [];
for (const fileName of exportFileNames) {
const response = await request.get(
`${baseURL}/api/jobs/${jobId}/exports/${fileName}`,
{ headers: { "x-api-key": apiAccessKey } },
);
const buffer = await response.body();
const outputPath = join(exportsDir, fileName);
writeFileSync(outputPath, buffer);
if (!response.ok()) {
results.push({
fileName,
status: "failed",
statusCode: response.status(),
error: response.statusText(),
});
continue;
}
const error = validateExportBody(fileName, buffer);
results.push({
fileName,
status: error ? "failed" : "passed",
statusCode: response.status(),
error,
});
}
return results;
}
function validateExportBody(
fileName: (typeof exportFileNames)[number],
buffer: Buffer,
) {
if (buffer.length === 0) return "empty export body";
if (fileName === "optimized.md") {
return buffer.toString("utf8").trim().length > 0
? undefined
: "optimized.md is blank";
}
if (fileName === "optimized.docx") {
return buffer.subarray(0, 2).toString("utf8") === "PK"
? undefined
: "optimized.docx does not look like a zip-based docx";
}
try {
const parsed = JSON.parse(buffer.toString("utf8")) as QaReportJson;
if (!parsed.overall_status || !Array.isArray(parsed.checks)) {
return "qa_report.json must contain overall_status and checks";
}
return undefined;
} catch {
return "qa_report.json is not valid JSON";
}
}
async function readQaReport(exportsDir: string): Promise<QaReportJson> {
const text = await import("node:fs").then(({ readFileSync }) =>
readFileSync(join(exportsDir, "qa_report.json"), "utf8"),
);
return JSON.parse(text) as QaReportJson;
}
function rulesWithStatus(report: QaReportJson, status: "warn" | "fail") {
return (report.checks ?? [])
.filter((check) => check.status === status && check.rule_id)
.map((check) => check.rule_id as string);
}
- Step 2: Type-check the helper
Run:
npx tsc --noEmit
Expected before all later files exist: either PASS, or fail only for files not yet created in later tasks. If TypeScript reports an error inside page-flow.ts, fix it before continuing.
- Step 3: Commit
Run:
git add tests/e2e/sample-flow/page-flow.ts
git commit -m "新增样例页面验收流程工具"
Task 4: Dedicated Playwright Sample Spec and Config
Files:
-
Create:
tests/e2e/sample-flow.spec.ts -
Create:
playwright.samples.config.ts -
Step 1: Create the live sample Playwright spec
Create tests/e2e/sample-flow.spec.ts:
import { mkdirSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { test } from "@playwright/test";
import { loadArticleSamples } from "./sample-flow/sample-loader";
import { writeSampleResult } from "./sample-flow/reporting";
import { runSamplePageFlow } from "./sample-flow/page-flow";
import type { ArticleSample, FailureCategory, SampleResult } from "./sample-flow/types";
const repoRoot = process.cwd();
const samplesDir = process.env.GEO_SAMPLE_DIR ?? join(repoRoot, "samples", "articles");
const reportDir =
process.env.GEO_SAMPLE_REPORT_DIR ??
join(repoRoot, "test-results", "geo-sample-flow", "manual");
const baseURL = process.env.GEO_SAMPLE_BASE_URL ?? "http://127.0.0.1:3000";
const apiAccessKey = process.env.API_ACCESS_KEY ?? "";
const timeoutMs = Number(process.env.GEO_SAMPLE_TIMEOUT_MS ?? "600000");
const filter = process.env.GEO_SAMPLE_FILTER;
const limit = Number(process.env.GEO_SAMPLE_LIMIT ?? "0");
const loaded = loadArticleSamples(samplesDir);
const selectedSamples = selectSamples(loaded.valid, filter, limit);
test.describe("GEO sample article live E2E flow", () => {
test.beforeAll(() => {
mkdirSync(reportDir, { recursive: true });
writeFileSync(
join(reportDir, "invalid-samples.json"),
`${JSON.stringify(loaded.invalid, null, 2)}\n`,
);
});
for (const sample of selectedSamples) {
test(sample.name, async ({ page, request }) => {
test.setTimeout(timeoutMs + 60_000);
const startedAt = Date.now();
const consoleLines: string[] = [];
const pageErrors: string[] = [];
page.on("console", (message) => {
if (message.type() === "error") {
consoleLines.push(`${message.type()}: ${message.text()}`);
}
});
page.on("pageerror", (error) => {
pageErrors.push(error.message);
});
try {
const result = await runSamplePageFlow({
page,
request,
sample,
baseURL,
apiAccessKey,
reportDir,
timeoutMs,
});
const enriched = enrichWithBrowserErrors(result, consoleLines, pageErrors);
writeSampleResult(reportDir, enriched);
if (enriched.status === "failed") {
throw new Error(enriched.failure_message ?? enriched.failure_category);
}
} catch (error) {
const failureScreenshot = join(
reportDir,
"samples",
sample.slug,
"failure.png",
);
await page.screenshot({ path: failureScreenshot, fullPage: true }).catch(() => {});
const failedResult = buildFailureResult({
sample,
error,
startedAt,
failureScreenshot,
consoleLines,
pageErrors,
});
writeSampleResult(reportDir, failedResult);
throw error;
}
});
}
});
function selectSamples(
samples: ArticleSample[],
sampleFilter: string | undefined,
sampleLimit: number,
) {
const filtered = sampleFilter
? samples.filter(
(sample) =>
sample.fileName.includes(sampleFilter) ||
sample.name.includes(sampleFilter) ||
sample.slug.includes(sampleFilter),
)
: samples;
return sampleLimit > 0 ? filtered.slice(0, sampleLimit) : filtered;
}
function enrichWithBrowserErrors(
result: SampleResult,
consoleLines: string[],
pageErrors: string[],
): SampleResult {
if (consoleLines.length === 0 && pageErrors.length === 0) return result;
return {
...result,
status: "failed",
failure_category: "console_error",
failure_message: [...consoleLines, ...pageErrors].join("\n"),
};
}
function buildFailureResult({
sample,
error,
startedAt,
failureScreenshot,
consoleLines,
pageErrors,
}: {
sample: ArticleSample;
error: unknown;
startedAt: number;
failureScreenshot: string;
consoleLines: string[];
pageErrors: string[];
}): SampleResult {
const message = error instanceof Error ? error.message : String(error);
return {
file: sample.filePath,
name: sample.name,
slug: sample.slug,
status: "failed",
duration_ms: Date.now() - startedAt,
qa_fail_rules: [],
qa_warn_rules: [],
expected_hard_failures: sample.expectedHardFailures,
expected_warnings: sample.expectedWarnings,
exports: {
"optimized.md": "failed",
"optimized.docx": "failed",
"qa_report.json": "failed",
},
llm_tasks: [],
failure_category: categorizeFailure(message),
failure_message: [...consoleLines, ...pageErrors, message].filter(Boolean).join("\n"),
artifacts: {
failure_screenshot: failureScreenshot,
},
};
}
function categorizeFailure(message: string): FailureCategory {
if (/timeout/i.test(message)) return "stream_timeout";
if (/LLM|provider|schema|quota|rate/i.test(message)) return "llm_failed";
if (/export|optimized\.md|optimized\.docx|qa_report\.json/i.test(message)) {
return "export_failed";
}
if (/console|pageerror/i.test(message)) return "console_error";
if (/failed|优化失败/i.test(message)) return "stream_failed";
return "page_flow_failed";
}
- Step 2: Create the dedicated Playwright config
Create playwright.samples.config.ts:
import { defineConfig, devices } from "@playwright/test";
const baseURL = process.env.GEO_SAMPLE_BASE_URL ?? "http://127.0.0.1:3000";
const timeout = Number(process.env.GEO_SAMPLE_TIMEOUT_MS ?? "600000") + 60_000;
export default defineConfig({
testDir: "./tests/e2e",
testMatch: "sample-flow.spec.ts",
fullyParallel: false,
workers: 1,
timeout,
retries: 0,
reporter: [["list"]],
use: {
baseURL,
trace: "retain-on-failure",
screenshot: "only-on-failure",
video: "retain-on-failure",
actionTimeout: 30_000,
navigationTimeout: 60_000,
},
projects: [
{
name: "chromium",
use: { ...devices["Desktop Chrome"] },
},
],
});
- Step 3: Run a type check
Run:
npx tsc --noEmit
Expected: PASS. If moduleResolution or Playwright import typing causes a TypeScript error, adjust imports rather than suppressing the error.
- Step 4: Commit
Run:
git add tests/e2e/sample-flow.spec.ts playwright.samples.config.ts
git commit -m "新增样例文章实时验收测试"
Task 5: CLI Runner, Environment Loading, and Summary Aggregation
Files:
-
Create:
scripts/run-geo-sample-e2e.mjs -
Modify:
package.json -
Step 1: Create the runner script
Create scripts/run-geo-sample-e2e.mjs:
#!/usr/bin/env node
import { createServer } from "node:net";
import {
existsSync,
mkdirSync,
readdirSync,
readFileSync,
writeFileSync,
} from "node:fs";
import { join, resolve } from "node:path";
import { spawn } from "node:child_process";
import { fileURLToPath } from "node:url";
const repoRoot = resolve(fileURLToPath(new URL("..", import.meta.url)));
const startedAt = new Date().toISOString();
const args = parseArgs(process.argv.slice(2));
const envFile = join(repoRoot, ".env.local");
const fileEnv = existsSync(envFile) ? parseEnvFile(readFileSync(envFile, "utf8")) : {};
const mergedEnv = { ...fileEnv, ...process.env };
const provider = (mergedEnv.LLM_PROVIDER || "deepseek").toLowerCase();
const model =
provider === "openai"
? mergedEnv.OPENAI_MODEL || "gpt-4.1-mini"
: mergedEnv.DEEPSEEK_MODEL || "deepseek-v4-pro";
const timestamp = startedAt.replace(/[:.]/g, "-");
const reportDir = resolve(
repoRoot,
"test-results",
"geo-sample-flow",
timestamp,
);
await main();
async function main() {
preflight(mergedEnv, provider);
mkdirSync(reportDir, { recursive: true });
const port = args.port ? Number(args.port) : await findAvailablePort(3000);
const baseURL = args.baseURL || `http://127.0.0.1:${port}`;
const appDataDir = join(reportDir, "app-data");
mkdirSync(appDataDir, { recursive: true });
const runEnv = {
...mergedEnv,
APP_DATA_DIR: appDataDir,
GEO_SAMPLE_BASE_URL: baseURL,
GEO_SAMPLE_REPORT_DIR: reportDir,
GEO_SAMPLE_TIMEOUT_MS: String(args.timeoutMs ?? 600000),
GEO_SAMPLE_FILTER: args.sample ?? "",
GEO_SAMPLE_LIMIT: args.limit ? String(args.limit) : "",
};
let serverProcess = null;
if (!args.reuseServer) {
serverProcess = spawn(
"npm",
["run", "dev", "--", "--port", String(port)],
{
cwd: repoRoot,
env: runEnv,
stdio: ["ignore", "pipe", "pipe"],
},
);
captureServerLogs(serverProcess, reportDir, runEnv);
await waitForServer(baseURL, 120000);
}
const status = await runPlaywright(runEnv);
await aggregateReport({
reportDir,
startedAt,
provider,
model,
baseURL,
invalidSamplesPath: join(reportDir, "invalid-samples.json"),
});
if (serverProcess) {
serverProcess.kill("SIGTERM");
}
process.exitCode = status;
}
function parseArgs(rawArgs) {
const parsed = {};
for (let index = 0; index < rawArgs.length; index += 1) {
const arg = rawArgs[index];
if (arg === "--sample") parsed.sample = rawArgs[++index];
else if (arg === "--limit") parsed.limit = Number(rawArgs[++index]);
else if (arg === "--timeout-ms") parsed.timeoutMs = Number(rawArgs[++index]);
else if (arg === "--port") parsed.port = Number(rawArgs[++index]);
else if (arg === "--base-url") parsed.baseURL = rawArgs[++index];
else if (arg === "--reuse-server") parsed.reuseServer = true;
else if (arg === "--headed") parsed.headed = true;
}
return parsed;
}
function parseEnvFile(content) {
const values = {};
for (const line of content.split(/\r?\n/)) {
const trimmed = line.trim();
if (!trimmed || trimmed.startsWith("#")) continue;
const match = trimmed.match(/^([A-Za-z_][A-Za-z0-9_]*)=(.*)$/);
if (!match) continue;
values[match[1]] = match[2].replace(/^['"]|['"]$/g, "");
}
return values;
}
function preflight(env, providerName) {
if (env.API_AUTH_DISABLED !== "true" && !env.API_ACCESS_KEY) {
throw new Error("API_ACCESS_KEY is required unless API_AUTH_DISABLED=true");
}
if (providerName === "openai" && !env.OPENAI_API_KEY) {
throw new Error("OPENAI_API_KEY is required for live sample E2E");
}
if (providerName !== "openai" && !env.DEEPSEEK_API_KEY) {
throw new Error("DEEPSEEK_API_KEY is required for live sample E2E");
}
}
async function findAvailablePort(startPort) {
for (let port = startPort; port < startPort + 50; port += 1) {
if (await canListen(port)) return port;
}
throw new Error(`no available port found from ${startPort}`);
}
function canListen(port) {
return new Promise((resolveCanListen) => {
const server = createServer();
server.once("error", () => resolveCanListen(false));
server.once("listening", () => {
server.close(() => resolveCanListen(true));
});
server.listen(port, "127.0.0.1");
});
}
function captureServerLogs(child, outputDir, env) {
const logPath = join(outputDir, "server.log");
const secrets = [
env.API_ACCESS_KEY,
env.DEEPSEEK_API_KEY,
env.OPENAI_API_KEY,
].filter(Boolean);
const append = (chunk) => {
const text = redact(String(chunk), secrets);
writeFileSync(logPath, text, { flag: "a" });
};
child.stdout.on("data", append);
child.stderr.on("data", append);
}
function redact(text, secrets) {
let next = text
.replace(/(x-api-key:\s*)([^\s]+)/gi, "$1[REDACTED]")
.replace(/(authorization:\s*bearer\s+)([^\s]+)/gi, "$1[REDACTED]");
for (const secret of secrets) {
next = next.split(secret).join("[REDACTED]");
}
return next;
}
async function waitForServer(baseURL, timeoutMs) {
const started = Date.now();
while (Date.now() - started < timeoutMs) {
try {
const response = await fetch(baseURL, { method: "HEAD" });
if (response.ok) return;
} catch {
await delay(1000);
}
}
throw new Error(`server did not become ready at ${baseURL}`);
}
function delay(ms) {
return new Promise((resolveDelay) => setTimeout(resolveDelay, ms));
}
function runPlaywright(env) {
return new Promise((resolveRun) => {
const argsForPlaywright = [
"playwright",
"test",
"--config",
"playwright.samples.config.ts",
];
if (args.headed) argsForPlaywright.push("--headed");
const child = spawn("npx", argsForPlaywright, {
cwd: repoRoot,
env,
stdio: "inherit",
});
child.on("close", (code) => resolveRun(code ?? 1));
});
}
async function aggregateReport({
reportDir: outputDir,
startedAt: startTime,
provider: providerName,
model: modelName,
baseURL,
invalidSamplesPath,
}) {
const invalidSamples = existsSync(invalidSamplesPath)
? JSON.parse(readFileSync(invalidSamplesPath, "utf8"))
: [];
const samples = readSampleResults(outputDir);
const summary = {
started_at: startTime,
finished_at: new Date().toISOString(),
mode: "live",
provider: providerName,
model: modelName,
base_url: baseURL,
report_dir: outputDir,
totals: {
passed: samples.filter((sample) => sample.status === "passed").length,
failed: samples.filter((sample) => sample.status === "failed").length,
skipped:
samples.filter((sample) => sample.status === "skipped").length +
invalidSamples.length,
},
samples,
invalid_samples: invalidSamples,
};
writeFileSync(join(outputDir, "summary.json"), `${JSON.stringify(summary, null, 2)}\n`);
writeFileSync(join(outputDir, "summary.md"), renderSummaryMarkdown(summary));
console.log(`GEO sample E2E report: ${join(outputDir, "summary.md")}`);
}
function readSampleResults(outputDir) {
const samplesDir = join(outputDir, "samples");
if (!existsSync(samplesDir)) return [];
return readdirSync(samplesDir, { withFileTypes: true })
.filter((entry) => entry.isDirectory())
.map((entry) => entry.name)
.map((slug) => join(samplesDir, slug, "result.json"))
.filter((filePath) => existsSync(filePath))
.map((filePath) => JSON.parse(readFileSync(filePath, "utf8")))
.sort((left, right) => left.file.localeCompare(right.file));
}
function renderSummaryMarkdown(summary) {
const rows = summary.samples.map((sample) =>
`| ${sample.name} | ${sample.status} | ${sample.qa_status ?? ""} | ${Math.round(sample.duration_ms / 1000)}s | ${sample.failure_category ?? ""} |`,
);
const lines = [
"# GEO 样例文章 E2E 测试报告",
"",
`- 模式:${summary.mode}`,
`- Provider:${summary.provider}`,
`- Model:${summary.model}`,
`- Base URL:${summary.base_url}`,
`- 报告目录:${summary.report_dir}`,
`- 结果:通过 ${summary.totals.passed},失败 ${summary.totals.failed},跳过 ${summary.totals.skipped}`,
"",
"| 样例 | 状态 | QA | 耗时 | 失败分类 |",
"| --- | --- | --- | ---: | --- |",
...rows,
];
if (summary.invalid_samples.length > 0) {
lines.push("", "## 无效样例", "");
for (const sample of summary.invalid_samples) {
lines.push(`- ${sample.fileName}: ${sample.reason}`);
}
}
lines.push("");
return lines.join("\n");
}
- Step 2: Verify preflight behavior
Run:
env DEEPSEEK_API_KEY= OPENAI_API_KEY= node scripts/run-geo-sample-e2e.mjs --limit 1 --reuse-server --base-url http://127.0.0.1:3000
Expected when env keys are missing: FAIL with DEEPSEEK_API_KEY is required for live sample E2E or OPENAI_API_KEY is required for live sample E2E.
- Step 3: Add npm script
Modify package.json scripts:
{
"scripts": {
"test:e2e:samples": "node scripts/run-geo-sample-e2e.mjs"
}
}
Keep existing scripts unchanged.
- Step 4: Run static checks
Run:
npx tsc --noEmit
npm run lint
Expected: both commands exit 0.
- Step 5: Commit
Run:
git add scripts/run-geo-sample-e2e.mjs package.json package-lock.json
git commit -m "新增样例文章验收运行脚本"
Task 6: Live Smoke Run With One Sample
Files:
-
Modify if failures require it:
tests/e2e/sample-flow/page-flow.ts -
Modify if failures require it:
tests/e2e/sample-flow.spec.ts -
Modify if failures require it:
scripts/run-geo-sample-e2e.mjs -
Step 1: Run one real sample
Run:
npm run test:e2e:samples -- --limit 1
Expected: the runner starts a local Next.js server, runs exactly one sample, writes test-results/geo-sample-flow/<timestamp>/summary.md, and exits 0 if the sample completes.
- Step 2: If the run fails, classify using captured evidence
Open:
ls -R test-results/geo-sample-flow | tail -80
Expected: latest report directory contains summary.json, summary.md, server.log, and at least one samples/<slug>/result.json.
Use the failure_category in result.json:
node -e 'const fs=require("fs"); const p=process.argv[1]; console.log(JSON.stringify(JSON.parse(fs.readFileSync(p,"utf8")), null, 2));' test-results/geo-sample-flow/<timestamp>/samples/<slug>/result.json
Expected categories are one of the values from FailureCategory.
- Step 3: Fix only runner/test bugs
If failure is caused by wrong selector, export validation, report writing, or server startup, patch the corresponding helper. Do not change production app behavior in this task unless the browser flow reveals a clear production regression that prevents the current UI from working.
For common selector drift, update fillFirstAvailable labels in tests/e2e/sample-flow/page-flow.ts:
await fillFirstAvailable(page, ["文章内容", "粘贴文章", "正文"], sample.input.body);
For completion text drift, update the regex in tests/e2e/sample-flow/page-flow.ts while keeping both current messages:
await expect(page.getByText(/优化完成|质检发现需要复核的问题/)).toBeVisible({
timeout: timeoutMs,
});
- Step 4: Re-run one sample
Run:
npm run test:e2e:samples -- --limit 1
Expected: PASS, or FAIL with a real LLM/provider/schema/page issue captured in the report. If it fails because of a real app or LLM issue, stop and report the evidence before broadening the test run.
- Step 5: Commit runner/test fixes
Run:
git add tests/e2e/sample-flow/page-flow.ts tests/e2e/sample-flow.spec.ts scripts/run-geo-sample-e2e.mjs
git commit -m "验证样例文章验收冒烟流程"
Task 7: README Documentation
Files:
-
Modify:
README.md -
Step 1: Document the command
Add this section after the existing Commands section:
## Sample Article E2E
Run the live sample-article browser workflow:
```bash
npm run test:e2e:samples
The runner reads .env.local, starts an isolated local Next.js server, loads
samples/articles/*.json, opens the web UI with Playwright, runs each sample
through the one-click optimization flow, validates authenticated exports, and
writes a report under:
test-results/geo-sample-flow/<timestamp>/
Useful options:
npm run test:e2e:samples -- --limit 1
npm run test:e2e:samples -- --sample title-quality
npm run test:e2e:samples -- --headed
npm run test:e2e:samples -- --reuse-server --base-url http://127.0.0.1:3000
This workflow uses the real configured LLM provider by default. It requires
API_ACCESS_KEY and the provider API key in .env.local or the shell
environment. Reports and exports are written to test-results/, which is
ignored by Git.
- [ ] **Step 2: Verify Markdown formatting**
Run:
```bash
git diff --check -- README.md
Expected: exit 0.
- Step 3: Commit
Run:
git add README.md
git commit -m "补充样例文章验收测试说明"
Task 8: Full Verification and Final Integration Commit
Files:
-
No new files expected.
-
Modify only files needed to fix verification failures.
-
Step 1: Run focused unit tests
Run:
npm test -- tests/e2e/sample-flow/__tests__/sample-loader.test.ts tests/e2e/sample-flow/__tests__/reporting.test.ts
Expected: PASS.
- Step 2: Run repository tests
Run:
npm test
Expected: PASS.
- Step 3: Run lint
Run:
npm run lint
Expected: PASS.
- Step 4: Run build
Run:
npm run build
Expected: PASS.
- Step 5: Run one live sample
Run:
npm run test:e2e:samples -- --limit 1
Expected: PASS, with a report path printed. If real LLM instability causes failure, the final response must include the report path, failure_category, and LLM/server error evidence instead of claiming the live sample passes.
- Step 6: Inspect Git status and ignored outputs
Run:
git status --short
git status --short --ignored=matching test-results
Expected: source files are clean or only intended source edits are staged; test-results/ appears as ignored output if reports were generated.
- Step 7: Commit remaining verification fixes
If any fixes were made during verification, run:
git add <changed-source-files>
git commit -m "完善样例文章自动验收工作流"
Do not commit test-results/, exported articles, local databases, .env.local, or provider logs.
Spec Coverage Self-Review
- Live LLM default: Task 5 preflight requires provider keys; Task 6 and Task 8 run live samples.
- Browser flow: Task 3 and Task 4 open the page, fill the current UI, click
开始优化, and wait for visible results. - Sample folder input: Task 1 reads and validates
samples/articles/*.json. - Report artifacts: Task 2 writes per-sample results and summaries; Task 3 stores screenshots and exports.
- Export validation: Task 3 validates
optimized.md,optimized.docx, andqa_report.jsonwithx-api-key. - Failure evidence: Task 4 captures console/page errors; Task 5 captures server logs; Task 6 uses
failure_category. - Git safety: Task 7 documents ignored reports; Task 8 checks ignored
test-results/. - Future skill boundary: Architecture keeps the behavior in repo scripts so a later skill can call
npm run test:e2e:samples.