diff --git a/docs/superpowers/plans/2026-06-16-deepseek-workflow-integration.md b/docs/superpowers/plans/2026-06-16-deepseek-workflow-integration.md new file mode 100644 index 0000000..f46dcfb --- /dev/null +++ b/docs/superpowers/plans/2026-06-16-deepseek-workflow-integration.md @@ -0,0 +1,1278 @@ +# DeepSeek Workflow Integration Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Wire the configured DeepSeek/OpenAI-compatible LLM client into the real optimization workflow while preserving deterministic local fallback behavior for tests and demos. + +**Architecture:** Keep the existing workflow modules and orchestrator shape. Each LLM-enabled node first tries `generateJson` when `isLlmConfigured()` is true, validates the model output with the existing Zod schemas, and falls back to the current deterministic implementation if the provider is unconfigured, errors, or returns invalid data. Quality inspection remains a hybrid gate: deterministic hard rules stay authoritative, while LLM output can enrich warnings, evidence, and suggestions without weakening hard failures. + +**Tech Stack:** Next.js route handlers, TypeScript, Zod, OpenAI SDK with DeepSeek base URL, Vitest module mocks, existing SQLite/export workflow. + +--- + +## Scope + +This plan implements live LLM use in the existing single-job workflow only. It does not add batch queues, user accounts, publishing integrations, `.docx` parsing, brand-template UI, streaming responses, or model-provider admin screens. + +## File Structure + +- Modify: `src/lib/llm/client.ts` + - Keep provider detection and `generateJson`. + - Export a typed helper `generateValidatedJson` that accepts a Zod schema and returns `null` when LLM use is unavailable or invalid. +- Modify: `src/lib/llm/prompts.ts` + - Add prompt builders for fact extraction, article optimization, QA enrichment, and targeted rewrite. +- Modify: `src/lib/workflow/fact-extractor.ts` + - Rename the current rules implementation to a fallback function. + - Try DeepSeek first and validate `CandidateFactCard`. +- Modify: `src/lib/workflow/article-optimizer.ts` + - Rename the current template implementation to a fallback function. + - Try DeepSeek first and validate `OptimizedArticle`. +- Modify: `src/lib/workflow/targeted-rewriter.ts` + - Make rewriting async. + - Try DeepSeek first for failed checks and fall back to current deterministic targeted fixes. +- Modify: `src/lib/workflow/orchestrator.ts` + - Await async targeted rewriting. +- Modify: `src/lib/workflow/quality-inspector.ts` + - Add optional async LLM enrichment through a new exported `inspectQualityWithLlm` while keeping `inspectQuality` as deterministic logic for unit tests and fallback. +- Modify: `src/lib/workflow/orchestrator.ts` + - Use `inspectQualityWithLlm` in the real workflow. +- Test: `src/lib/llm/__tests__/client.test.ts` + - Verify invalid LLM data returns `null` from the helper. +- Test: `src/lib/workflow/__tests__/llm-integration.test.ts` + - Mock the LLM client and prove each workflow node calls it when configured. + - Prove invalid or rejected LLM calls fall back to deterministic behavior. +- Modify: `src/lib/workflow/__tests__/workflow.test.ts` + - Update targeted rewriter tests to await the async function. +- Modify: `src/app/api/__tests__/jobs.test.ts` + - Add an API-level test proving a mocked LLM article can flow through `/optimize`. +- Modify: `README.md` + - Clarify that configured DeepSeek is used by workflow nodes, with deterministic fallback when unavailable. + +## Task 1: Add Validated LLM Helper + +**Files:** + +- Modify: `src/lib/llm/client.ts` +- Create: `src/lib/llm/__tests__/client.test.ts` + +- [ ] **Step 1: Write the failing tests** + + Create `src/lib/llm/__tests__/client.test.ts`: + + ```ts + import { afterEach, describe, expect, it, vi } from "vitest"; + import { z } from "zod"; + + import * as client from "../client"; + + describe("generateValidatedJson", () => { + const originalProvider = process.env.LLM_PROVIDER; + const originalDeepSeekKey = process.env.DEEPSEEK_API_KEY; + + afterEach(() => { + process.env.LLM_PROVIDER = originalProvider; + process.env.DEEPSEEK_API_KEY = originalDeepSeekKey; + vi.restoreAllMocks(); + }); + + it("returns null when no provider key is configured", async () => { + process.env.LLM_PROVIDER = "deepseek"; + delete process.env.DEEPSEEK_API_KEY; + + const result = await client.generateValidatedJson({ + schema: z.object({ value: z.string() }), + prompt: "Return JSON.", + }); + + expect(result).toBeNull(); + }); + + it("returns parsed data when the model response matches the schema", async () => { + process.env.LLM_PROVIDER = "deepseek"; + process.env.DEEPSEEK_API_KEY = "test-key"; + vi.spyOn(client, "generateJson").mockResolvedValue({ value: "from-llm" }); + + const result = await client.generateValidatedJson({ + schema: z.object({ value: z.string() }), + prompt: "Return JSON.", + }); + + expect(result).toEqual({ value: "from-llm" }); + }); + + it("returns null when the model response fails schema validation", async () => { + process.env.LLM_PROVIDER = "deepseek"; + process.env.DEEPSEEK_API_KEY = "test-key"; + vi.spyOn(client, "generateJson").mockResolvedValue({ value: 42 }); + + const result = await client.generateValidatedJson({ + schema: z.object({ value: z.string() }), + prompt: "Return JSON.", + }); + + expect(result).toBeNull(); + }); + + it("returns null when the provider call rejects", async () => { + process.env.LLM_PROVIDER = "deepseek"; + process.env.DEEPSEEK_API_KEY = "test-key"; + vi.spyOn(client, "generateJson").mockRejectedValue(new Error("provider down")); + + const result = await client.generateValidatedJson({ + schema: z.object({ value: z.string() }), + prompt: "Return JSON.", + }); + + expect(result).toBeNull(); + }); + }); + ``` + +- [ ] **Step 2: Run the test to verify it fails** + + Run: + + ```bash + npm test -- src/lib/llm/__tests__/client.test.ts + ``` + + Expected: FAIL with a TypeScript or runtime error indicating `generateValidatedJson` is not exported. + +- [ ] **Step 3: Implement the helper** + + Modify `src/lib/llm/client.ts`. + + Add this import below the existing `openai` import: + + ```ts + import type { z } from "zod"; + ``` + + Add these interfaces after `GenerateInput`: + + ```ts + export interface GenerateValidatedJsonInput extends GenerateInput { + schema: z.ZodType; + } + ``` + + Add this exported function after `generateJson`: + + ```ts + export async function generateValidatedJson({ + schema, + ...input + }: GenerateValidatedJsonInput): Promise { + if (!isLlmConfigured()) { + return null; + } + + try { + const generated = await generateJson(input); + return schema.parse(generated); + } catch { + return null; + } + } + ``` + +- [ ] **Step 4: Run the test to verify it passes** + + Run: + + ```bash + npm test -- src/lib/llm/__tests__/client.test.ts + ``` + + Expected: PASS for all four tests. + +- [ ] **Step 5: Commit** + + ```bash + git add src/lib/llm/client.ts src/lib/llm/__tests__/client.test.ts + git commit -m "feat: add validated llm json helper" + ``` + +## Task 2: Add Prompt Builders + +**Files:** + +- Modify: `src/lib/llm/prompts.ts` + +- [ ] **Step 1: Replace `prompts.ts` with explicit builders** + + Replace `src/lib/llm/prompts.ts` with: + + ```ts + import type { + ArticleInput, + ConfirmedFactCard, + OptimizedArticle, + PublishPlatform, + QaCheck, + } from "../domain/types"; + + export const JSON_ONLY_PROMPT = + "Return valid JSON only. Do not include markdown fences or commentary."; + + export const ARTICLE_OPTIMIZER_SYSTEM_PROMPT = [ + "You optimize GEO-related articles under a confirmed fact card.", + "Never invent numbers, cases, qualifications, company names, products, or years.", + "Use only facts present in the confirmed fact card or source article.", + JSON_ONLY_PROMPT, + ].join(" "); + + export const QUALITY_INSPECTOR_SYSTEM_PROMPT = [ + "Evaluate article quality against the confirmed fact card and target platform.", + "Return one structured check per required quality gate when asked.", + "Do not downgrade deterministic hard failures supplied by the application.", + JSON_ONLY_PROMPT, + ].join(" "); + + export const FACT_EXTRACTOR_SYSTEM_PROMPT = [ + "Extract a candidate fact card from the source article.", + "Do not mark uncertain facts as confirmed.", + "Put missing or conflicting facts in uncertain_items.", + JSON_ONLY_PROMPT, + ].join(" "); + + export const TARGETED_REWRITER_SYSTEM_PROMPT = [ + "Rewrite only the fields needed to resolve the provided failed QA checks.", + "Keep confirmed facts unchanged.", + "Preserve sections that are unrelated to failed checks.", + JSON_ONLY_PROMPT, + ].join(" "); + + export function buildFactExtractorPrompt(input: ArticleInput) { + return [ + "Return a CandidateFactCard JSON object with these exact keys:", + "company_full_name, company_short_names, brand_names, product_names, target_industry, target_audience, experience_years, core_claims, forbidden_claims, image_topics, uncertain_items.", + "Do not include confirmed_by_user.", + "", + "Article input:", + JSON.stringify(input, null, 2), + ].join("\n"); + } + + export function buildArticleOptimizerPrompt(input: ArticleInput, factCard: ConfirmedFactCard) { + return [ + "Return an OptimizedArticle JSON object with these exact keys:", + "title, summary, body_markdown, image_suggestions, changed_sections, requires_user_confirmation.", + "Use Markdown in body_markdown.", + "If the user instruction asks for unsupported facts, omit them from the article and add them to requires_user_confirmation.", + "", + "Confirmed fact card:", + JSON.stringify(factCard, null, 2), + "", + "Article input:", + JSON.stringify(input, null, 2), + ].join("\n"); + } + + export function buildQualityInspectorPrompt(input: { + article: OptimizedArticle; + factCard: ConfirmedFactCard; + platform: PublishPlatform; + deterministicChecks: QaCheck[]; + }) { + return [ + "Return a JSON object with a checks array.", + "Each check must include rule_id, status, evidence, reason, suggested_fix, and target_agent.", + "Only use rule_id values already present in deterministicChecks.", + "If a deterministic check has status fail, keep it fail.", + "", + "Target platform:", + input.platform, + "", + "Confirmed fact card:", + JSON.stringify(input.factCard, null, 2), + "", + "Optimized article:", + JSON.stringify(input.article, null, 2), + "", + "Deterministic checks:", + JSON.stringify(input.deterministicChecks, null, 2), + ].join("\n"); + } + + export function buildTargetedRewritePrompt(input: { + article: OptimizedArticle; + factCard: ConfirmedFactCard; + failedChecks: QaCheck[]; + }) { + return [ + "Return an OptimizedArticle JSON object.", + "Rewrite only the fields needed for failedChecks.", + "Do not add unconfirmed numbers, customer names, qualifications, awards, or capabilities.", + "", + "Confirmed fact card:", + JSON.stringify(input.factCard, null, 2), + "", + "Current optimized article:", + JSON.stringify(input.article, null, 2), + "", + "Failed checks:", + JSON.stringify(input.failedChecks, null, 2), + ].join("\n"); + } + ``` + +- [ ] **Step 2: Run existing tests** + + Run: + + ```bash + npm test + ``` + + Expected: PASS. No workflow code uses the new builders yet. + +- [ ] **Step 3: Commit** + + ```bash + git add src/lib/llm/prompts.ts + git commit -m "feat: add llm workflow prompt builders" + ``` + +## Task 3: Connect Fact Extraction To LLM + +**Files:** + +- Modify: `src/lib/workflow/fact-extractor.ts` +- Create: `src/lib/workflow/__tests__/llm-integration.test.ts` + +- [ ] **Step 1: Write failing fact-extractor tests** + + Create `src/lib/workflow/__tests__/llm-integration.test.ts`: + + ```ts + import { afterEach, describe, expect, it, vi } from "vitest"; + + import { extractCandidateFactCard } from "../fact-extractor"; + + vi.mock("../../llm/client", async () => { + const actual = await vi.importActual( + "../../llm/client", + ); + return { + ...actual, + generateValidatedJson: vi.fn(), + }; + }); + + const llmClient = await import("../../llm/client"); + + describe("LLM workflow integration", () => { + afterEach(() => { + vi.mocked(llmClient.generateValidatedJson).mockReset(); + }); + + it("uses LLM output for candidate fact extraction when valid", async () => { + vi.mocked(llmClient.generateValidatedJson).mockResolvedValueOnce({ + company_full_name: "DeepSeek Example Co., Ltd.", + company_short_names: ["DeepSeek Example"], + brand_names: ["DSExample"], + product_names: ["DS GEO"], + target_industry: "GEO optimization", + target_audience: "Marketing teams", + experience_years: 9, + core_claims: ["9 years of GEO optimization experience"], + forbidden_claims: ["industry first"], + image_topics: ["dashboard"], + uncertain_items: [], + is_ready_for_optimization: true, + }); + + const card = await extractCandidateFactCard({ + title: "Example source", + body: "Fallback Technology Co., Ltd. has 8 years of GEO optimization experience.", + images: [{ type: "description", content: "dashboard" }], + platform: "official_site", + user_instructions: "", + }); + + expect(card.company_full_name).toBe("DeepSeek Example Co., Ltd."); + expect(card.experience_years).toBe(9); + expect(llmClient.generateValidatedJson).toHaveBeenCalledOnce(); + }); + + it("falls back to deterministic candidate extraction when LLM returns null", async () => { + vi.mocked(llmClient.generateValidatedJson).mockResolvedValueOnce(null); + + const card = await extractCandidateFactCard({ + title: "Fallback Technology Co., Ltd. GEO guide", + body: "Fallback Technology Co., Ltd. has 8 years of GEO optimization experience.", + images: [{ type: "description", content: "dashboard" }], + platform: "official_site", + user_instructions: "", + }); + + expect(card.company_full_name).toBe("Fallback Technology Co., Ltd."); + expect(card.experience_years).toBe(8); + }); + }); + ``` + +- [ ] **Step 2: Run the test to verify it fails** + + Run: + + ```bash + npm test -- src/lib/workflow/__tests__/llm-integration.test.ts + ``` + + Expected: FAIL because `extractCandidateFactCard` does not call `generateValidatedJson`. + +- [ ] **Step 3: Implement LLM-first fact extraction** + + Modify the top of `src/lib/workflow/fact-extractor.ts` to add imports: + + ```ts + import { generateValidatedJson } from "../llm/client"; + import { + FACT_EXTRACTOR_SYSTEM_PROMPT, + buildFactExtractorPrompt, + } from "../llm/prompts"; + ``` + + Replace the current exported `extractCandidateFactCard` body with: + + ```ts + export async function extractCandidateFactCard( + input: ArticleInput, + ): Promise { + const llmCard = await generateValidatedJson({ + schema: candidateFactCardSchema, + system: FACT_EXTRACTOR_SYSTEM_PROMPT, + prompt: buildFactExtractorPrompt(input), + temperature: 0.1, + }); + + return llmCard ?? extractCandidateFactCardFallback(input); + } + + function extractCandidateFactCardFallback(input: ArticleInput): CandidateFactCard { + const text = `${input.title}\n${input.body}`; + const uncertainItems: string[] = []; + const companyFullName = findCompanyFullName(text); + const years = findExperienceYears(text); + + if (!companyFullName) { + uncertainItems.push("Missing company full name"); + } + if (years.length > 1) { + uncertainItems.push(`Conflicting experience years: ${years.join(", ")}`); + } + if (input.images.length === 0) { + uncertainItems.push("Image description is missing"); + } + + const industry = inferIndustry(text); + + return candidateFactCardSchema.parse({ + company_full_name: companyFullName ?? "", + company_short_names: inferCompanyShortNames(companyFullName), + brand_names: inferBrandNames(text, companyFullName), + product_names: inferProducts(text), + target_industry: industry, + target_audience: text.toLowerCase().includes("marketing") + ? "Marketing teams" + : inferTargetAudience(text), + experience_years: years.length === 1 ? years[0] : null, + core_claims: years.length === 1 ? [`${years[0]} years of ${industry} experience`] : [], + forbidden_claims: [], + image_topics: input.images.map((image) => image.content), + uncertain_items: uncertainItems, + }); + } + ``` + + Keep the helper functions `findCompanyFullName`, `findExperienceYears`, `inferIndustry`, `inferBrandNames`, `inferProducts`, `inferCompanyShortNames`, `inferChineseShortName`, and `inferTargetAudience` unchanged below the fallback function. + +- [ ] **Step 4: Run targeted and existing workflow tests** + + Run: + + ```bash + npm test -- src/lib/workflow/__tests__/llm-integration.test.ts src/lib/workflow/__tests__/workflow.test.ts + ``` + + Expected: PASS. + +- [ ] **Step 5: Commit** + + ```bash + git add src/lib/workflow/fact-extractor.ts src/lib/workflow/__tests__/llm-integration.test.ts + git commit -m "feat: use llm for fact extraction" + ``` + +## Task 4: Connect Article Optimization To LLM + +**Files:** + +- Modify: `src/lib/workflow/article-optimizer.ts` +- Modify: `src/lib/workflow/__tests__/llm-integration.test.ts` + +- [ ] **Step 1: Add failing optimizer tests** + + Append these imports to `src/lib/workflow/__tests__/llm-integration.test.ts`: + + ```ts + import { optimizeArticle } from "../article-optimizer"; + ``` + + Add this shared fact card inside the `describe` block: + + ```ts + const confirmedFactCard = { + company_full_name: "Example Technology Co., Ltd.", + company_short_names: ["Example Tech"], + brand_names: ["Example"], + product_names: ["Example GEO"], + target_industry: "GEO optimization", + target_audience: "Marketing teams", + experience_years: 8, + core_claims: ["8 years of GEO optimization experience"], + forbidden_claims: ["industry first"], + image_topics: ["dashboard"], + uncertain_items: [], + is_ready_for_optimization: true, + confirmed_by_user: true, + } as const; + ``` + + Add these tests inside the `describe` block: + + ```ts + it("uses LLM output for article optimization when valid", async () => { + vi.mocked(llmClient.generateValidatedJson).mockResolvedValueOnce({ + title: "LLM Optimized GEO Article", + summary: "LLM summary constrained by the fact card.", + body_markdown: "## LLM Body\nExample Technology Co., Ltd. keeps claims factual.", + image_suggestions: [{ source: "image_1", suggestion: "Use dashboard." }], + changed_sections: ["title", "body"], + requires_user_confirmation: [], + }); + + const article = await optimizeArticle({ + input: { + title: "Original", + body: "Example Technology Co., Ltd. has 8 years of GEO optimization experience.", + images: [{ type: "description", content: "dashboard" }], + platform: "official_site", + user_instructions: "", + }, + factCard: confirmedFactCard, + }); + + expect(article.title).toBe("LLM Optimized GEO Article"); + expect(article.body_markdown).toContain("LLM Body"); + expect(llmClient.generateValidatedJson).toHaveBeenCalledOnce(); + }); + + it("falls back to deterministic article optimization when LLM returns null", async () => { + vi.mocked(llmClient.generateValidatedJson).mockResolvedValueOnce(null); + + const article = await optimizeArticle({ + input: { + title: "Original", + body: "Example Technology Co., Ltd. has 8 years of GEO optimization experience.", + images: [{ type: "description", content: "dashboard" }], + platform: "official_site", + user_instructions: "Say we have 99 patents.", + }, + factCard: confirmedFactCard, + }); + + expect(article.title).toContain("GEO optimization Guide"); + expect(article.requires_user_confirmation).toContain( + "Unsupported requested claim: 99 patents", + ); + }); + ``` + +- [ ] **Step 2: Run the test to verify it fails** + + Run: + + ```bash + npm test -- src/lib/workflow/__tests__/llm-integration.test.ts + ``` + + Expected: FAIL because `optimizeArticle` does not call `generateValidatedJson`. + +- [ ] **Step 3: Implement LLM-first optimization** + + Modify the top of `src/lib/workflow/article-optimizer.ts` to add imports: + + ```ts + import { generateValidatedJson } from "../llm/client"; + import { + ARTICLE_OPTIMIZER_SYSTEM_PROMPT, + buildArticleOptimizerPrompt, + } from "../llm/prompts"; + ``` + + Replace the current exported `optimizeArticle` body with: + + ```ts + export async function optimizeArticle({ + input, + factCard, + }: OptimizeArticleInput): Promise { + const llmArticle = await generateValidatedJson({ + schema: optimizedArticleSchema, + system: ARTICLE_OPTIMIZER_SYSTEM_PROMPT, + prompt: buildArticleOptimizerPrompt(input, factCard), + temperature: 0.2, + }); + + return llmArticle ?? optimizeArticleFallback({ input, factCard }); + } + + function optimizeArticleFallback({ + input, + factCard, + }: OptimizeArticleInput): OptimizedArticle { + const unsupported = findUnsupportedInstructionClaims( + input.user_instructions, + factCard, + ); + const title = `${factCard.brand_names[0] ?? factCard.company_short_names[0] ?? factCard.company_full_name} ${factCard.target_industry} Guide`; + const coreClaims = + factCard.core_claims.length > 0 + ? factCard.core_claims.map((claim) => `- ${claim}`).join("\n") + : "- Confirmed facts only; no extra claims added."; + const body = [ + `## ${factCard.company_full_name}`, + cleanBody(input.body, factCard), + "", + "### Confirmed Facts", + coreClaims, + ].join("\n"); + + return optimizedArticleSchema.parse({ + title, + summary: `A ${input.platform.replace(/_/g, " ")} article for ${factCard.target_audience} about ${factCard.target_industry}.`, + body_markdown: body, + image_suggestions: factCard.image_topics.map((topic, index) => ({ + source: `image_${index + 1}`, + suggestion: `Use image content related to ${topic}.`, + })), + changed_sections: ["title", "body structure", "summary"], + requires_user_confirmation: unsupported, + }); + } + ``` + + Keep `cleanBody`, `findUnsupportedInstructionClaims`, and `escapeRegExp` unchanged below the fallback function. + +- [ ] **Step 4: Run targeted and workflow tests** + + Run: + + ```bash + npm test -- src/lib/workflow/__tests__/llm-integration.test.ts src/lib/workflow/__tests__/workflow.test.ts + ``` + + Expected: PASS. + +- [ ] **Step 5: Commit** + + ```bash + git add src/lib/workflow/article-optimizer.ts src/lib/workflow/__tests__/llm-integration.test.ts + git commit -m "feat: use llm for article optimization" + ``` + +## Task 5: Connect Targeted Rewrite To LLM + +**Files:** + +- Modify: `src/lib/workflow/targeted-rewriter.ts` +- Modify: `src/lib/workflow/orchestrator.ts` +- Modify: `src/lib/workflow/__tests__/workflow.test.ts` +- Modify: `src/lib/workflow/__tests__/llm-integration.test.ts` + +- [ ] **Step 1: Update and add failing rewrite tests** + + In `src/lib/workflow/__tests__/workflow.test.ts`, change the existing targeted rewrite test line: + + ```ts + const rewritten = rewriteFailedSections({ + ``` + + to: + + ```ts + const rewritten = await rewriteFailedSections({ + ``` + + Append this import to `src/lib/workflow/__tests__/llm-integration.test.ts`: + + ```ts + import { rewriteFailedSections } from "../targeted-rewriter"; + ``` + + Add these tests inside the existing `describe` block: + + ```ts + it("uses LLM output for targeted rewrite when valid", async () => { + vi.mocked(llmClient.generateValidatedJson).mockResolvedValueOnce({ + title: "Rewritten By LLM", + summary: "Original summary", + body_markdown: "## Body\nExample Technology Co., Ltd. focuses on GEO optimization.", + image_suggestions: [], + changed_sections: ["title"], + requires_user_confirmation: [], + }); + + const rewritten = await rewriteFailedSections({ + article: { + title: "Bad title!!!", + summary: "Original summary", + body_markdown: "## Body\nOriginal body", + image_suggestions: [], + changed_sections: [], + requires_user_confirmation: [], + }, + factCard: confirmedFactCard, + failedChecks: [ + { + rule_id: "title_quality", + status: "fail", + evidence: "Bad title!!!", + reason: "Title has punctuation stuffing.", + suggested_fix: "Rewrite title.", + target_agent: "title", + }, + ], + }); + + expect(rewritten.title).toBe("Rewritten By LLM"); + expect(llmClient.generateValidatedJson).toHaveBeenCalledOnce(); + }); + + it("falls back to deterministic targeted rewrite when LLM returns null", async () => { + vi.mocked(llmClient.generateValidatedJson).mockResolvedValueOnce(null); + + const rewritten = await rewriteFailedSections({ + article: { + title: "Bad title!!!", + summary: "Original summary", + body_markdown: "## Body\nOriginal body", + image_suggestions: [], + changed_sections: [], + requires_user_confirmation: [], + }, + factCard: confirmedFactCard, + failedChecks: [ + { + rule_id: "title_quality", + status: "fail", + evidence: "Bad title!!!", + reason: "Title has punctuation stuffing.", + suggested_fix: "Rewrite title.", + target_agent: "title", + }, + ], + }); + + expect(rewritten.title).toContain("GEO optimization Guide"); + expect(rewritten.summary).toBe("Original summary"); + }); + ``` + +- [ ] **Step 2: Run tests to verify failure** + + Run: + + ```bash + npm test -- src/lib/workflow/__tests__/llm-integration.test.ts src/lib/workflow/__tests__/workflow.test.ts + ``` + + Expected: FAIL because `rewriteFailedSections` is still synchronous and does not call `generateValidatedJson`. + +- [ ] **Step 3: Implement async LLM-first targeted rewrite** + + Add imports to `src/lib/workflow/targeted-rewriter.ts`: + + ```ts + import { optimizedArticleSchema } from "../domain/validation"; + import { generateValidatedJson } from "../llm/client"; + import { + TARGETED_REWRITER_SYSTEM_PROMPT, + buildTargetedRewritePrompt, + } from "../llm/prompts"; + ``` + + Change the exported function to: + + ```ts + export async function rewriteFailedSections({ + article, + factCard, + failedChecks, + }: RewriteFailedSectionsInput): Promise { + const llmArticle = await generateValidatedJson({ + schema: optimizedArticleSchema, + system: TARGETED_REWRITER_SYSTEM_PROMPT, + prompt: buildTargetedRewritePrompt({ article, factCard, failedChecks }), + temperature: 0.15, + }); + + return llmArticle ?? rewriteFailedSectionsFallback({ article, factCard, failedChecks }); + } + + function rewriteFailedSectionsFallback({ + article, + factCard, + failedChecks, + }: RewriteFailedSectionsInput): OptimizedArticle { + let rewritten = { ...article }; + + for (const check of failedChecks) { + if (check.target_agent === "title") { + rewritten = { + ...rewritten, + title: `${factCard.brand_names[0] ?? factCard.company_short_names[0]} ${factCard.target_industry} Guide`, + changed_sections: [...new Set([...rewritten.changed_sections, "title"])], + }; + } + + if (check.target_agent === "body" && check.rule_id === "company_name_integrity") { + rewritten = { + ...rewritten, + body_markdown: `${factCard.company_full_name}\n\n${rewritten.body_markdown}`, + changed_sections: [...new Set([...rewritten.changed_sections, "company name"])], + }; + } + + if (check.target_agent === "body" && check.rule_id === "claim_consistency") { + rewritten = { + ...rewritten, + body_markdown: rewritten.body_markdown.replace( + /\b\d{1,3}\s*(?:years?|年)\b/gi, + factCard.experience_years === null + ? "confirmed experience" + : `${factCard.experience_years} years`, + ), + changed_sections: [...new Set([...rewritten.changed_sections, "claim consistency"])], + }; + } + } + + return rewritten; + } + ``` + + In `src/lib/workflow/orchestrator.ts`, change: + + ```ts + article = rewriteFailedSections({ article, factCard, failedChecks }); + ``` + + to: + + ```ts + article = await rewriteFailedSections({ article, factCard, failedChecks }); + ``` + +- [ ] **Step 4: Run workflow tests** + + Run: + + ```bash + npm test -- src/lib/workflow/__tests__/llm-integration.test.ts src/lib/workflow/__tests__/workflow.test.ts + ``` + + Expected: PASS. + +- [ ] **Step 5: Commit** + + ```bash + git add src/lib/workflow/targeted-rewriter.ts src/lib/workflow/orchestrator.ts src/lib/workflow/__tests__/workflow.test.ts src/lib/workflow/__tests__/llm-integration.test.ts + git commit -m "feat: use llm for targeted rewrite" + ``` + +## Task 6: Add Hybrid LLM Quality Inspection + +**Files:** + +- Modify: `src/lib/workflow/quality-inspector.ts` +- Modify: `src/lib/workflow/orchestrator.ts` +- Modify: `src/lib/workflow/__tests__/llm-integration.test.ts` + +- [ ] **Step 1: Add failing QA enrichment tests** + + Append this import to `src/lib/workflow/__tests__/llm-integration.test.ts`: + + ```ts + import { inspectQualityWithLlm } from "../quality-inspector"; + ``` + + Add these tests inside the existing `describe` block: + + ```ts + it("uses LLM quality checks to enrich non-failing deterministic checks", async () => { + vi.mocked(llmClient.generateValidatedJson).mockResolvedValueOnce({ + checks: [ + { + rule_id: "platform_fit", + status: "warn", + evidence: "LLM noticed the article reads like a generic blog post.", + reason: "The structure is not specific enough for an official site.", + suggested_fix: "Add a clearer brand-owned introduction.", + target_agent: "body", + }, + ], + }); + + const report = await inspectQualityWithLlm({ + article: { + title: "Example GEO Optimization Guide", + summary: "A official site article for Marketing teams about GEO optimization.", + body_markdown: "Example Technology Co., Ltd. has 8 years of GEO optimization experience.", + image_suggestions: [{ source: "image_1", suggestion: "Use dashboard." }], + changed_sections: [], + requires_user_confirmation: [], + }, + factCard: confirmedFactCard, + platform: "official_site", + sourceImages: [{ type: "description", content: "dashboard" }], + }); + + const platformCheck = report.checks.find((check) => check.rule_id === "platform_fit"); + expect(platformCheck?.status).toBe("warn"); + expect(platformCheck?.evidence).toContain("LLM noticed"); + }); + + it("does not let LLM downgrade deterministic hard failures", async () => { + vi.mocked(llmClient.generateValidatedJson).mockResolvedValueOnce({ + checks: [ + { + rule_id: "company_name_integrity", + status: "pass", + evidence: "LLM says it is fine.", + reason: "LLM attempted to downgrade a failure.", + suggested_fix: "", + target_agent: null, + }, + ], + }); + + const report = await inspectQualityWithLlm({ + article: { + title: "Example GEO Optimization Guide", + summary: "A official site article for Marketing teams about GEO optimization.", + body_markdown: "Example has 8 years of GEO optimization experience.", + image_suggestions: [{ source: "image_1", suggestion: "Use dashboard." }], + changed_sections: [], + requires_user_confirmation: [], + }, + factCard: confirmedFactCard, + platform: "official_site", + sourceImages: [{ type: "description", content: "dashboard" }], + }); + + const companyCheck = report.checks.find( + (check) => check.rule_id === "company_name_integrity", + ); + expect(companyCheck?.status).toBe("fail"); + expect(companyCheck?.reason).toContain("公司"); + }); + ``` + +- [ ] **Step 2: Run tests to verify failure** + + Run: + + ```bash + npm test -- src/lib/workflow/__tests__/llm-integration.test.ts + ``` + + Expected: FAIL because `inspectQualityWithLlm` is not exported. + +- [ ] **Step 3: Implement hybrid inspection** + + Add imports to `src/lib/workflow/quality-inspector.ts`: + + ```ts + import { z } from "zod"; + import { generateValidatedJson } from "../llm/client"; + import { + QUALITY_INSPECTOR_SYSTEM_PROMPT, + buildQualityInspectorPrompt, + } from "../llm/prompts"; + ``` + + Add this schema after `REQUIRED_RULES`: + + ```ts + const llmQaPatchSchema = z.object({ + checks: z.array(qaCheckSchema).default([]), + }); + ``` + + Change the validation import from: + + ```ts + import { qaReportSchema } from "../domain/validation"; + ``` + + to: + + ```ts + import { qaCheckSchema, qaReportSchema } from "../domain/validation"; + ``` + + Add this exported function after `inspectQuality`: + + ```ts + export async function inspectQualityWithLlm(input: InspectQualityInput): Promise { + const deterministicReport = inspectQuality(input); + const llmPatch = await generateValidatedJson({ + schema: llmQaPatchSchema, + system: QUALITY_INSPECTOR_SYSTEM_PROMPT, + prompt: buildQualityInspectorPrompt({ + article: input.article, + factCard: input.factCard, + platform: input.platform, + deterministicChecks: deterministicReport.checks, + }), + temperature: 0.1, + }); + + if (!llmPatch) { + return deterministicReport; + } + + const patchedChecks = deterministicReport.checks.map((deterministicCheck) => { + const llmCheck = llmPatch.checks.find( + (check) => check.rule_id === deterministicCheck.rule_id, + ); + if (!llmCheck) { + return deterministicCheck; + } + if (deterministicCheck.status === "fail") { + return deterministicCheck; + } + return { + ...deterministicCheck, + status: llmCheck.status, + evidence: llmCheck.evidence, + reason: llmCheck.reason, + suggested_fix: llmCheck.suggested_fix, + target_agent: llmCheck.target_agent, + }; + }); + + const overall_status: CheckStatus = patchedChecks.some((check) => check.status === "fail") + ? "fail" + : patchedChecks.some((check) => check.status === "warn") + ? "warn" + : "pass"; + + return qaReportSchema.parse({ overall_status, checks: patchedChecks }); + } + ``` + + In `src/lib/workflow/orchestrator.ts`, change the import: + + ```ts + import { inspectQuality } from "./quality-inspector"; + ``` + + to: + + ```ts + import { inspectQualityWithLlm } from "./quality-inspector"; + ``` + + Change both calls to `inspectQuality({ ... })` in `runOptimizationWorkflow` to: + + ```ts + await inspectQualityWithLlm({ + article, + factCard, + platform: input.platform, + sourceImages: input.images, + }) + ``` + +- [ ] **Step 4: Run workflow tests** + + Run: + + ```bash + npm test -- src/lib/workflow/__tests__/llm-integration.test.ts src/lib/workflow/__tests__/workflow.test.ts + ``` + + Expected: PASS. + +- [ ] **Step 5: Commit** + + ```bash + git add src/lib/workflow/quality-inspector.ts src/lib/workflow/orchestrator.ts src/lib/workflow/__tests__/llm-integration.test.ts + git commit -m "feat: add hybrid llm quality inspection" + ``` + +## Task 7: Add API-Level LLM Coverage And Docs + +**Files:** + +- Modify: `src/app/api/__tests__/jobs.test.ts` +- Modify: `README.md` + +- [ ] **Step 1: Add API test with mocked LLM output** + + Add this mock near the top of `src/app/api/__tests__/jobs.test.ts`, after imports: + + ```ts + import { vi } from "vitest"; + + vi.mock("../../../lib/llm/client", async () => { + const actual = await vi.importActual( + "../../../lib/llm/client", + ); + return { + ...actual, + generateValidatedJson: vi.fn(), + }; + }); + + const llmClient = await import("../../../lib/llm/client"); + ``` + + Add this line inside the existing `afterEach` block: + + ```ts + vi.mocked(llmClient.generateValidatedJson).mockReset(); + ``` + + Add this test inside `describe("job API routes", () => { ... })`: + + ```ts + it("uses mocked LLM article output during optimize route", async () => { + const { job } = await createJobFixture(); + await confirmFactCard(request(validFactCard), params({ jobId: job.id })); + + vi.mocked(llmClient.generateValidatedJson) + .mockResolvedValueOnce({ + title: "API LLM Optimized GEO Article", + summary: "A official site article for Marketing teams about GEO optimization.", + body_markdown: + "Example Technology Co., Ltd. has 8 years of GEO optimization experience.", + image_suggestions: [{ source: "image_1", suggestion: "Use Product dashboard." }], + changed_sections: ["title", "body"], + requires_user_confirmation: [], + }) + .mockResolvedValue(null); + + const response = await optimizeJob(request({}), params({ jobId: job.id })); + const body = await response.json(); + + expect(response.status).toBe(200); + expect(body.optimizedArticle.title).toBe("API LLM Optimized GEO Article"); + expect(llmClient.generateValidatedJson).toHaveBeenCalled(); + }); + ``` + +- [ ] **Step 2: Run API tests to verify they pass** + + Run: + + ```bash + npm test -- src/app/api/__tests__/jobs.test.ts + ``` + + Expected: PASS. + +- [ ] **Step 3: Update README environment description** + + In `README.md`, replace: + + ```md + When no API key is configured, deterministic local fallbacks keep the workflow + usable for tests and local review. + ``` + + with: + + ```md + When a DeepSeek or OpenAI-compatible key is configured, the workflow uses the + provider for fact extraction, article optimization, QA enrichment, and targeted + rewrite. Every LLM response is validated with Zod before use. When no API key is + configured, or when the provider response is invalid, deterministic local + fallbacks keep the workflow usable for tests and local review. + ``` + +- [ ] **Step 4: Run full verification** + + Run: + + ```bash + npm test + npm run build + ``` + + Expected: both commands PASS. + +- [ ] **Step 5: Commit** + + ```bash + git add src/app/api/__tests__/jobs.test.ts README.md + git commit -m "test: cover llm workflow through api" + ``` + +## Task 8: Optional Manual DeepSeek Smoke Test + +**Files:** + +- Verify: `.env.local` +- Verify: `src/lib/workflow/**` + +- [ ] **Step 1: Confirm environment values are present locally** + + Run: + + ```bash + test -n "$DEEPSEEK_API_KEY" || test -n "$(grep '^DEEPSEEK_API_KEY=.' .env.local 2>/dev/null)" + ``` + + Expected: command exits successfully. If it fails, add a real `DEEPSEEK_API_KEY` to `.env.local` on the local machine only. + +- [ ] **Step 2: Start the app** + + Run: + + ```bash + npm run dev + ``` + + Expected: Next.js starts and prints a local URL such as `http://localhost:3000`. + +- [ ] **Step 3: Exercise the UI** + + In the browser: + + - Open `http://localhost:3000`. + - Enter title: `Example Technology Co., Ltd. GEO 指南`. + - Enter body: `Example Technology Co., Ltd. has 8 years of GEO optimization experience. Example GEO 帮助市场团队优化内容结构。` + - Enter image description: `产品仪表盘截图`. + - Enter user instruction: `保持事实准确,语气自然,不要新增未经确认的客户案例。` + - Click `分析文章`. + - Clear any `待确认事项` after reviewing the fact card. + - Click `确认事实卡`. + - Click `开始优化`. + + Expected: the generated title/body should no longer look like the deterministic fallback template unless the provider call failed. Export links should appear when QA is `pass` or `warn`. + +- [ ] **Step 4: Stop the dev server** + + Press `Ctrl-C` in the terminal running `npm run dev`. + +## Self-Review + +- Spec coverage: The plan connects configured DeepSeek/OpenAI-compatible credentials into fact extraction, article optimization, targeted rewrite, and QA enrichment. It preserves hard QA failures, export blocking, Zod validation, and local deterministic fallback. +- Placeholder scan: The plan contains concrete file paths, commands, code snippets, and expected outputs. It does not rely on unspecified behavior. +- Type consistency: All new calls use existing domain schemas and types: `candidateFactCardSchema`, `optimizedArticleSchema`, `qaCheckSchema`, `qaReportSchema`, `ArticleInput`, `ConfirmedFactCard`, `OptimizedArticle`, and `QaCheck`.