Files

T

Codex f017ce1bba docs: add deepseek workflow integration plan

2026-06-21 23:43:02 +08:00

40 KiB

Raw Blame History

DeepSeek Workflow Integration Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Wire the configured DeepSeek/OpenAI-compatible LLM client into the real optimization workflow while preserving deterministic local fallback behavior for tests and demos.

Architecture: Keep the existing workflow modules and orchestrator shape. Each LLM-enabled node first tries generateJson when isLlmConfigured() is true, validates the model output with the existing Zod schemas, and falls back to the current deterministic implementation if the provider is unconfigured, errors, or returns invalid data. Quality inspection remains a hybrid gate: deterministic hard rules stay authoritative, while LLM output can enrich warnings, evidence, and suggestions without weakening hard failures.

Tech Stack: Next.js route handlers, TypeScript, Zod, OpenAI SDK with DeepSeek base URL, Vitest module mocks, existing SQLite/export workflow.

Scope

This plan implements live LLM use in the existing single-job workflow only. It does not add batch queues, user accounts, publishing integrations, .docx parsing, brand-template UI, streaming responses, or model-provider admin screens.

File Structure

Modify: src/lib/llm/client.ts
- Keep provider detection and generateJson.
- Export a typed helper generateValidatedJson that accepts a Zod schema and returns null when LLM use is unavailable or invalid.
Modify: src/lib/llm/prompts.ts
- Add prompt builders for fact extraction, article optimization, QA enrichment, and targeted rewrite.
Modify: src/lib/workflow/fact-extractor.ts
- Rename the current rules implementation to a fallback function.
- Try DeepSeek first and validate CandidateFactCard.
Modify: src/lib/workflow/article-optimizer.ts
- Rename the current template implementation to a fallback function.
- Try DeepSeek first and validate OptimizedArticle.
Modify: src/lib/workflow/targeted-rewriter.ts
- Make rewriting async.
- Try DeepSeek first for failed checks and fall back to current deterministic targeted fixes.
Modify: src/lib/workflow/orchestrator.ts
- Await async targeted rewriting.
Modify: src/lib/workflow/quality-inspector.ts
- Add optional async LLM enrichment through a new exported inspectQualityWithLlm while keeping inspectQuality as deterministic logic for unit tests and fallback.
Modify: src/lib/workflow/orchestrator.ts
- Use inspectQualityWithLlm in the real workflow.
Test: src/lib/llm/__tests__/client.test.ts
- Verify invalid LLM data returns null from the helper.
Test: src/lib/workflow/__tests__/llm-integration.test.ts
- Mock the LLM client and prove each workflow node calls it when configured.
- Prove invalid or rejected LLM calls fall back to deterministic behavior.
Modify: src/lib/workflow/__tests__/workflow.test.ts
- Update targeted rewriter tests to await the async function.
Modify: src/app/api/__tests__/jobs.test.ts
- Add an API-level test proving a mocked LLM article can flow through /optimize.
Modify: README.md
- Clarify that configured DeepSeek is used by workflow nodes, with deterministic fallback when unavailable.

Task 1: Add Validated LLM Helper

Files:

Modify: src/lib/llm/client.ts
Create: src/lib/llm/__tests__/client.test.ts

Step 1: Write the failing tests

Create src/lib/llm/__tests__/client.test.ts:

import { afterEach, describe, expect, it, vi } from "vitest";
import { z } from "zod";

import * as client from "../client";

describe("generateValidatedJson", () => {
  const originalProvider = process.env.LLM_PROVIDER;
  const originalDeepSeekKey = process.env.DEEPSEEK_API_KEY;

  afterEach(() => {
    process.env.LLM_PROVIDER = originalProvider;
    process.env.DEEPSEEK_API_KEY = originalDeepSeekKey;
    vi.restoreAllMocks();
  });

  it("returns null when no provider key is configured", async () => {
    process.env.LLM_PROVIDER = "deepseek";
    delete process.env.DEEPSEEK_API_KEY;

    const result = await client.generateValidatedJson({
      schema: z.object({ value: z.string() }),
      prompt: "Return JSON.",
    });

    expect(result).toBeNull();
  });

  it("returns parsed data when the model response matches the schema", async () => {
    process.env.LLM_PROVIDER = "deepseek";
    process.env.DEEPSEEK_API_KEY = "test-key";
    vi.spyOn(client, "generateJson").mockResolvedValue({ value: "from-llm" });

    const result = await client.generateValidatedJson({
      schema: z.object({ value: z.string() }),
      prompt: "Return JSON.",
    });

    expect(result).toEqual({ value: "from-llm" });
  });

  it("returns null when the model response fails schema validation", async () => {
    process.env.LLM_PROVIDER = "deepseek";
    process.env.DEEPSEEK_API_KEY = "test-key";
    vi.spyOn(client, "generateJson").mockResolvedValue({ value: 42 });

    const result = await client.generateValidatedJson({
      schema: z.object({ value: z.string() }),
      prompt: "Return JSON.",
    });

    expect(result).toBeNull();
  });

  it("returns null when the provider call rejects", async () => {
    process.env.LLM_PROVIDER = "deepseek";
    process.env.DEEPSEEK_API_KEY = "test-key";
    vi.spyOn(client, "generateJson").mockRejectedValue(new Error("provider down"));

    const result = await client.generateValidatedJson({
      schema: z.object({ value: z.string() }),
      prompt: "Return JSON.",
    });

    expect(result).toBeNull();
  });
});

Step 2: Run the test to verify it fails

Run:
```
npm test -- src/lib/llm/__tests__/client.test.ts
```
Expected: FAIL with a TypeScript or runtime error indicating generateValidatedJson is not exported.

Step 3: Implement the helper

Modify src/lib/llm/client.ts.

Add this import below the existing openai import:

import type { z } from "zod";

Add these interfaces after GenerateInput:

export interface GenerateValidatedJsonInput<T> extends GenerateInput {
  schema: z.ZodType<T>;
}

Add this exported function after generateJson:

export async function generateValidatedJson<T>({
  schema,
  ...input
}: GenerateValidatedJsonInput<T>): Promise<T | null> {
  if (!isLlmConfigured()) {
    return null;
  }

  try {
    const generated = await generateJson<unknown>(input);
    return schema.parse(generated);
  } catch {
    return null;
  }
}

Step 4: Run the test to verify it passes

Run:
```
npm test -- src/lib/llm/__tests__/client.test.ts
```
Expected: PASS for all four tests.

Step 5: Commit

git add src/lib/llm/client.ts src/lib/llm/__tests__/client.test.ts
git commit -m "feat: add validated llm json helper"

Task 2: Add Prompt Builders

Files:

Modify: src/lib/llm/prompts.ts

Step 1: Replace prompts.ts with explicit builders

Replace src/lib/llm/prompts.ts with:

import type {
  ArticleInput,
  ConfirmedFactCard,
  OptimizedArticle,
  PublishPlatform,
  QaCheck,
} from "../domain/types";

export const JSON_ONLY_PROMPT =
  "Return valid JSON only. Do not include markdown fences or commentary.";

export const ARTICLE_OPTIMIZER_SYSTEM_PROMPT = [
  "You optimize GEO-related articles under a confirmed fact card.",
  "Never invent numbers, cases, qualifications, company names, products, or years.",
  "Use only facts present in the confirmed fact card or source article.",
  JSON_ONLY_PROMPT,
].join(" ");

export const QUALITY_INSPECTOR_SYSTEM_PROMPT = [
  "Evaluate article quality against the confirmed fact card and target platform.",
  "Return one structured check per required quality gate when asked.",
  "Do not downgrade deterministic hard failures supplied by the application.",
  JSON_ONLY_PROMPT,
].join(" ");

export const FACT_EXTRACTOR_SYSTEM_PROMPT = [
  "Extract a candidate fact card from the source article.",
  "Do not mark uncertain facts as confirmed.",
  "Put missing or conflicting facts in uncertain_items.",
  JSON_ONLY_PROMPT,
].join(" ");

export const TARGETED_REWRITER_SYSTEM_PROMPT = [
  "Rewrite only the fields needed to resolve the provided failed QA checks.",
  "Keep confirmed facts unchanged.",
  "Preserve sections that are unrelated to failed checks.",
  JSON_ONLY_PROMPT,
].join(" ");

export function buildFactExtractorPrompt(input: ArticleInput) {
  return [
    "Return a CandidateFactCard JSON object with these exact keys:",
    "company_full_name, company_short_names, brand_names, product_names, target_industry, target_audience, experience_years, core_claims, forbidden_claims, image_topics, uncertain_items.",
    "Do not include confirmed_by_user.",
    "",
    "Article input:",
    JSON.stringify(input, null, 2),
  ].join("\n");
}

export function buildArticleOptimizerPrompt(input: ArticleInput, factCard: ConfirmedFactCard) {
  return [
    "Return an OptimizedArticle JSON object with these exact keys:",
    "title, summary, body_markdown, image_suggestions, changed_sections, requires_user_confirmation.",
    "Use Markdown in body_markdown.",
    "If the user instruction asks for unsupported facts, omit them from the article and add them to requires_user_confirmation.",
    "",
    "Confirmed fact card:",
    JSON.stringify(factCard, null, 2),
    "",
    "Article input:",
    JSON.stringify(input, null, 2),
  ].join("\n");
}

export function buildQualityInspectorPrompt(input: {
  article: OptimizedArticle;
  factCard: ConfirmedFactCard;
  platform: PublishPlatform;
  deterministicChecks: QaCheck[];
}) {
  return [
    "Return a JSON object with a checks array.",
    "Each check must include rule_id, status, evidence, reason, suggested_fix, and target_agent.",
    "Only use rule_id values already present in deterministicChecks.",
    "If a deterministic check has status fail, keep it fail.",
    "",
    "Target platform:",
    input.platform,
    "",
    "Confirmed fact card:",
    JSON.stringify(input.factCard, null, 2),
    "",
    "Optimized article:",
    JSON.stringify(input.article, null, 2),
    "",
    "Deterministic checks:",
    JSON.stringify(input.deterministicChecks, null, 2),
  ].join("\n");
}

export function buildTargetedRewritePrompt(input: {
  article: OptimizedArticle;
  factCard: ConfirmedFactCard;
  failedChecks: QaCheck[];
}) {
  return [
    "Return an OptimizedArticle JSON object.",
    "Rewrite only the fields needed for failedChecks.",
    "Do not add unconfirmed numbers, customer names, qualifications, awards, or capabilities.",
    "",
    "Confirmed fact card:",
    JSON.stringify(input.factCard, null, 2),
    "",
    "Current optimized article:",
    JSON.stringify(input.article, null, 2),
    "",
    "Failed checks:",
    JSON.stringify(input.failedChecks, null, 2),
  ].join("\n");
}

Step 2: Run existing tests

Run:
```
npm test
```
Expected: PASS. No workflow code uses the new builders yet.

Step 3: Commit

git add src/lib/llm/prompts.ts
git commit -m "feat: add llm workflow prompt builders"

Task 3: Connect Fact Extraction To LLM

Files:

Modify: src/lib/workflow/fact-extractor.ts
Create: src/lib/workflow/__tests__/llm-integration.test.ts

Step 1: Write failing fact-extractor tests

Create src/lib/workflow/__tests__/llm-integration.test.ts:

import { afterEach, describe, expect, it, vi } from "vitest";

import { extractCandidateFactCard } from "../fact-extractor";

vi.mock("../../llm/client", async () => {
  const actual = await vi.importActual<typeof import("../../llm/client")>(
    "../../llm/client",
  );
  return {
    ...actual,
    generateValidatedJson: vi.fn(),
  };
});

const llmClient = await import("../../llm/client");

describe("LLM workflow integration", () => {
  afterEach(() => {
    vi.mocked(llmClient.generateValidatedJson).mockReset();
  });

  it("uses LLM output for candidate fact extraction when valid", async () => {
    vi.mocked(llmClient.generateValidatedJson).mockResolvedValueOnce({
      company_full_name: "DeepSeek Example Co., Ltd.",
      company_short_names: ["DeepSeek Example"],
      brand_names: ["DSExample"],
      product_names: ["DS GEO"],
      target_industry: "GEO optimization",
      target_audience: "Marketing teams",
      experience_years: 9,
      core_claims: ["9 years of GEO optimization experience"],
      forbidden_claims: ["industry first"],
      image_topics: ["dashboard"],
      uncertain_items: [],
      is_ready_for_optimization: true,
    });

    const card = await extractCandidateFactCard({
      title: "Example source",
      body: "Fallback Technology Co., Ltd. has 8 years of GEO optimization experience.",
      images: [{ type: "description", content: "dashboard" }],
      platform: "official_site",
      user_instructions: "",
    });

    expect(card.company_full_name).toBe("DeepSeek Example Co., Ltd.");
    expect(card.experience_years).toBe(9);
    expect(llmClient.generateValidatedJson).toHaveBeenCalledOnce();
  });

  it("falls back to deterministic candidate extraction when LLM returns null", async () => {
    vi.mocked(llmClient.generateValidatedJson).mockResolvedValueOnce(null);

    const card = await extractCandidateFactCard({
      title: "Fallback Technology Co., Ltd. GEO guide",
      body: "Fallback Technology Co., Ltd. has 8 years of GEO optimization experience.",
      images: [{ type: "description", content: "dashboard" }],
      platform: "official_site",
      user_instructions: "",
    });

    expect(card.company_full_name).toBe("Fallback Technology Co., Ltd.");
    expect(card.experience_years).toBe(8);
  });
});

Step 2: Run the test to verify it fails

Run:
```
npm test -- src/lib/workflow/__tests__/llm-integration.test.ts
```
Expected: FAIL because extractCandidateFactCard does not call generateValidatedJson.

Step 3: Implement LLM-first fact extraction

Modify the top of src/lib/workflow/fact-extractor.ts to add imports:

import { generateValidatedJson } from "../llm/client";
import {
  FACT_EXTRACTOR_SYSTEM_PROMPT,
  buildFactExtractorPrompt,
} from "../llm/prompts";

Replace the current exported extractCandidateFactCard body with:

export async function extractCandidateFactCard(
  input: ArticleInput,
): Promise<CandidateFactCard> {
  const llmCard = await generateValidatedJson({
    schema: candidateFactCardSchema,
    system: FACT_EXTRACTOR_SYSTEM_PROMPT,
    prompt: buildFactExtractorPrompt(input),
    temperature: 0.1,
  });

  return llmCard ?? extractCandidateFactCardFallback(input);
}

function extractCandidateFactCardFallback(input: ArticleInput): CandidateFactCard {
  const text = `${input.title}\n${input.body}`;
  const uncertainItems: string[] = [];
  const companyFullName = findCompanyFullName(text);
  const years = findExperienceYears(text);

  if (!companyFullName) {
    uncertainItems.push("Missing company full name");
  }
  if (years.length > 1) {
    uncertainItems.push(`Conflicting experience years: ${years.join(", ")}`);
  }
  if (input.images.length === 0) {
    uncertainItems.push("Image description is missing");
  }

  const industry = inferIndustry(text);

  return candidateFactCardSchema.parse({
    company_full_name: companyFullName ?? "",
    company_short_names: inferCompanyShortNames(companyFullName),
    brand_names: inferBrandNames(text, companyFullName),
    product_names: inferProducts(text),
    target_industry: industry,
    target_audience: text.toLowerCase().includes("marketing")
      ? "Marketing teams"
      : inferTargetAudience(text),
    experience_years: years.length === 1 ? years[0] : null,
    core_claims: years.length === 1 ? [`${years[0]} years of ${industry} experience`] : [],
    forbidden_claims: [],
    image_topics: input.images.map((image) => image.content),
    uncertain_items: uncertainItems,
  });
}

Keep the helper functions findCompanyFullName, findExperienceYears, inferIndustry, inferBrandNames, inferProducts, inferCompanyShortNames, inferChineseShortName, and inferTargetAudience unchanged below the fallback function.

Step 4: Run targeted and existing workflow tests

Run:

npm test -- src/lib/workflow/__tests__/llm-integration.test.ts src/lib/workflow/__tests__/workflow.test.ts

Expected: PASS.

Step 5: Commit

git add src/lib/workflow/fact-extractor.ts src/lib/workflow/__tests__/llm-integration.test.ts
git commit -m "feat: use llm for fact extraction"

Task 4: Connect Article Optimization To LLM

Files:

Modify: src/lib/workflow/article-optimizer.ts
Modify: src/lib/workflow/__tests__/llm-integration.test.ts

Step 1: Add failing optimizer tests

Append these imports to src/lib/workflow/__tests__/llm-integration.test.ts:

import { optimizeArticle } from "../article-optimizer";

Add this shared fact card inside the describe block:

const confirmedFactCard = {
  company_full_name: "Example Technology Co., Ltd.",
  company_short_names: ["Example Tech"],
  brand_names: ["Example"],
  product_names: ["Example GEO"],
  target_industry: "GEO optimization",
  target_audience: "Marketing teams",
  experience_years: 8,
  core_claims: ["8 years of GEO optimization experience"],
  forbidden_claims: ["industry first"],
  image_topics: ["dashboard"],
  uncertain_items: [],
  is_ready_for_optimization: true,
  confirmed_by_user: true,
} as const;

Add these tests inside the describe block:

it("uses LLM output for article optimization when valid", async () => {
  vi.mocked(llmClient.generateValidatedJson).mockResolvedValueOnce({
    title: "LLM Optimized GEO Article",
    summary: "LLM summary constrained by the fact card.",
    body_markdown: "## LLM Body\nExample Technology Co., Ltd. keeps claims factual.",
    image_suggestions: [{ source: "image_1", suggestion: "Use dashboard." }],
    changed_sections: ["title", "body"],
    requires_user_confirmation: [],
  });

  const article = await optimizeArticle({
    input: {
      title: "Original",
      body: "Example Technology Co., Ltd. has 8 years of GEO optimization experience.",
      images: [{ type: "description", content: "dashboard" }],
      platform: "official_site",
      user_instructions: "",
    },
    factCard: confirmedFactCard,
  });

  expect(article.title).toBe("LLM Optimized GEO Article");
  expect(article.body_markdown).toContain("LLM Body");
  expect(llmClient.generateValidatedJson).toHaveBeenCalledOnce();
});

it("falls back to deterministic article optimization when LLM returns null", async () => {
  vi.mocked(llmClient.generateValidatedJson).mockResolvedValueOnce(null);

  const article = await optimizeArticle({
    input: {
      title: "Original",
      body: "Example Technology Co., Ltd. has 8 years of GEO optimization experience.",
      images: [{ type: "description", content: "dashboard" }],
      platform: "official_site",
      user_instructions: "Say we have 99 patents.",
    },
    factCard: confirmedFactCard,
  });

  expect(article.title).toContain("GEO optimization Guide");
  expect(article.requires_user_confirmation).toContain(
    "Unsupported requested claim: 99 patents",
  );
});

Step 2: Run the test to verify it fails

Run:
```
npm test -- src/lib/workflow/__tests__/llm-integration.test.ts
```
Expected: FAIL because optimizeArticle does not call generateValidatedJson.

Step 3: Implement LLM-first optimization

Modify the top of src/lib/workflow/article-optimizer.ts to add imports:

import { generateValidatedJson } from "../llm/client";
import {
  ARTICLE_OPTIMIZER_SYSTEM_PROMPT,
  buildArticleOptimizerPrompt,
} from "../llm/prompts";

Replace the current exported optimizeArticle body with:

export async function optimizeArticle({
  input,
  factCard,
}: OptimizeArticleInput): Promise<OptimizedArticle> {
  const llmArticle = await generateValidatedJson({
    schema: optimizedArticleSchema,
    system: ARTICLE_OPTIMIZER_SYSTEM_PROMPT,
    prompt: buildArticleOptimizerPrompt(input, factCard),
    temperature: 0.2,
  });

  return llmArticle ?? optimizeArticleFallback({ input, factCard });
}

function optimizeArticleFallback({
  input,
  factCard,
}: OptimizeArticleInput): OptimizedArticle {
  const unsupported = findUnsupportedInstructionClaims(
    input.user_instructions,
    factCard,
  );
  const title = `${factCard.brand_names[0] ?? factCard.company_short_names[0] ?? factCard.company_full_name} ${factCard.target_industry} Guide`;
  const coreClaims =
    factCard.core_claims.length > 0
      ? factCard.core_claims.map((claim) => `- ${claim}`).join("\n")
      : "- Confirmed facts only; no extra claims added.";
  const body = [
    `## ${factCard.company_full_name}`,
    cleanBody(input.body, factCard),
    "",
    "### Confirmed Facts",
    coreClaims,
  ].join("\n");

  return optimizedArticleSchema.parse({
    title,
    summary: `A ${input.platform.replace(/_/g, " ")} article for ${factCard.target_audience} about ${factCard.target_industry}.`,
    body_markdown: body,
    image_suggestions: factCard.image_topics.map((topic, index) => ({
      source: `image_${index + 1}`,
      suggestion: `Use image content related to ${topic}.`,
    })),
    changed_sections: ["title", "body structure", "summary"],
    requires_user_confirmation: unsupported,
  });
}

Keep cleanBody, findUnsupportedInstructionClaims, and escapeRegExp unchanged below the fallback function.

Step 4: Run targeted and workflow tests

Run:

npm test -- src/lib/workflow/__tests__/llm-integration.test.ts src/lib/workflow/__tests__/workflow.test.ts

Expected: PASS.

Step 5: Commit

git add src/lib/workflow/article-optimizer.ts src/lib/workflow/__tests__/llm-integration.test.ts
git commit -m "feat: use llm for article optimization"

Task 5: Connect Targeted Rewrite To LLM

Files:

Modify: src/lib/workflow/targeted-rewriter.ts
Modify: src/lib/workflow/orchestrator.ts
Modify: src/lib/workflow/__tests__/workflow.test.ts
Modify: src/lib/workflow/__tests__/llm-integration.test.ts

Step 1: Update and add failing rewrite tests

In src/lib/workflow/__tests__/workflow.test.ts, change the existing targeted rewrite test line:

const rewritten = rewriteFailedSections({

to:

const rewritten = await rewriteFailedSections({

Append this import to src/lib/workflow/__tests__/llm-integration.test.ts:

import { rewriteFailedSections } from "../targeted-rewriter";

Add these tests inside the existing describe block:

it("uses LLM output for targeted rewrite when valid", async () => {
  vi.mocked(llmClient.generateValidatedJson).mockResolvedValueOnce({
    title: "Rewritten By LLM",
    summary: "Original summary",
    body_markdown: "## Body\nExample Technology Co., Ltd. focuses on GEO optimization.",
    image_suggestions: [],
    changed_sections: ["title"],
    requires_user_confirmation: [],
  });

  const rewritten = await rewriteFailedSections({
    article: {
      title: "Bad title!!!",
      summary: "Original summary",
      body_markdown: "## Body\nOriginal body",
      image_suggestions: [],
      changed_sections: [],
      requires_user_confirmation: [],
    },
    factCard: confirmedFactCard,
    failedChecks: [
      {
        rule_id: "title_quality",
        status: "fail",
        evidence: "Bad title!!!",
        reason: "Title has punctuation stuffing.",
        suggested_fix: "Rewrite title.",
        target_agent: "title",
      },
    ],
  });

  expect(rewritten.title).toBe("Rewritten By LLM");
  expect(llmClient.generateValidatedJson).toHaveBeenCalledOnce();
});

it("falls back to deterministic targeted rewrite when LLM returns null", async () => {
  vi.mocked(llmClient.generateValidatedJson).mockResolvedValueOnce(null);

  const rewritten = await rewriteFailedSections({
    article: {
      title: "Bad title!!!",
      summary: "Original summary",
      body_markdown: "## Body\nOriginal body",
      image_suggestions: [],
      changed_sections: [],
      requires_user_confirmation: [],
    },
    factCard: confirmedFactCard,
    failedChecks: [
      {
        rule_id: "title_quality",
        status: "fail",
        evidence: "Bad title!!!",
        reason: "Title has punctuation stuffing.",
        suggested_fix: "Rewrite title.",
        target_agent: "title",
      },
    ],
  });

  expect(rewritten.title).toContain("GEO optimization Guide");
  expect(rewritten.summary).toBe("Original summary");
});

Step 2: Run tests to verify failure

Run:
```
npm test -- src/lib/workflow/__tests__/llm-integration.test.ts src/lib/workflow/__tests__/workflow.test.ts
```
Expected: FAIL because rewriteFailedSections is still synchronous and does not call generateValidatedJson.

Step 3: Implement async LLM-first targeted rewrite

Add imports to src/lib/workflow/targeted-rewriter.ts:

import { optimizedArticleSchema } from "../domain/validation";
import { generateValidatedJson } from "../llm/client";
import {
  TARGETED_REWRITER_SYSTEM_PROMPT,
  buildTargetedRewritePrompt,
} from "../llm/prompts";

Change the exported function to:

export async function rewriteFailedSections({
  article,
  factCard,
  failedChecks,
}: RewriteFailedSectionsInput): Promise<OptimizedArticle> {
  const llmArticle = await generateValidatedJson({
    schema: optimizedArticleSchema,
    system: TARGETED_REWRITER_SYSTEM_PROMPT,
    prompt: buildTargetedRewritePrompt({ article, factCard, failedChecks }),
    temperature: 0.15,
  });

  return llmArticle ?? rewriteFailedSectionsFallback({ article, factCard, failedChecks });
}

function rewriteFailedSectionsFallback({
  article,
  factCard,
  failedChecks,
}: RewriteFailedSectionsInput): OptimizedArticle {
  let rewritten = { ...article };

  for (const check of failedChecks) {
    if (check.target_agent === "title") {
      rewritten = {
        ...rewritten,
        title: `${factCard.brand_names[0] ?? factCard.company_short_names[0]} ${factCard.target_industry} Guide`,
        changed_sections: [...new Set([...rewritten.changed_sections, "title"])],
      };
    }

    if (check.target_agent === "body" && check.rule_id === "company_name_integrity") {
      rewritten = {
        ...rewritten,
        body_markdown: `${factCard.company_full_name}\n\n${rewritten.body_markdown}`,
        changed_sections: [...new Set([...rewritten.changed_sections, "company name"])],
      };
    }

    if (check.target_agent === "body" && check.rule_id === "claim_consistency") {
      rewritten = {
        ...rewritten,
        body_markdown: rewritten.body_markdown.replace(
          /\b\d{1,3}\s*(?:years?|年)\b/gi,
          factCard.experience_years === null
            ? "confirmed experience"
            : `${factCard.experience_years} years`,
        ),
        changed_sections: [...new Set([...rewritten.changed_sections, "claim consistency"])],
      };
    }
  }

  return rewritten;
}

In src/lib/workflow/orchestrator.ts, change:

article = rewriteFailedSections({ article, factCard, failedChecks });

to:

article = await rewriteFailedSections({ article, factCard, failedChecks });

Step 4: Run workflow tests

Run:

npm test -- src/lib/workflow/__tests__/llm-integration.test.ts src/lib/workflow/__tests__/workflow.test.ts

Expected: PASS.

Step 5: Commit

git add src/lib/workflow/targeted-rewriter.ts src/lib/workflow/orchestrator.ts src/lib/workflow/__tests__/workflow.test.ts src/lib/workflow/__tests__/llm-integration.test.ts
git commit -m "feat: use llm for targeted rewrite"

Task 6: Add Hybrid LLM Quality Inspection

Files:

Modify: src/lib/workflow/quality-inspector.ts
Modify: src/lib/workflow/orchestrator.ts
Modify: src/lib/workflow/__tests__/llm-integration.test.ts

Step 1: Add failing QA enrichment tests

Append this import to src/lib/workflow/__tests__/llm-integration.test.ts:

import { inspectQualityWithLlm } from "../quality-inspector";

Add these tests inside the existing describe block:

it("uses LLM quality checks to enrich non-failing deterministic checks", async () => {
  vi.mocked(llmClient.generateValidatedJson).mockResolvedValueOnce({
    checks: [
      {
        rule_id: "platform_fit",
        status: "warn",
        evidence: "LLM noticed the article reads like a generic blog post.",
        reason: "The structure is not specific enough for an official site.",
        suggested_fix: "Add a clearer brand-owned introduction.",
        target_agent: "body",
      },
    ],
  });

  const report = await inspectQualityWithLlm({
    article: {
      title: "Example GEO Optimization Guide",
      summary: "A official site article for Marketing teams about GEO optimization.",
      body_markdown: "Example Technology Co., Ltd. has 8 years of GEO optimization experience.",
      image_suggestions: [{ source: "image_1", suggestion: "Use dashboard." }],
      changed_sections: [],
      requires_user_confirmation: [],
    },
    factCard: confirmedFactCard,
    platform: "official_site",
    sourceImages: [{ type: "description", content: "dashboard" }],
  });

  const platformCheck = report.checks.find((check) => check.rule_id === "platform_fit");
  expect(platformCheck?.status).toBe("warn");
  expect(platformCheck?.evidence).toContain("LLM noticed");
});

it("does not let LLM downgrade deterministic hard failures", async () => {
  vi.mocked(llmClient.generateValidatedJson).mockResolvedValueOnce({
    checks: [
      {
        rule_id: "company_name_integrity",
        status: "pass",
        evidence: "LLM says it is fine.",
        reason: "LLM attempted to downgrade a failure.",
        suggested_fix: "",
        target_agent: null,
      },
    ],
  });

  const report = await inspectQualityWithLlm({
    article: {
      title: "Example GEO Optimization Guide",
      summary: "A official site article for Marketing teams about GEO optimization.",
      body_markdown: "Example has 8 years of GEO optimization experience.",
      image_suggestions: [{ source: "image_1", suggestion: "Use dashboard." }],
      changed_sections: [],
      requires_user_confirmation: [],
    },
    factCard: confirmedFactCard,
    platform: "official_site",
    sourceImages: [{ type: "description", content: "dashboard" }],
  });

  const companyCheck = report.checks.find(
    (check) => check.rule_id === "company_name_integrity",
  );
  expect(companyCheck?.status).toBe("fail");
  expect(companyCheck?.reason).toContain("公司");
});

Step 2: Run tests to verify failure

Run:
```
npm test -- src/lib/workflow/__tests__/llm-integration.test.ts
```
Expected: FAIL because inspectQualityWithLlm is not exported.

Step 3: Implement hybrid inspection

Add imports to src/lib/workflow/quality-inspector.ts:

import { z } from "zod";
import { generateValidatedJson } from "../llm/client";
import {
  QUALITY_INSPECTOR_SYSTEM_PROMPT,
  buildQualityInspectorPrompt,
} from "../llm/prompts";

Add this schema after REQUIRED_RULES:

const llmQaPatchSchema = z.object({
  checks: z.array(qaCheckSchema).default([]),
});

Change the validation import from:

import { qaReportSchema } from "../domain/validation";

to:

import { qaCheckSchema, qaReportSchema } from "../domain/validation";

Add this exported function after inspectQuality:

export async function inspectQualityWithLlm(input: InspectQualityInput): Promise<QaReport> {
  const deterministicReport = inspectQuality(input);
  const llmPatch = await generateValidatedJson({
    schema: llmQaPatchSchema,
    system: QUALITY_INSPECTOR_SYSTEM_PROMPT,
    prompt: buildQualityInspectorPrompt({
      article: input.article,
      factCard: input.factCard,
      platform: input.platform,
      deterministicChecks: deterministicReport.checks,
    }),
    temperature: 0.1,
  });

  if (!llmPatch) {
    return deterministicReport;
  }

  const patchedChecks = deterministicReport.checks.map((deterministicCheck) => {
    const llmCheck = llmPatch.checks.find(
      (check) => check.rule_id === deterministicCheck.rule_id,
    );
    if (!llmCheck) {
      return deterministicCheck;
    }
    if (deterministicCheck.status === "fail") {
      return deterministicCheck;
    }
    return {
      ...deterministicCheck,
      status: llmCheck.status,
      evidence: llmCheck.evidence,
      reason: llmCheck.reason,
      suggested_fix: llmCheck.suggested_fix,
      target_agent: llmCheck.target_agent,
    };
  });

  const overall_status: CheckStatus = patchedChecks.some((check) => check.status === "fail")
    ? "fail"
    : patchedChecks.some((check) => check.status === "warn")
      ? "warn"
      : "pass";

  return qaReportSchema.parse({ overall_status, checks: patchedChecks });
}

In src/lib/workflow/orchestrator.ts, change the import:

import { inspectQuality } from "./quality-inspector";

to:

import { inspectQualityWithLlm } from "./quality-inspector";

Change both calls to inspectQuality({ ... }) in runOptimizationWorkflow to:

await inspectQualityWithLlm({
  article,
  factCard,
  platform: input.platform,
  sourceImages: input.images,
})

Step 4: Run workflow tests

Run:

npm test -- src/lib/workflow/__tests__/llm-integration.test.ts src/lib/workflow/__tests__/workflow.test.ts

Expected: PASS.

Step 5: Commit

git add src/lib/workflow/quality-inspector.ts src/lib/workflow/orchestrator.ts src/lib/workflow/__tests__/llm-integration.test.ts
git commit -m "feat: add hybrid llm quality inspection"

Task 7: Add API-Level LLM Coverage And Docs

Files:

Modify: src/app/api/__tests__/jobs.test.ts
Modify: README.md

Step 1: Add API test with mocked LLM output

Add this mock near the top of src/app/api/__tests__/jobs.test.ts, after imports:

import { vi } from "vitest";

vi.mock("../../../lib/llm/client", async () => {
  const actual = await vi.importActual<typeof import("../../../lib/llm/client")>(
    "../../../lib/llm/client",
  );
  return {
    ...actual,
    generateValidatedJson: vi.fn(),
  };
});

const llmClient = await import("../../../lib/llm/client");

Add this line inside the existing afterEach block:

vi.mocked(llmClient.generateValidatedJson).mockReset();

Add this test inside describe("job API routes", () => { ... }):

it("uses mocked LLM article output during optimize route", async () => {
  const { job } = await createJobFixture();
  await confirmFactCard(request(validFactCard), params({ jobId: job.id }));

  vi.mocked(llmClient.generateValidatedJson)
    .mockResolvedValueOnce({
      title: "API LLM Optimized GEO Article",
      summary: "A official site article for Marketing teams about GEO optimization.",
      body_markdown:
        "Example Technology Co., Ltd. has 8 years of GEO optimization experience.",
      image_suggestions: [{ source: "image_1", suggestion: "Use Product dashboard." }],
      changed_sections: ["title", "body"],
      requires_user_confirmation: [],
    })
    .mockResolvedValue(null);

  const response = await optimizeJob(request({}), params({ jobId: job.id }));
  const body = await response.json();

  expect(response.status).toBe(200);
  expect(body.optimizedArticle.title).toBe("API LLM Optimized GEO Article");
  expect(llmClient.generateValidatedJson).toHaveBeenCalled();
});

Step 2: Run API tests to verify they pass

Run:
```
npm test -- src/app/api/__tests__/jobs.test.ts
```
Expected: PASS.

Step 3: Update README environment description

In README.md, replace:

When no API key is configured, deterministic local fallbacks keep the workflow
usable for tests and local review.

with:

When a DeepSeek or OpenAI-compatible key is configured, the workflow uses the
provider for fact extraction, article optimization, QA enrichment, and targeted
rewrite. Every LLM response is validated with Zod before use. When no API key is
configured, or when the provider response is invalid, deterministic local
fallbacks keep the workflow usable for tests and local review.

Step 4: Run full verification

Run:
```
npm test
npm run build
```
Expected: both commands PASS.

Step 5: Commit

git add src/app/api/__tests__/jobs.test.ts README.md
git commit -m "test: cover llm workflow through api"

Task 8: Optional Manual DeepSeek Smoke Test

Files:

Verify: .env.local
Verify: src/lib/workflow/**
Step 1: Confirm environment values are present locally

Run:
```
test -n "$DEEPSEEK_API_KEY" || test -n "$(grep '^DEEPSEEK_API_KEY=.' .env.local 2>/dev/null)"
```
Expected: command exits successfully. If it fails, add a real DEEPSEEK_API_KEY to .env.local on the local machine only.
Step 2: Start the app

Run:
```
npm run dev
```
Expected: Next.js starts and prints a local URL such as http://localhost:3000.
Step 3: Exercise the UI

In the browser:
- Open http://localhost:3000.
- Enter title: Example Technology Co., Ltd. GEO 指南.
- Enter body: Example Technology Co., Ltd. has 8 years of GEO optimization experience. Example GEO 帮助市场团队优化内容结构。
- Enter image description: 产品仪表盘截图.
- Enter user instruction: 保持事实准确，语气自然，不要新增未经确认的客户案例。
- Click 分析文章.
- Clear any 待确认事项 after reviewing the fact card.
- Click 确认事实卡.
- Click 开始优化.
Expected: the generated title/body should no longer look like the deterministic fallback template unless the provider call failed. Export links should appear when QA is pass or warn.
Step 4: Stop the dev server

Press Ctrl-C in the terminal running npm run dev.

Self-Review

Spec coverage: The plan connects configured DeepSeek/OpenAI-compatible credentials into fact extraction, article optimization, targeted rewrite, and QA enrichment. It preserves hard QA failures, export blocking, Zod validation, and local deterministic fallback.
Placeholder scan: The plan contains concrete file paths, commands, code snippets, and expected outputs. It does not rely on unspecified behavior.
Type consistency: All new calls use existing domain schemas and types: candidateFactCardSchema, optimizedArticleSchema, qaCheckSchema, qaReportSchema, ArticleInput, ConfirmedFactCard, OptimizedArticle, and QaCheck.

40 KiB Raw Blame History

DeepSeek Workflow Integration Implementation Plan

Scope

File Structure

Task 1: Add Validated LLM Helper

Task 2: Add Prompt Builders

Task 3: Connect Fact Extraction To LLM

Task 4: Connect Article Optimization To LLM

Task 5: Connect Targeted Rewrite To LLM

Task 6: Add Hybrid LLM Quality Inspection

Task 7: Add API-Level LLM Coverage And Docs

Task 8: Optional Manual DeepSeek Smoke Test

Self-Review

40 KiB

Raw Blame History