fix(wechat): 公众号新版页面抓不到正文,改从 cgiDataNew 里读
CI / backend (push) Canceled after 0s

用户发来 https://mp.weixin.qq.com/s/ZEOsi4R4JcmZ_mhS6qpcww,存进库的正文只有 67 个字符
("原文链接 + 一个句号")。复盘发现公众号换了渲染方式:新版页面把整篇塞进
window.cgiDataNew 这个 JS 对象由前端渲染,页面里根本没有 id="js_content";
三种 UA 拿到的字节数完全一样,也不是 UA 的问题。

- 从 JS 字符串字面量里读正文(content_noencode / desc),按 \xNN、\uNNNN 反转义,
  话题标签是双重转义过的 HTML,先 unescape 再走 DOM。
- 配图字段排在文字之后,所以前后各找 2500 字符合图;剩下没用过的 mmbiz 图按
  12 张上限补在末尾(图文型文章靠就近配对会漏)。
- 解析顺序:旧 DOM → 新版 JS 数据 → 通用抽取,一层层退。

实测两篇:2373 字符 + 12 张图、2772 字符 + 13 张图,作者也都取到了。

同时补上用户提的两件:

- 知识能删了:DELETE /knowledge/{id}?confirm=true,软删 + 真删检索切片,
  只有作者或 owner/admin 能删;列表卡片(悬停出现)和阅读页动作行各一个入口。
  删完详情也回 7001,否则收藏的链接还能打开一篇已删除的知识。
- 分类下面有标签清单了:GET /knowledge/tags 按标签数"有多少条知识用了它",
  侧栏渲染成 #标签 + 条数,点一下筛选、再点取消。
This commit is contained in:
Steven
2026-11-06 02:08:36 +08:00
parent 626f83fb9c
commit 690779c5fb
10 changed files with 720 additions and 15 deletions
+26 -2
View File
@@ -513,7 +513,30 @@ Web 端未读数由 `AppShell` 每 30 秒轮询一次;在通知页标已读后
### 5.2 `GET /knowledge/{item_id}` ### 5.2 `GET /knowledge/{item_id}`
详情,附最近 50 条 `activities`。不存在或跨租户统一 `7001`。 详情,附最近 50 条 `activities`。不存在、跨租户、**已经删掉**的统一 `7001`
(删除是软删,但对读的人来说就是"没了",否则收藏的链接还会打开一篇已删除的知识)。
### 5.2.1 `GET /knowledge/tags` — 侧栏那串标签
```
GET /knowledge/tags?status=published&status=in_review
```
```json
{"items": [{"name": "发布", "count": 12}, {"name": "回滚", "count": 3}]}
```
- `count` 是**有多少条知识用了它**(`count(distinct item_id)`),不是引用次数。
- 只统计未删除的知识;带了 `status` 就只数这些状态,和列表的 tab 对得上。
- 按条数从多到少排,同数按名字。标签是分类之外的第二根轴:分类回答"属于哪一块",标签回答"跟什么有关"。
### 5.2.2 `DELETE /knowledge/{item_id}?confirm=true` — 删除
- 必须显式带 `confirm=true`,否则 `1002`「删除不能撤销,带 confirm=true 再来一次」(防手滑)。
- 只有**作者本人**或 `owner`/`admin` 能删,否则 `3004`。
- 软删:`item.deleted_at` 落时间戳,列表、详情、检索、标签统计里都不再出现;
**检索切片真删**(`chunk` 表),否则"删掉的知识还能被搜出来"比删不掉更吓人。
- 重复删不回错误,回 `message: "已经删过了"`,让重试的客户端不必区分。
### 5.3 `POST /knowledge` — 手记或指定来源 ### 5.3 `POST /knowledge` — 手记或指定来源
@@ -758,7 +781,8 @@ Query:
|---|---| |---|---|
| 判断 | `extract_url()`:去掉首尾标点后只剩一个链接;带了别的话("看下 https://… 这个")仍按任务处理 | | 判断 | `extract_url()`:去掉首尾标点后只剩一个链接;带了别的话("看下 https://… 这个")仍按任务处理 |
| 抓取 | `httpx` 取页面(25 秒超时、8 MB 上限),`trafilatura` 抽正文;标题取 `og:title`/`<h1>`/`<title>` | | 抓取 | `httpx` 取页面(25 秒超时、8 MB 上限),`trafilatura` 抽正文;标题取 `og:title`/`<h1>`/`<title>` |
| 公众号 | `mp.weixin.qq.com` 单独走 DOM:标题 `#activity-name`、作者 `#js_name`、正文 `#js_content`,图片取 `data-src`(通用抽取器会把公众号的图丢光) | | 公众号(旧页) | `mp.weixin.qq.com` 且页面有 `#js_content`:标题 `#activity-name`、作者 `#js_name`、正文 `#js_content`,图片取 `data-src`(通用抽取器会把公众号的图丢光) |
| 公众号(新版页) | 没有 `#js_content` 时(`appmsg_type = '9'` / `'10002'`)整篇在 `window.cgiDataNew` 里:从 `content_noencode` / `desc` 的 JS 字符串字面量读正文,按 `\xNN`、`\uNNNN` 反转义,话题标签先 `html.unescape` 再走 DOM;配图字段(`cdn_url`/`img_url`/`pic_url`)排在文字之后,前后各找 2500 字符合图;作者兜底取 `nick_name` |
| 清洗 | 导航/页脚按行边界裁掉(`_trim_chrome`);`trafilatura` 结果太短(<120 字)就退到容器打分兜底 | | 清洗 | 导航/页脚按行边界裁掉(`_trim_chrome`);`trafilatura` 结果太短(<120 字)就退到容器打分兜底 |
| 配图 | 正文里的外链图下载进我们自己的存储,最多 12 张、单张 8 MB,正文改写成 `/attachments/{id}/raw?签名`——公众号一换图,外链就是一排红叉 | | 配图 | 正文里的外链图下载进我们自己的存储,最多 12 张、单张 8 MB,正文改写成 `/attachments/{id}/raw?签名`——公众号一换图,外链就是一排红叉 |
| 落库 | 知识草稿(`status="draft"`,`doc_type="note"`),正文首行是 `> 原文:<url>` | | 落库 | 知识草稿(`status="draft"`,`doc_type="note"`),正文首行是 `> 原文:<url>` |
+31 -1
View File
@@ -857,7 +857,37 @@
`lxml / courlan / htmldate / jusText / lxml_html_clean / …` 再 `pip install --no-deps trafilatura`; `lxml / courlan / htmldate / jusText / lxml_html_clean / …` 再 `pip install --no-deps trafilatura`;
`lxml` 6 起 `lxml.html.clean` 拆成 `lxml_html_clean`,不装就 `ImportError`。 `lxml` 6 起 `lxml.html.clean` 拆成 `lxml_html_clean`,不装就 `ImportError`。
### 13.9 还没做 #### 13.8.1 用户发来真链接,结果只抓到一个「。」
上线后用户发了 `https://mp.weixin.qq.com/s/ZEOsi4R4JcmZ_mhS6qpcww`,库里只有 67 个字符的正文。
复盘:**公众号换了渲染方式**——新版页面(`window.appmsg_type = '9'` / `'10002'`)整篇塞进
`window.cgiDataNew` 这个 JS 对象、由前端渲染,页面里根本没有 `id="js_content"`;
三种 UA(桌面 / iPhone Safari / MicroMessenger)拿到的字节数完全一样,所以也不是 UA 的问题。
修法(`backend/app/services/article.py`):
- 从 JS 字符串字面量里读正文:`content_noencode` / `desc`,按 `\xNN`、`\uNNNN` 反转义;
话题标签是双重转义的 HTML(`\x26lt;a class=\x26quot;wx_topic_link\x26quot;\x26gt;#调料…`),
先 `html.unescape` 再走 DOM。
- 配图字段(`cdn_url` / `img_url` / `pic_url`)**排在文字字段之后**,所以配图要前后各找 2500 字符;
剩下没用过的 `mmbiz.qpic.cn` 图按 12 张上限补在末尾(图文型「小绿书」的图不在文字附近)。
- 解析顺序:旧 DOM(`#js_content`)→ 新版 JS 数据 → 通用抽取,一层层退。
- 作者兜底从 `nick_name` 取,标题取 `og:title`。
结果:那篇 2373 字符 + 12 张图,蒜蓉酱那篇 2772 字符 + 13 张图,作者都取到了。
**教训:抓取器写死在一种页面结构上,等于给自己埋一个定时炸弹;线上先看真实 HTML 再改。**
### 13.9 知识库:标签清单与删除
用户提两件:"文章都缺少删除功能"、"分类有列表显示,但是分类的下面应该还有标签的列表"。
- `GET /knowledge/tags`:按标签统计"有多少条知识用了它",带 `status` 就跟着 tab 数;
侧栏在分类树下面渲染成一排 `#标签 + 条数`,点一下筛选、再点取消。
- `DELETE /knowledge/{item_id}?confirm=true`:软删 + 真删检索切片;只有作者或 `owner`/`admin` 能删;
列表卡片(悬停出现)和阅读页动作行各有一个入口。删完详情也回 `7001`,
否则收藏的链接还能打开一篇"已经删掉的知识"。
### 13.10 还没做
- 只做了"进"(收消息建事项),没做"出"(任务有进展回一条微信通知)。 - 只做了"进"(收消息建事项),没做"出"(任务有进展回一条微信通知)。
- 消息不认人:按通道创建者记账;要认人得先做微信身份绑定。 - 消息不认人:按通道创建者记账;要认人得先做微信身份绑定。
+89 -5
View File
@@ -22,7 +22,7 @@ from typing import Annotated, Any
from fastapi import APIRouter, Query from fastapi import APIRouter, Query
from pydantic import BaseModel, Field from pydantic import BaseModel, Field
from sqlalchemy import Select, and_, func, or_, select, update from sqlalchemy import Select, and_, delete, func, or_, select, update
from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.ext.asyncio import AsyncSession
from ..audit import record as audit_record from ..audit import record as audit_record
@@ -33,6 +33,7 @@ from ..models_knowledge import (
DOC_TYPES, DOC_TYPES,
KNOWLEDGE_STATUSES, KNOWLEDGE_STATUSES,
MAX_CATEGORY_DEPTH, MAX_CATEGORY_DEPTH,
Chunk,
KnowledgeCategory, KnowledgeCategory,
KnowledgeDoc, KnowledgeDoc,
) )
@@ -96,11 +97,23 @@ class StatusActionIn(BaseModel):
# ── 内部工具 ──────────────────────────────────────────────────────────── # ── 内部工具 ────────────────────────────────────────────────────────────
async def _load_doc( async def _load_doc(
session: AsyncSession, ws_id: uuid.UUID, item_id: uuid.UUID session: AsyncSession, ws_id: uuid.UUID, item_id: uuid.UUID, *, include_deleted: bool = False
) -> tuple[Item, KnowledgeDoc]: ) -> tuple[Item, KnowledgeDoc]:
"""取一条知识。
删除是软删(`deleted_at`),所以默认把已删除的当成"不存在":
否则删完还能从收藏的链接、浏览器历史里点开,用户会以为删除没生效。
只有删除接口自己需要 `include_deleted=True`,好让重复删回一句"已经删过了"。
"""
item = await session.get(Item, item_id) item = await session.get(Item, item_id)
doc = await session.get(KnowledgeDoc, item_id) doc = await session.get(KnowledgeDoc, item_id)
if item is None or doc is None or item.workspace_id != ws_id or item.kind != "knowledge": if (
item is None
or doc is None
or item.workspace_id != ws_id
or item.kind != "knowledge"
or (item.deleted_at is not None and not include_deleted)
):
raise AppError(ErrorCode.KNOWLEDGE_NOT_FOUND, "这份知识不存在") raise AppError(ErrorCode.KNOWLEDGE_NOT_FOUND, "这份知识不存在")
return item, doc return item, doc
@@ -595,8 +608,39 @@ async def delete_category(
return ok({"deleted": str(category_id), "uncategorized_docs": docs}) return ok({"deleted": str(category_id), "uncategorized_docs": docs})
# 注意路由顺序:`/knowledge/search` 必须排在 `/knowledge/{item_id}` 之前, # 注意路由顺序:`/knowledge/search` 与 `/knowledge/tags` 必须排在
# 否则 "search" 会被当成 uuid 解析,返回 422。 # `/knowledge/{item_id}` 之前,否则会被当成 uuid 解析,返回 422。
@router.get("/knowledge/tags", summary="知识用到的标签(带条数)")
async def list_knowledge_tags(
ctx: MemberDep,
session: SessionDep,
status: Annotated[list[str] | None, Query()] = None,
) -> dict:
"""只统计"还在用"的标签,给右侧那串标签用,点一下按标签筛。
条数是**知识条数**,不是引用次数——同一个标签在一条知识里挂两次也只算一条。
"""
stmt: Select = (
select(Tag.name, func.count(func.distinct(Item.id)).label("docs"))
.join(ItemTag, ItemTag.tag_id == Tag.id)
.join(Item, Item.id == ItemTag.item_id)
.join(KnowledgeDoc, KnowledgeDoc.item_id == Item.id)
.where(
Item.workspace_id == ctx.ws_id,
Item.kind == "knowledge",
Item.deleted_at.is_(None),
Tag.workspace_id == ctx.ws_id,
)
.group_by(Tag.name)
.order_by(func.count(func.distinct(Item.id)).desc(), Tag.name)
)
wanted = [s for s in (status or []) if s in KNOWLEDGE_STATUSES]
if wanted:
stmt = stmt.where(KnowledgeDoc.status_hint.in_(wanted))
rows = (await session.execute(stmt)).all()
return ok({"items": [{"name": name, "count": count} for name, count in rows]})
@router.get("/knowledge/search", summary="检索已发布知识") @router.get("/knowledge/search", summary="检索已发布知识")
async def search( async def search(
ctx: MemberDep, ctx: MemberDep,
@@ -756,6 +800,46 @@ async def patch_knowledge(
return ok(await _knowledge_out(session, ws_id, item, doc)) return ok(await _knowledge_out(session, ws_id, item, doc))
@router.delete("/knowledge/{item_id}", summary="删除知识(软删,连带删掉检索切片)")
async def delete_knowledge(
item_id: uuid.UUID,
ctx: MemberDep,
session: SessionDep,
confirm: Annotated[bool, Query(description="必须显式带 true,防手滑")] = False,
) -> dict:
"""删除是软删:条目留在库里(活动记录、来源关系都还在),只是不再出现在列表与检索里。
切片必须真删:检索只查 `chunk` 表,留着的话"已经删掉的知识"还能被搜出来,
那比删不掉更吓人。
"""
ws_id = ctx.ws_id
if not confirm:
raise AppError(ErrorCode.VALIDATION_FAILED, "删除不能撤销,带 confirm=true 再来一次")
item, doc = await _load_doc(session, ws_id, item_id, include_deleted=True)
if item.deleted_at is not None:
return ok({"id": str(item.id)}, message="已经删过了")
if item.creator_id != ctx.user.id and ctx.role not in ("owner", "admin"):
raise AppError(ErrorCode.INSUFFICIENT_ROLE, "只有作者本人或管理员能删这条知识")
item.deleted_at = datetime.now(timezone.utc)
item.version = (item.version or 1) + 1
await session.execute(
delete(Chunk).where(Chunk.workspace_id == ws_id, Chunk.item_id == item.id)
)
doc.chunk_count = 0
await session.flush()
await audit_record(
session,
workspace_id=ws_id,
actor_user_id=ctx.user.id,
action="knowledge.deleted",
target_type="item",
target_id=str(item.id),
payload={"title": item.title, "status": doc.status_hint},
)
return ok({"id": str(item.id)}, message="已删除")
# ── 状态动作 ──────────────────────────────────────────────────────────── # ── 状态动作 ────────────────────────────────────────────────────────────
@router.post("/knowledge/{item_id}/status", summary="提交审议 / 发布 / 归档 / 恢复") @router.post("/knowledge/{item_id}/status", summary="提交审议 / 发布 / 归档 / 恢复")
async def change_status( async def change_status(
+124 -1
View File
@@ -15,6 +15,7 @@
from __future__ import annotations from __future__ import annotations
import html as html_mod
import ipaddress import ipaddress
import re import re
import socket import socket
@@ -55,6 +56,15 @@ WECHAT_UA = (
IMAGE_EXTS = {"jpeg": "jpg", "jpg": "jpg", "png": "png", "gif": "gif", "webp": "webp"} IMAGE_EXTS = {"jpeg": "jpg", "jpg": "jpg", "png": "png", "gif": "gif", "webp": "webp"}
# 公众号新版页面(2024 之后)不再把正文放进 `#js_content`,而是把整篇塞进
# `window.cgiDataNew` 这个 JS 对象里,由前端渲染。所以还得会读 JS 字符串字面量:
# 换行写成 `\x0a`,话题链接是 `\x26lt;a …\x26gt;`(先转义成实体、再转义成 JS)。
_WECHAT_FIELD_RE = re.compile(r"(?:content_noencode|desc)\s*:\s*'")
_WECHAT_IMAGE_RE = re.compile(
r"""(?:cdn_url|img_url|pic_url)\s*:\s*['"]((?:https?:)?//[^'"]*mmbiz\.qpic\.cn[^'"]*)['"]"""
)
_JS_ESCAPE_RE = re.compile(r"\\(x[0-9a-fA-F]{2}|u[0-9a-fA-F]{4}|.)", re.S)
class ArticleError(RuntimeError): class ArticleError(RuntimeError):
"""抓取/解析没成功。调用方据此决定降级方式。""" """抓取/解析没成功。调用方据此决定降级方式。"""
@@ -174,6 +184,10 @@ def _wechat_meta(html: str) -> tuple[str, str, list[tuple[str, str]]]:
author = " ".join(p for p in parts if p) author = " ".join(p for p in parts if p)
if author: if author:
break break
if not author:
# 新版页面没有 #js_name,公众号名在 cgiDataNew 的 nick_name 字段里
found = re.search(r"nick_name\s*:\s*'([^']{1,80})'", html)
author = _clean_text(found.group(1)) if found else ""
return title, author, tree return title, author, tree
@@ -244,6 +258,115 @@ def _wechat_body(html: str) -> str:
return "\n\n".join(s.strip() for s in "\n".join(out).split("\n\n") if s.strip()) return "\n\n".join(s.strip() for s in "\n".join(out).split("\n\n") if s.strip())
def _js_literal(text: str, quote_pos: int) -> str:
"""读一个单引号 JS 字符串的原始内容(`quote_pos` 指向那个开引号)。"""
out: list[str] = []
i = quote_pos + 1
while i < len(text):
ch = text[i]
if ch == "\\":
out.append(text[i : i + 2])
i += 2
continue
if ch == "'":
break
out.append(ch)
i += 1
return "".join(out)
def _unescape_js(raw: str) -> str:
def repl(match: re.Match[str]) -> str:
seq = match.group(1)
if seq[0] in "xu" and len(seq) > 1:
return chr(int(seq[1:], 16))
return {"n": "\n", "t": "\t", "r": "\r", "'": "'", '"': '"', "\\": "\\"}.get(seq, seq)
return _JS_ESCAPE_RE.sub(repl, raw)
def _wechat_data_body(html_text: str) -> str:
"""新版公众号:从 `window.cgiDataNew` 的 JS 字段里把正文和配图捞出来。
一篇文章是若干「图片 + 一段说明文字」拼起来的,JS 里是
`cdn_url: '…'` 后面紧跟 `desc` / `content_noencode`,所以按出现顺序取文字,
再往回找它自己那张图,配图就跟着段落走。
"""
blocks: list[str] = []
seen_text: set[str] = set()
used_images: set[str] = set()
for match in _WECHAT_FIELD_RE.finditer(html_text):
piece = _fragment_to_markdown(_unescape_js(_js_literal(html_text, match.end() - 1)))
if len(piece) < 30 or piece in seen_text:
continue
seen_text.add(piece)
image = _nearby_image(html_text, match.start(), used_images)
if image and "![](" not in piece:
used_images.add(image)
piece = f"{piece}\n\n![]({image})"
blocks.append(piece)
# 图文型文章("小绿书")的图不在文字附近,靠上面的就近配对会漏掉;
# 剩下的正文图统一补在末尾,宁可顺序差点也别丢图。
leftover: list[str] = []
for match in _WECHAT_IMAGE_RE.finditer(html_text):
image = match.group(1)
image = image if image.startswith("http") else f"https:{image}"
if image in used_images:
continue
used_images.add(image)
leftover.append(f"![]({image})")
if len(leftover) >= MAX_IMAGES:
break
if leftover:
blocks.append("\n\n".join(leftover))
return "\n\n".join(blocks).strip()
def _fragment_to_markdown(fragment: str) -> str:
"""把「实体转义 + 夹着标签」的正文片段变成 markdown 文本。
公众号把话题标签写成 `&lt;a class="wx_topic_link" …&gt;#调料&lt;/a&gt;`,
先 unescape 再当 HTML 处理,标签会被去掉、只留文字;`<img>` 转成 markdown。
"""
from lxml import html as lxml_html
fragment = html_mod.unescape(fragment or "")
fragment = re.sub(
r"""<img[^>]*?(?:data-src|src)\s*=\s*["']([^"']+)["'][^>]*>""",
lambda m: f"\n\n![]({m.group(1)})\n\n",
fragment,
flags=re.I,
)
fragment = re.sub(r"<br\s*/?>", "\n", fragment, flags=re.I)
try:
tree = lxml_html.fragment_fromstring(fragment, create_parent="div")
text = tree.text_content()
except Exception: # 片段结构烂到 lxml 都不认,直接剥标签
text = re.sub(r"<[^>]+>", "", fragment)
text = text.replace("\u00a0", " ")
lines = [re.sub(r"[ \t\u3000]+", " ", line).strip() for line in text.splitlines()]
out: list[str] = []
for line in lines:
if not line and (not out or not out[-1]):
continue
out.append(line)
return "\n".join(out).strip()
def _nearby_image(html_text: str, pos: int, used: set[str]) -> str | None:
"""这条文字对应的配图:先往后找(新版页面图片字段排在文字之后),再往回找。"""
for start, end in ((pos, pos + 2500), (max(0, pos - 2500), pos)):
found = [
m.group(1)
for m in _WECHAT_IMAGE_RE.finditer(html_text, start, min(end, len(html_text)))
if m.group(1) not in used
]
if found:
image = found[0]
return image if image.startswith("http") else f"https:{image}"
return None
def _block_markdown(html: str) -> str: def _block_markdown(html: str) -> str:
"""兜底的正文提取:找出"装着最多段文字"的那个容器,再按段落拼成 markdown。 """兜底的正文提取:找出"装着最多段文字"的那个容器,再按段落拼成 markdown。
@@ -371,7 +494,7 @@ def parse_article(html: str, url: str) -> Article:
raise ArticleError("微信把这篇拦下来了(需要人工验证或链接已失效)") raise ArticleError("微信把这篇拦下来了(需要人工验证或链接已失效)")
if is_wechat_article(url): if is_wechat_article(url):
title, author, _tree = _wechat_meta(html) title, author, _tree = _wechat_meta(html)
body = _wechat_body(html) body = _wechat_body(html) or _wechat_data_body(html)
if not body.strip(): if not body.strip():
# 公众号偶尔会换结构,退回通用抽取,宁可少图也别丢正文 # 公众号偶尔会换结构,退回通用抽取,宁可少图也别丢正文
article = _generic_article(html, url) article = _generic_article(html, url)
+115
View File
@@ -423,3 +423,118 @@ async def test_list_pages_through_every_document(client: AsyncClient) -> None:
assert cursor is None assert cursor is None
assert sorted(seen) == titles assert sorted(seen) == titles
assert len(seen) == len(set(seen)), "翻页翻出了重复条目" assert len(seen) == len(set(seen)), "翻页翻出了重复条目"
async def test_delete_removes_from_list_and_search(client: AsyncClient) -> None:
"""删掉的知识不能还留在列表和检索里——切片要一起删。"""
owner = await login(client)
ws = await create_workspace(client, owner, "删除知识")
note = unwrap(
await client.post(
f"{V1}/workspaces/{ws['id']}/knowledge",
json={
"title": "回滚前先停灰度",
"body_md": "灰度流量上涨后错误率飙升,第一步先停灰度。",
"tag_names": ["发布", "回滚"],
},
headers=owner["headers"],
)
)
unwrap(
await client.post(
f"{V1}/workspaces/{ws['id']}/knowledge/{note['id']}/status",
json={"action": "publish", "version": note["version"]},
headers=owner["headers"],
)
)
# 不带 confirm 不给删,防手滑
resp = await client.delete(
f"{V1}/workspaces/{ws['id']}/knowledge/{note['id']}", headers=owner["headers"]
)
assert resp.json()["code"] == 1002
unwrap(
await client.delete(
f"{V1}/workspaces/{ws['id']}/knowledge/{note['id']}",
params={"confirm": "true"},
headers=owner["headers"],
)
)
page = unwrap(
await client.get(f"{V1}/workspaces/{ws['id']}/knowledge", headers=owner["headers"])
)
assert page["items"] == []
hits = unwrap(
await client.get(
f"{V1}/workspaces/{ws['id']}/knowledge/search",
params={"q": "灰度"},
headers=owner["headers"],
)
)
assert hits["items"] == []
tags = unwrap(
await client.get(f"{V1}/workspaces/{ws['id']}/knowledge/tags", headers=owner["headers"])
)
assert tags["items"] == [], "删掉的知识不该再给标签贡献条数"
# 删完再从收藏的链接点进来:要当它不存在,不能还渲染出一篇"删掉了的知识"
detail = await client.get(
f"{V1}/workspaces/{ws['id']}/knowledge/{note['id']}", headers=owner["headers"]
)
assert detail.json()["code"] == 7001
again = await client.delete(
f"{V1}/workspaces/{ws['id']}/knowledge/{note['id']}",
params={"confirm": "true"},
headers=owner["headers"],
)
assert again.json()["message"] == "已经删过了", "重复删不该报错,回一句就够"
async def test_only_author_or_admin_can_delete(client: AsyncClient) -> None:
owner = await login(client)
ws = await create_workspace(client, owner, "谁能删")
member = await _member(client, owner, ws)
note = unwrap(
await client.post(
f"{V1}/workspaces/{ws['id']}/knowledge",
json={"title": "别人的笔记", "body_md": "随手记的"},
headers=owner["headers"],
)
)
resp = await client.delete(
f"{V1}/workspaces/{ws['id']}/knowledge/{note['id']}",
params={"confirm": "true"},
headers=member["headers"],
)
assert resp.json()["code"] == 3004
# 作者自己能删
unwrap(
await client.delete(
f"{V1}/workspaces/{ws['id']}/knowledge/{note['id']}",
params={"confirm": "true"},
headers=owner["headers"],
)
)
async def test_tags_endpoint_counts_documents(client: AsyncClient) -> None:
owner = await login(client)
ws = await create_workspace(client, owner, "标签统计")
for title, tags in (("甲", ["发布", "回滚"]), ("乙", ["发布"])):
unwrap(
await client.post(
f"{V1}/workspaces/{ws['id']}/knowledge",
json={"title": title, "tag_names": tags},
headers=owner["headers"],
)
)
got = unwrap(
await client.get(f"{V1}/workspaces/{ws['id']}/knowledge/tags", headers=owner["headers"])
)
counts = {row["name"]: row["count"] for row in got["items"]}
assert counts == {"发布": 2, "回滚": 1}
+40
View File
@@ -43,6 +43,28 @@ BLOCKED_HTML = """
<html><body><div class="weui-msg"><p>环境异常</p><p>完成验证后即可继续访问</p></div></body></html> <html><body><div class="weui-msg"><p>环境异常</p><p>完成验证后即可继续访问</p></div></body></html>
""" """
# 新版公众号页面:没有 #js_content,正文全在 window.cgiDataNew 里;
# 换行是 \x0a,话题标签是双重转义过的 HTML 片段,图片字段排在文字前面。
# 正文抽成常量:desc 与 content_noencode 同文(真实页面也是这样,后者更权威),
# 写成一行会超行宽,所以拆开拼。
_WECHAT_DATA_BODY = (
r"食材: 蒜瓣500g、小米辣60g\x0a\x0a步骤:\x0a1\u20e3蒜瓣打碎,"
r"放在冷水里泡10分钟(泡一下蒜不会发苦)\x0a2\u20e3油烧热,转小火把蒜泥炒香"
r"\x0a\x0a注意\x26lt;a class=\x26quot;wx_topic_link\x26quot;\x26gt;"
r"#调料\x26lt;/a\x26gt;"
)
WECHAT_DATA_HTML = (
"<html><head>\n"
'<meta property="og:title" content="自制万能蒜蓉酱" />\n'
"</head><body>\n<script>\n window.cgiDataNew = {\n"
" nick_name: '破冰记的随笔',\n item_list: [{\n"
" cdn_url: 'http://mmbiz.qpic.cn/mmbiz_jpg/xyz/640?wx_fmt=jpeg',\n"
" title: '自制万能蒜蓉酱',\n"
" desc: '" + _WECHAT_DATA_BODY + "',\n"
" content_noencode: '" + _WECHAT_DATA_BODY + "'\n"
" }]\n };\n</script>\n</body></html>\n"
)
def test_extract_url_only_when_message_is_a_link() -> None: def test_extract_url_only_when_message_is_a_link() -> None:
assert article_service.extract_url("https://mp.weixin.qq.com/s/abc") == ( assert article_service.extract_url("https://mp.weixin.qq.com/s/abc") == (
@@ -71,6 +93,24 @@ def test_wechat_article_parses_title_body_and_images() -> None:
assert "扫码关注" not in got.body_md assert "扫码关注" not in got.body_md
def test_new_wechat_page_reads_js_data() -> None:
"""新版页面没有 #js_content:正文在 cgiDataNew 的 JS 字符串里,得能读出来。"""
got = article_service.parse_article(WECHAT_DATA_HTML, "https://mp.weixin.qq.com/s/abc")
assert got.title == "自制万能蒜蓉酱"
assert got.author == "破冰记的随笔"
assert "蒜瓣打碎" in got.body_md
# \x0a 要还原成换行,否则整篇挤成一行
assert "\n" in got.body_md
# 双重转义的话题标签还原成文字,不能留下 &lt;a 这种碎片
assert "#调料" in got.body_md
assert "&lt;" not in got.body_md
# 配图要跟着正文进来(正文里换成本地地址之前,先能取到这张图)
assert "mmbiz.qpic.cn/mmbiz_jpg/xyz/640" in got.body_md
# desc 与 content_noencode 是同一段文字,不能重复两遍
assert got.body_md.count("蒜瓣打碎") == 1
def test_blocked_wechat_page_is_reported() -> None: def test_blocked_wechat_page_is_reported() -> None:
with pytest.raises(article_service.ArticleError): with pytest.raises(article_service.ArticleError):
article_service.parse_article(BLOCKED_HTML, "https://mp.weixin.qq.com/s/abc") article_service.parse_article(BLOCKED_HTML, "https://mp.weixin.qq.com/s/abc")
+160
View File
@@ -0,0 +1,160 @@
/**
* 知识库的两件小事:分类下面的标签清单能点着筛,文章能删。
*
* 为什么单独写一条:标签统计和删除都是"后端好了但界面没入口"的典型,
* 看不见就等于没有;所以这里从界面点,最后用接口回查确认真的生效。
*
* 用法:
* node web/e2e/knowledge-tags-delete.mjs # 默认 dev 5180
* WEB=http://192.168.8.188:8200 PHONE=13605513683 PASSWORD=... node web/e2e/knowledge-tags-delete.mjs
*/
import { readFile } from "node:fs/promises";
import { chromium } from "playwright-core";
const WEB = process.env.WEB ?? "http://localhost:5180";
const API = process.env.API ?? `${WEB}/api/v1`;
const PHONE = process.env.PHONE ?? "13900000001";
const CHROME =
process.env.CHROME ??
`${process.env.LOCALAPPDATA}\\ms-playwright\\chromium-1223\\chrome-win64\\chrome.exe`;
let pass = 0;
let fail = 0;
function check(name, ok, extra = "") {
if (ok) {
pass += 1;
console.log(`PASS ${name}`);
} else {
fail += 1;
console.log(`FAIL ${name} ${extra}`);
}
}
async function readPassword() {
if (process.env.PASSWORD) return process.env.PASSWORD;
try {
const file = new URL("../../.local/e2e-account-password.txt", import.meta.url);
const text = (await readFile(file, "utf8")).trim();
return text || null;
} catch {
return null;
}
}
async function call(path, { method = "GET", body, token } = {}) {
const headers = { "Content-Type": "application/json" };
if (token) headers.Authorization = `Bearer ${token}`;
const r = await fetch(`${API}${path}`, {
method,
headers,
body: body === undefined ? undefined : JSON.stringify(body),
});
return r.json();
}
async function main() {
const password = await readPassword();
if (!password) throw new Error("没有密码:配 PASSWORD 或写 .local/e2e-account-password.txt");
const login = await call("/auth/login/password", {
method: "POST",
body: { phone: PHONE, password },
});
if (login.code !== 0) throw new Error(`登录失败:${login.message}`);
const token = login.data.tokens.access_token;
const spaces = (await call("/workspaces", { token })).data?.items ?? [];
if (!spaces.length) throw new Error("这个账号没有工作区");
const ws = spaces[0].id;
// 自己造数据:标签名带时间戳,避免和库里已有的同名标签混在一起数不准
const stamp = Date.now().toString().slice(-6);
const tag = `核对${stamp}`;
const created = await call(`/workspaces/${ws}/knowledge`, {
method: "POST",
token,
body: { title: `临时核对:标签与删除 ${stamp}`, body_md: "这条是 e2e 建的,跑完会删掉。", tag_names: [tag] },
});
if (created.code !== 0) throw new Error(`建知识失败:${created.message}`);
const doc = created.data;
await call(`/workspaces/${ws}/knowledge/${doc.id}/status`, {
method: "POST",
token,
body: { action: "publish", version: doc.version },
});
const browser = await chromium.launch({ executablePath: CHROME });
const context = await browser.newContext({ viewport: { width: 1440, height: 960 } });
await context.addInitScript(
([access, refresh]) => {
localStorage.setItem("dadao.access", access);
localStorage.setItem("dadao.refresh", refresh);
},
[token, login.data.tokens.refresh_token],
);
const page = await context.newPage();
const errors = [];
page.on("pageerror", (e) => errors.push(String(e)));
page.on("dialog", (d) => void d.accept());
await page.goto(`${WEB}/w/${ws}/knowledge`, { waitUntil: "networkidle" });
// ① 分类下面有「标签」块,且自己建的那个标签在里面
const tagBox = page.locator("section", { hasText: "标签" }).last();
const tagBtn = page.getByRole("button", { name: new RegExp(`#${tag}`) }).first();
await tagBtn.waitFor({ timeout: 10000 });
check("分类下面是标签清单", (await tagBox.count()) > 0);
check("刚建的标签出现在侧栏(带条数)", (await tagBtn.count()) > 0);
// ② 点标签 → 列表只剩这一类
await tagBtn.click();
await page.waitForTimeout(800);
const titles = await page.locator("li p.truncate").allInnerTexts();
check(
"按标签筛出的正好是自己那条",
titles.filter((t) => t.includes(stamp)).length === 1 && titles.length >= 1,
`titles=${titles.length}`,
);
check("标签被选中", (await page.getByRole("button", { name: new RegExp(`#${tag}`) }).first().getAttribute("aria-pressed")) === "true");
// ③ 再点一次取消筛选
await tagBtn.click();
await page.waitForTimeout(600);
check(
"再点一次取消标签筛选",
(await page.getByRole("button", { name: new RegExp(`#${tag}`) }).first().getAttribute("aria-pressed")) === "false",
);
// ④ 卡片上的删除:确认框点"确定",这条从列表消失
const row = page.locator("li").filter({ hasText: stamp }).first();
await row.hover();
await page.getByRole("button", { name: `删除 临时核对:标签与删除 ${stamp}` }).first().click();
await page.waitForTimeout(1200);
check("列表里已经看不到这条", (await page.locator("li").filter({ hasText: stamp }).count()) === 0);
const afterDelete = await call(`/workspaces/${ws}/knowledge/${doc.id}`, { token });
check("接口回查:这条确实没了", afterDelete.code !== 0, `code=${afterDelete.code}`);
// ⑤ 阅读页也有删除入口(换一条已存在的打开看看)
const page2 = await call(`/workspaces/${ws}/knowledge?limit=1`, { token });
const anyDoc = page2.data?.items?.[0];
if (anyDoc) {
await page.goto(`${WEB}/w/${ws}/knowledge/${anyDoc.id}`, { waitUntil: "networkidle" });
check("阅读页有「删除」按钮", (await page.getByRole("button", { name: "删除" }).count()) > 0);
}
check("控制台零报错", errors.length === 0, errors.slice(0, 2).join(" | "));
await page.goto(`${WEB}/w/${ws}/knowledge`, { waitUntil: "networkidle" });
await page.screenshot({ path: "e2e/artifacts/knowledge-tags.png" });
await browser.close();
// 收尾:万一界面那步没删掉,这里补一刀,别给下次跑留垃圾
await call(`/workspaces/${ws}/knowledge/${doc.id}?confirm=true`, { method: "DELETE", token });
console.log(`\n${fail === 0 ? "ALL PASS" : "FAILED"} — pass ${pass}, fail ${fail}`);
if (fail > 0) process.exitCode = 1;
}
main().catch((err) => {
console.error("FAIL 脚本异常 —", err);
process.exitCode = 1;
});
+6
View File
@@ -191,6 +191,12 @@ export interface KnowledgeCategoryNode extends KnowledgeCategory {
doc_count: number; doc_count: number;
} }
/** 侧栏那串标签:`count` 是"有多少条知识用了它",不是引用次数。 */
export interface KnowledgeTagCount {
name: string;
count: number;
}
export interface Knowledge extends Item { export interface Knowledge extends Item {
knowledge: KnowledgeInfo; knowledge: KnowledgeInfo;
category: KnowledgeCategory | null; category: KnowledgeCategory | null;
+107 -5
View File
@@ -28,6 +28,7 @@ import type {
Attachment, Attachment,
Knowledge, Knowledge,
KnowledgeCategoryNode, KnowledgeCategoryNode,
KnowledgeTagCount,
SearchHit, SearchHit,
} from "../lib/types"; } from "../lib/types";
@@ -76,6 +77,9 @@ export function KnowledgePage() {
/** null = 不按分类筛(全部);"none" = 未分类;其它是分类 id(含子分类) */ /** null = 不按分类筛(全部);"none" = 未分类;其它是分类 id(含子分类) */
const [catFilter, setCatFilter] = useState<string | null>(null); const [catFilter, setCatFilter] = useState<string | null>(null);
const [categories, setCategories] = useState<KnowledgeCategoryNode[]>([]); const [categories, setCategories] = useState<KnowledgeCategoryNode[]>([]);
/** 分类下面的标签清单:知识库就靠分类 + 标签两条轴找东西 */
const [tags, setTags] = useState<KnowledgeTagCount[]>([]);
const [tagFilter, setTagFilter] = useState<string | null>(null);
const [catBusy, setCatBusy] = useState(false); const [catBusy, setCatBusy] = useState(false);
const reqId = useRef(0); const reqId = useRef(0);
@@ -90,10 +94,28 @@ export function KnowledgePage() {
} }
}, [wsId]); }, [wsId]);
const loadTags = useCallback(async () => {
try {
const qs = new URLSearchParams();
for (const s of TABS.find((t) => t.key === tab)!.statuses) qs.append("status", s);
const page = await api.get<{ items: KnowledgeTagCount[] }>(
`/workspaces/${wsId}/knowledge/tags?${qs.toString()}`,
);
setTags(page.items);
} catch {
// 标签只是辅助导航,取不到不该挡住列表
setTags([]);
}
}, [wsId, tab]);
useEffect(() => { useEffect(() => {
void loadCategories(); void loadCategories();
}, [loadCategories]); }, [loadCategories]);
useEffect(() => {
void loadTags();
}, [loadTags]);
async function createCategory(name: string, parentId: string | null, color: string | null = null) { async function createCategory(name: string, parentId: string | null, color: string | null = null) {
setCatBusy(true); setCatBusy(true);
setError(null); setError(null);
@@ -186,6 +208,7 @@ export function KnowledgePage() {
if (q.trim()) qs.set("q", q.trim()); if (q.trim()) qs.set("q", q.trim());
if (catFilter === "none") qs.set("uncategorized", "true"); if (catFilter === "none") qs.set("uncategorized", "true");
else if (catFilter) qs.set("category_id", catFilter); else if (catFilter) qs.set("category_id", catFilter);
if (tagFilter) qs.set("tag", tagFilter);
if (nextCursor) qs.set("cursor", nextCursor); if (nextCursor) qs.set("cursor", nextCursor);
const page = await api.get<{ const page = await api.get<{
items: Knowledge[]; items: Knowledge[];
@@ -202,7 +225,7 @@ export function KnowledgePage() {
if (mine === reqId.current) setLoading(false); if (mine === reqId.current) setLoading(false);
} }
}, },
[wsId, tab, q, catFilter], [wsId, tab, q, catFilter, tagFilter],
); );
useEffect(() => { useEffect(() => {
@@ -239,11 +262,30 @@ export function KnowledgePage() {
setComposing(false); setComposing(false);
setTab("draft"); setTab("draft");
setCatFilter(null); setCatFilter(null);
setTagFilter(null);
openDoc(doc.id); openDoc(doc.id);
void load(null); void load(null);
void loadCategories(); void loadCategories();
} }
/**
* 删除是软删(后端 `deleted_at`),但界面上一旦删了就当它不在了:
* 所以列表重拉、标签计数重算,别留一条需要刷新才消失的幽灵。
*/
async function removeDoc(doc: Knowledge) {
if (!window.confirm(`删除「${doc.title}」?删了不会回来,检索里也会一起消失。`)) return;
setError(null);
try {
await api.del(`/workspaces/${wsId}/knowledge/${doc.id}?confirm=true`);
setDocs((prev) => prev.filter((d) => d.id !== doc.id));
void load(null);
void loadCategories();
void loadTags();
} catch (err) {
setError(err instanceof ApiError ? err.message : "删除失败");
}
}
const pendingCount = docs.filter((d) => d.knowledge.status_hint !== "published").length; const pendingCount = docs.filter((d) => d.knowledge.status_hint !== "published").length;
const activeCategory = categories.find((c) => c.id === catFilter) ?? null; const activeCategory = categories.find((c) => c.id === catFilter) ?? null;
@@ -291,6 +333,13 @@ export function KnowledgePage() {
loading={loading} loading={loading}
onSelect={(value) => { onSelect={(value) => {
setCatFilter(value); setCatFilter(value);
setTagFilter(null);
setHits(null);
}}
tags={tags}
tagSelected={tagFilter}
onSelectTag={(name) => {
setTagFilter((prev) => (prev === name ? null : name));
setHits(null); setHits(null);
}} }}
busy={catBusy} busy={catBusy}
@@ -486,9 +535,20 @@ export function KnowledgePage() {
</div> </div>
) : null} ) : null}
</button> </button>
<Button variant="ghost" size="sm" className="shrink-0" onClick={() => openDoc(d.id)}> <div className="flex shrink-0 items-center gap-1">
打开 <Button variant="ghost" size="sm" onClick={() => openDoc(d.id)}>
</Button> 打开
</Button>
{/* 删除只在悬停/聚焦时露出来:这一列动作不该比标题更抢眼 */}
<button
className="rounded-md p-1.5 text-ink-400 opacity-0 transition hover:bg-red-50 hover:text-red-600 focus:opacity-100 group-hover:opacity-100"
title="删除这条知识"
aria-label={`删除 ${d.title}`}
onClick={() => void removeDoc(d)}
>
<Trash size={14} />
</button>
</div>
</li> </li>
))} ))}
</ul> </ul>
@@ -548,12 +608,15 @@ function ColorSwatch({
); );
} }
/** 左侧分类树:一层一层缩进,点一下就把列表收窄到该分类(含子分类)。 */ /** 左侧分类树:一层一层缩进,点一下就把列表收窄到该分类(含子分类);分类下面是标签。 */
function CategorySidebar({ function CategorySidebar({
categories, categories,
selected, selected,
loading, loading,
onSelect, onSelect,
tags,
tagSelected,
onSelectTag,
busy, busy,
onCreate, onCreate,
onRename, onRename,
@@ -564,6 +627,9 @@ function CategorySidebar({
selected: string | null; selected: string | null;
loading: boolean; loading: boolean;
onSelect: (value: string | null) => void; onSelect: (value: string | null) => void;
tags: KnowledgeTagCount[];
tagSelected: string | null;
onSelectTag: (name: string) => void;
busy: boolean; busy: boolean;
onCreate: (name: string, parentId: string | null, color: string | null) => void; onCreate: (name: string, parentId: string | null, color: string | null) => void;
onRename: (cat: KnowledgeCategoryNode, name: string) => void; onRename: (cat: KnowledgeCategoryNode, name: string) => void;
@@ -686,6 +752,42 @@ function CategorySidebar({
还没有分类。先建一层「研发 / 部署」这样的目录,知识才有地方落脚。 还没有分类。先建一层「研发 / 部署」这样的目录,知识才有地方落脚。
</p> </p>
) : null} ) : null}
{/* 标签是第二根轴:分类回答"这属于哪一块",标签回答"它跟什么有关"。
按条数从多到少排,点一下切换筛选(再点一次取消)。 */}
{tags.length > 0 ? (
<section className="mt-4 border-t border-line-200 pt-3">
<div className="flex items-center justify-between px-1 pb-1">
<span className="text-2xs font-medium uppercase tracking-wide text-ink-400">标签</span>
{tagSelected ? (
<button className="link text-2xs" onClick={() => onSelectTag(tagSelected)}>
清除
</button>
) : null}
</div>
<div className="flex flex-wrap gap-1 px-1">
{tags.map((t) => {
const active = tagSelected === t.name;
return (
<button
key={t.name}
aria-pressed={active}
onClick={() => onSelectTag(t.name)}
className={`inline-flex max-w-full items-center gap-1 rounded-full px-2 py-0.5 text-2xs transition-colors ${
active
? "bg-primary-600 text-white"
: "bg-primary-50 text-primary-700 hover:bg-primary-100"
}`}
title={`${t.count} 条知识用了这个标签`}
>
<span className="truncate">#{t.name}</span>
<span className={active ? "text-white/70" : "text-primary-400"}>{t.count}</span>
</button>
);
})}
</div>
</section>
) : null}
</aside> </aside>
); );
} }
+22 -1
View File
@@ -11,7 +11,7 @@ import { Link, useNavigate, useParams } from "react-router-dom";
import { AppShell } from "../components/AppShell"; import { AppShell } from "../components/AppShell";
import { AttachmentPanel } from "../components/AttachmentPanel"; import { AttachmentPanel } from "../components/AttachmentPanel";
import { Markdown } from "../components/Markdown"; import { Markdown } from "../components/Markdown";
import { ChevronLeft, Link2, Pencil } from "../components/icons"; import { ChevronLeft, Link2, Pencil, Trash } from "../components/icons";
import { Button, InlineAlert, SkeletonRows, StatusPill } from "../components/ui"; import { Button, InlineAlert, SkeletonRows, StatusPill } from "../components/ui";
import { ApiError, api } from "../lib/client"; import { ApiError, api } from "../lib/client";
import { ACTION_LABEL, DOC_TYPE_LABEL, STATUS_LABEL, categoryColor } from "../lib/knowledge"; import { ACTION_LABEL, DOC_TYPE_LABEL, STATUS_LABEL, categoryColor } from "../lib/knowledge";
@@ -92,6 +92,22 @@ export function KnowledgeDocPage() {
const status = doc?.knowledge.status_hint; const status = doc?.knowledge.status_hint;
const canEdit = status === "draft" || status === "in_review" || status === "published"; const canEdit = status === "draft" || status === "in_review" || status === "published";
/** 删除:确认后回列表页。后端是软删,但界面按"没了"处理最不容易让人误会。 */
async function remove() {
if (!doc) return;
if (!window.confirm(`删除「${doc.title}」?删了不会回来,检索里也会一起消失。`)) return;
setBusy(true);
setError(null);
try {
await api.del(`/workspaces/${wsId}/knowledge/${itemId}?confirm=true`);
navigate(`/w/${wsId}/knowledge`, { replace: true });
} catch (err) {
setError(err instanceof ApiError ? err.message : "删除失败");
} finally {
setBusy(false);
}
}
const actions: { key: string; label: string; show: boolean }[] = [ const actions: { key: string; label: string; show: boolean }[] = [
{ key: "submit_review", label: ACTION_LABEL.submit_review, show: status === "draft" }, { key: "submit_review", label: ACTION_LABEL.submit_review, show: status === "draft" },
{ key: "withdraw", label: ACTION_LABEL.withdraw, show: status === "in_review" }, { key: "withdraw", label: ACTION_LABEL.withdraw, show: status === "in_review" },
@@ -210,6 +226,11 @@ export function KnowledgeDocPage() {
{a.label} {a.label}
</Button> </Button>
))} ))}
{/* 删除放最后、用弱色 + 垃圾桶:它是不可逆的,不该和"发布"抢注意力 */}
<Button variant="ghost" disabled={busy} className="ml-auto text-ink-400 hover:text-red-600" onClick={() => void remove()}>
<Trash size={13} />
删除
</Button>
</div> </div>
<div className="mt-6"> <div className="mt-6">