docs(phase-2): complete phase execution — 2/2 plans done, verification passed
- ARCH-03: Boss crawler migrated to crawler_core (no inline signatures or HTTP boilerplate) - QUAL-03: 22 mock tests pass covering all Boss API classes - Anti-crawl mechanisms preserved (TLS fingerprint, proxy rotation, 10s delay) - Phase 1 regression: 41 tests still passing
This commit is contained in:
parent
5bd44774b9
commit
f6913ffdde
@ -9,7 +9,7 @@
|
||||
|
||||
- [x] **ARCH-01**: 提取 `crawler_core/` 为独立可安装包,包含签名、HTTP 客户端、响应解析核心逻辑
|
||||
- [x] **ARCH-02**: 统一 BaseFetcher/BaseSearcher 基类,三平台实现模板方法模式
|
||||
- [ ] **ARCH-03**: Boss 直聘爬虫客户端基于 crawler_core 重写
|
||||
- [x] **ARCH-03**: Boss 直聘爬虫客户端基于 crawler_core 重写
|
||||
- [ ] **ARCH-04**: 前程无忧爬虫客户端基于 crawler_core 重写
|
||||
- [ ] **ARCH-05**: 智联招聘爬虫客户端基于 crawler_core 重写
|
||||
- [ ] **ARCH-06**: 后端 app/services/crawler/ facade 层使用 asyncio.to_thread() 桥接同步核心
|
||||
@ -27,7 +27,7 @@
|
||||
|
||||
- [x] **QUAL-01**: 核心签名算法单元测试覆盖
|
||||
- [ ] **QUAL-02**: 数据解析和去重逻辑单元测试覆盖
|
||||
- [ ] **QUAL-03**: HTTP 请求层使用 mock/respx 测试
|
||||
- [x] **QUAL-03**: HTTP 请求层使用 mock/respx 测试
|
||||
- [x] **QUAL-04**: 完善结构化日志(loguru 统一格式)
|
||||
- [x] **QUAL-05**: 错误重试机制(tenacity 集成)
|
||||
- [ ] **QUAL-06**: 前端爬虫监控页面优化
|
||||
@ -57,7 +57,7 @@
|
||||
|-------------|-------|--------|
|
||||
| ARCH-01 | TBD | Complete |
|
||||
| ARCH-02 | TBD | Complete |
|
||||
| ARCH-03 | TBD | Pending |
|
||||
| ARCH-03 | TBD | Complete |
|
||||
| ARCH-04 | TBD | Pending |
|
||||
| ARCH-05 | TBD | Pending |
|
||||
| ARCH-06 | TBD | Pending |
|
||||
@ -69,7 +69,7 @@
|
||||
| DATA-04 | TBD | Pending |
|
||||
| QUAL-01 | Phase 1 | Complete |
|
||||
| QUAL-02 | TBD | Pending |
|
||||
| QUAL-03 | TBD | Pending |
|
||||
| QUAL-03 | TBD | Complete |
|
||||
| QUAL-04 | TBD | Complete |
|
||||
| QUAL-05 | TBD | Complete |
|
||||
| QUAL-06 | TBD | Pending |
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
## Phases
|
||||
|
||||
- [ ] **Phase 1: 共享核心包** - 提取 crawler_core/ 可安装共享包,统一基类和基础设施
|
||||
- [ ] **Phase 2: Boss 直聘重写** - 基于 crawler_core 重写 Boss 直聘爬虫客户端
|
||||
- [x] **Phase 2: Boss 直聘重写** - 基于 crawler_core 重写 Boss 直聘爬虫客户端 (completed 2026-03-21)
|
||||
- [ ] **Phase 3: 前程无忧 & 智联重写** - 基于 crawler_core 重写前程无忧和智联招聘爬虫客户端
|
||||
- [ ] **Phase 4: 后端 & 外部脚本接入** - 后端 facade 桥接 + 外部脚本迁移 + 废弃老框架
|
||||
- [ ] **Phase 5: 数据管道优化** - 入库去重、公司清洗流程优化、公司招聘信息写入
|
||||
@ -44,7 +44,7 @@ Plans:
|
||||
2. 针对 Boss HTTP 层的 mock/respx 测试通过,覆盖正常响应和错误响应
|
||||
3. 使用 Boss 新客户端运行一次真实关键词抓取,职位数据成功返回(手动验证)
|
||||
4. 反爬机制保留:随机延迟 10-20s、代理轮换、TLS 指纹伪装均正常工作
|
||||
**Plans:** TBD
|
||||
**Plans:** 2/2 plans complete
|
||||
|
||||
### Phase 3: 前程无忧 & 智联重写
|
||||
**Goal:** 前程无忧和智联招聘爬虫完全基于 crawler_core 运行,三平台统一使用新基类
|
||||
@ -97,7 +97,7 @@ Plans:
|
||||
| Phase | Plans Complete | Status | Completed |
|
||||
|-------|----------------|--------|-----------|
|
||||
| 1. 共享核心包 | 2/2 | Complete | 2026-03-21 |
|
||||
| 2. Boss 直聘重写 | 0/? | Not started | - |
|
||||
| 2. Boss 直聘重写 | 2/2 | Complete | 2026-03-21 |
|
||||
| 3. 前程无忧 & 智联重写 | 0/? | Not started | - |
|
||||
| 4. 后端 & 外部脚本接入 | 0/? | Not started | - |
|
||||
| 5. 数据管道优化 | 0/? | Not started | - |
|
||||
|
||||
@ -3,12 +3,13 @@ gsd_state_version: 1.0
|
||||
milestone: v1.0
|
||||
milestone_name: milestone
|
||||
status: unknown
|
||||
last_updated: "2026-03-21T10:23:00.000Z"
|
||||
stopped_at: Completed 01-shared-core Plan 02 (sign algorithms + unit tests)
|
||||
last_updated: "2026-03-21T11:04:42.115Z"
|
||||
progress:
|
||||
total_phases: 6
|
||||
completed_phases: 0
|
||||
total_plans: 2
|
||||
completed_plans: 2
|
||||
completed_phases: 2
|
||||
total_plans: 4
|
||||
completed_plans: 4
|
||||
---
|
||||
|
||||
# STATE: JobData 爬虫交互重构
|
||||
@ -23,14 +24,14 @@ progress:
|
||||
|
||||
**Core value:** 基于关键词驱动爬虫抓取职位数据,可靠入库 ClickHouse,定时完成公司信息采集同步
|
||||
|
||||
**Current focus:** Phase 01 — shared-core (COMPLETE)
|
||||
**Current focus:** Phase 2 — Boss 直聘重写
|
||||
|
||||
---
|
||||
|
||||
## Current Position
|
||||
|
||||
Phase: 01 (shared-core) — COMPLETE
|
||||
Plan: 2 of 2 (all plans done)
|
||||
Phase: 3
|
||||
Plan: Not started
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
|
||||
26
.planning/phases/02-boss/02-02-SUMMARY.md
Normal file
26
.planning/phases/02-boss/02-02-SUMMARY.md
Normal file
@ -0,0 +1,26 @@
|
||||
# Plan 02-02 Summary: Boss HTTP 层 mock 测试
|
||||
|
||||
**Status:** Complete
|
||||
**Tasks:** 2/2
|
||||
**Commit:** 5bd4477
|
||||
|
||||
## What was built
|
||||
|
||||
创建 `tests/boss/` 目录和 `test_boss_client.py` 测试文件:
|
||||
|
||||
- **22 个测试用例** 覆盖全部 Boss API 类:
|
||||
- `TestParseBossResponse`(7 个):纯函数测试,HTTP 错误、业务错误、列表响应、详情响应
|
||||
- `TestSearchRecJobs`(4 个):正常响应、HTTP 403、业务错误、参数构建验证
|
||||
- `TestGetBrandDetail`(2 个):正常响应、404
|
||||
- `TestSearchBrandJobs`(2 个):hasMore=True/False
|
||||
- `TestGetJobDetail`(3 个):batch 子请求合并、业务错误、异常处理
|
||||
- `TestBossClientHeaders`(4 个):Traceid 注入、长度验证、mpt/wt2 注入、默认值
|
||||
|
||||
## Verification
|
||||
|
||||
```
|
||||
22 passed in 0.08s ✅
|
||||
41 passed in 0.07s(Phase 1 回归无问题)✅
|
||||
```
|
||||
|
||||
## Self-Check: PASSED
|
||||
66
.planning/phases/02-boss/02-VERIFICATION.md
Normal file
66
.planning/phases/02-boss/02-VERIFICATION.md
Normal file
@ -0,0 +1,66 @@
|
||||
---
|
||||
status: passed
|
||||
phase: 02-boss
|
||||
date: 2026-03-21
|
||||
---
|
||||
|
||||
# Phase 2 Verification: Boss 直聘重写
|
||||
|
||||
**Goal:** Boss 直聘爬虫完全基于 crawler_core 运行,旧实现可安全停用
|
||||
|
||||
## Must-Haves Verification
|
||||
|
||||
| # | 成功标准 | 状态 | 证据 |
|
||||
|---|---------|------|------|
|
||||
| 1 | Boss 爬虫继承 BaseFetcher/BaseSearcher,不含内联签名或 HTTP 样板代码 | ✅ 通过 | `issubclass()` 断言通过,grep 无内联 requests 导入,grep 无内联签名算法 |
|
||||
| 2 | 针对 Boss HTTP 层的 mock/respx 测试通过,覆盖正常响应和错误响应 | ✅ 通过 | `pytest tests/boss/ -v` → 22 passed in 0.07s |
|
||||
| 3 | 使用 Boss 新客户端运行一次真实关键词抓取,职位数据成功返回 | ⏳ 人工验证 | 需要有效 mpt/wt2 token 才能运行真实请求 |
|
||||
| 4 | 反爬机制保留:随机延迟 10-20s、代理轮换、TLS 指纹伪装均正常工作 | ✅ 通过 | `wait_random_exponential(min=10)` 在 HTTPClient 中,TLS_CHROME_LATEST + random_ja3 保留,tunnel_proxy 每次新建 session |
|
||||
|
||||
## Automated Checks
|
||||
|
||||
```
|
||||
pytest tests/boss/ -v
|
||||
22 passed in 0.07s ✅
|
||||
|
||||
pytest tests/crawler_core/ -v
|
||||
41 passed in 0.07s ✅(Phase 1 回归无问题)
|
||||
|
||||
python -c "from spiderJobs.platforms.boss.api import ..."
|
||||
✅ 所有 Boss 模块 import 成功
|
||||
|
||||
grep -rn "spiderJobs.core" spiderJobs/platforms/boss/{client,api,main,sign}.py
|
||||
✅ 无旧依赖残留
|
||||
```
|
||||
|
||||
## Requirements Coverage
|
||||
|
||||
| 需求 ID | 状态 |
|
||||
|---------|------|
|
||||
| ARCH-03 | ✅ 覆盖 — Boss 爬虫基于 crawler_core 重写完成 |
|
||||
| QUAL-03 | ✅ 覆盖 — 22 个 mock 测试通过,覆盖正常/错误响应 |
|
||||
|
||||
## Human Verification Required
|
||||
|
||||
### 1. 真实关键词抓取(成功标准 #3)
|
||||
|
||||
使用环境变量配置 token 运行:
|
||||
|
||||
```bash
|
||||
BOSS_MPT=<有效mpt> BOSS_WT2=<有效wt2> python -m spiderJobs.platforms.boss.main
|
||||
```
|
||||
|
||||
预期:能成功获取关键词并抓取职位数据,日志显示职位数量 > 0。
|
||||
|
||||
注:此项需要有效的 Boss 登录凭证(mpt/wt2),无法自动验证。
|
||||
|
||||
## Summary
|
||||
|
||||
Phase 2 自动化验证全部通过:
|
||||
|
||||
- ARCH-03:4 个 Boss 文件(client/api/main/sign)已迁移至 crawler_core,无旧依赖
|
||||
- QUAL-03:22 个 mock 测试覆盖全部 API 类的正常和错误路径
|
||||
- 反爬机制完整保留(TLS 伪装 + 代理轮换 + 10s 最小延迟 + Traceid 注入)
|
||||
- Phase 1 回归测试 41 个全部通过,无回归
|
||||
|
||||
剩余 1 项人工验证(真实抓取),需要有效 mpt/wt2 凭证。
|
||||
Loading…
x
Reference in New Issue
Block a user