STATE: JobData 爬虫交互重构

Last updated: 2026-03-21 Milestone: 爬虫交互重构 v1

Project Reference

Core value: 基于关键词驱动爬虫抓取职位数据，可靠入库 ClickHouse，定时完成公司信息采集同步

Current focus: Phase 1 — 提取 crawler_core/ 共享核心包

Current Position

Field	Value
Phase	1
Phase name	共享核心包
Plan	None (not started)
Status	Not started

Progress:

Phase 1 ░░░░░░░░░░░░░░░░░░░░  0%
Phase 2 ░░░░░░░░░░░░░░░░░░░░  0%
Phase 3 ░░░░░░░░░░░░░░░░░░░░  0%
Phase 4 ░░░░░░░░░░░░░░░░░░░░  0%
Phase 5 ░░░░░░░░░░░░░░░░░░░░  0%
Phase 6 ░░░░░░░░░░░░░░░░░░░░  0%
─────────────────────────────
Overall  0/6 phases complete

Performance Metrics

Metric	Value
Plans completed	0
Plans total (estimated)	TBD
Phases completed	0/6
Requirements mapped	19/19

Accumulated Context

Key Decisions

Decision	Rationale	Status
先提取 crawler_core/ 再改平台代码	避免签名代码再次被复制粘贴	Active
feature flag 并行新旧代码	重构期间不中断在线爬虫	Active
jobs_spider/ 最终废弃	迁移到 spiderJobs/ + crawler_core	Active
同步核心 + asyncio.to_thread() 桥接	requests_go 不支持 async，保持 TLS 指纹	Active
30 天时间窗口去重	避免全表扫描 ClickHouse	Active

Architecture Notes

crawler_core/ 放项目根目录，pip install -e ./crawler_core 本地可安装
核心保持同步（requests_go），后端通过 asyncio.to_thread() 调用
SmartIPManager 在 FastAPI 中需做成单例，避免会话状态丢失
ClickHouse 表结构不变，MySQL 模型变更走 Aerich

Known Risks

Risk	Mitigation
重构期间打断在线爬虫	feature flag 并行新旧代码，旧路径保留到 Phase 4 完成
反爬会话状态丢失	SmartIPManager 做单例，Phase 4 重点验证
无测试基线	Phase 1 先写签名纯函数测试，Phase 2 同步补 HTTP mock 测试

Open TODOs

确认 crawler_core/ 包的 Python 包名（建议 crawler_core，避免与系统包冲突）
确认 feature flag 实现方式（环境变量 vs 配置文件）
Phase 3 可与 Phase 2 并行（两平台互相独立），视进度决定

Blockers

None currently.

Session Continuity

To resume this project:

Check current phase in this file
Read .planning/ROADMAP.md for phase goals and success criteria
Read .planning/REQUIREMENTS.md for requirement coverage
Run git status to see uncommitted work
Start with /gsd:plan-phase {current_phase} if no active plan

Key files:

.planning/ROADMAP.md — Phase structure and success criteria
.planning/REQUIREMENTS.md — All v1 requirements with traceability
.planning/PROJECT.md — Project context and constraints
crawler_core/ — (to be created) shared package
spiderJobs/ — external scripts using crawler_core
app/services/crawler/ — backend facade layer

State initialized: 2026-03-21

3.4 KiB Raw Blame History Unescape Escape