9756df6044
Summary files and roadmap/STATE updated. Phase 5 complete: DATA-01 (30-day dedup window) + DATA-04 (company channel) DATA-02 + DATA-03 confirmed already complete.
48 lines
1.4 KiB
Markdown
48 lines
1.4 KiB
Markdown
# Phase 5 Summary: 数据管道优化
|
||
|
||
**Status:** Complete ✅
|
||
**Tests:** 112 passed (6 new)
|
||
**Commit:** 3d202c3
|
||
|
||
## Plan 01 — DATA-01: dedup 30天窗口（`dedup.py`）
|
||
|
||
**修复前：** 查全量历史数据去重，同一职位 ID 永久只入库一次
|
||
|
||
**修复后：**
|
||
```sql
|
||
-- 单字段（boss job_id 等）
|
||
SELECT job_id FROM boss_job
|
||
WHERE job_id IN {keys} AND created_at > now() - INTERVAL 30 DAY
|
||
|
||
-- 双字段（qcwy job_id + update_date_time）
|
||
SELECT job_id, update_date_time FROM qcwy_job
|
||
WHERE job_id IN {keys} AND created_at > now() - INTERVAL 30 DAY
|
||
```
|
||
|
||
**新增测试（`tests/ingest/test_dedup.py`，6 个）：**
|
||
- 30 天内有记录 → 过滤（1 个验证 SQL 窗口）
|
||
- 无记录 → 允许入库
|
||
- 空输入 → 不查 ClickHouse
|
||
- build_insert_row 含 channel 字段
|
||
|
||
---
|
||
|
||
## Plan 02 — DATA-04: company_job channel 区分
|
||
|
||
**目的：** 区分`搜索职位（channel=mini）`和`公司关联职位（channel=company）`
|
||
|
||
| 文件 | 变更 |
|
||
|------|------|
|
||
| `schemas/ingest.py` | `ChannelType.COMPANY = "company"` |
|
||
| `configs/boss.py` | 新注册 `channel=company, data_type=job` |
|
||
| `configs/qcwy.py` | 同上 |
|
||
| `configs/zhilian.py` | 同上 |
|
||
| `company_jobs_sync.py` | `"mini"` → `"company"` |
|
||
|
||
---
|
||
|
||
## DATA-02 + DATA-03 确认（无需改动）
|
||
|
||
- **DATA-02：** `job.py` 已有 `/data/batch-async` 端点，spiderJobs 直接调用即可 ✅
|
||
- **DATA-03：** `company_cleaner.py` 已完整实现 ClickHouse→爬详情→MySQL 全链路 ✅
|