111 lines
3.0 KiB
Markdown
111 lines
3.0 KiB
Markdown
---
|
||
phase: 5
|
||
plan: 1
|
||
wave: 1
|
||
title: "30 天窗口去重修复（DATA-01）"
|
||
depends_on: []
|
||
files_modified:
|
||
- app/services/ingest/dedup.py
|
||
- tests/ingest/test_dedup.py # NEW
|
||
autonomous: true
|
||
requirements:
|
||
- DATA-01
|
||
---
|
||
|
||
# Phase 5 Plan 01: 30 天窗口去重修复（DATA-01）
|
||
|
||
## Objective
|
||
|
||
`dedup.py` 的 `batch_dedup_filter()` 目前查全量历史数据去重，导致同一职位超过 30 天后无法重新入库。
|
||
本 Plan 在去重查询 SQL 中加入 30 天时间窗口。
|
||
|
||
## Must Haves
|
||
|
||
- [ ] 单字段 dedup SQL 加 `AND created_at > now() - INTERVAL 30 DAY`
|
||
- [ ] 双字段 dedup SQL 同样加 30 天窗口
|
||
- [ ] 新增 Mock 测试（用 `AsyncMock` 模拟 ClickHouse 客户端）覆盖：
|
||
- 30 天内有记录 → 视为重复，过滤
|
||
- 30 天外有记录 → 不重复，允许入库
|
||
- 无记录 → 允许入库
|
||
- [ ] `pipenv run python -m pytest tests/ingest/test_dedup.py -v` 全部通过
|
||
- [ ] 全量回归 `pytest tests/ -v` 无失败
|
||
|
||
---
|
||
|
||
## Wave 1
|
||
|
||
### Task 1.1: 修改 dedup.py
|
||
|
||
<read_first>
|
||
- `app/services/ingest/dedup.py`（当前 81 行）
|
||
</read_first>
|
||
|
||
<action>
|
||
修改 `batch_dedup_filter()` 中的两个 SQL 查询：
|
||
|
||
**单字段去重（第 51 行附近）：**
|
||
```python
|
||
# 修改前
|
||
query = f"SELECT {key_col} FROM {table} WHERE {key_col} IN {{keys:Array(String)}}"
|
||
|
||
# 修改后
|
||
query = (
|
||
f"SELECT {key_col} FROM {table} "
|
||
f"WHERE {key_col} IN {{keys:Array(String)}} "
|
||
f"AND created_at > now() - INTERVAL 30 DAY"
|
||
)
|
||
```
|
||
|
||
**双字段去重（第 65 行附近）：**
|
||
```python
|
||
# 修改前
|
||
query = f"SELECT {c1}, {c2} FROM {table} WHERE {c1} IN {{keys:Array(String)}}"
|
||
|
||
# 修改后
|
||
query = (
|
||
f"SELECT {c1}, {c2} FROM {table} "
|
||
f"WHERE {c1} IN {{keys:Array(String)}} "
|
||
f"AND created_at > now() - INTERVAL 30 DAY"
|
||
)
|
||
```
|
||
</action>
|
||
|
||
<acceptance_criteria>
|
||
- `grep "INTERVAL 30 DAY" app/services/ingest/dedup.py` 有两处输出
|
||
- `grep "INTERVAL 30 DAY" app/services/ingest/dedup.py | wc -l` 输出为 2
|
||
</acceptance_criteria>
|
||
|
||
---
|
||
|
||
### Task 1.2: 新增 Mock 测试
|
||
|
||
<action>
|
||
创建 `tests/ingest/__init__.py`（空文件）和 `tests/ingest/test_dedup.py`：
|
||
|
||
测试覆盖：
|
||
1. `test_single_field_dedup_within_30_days`：30 天内有相同 job_id → 过滤（duplicate=1）
|
||
2. `test_single_field_dedup_outside_30_days`：先不存在 → 允许入库（duplicate=0）（模拟 30 天外无记录）
|
||
3. `test_double_field_dedup_within_30_days`：双字段在 30 天内有记录 → 过滤
|
||
4. `test_dedup_empty_input`：空输入 → 直接返回，不查 ClickHouse
|
||
5. `test_dedup_no_dedup_columns`：无 dedup 字段 → 跳过过滤
|
||
6. `test_build_insert_row_has_channel`：验证 build_insert_row 生成的行包含 channel 列
|
||
|
||
SQL 验证：用 `mock_client.query.call_args[0][0]` 断言 SQL 中包含 `INTERVAL 30 DAY`
|
||
</action>
|
||
|
||
---
|
||
|
||
## Verification
|
||
|
||
```bash
|
||
# 1. 验证 SQL 变更
|
||
grep "INTERVAL 30 DAY" app/services/ingest/dedup.py
|
||
# 预期输出：2 行
|
||
|
||
# 2. 运行新增测试
|
||
pipenv run python -m pytest tests/ingest/test_dedup.py -v
|
||
|
||
# 3. 全量回归
|
||
pipenv run python -m pytest tests/ -v --tb=short
|
||
```
|