356 lines
13 KiB
Markdown
356 lines
13 KiB
Markdown
---
|
||
phase: 2
|
||
plan: 2
|
||
wave: 1
|
||
title: "新增 Boss HTTP 层 mock 测试"
|
||
depends_on:
|
||
- "02-01-PLAN.md"
|
||
files_modified:
|
||
- tests/boss/__init__.py
|
||
- tests/boss/test_boss_client.py
|
||
autonomous: true
|
||
requirements:
|
||
- QUAL-03
|
||
---
|
||
|
||
# Phase 2 Plan 02: Boss HTTP 层 mock 测试
|
||
|
||
## Objective
|
||
|
||
使用 `unittest.mock.MagicMock` 为 Boss HTTP 层添加 mock 测试,
|
||
覆盖正常响应和错误响应两种场景,满足 QUAL-03。
|
||
|
||
测试文件位置:`tests/boss/test_boss_client.py`
|
||
|
||
**为什么不用 respx:** `crawler_core.HTTPClient` 底层使用 `requests_go`(非 httpx),
|
||
respx 只拦截 httpx,无法适用。使用 `MagicMock` 直接 mock `http_client` 接口,
|
||
不依赖网络,测试更快更稳定。
|
||
|
||
## Must Haves
|
||
|
||
- [ ] `tests/boss/__init__.py` 存在(空文件即可)
|
||
- [ ] `tests/boss/test_boss_client.py` 存在并包含 mock 测试
|
||
- [ ] `pytest tests/boss/ -v` 全部通过(无 ImportError,无失败)
|
||
- [ ] 测试组:SearchRecJobs(正常响应、业务错误、HTTP 非 200)
|
||
- [ ] 测试组:GetBrandDetail(正常响应)
|
||
- [ ] 测试组:SearchBrandJobs(正常响应、hasMore=True 时 is_end_page=False)
|
||
- [ ] 测试组:GetJobDetail.fetch()(通过 batch 接口,正常合并子请求)
|
||
- [ ] 测试组:BossClient(验证每次请求注入 Traceid 头、mpt/wt2 头)
|
||
|
||
---
|
||
|
||
## Wave 1
|
||
|
||
### Task 2.1: 创建 tests/boss/ 目录和 __init__.py
|
||
|
||
<read_first>
|
||
- `tests/crawler_core/test_boss_sign.py`(参考现有测试风格和 conftest 依赖)
|
||
- `conftest.py`(项目根目录,确认 sys.path 设置)
|
||
</read_first>
|
||
|
||
<action>
|
||
1. 创建 `tests/boss/__init__.py`,内容为空文件(或仅含 `# tests/boss/`)。
|
||
|
||
确认 `conftest.py` 已在项目根目录(Phase 1 已创建),无需重复创建。
|
||
</action>
|
||
|
||
<acceptance_criteria>
|
||
- `test -f tests/boss/__init__.py && echo "OK"` 输出 OK
|
||
- `pytest tests/boss/ --collect-only 2>&1 | head -5` 无 "ERROR collecting" 字样
|
||
</acceptance_criteria>
|
||
|
||
---
|
||
|
||
### Task 2.2: 编写 test_boss_client.py
|
||
|
||
<read_first>
|
||
- `spiderJobs/platforms/boss/api.py`(迁移后版本,Task 1.2 产物)
|
||
- `spiderJobs/platforms/boss/client.py`(迁移后版本)
|
||
- `tests/crawler_core/test_boss_sign.py`(参考测试风格)
|
||
- `crawler_core/base.py`(Result 结构)
|
||
</read_first>
|
||
|
||
<action>
|
||
创建 `tests/boss/test_boss_client.py`,内容如下(完整内容,直接写入文件):
|
||
|
||
```python
|
||
"""
|
||
Boss 直聘 HTTP 层 mock 测试(QUAL-03)
|
||
|
||
使用 unittest.mock.MagicMock 替代真实 HTTP 客户端,
|
||
覆盖正常响应和错误响应场景,无网络依赖。
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
from unittest.mock import MagicMock, patch, call
|
||
import pytest
|
||
|
||
from spiderJobs.platforms.boss.api import (
|
||
SearchRecJobs,
|
||
GetBrandDetail,
|
||
SearchBrandJobs,
|
||
GetJobDetail,
|
||
_parse_boss_response,
|
||
)
|
||
from spiderJobs.platforms.boss.client import BossClient
|
||
from crawler_core.base import Result
|
||
|
||
|
||
# ─────────────────────────────────────────────────────────
|
||
# 1. _parse_boss_response 纯函数测试
|
||
# ─────────────────────────────────────────────────────────
|
||
|
||
class TestParseBossResponse:
|
||
|
||
def test_http_error_returns_failure(self):
|
||
result = _parse_boss_response(500, {})
|
||
assert result.success is False
|
||
assert result.status_code == 500
|
||
|
||
def test_non_dict_raw_returns_failure(self):
|
||
result = _parse_boss_response(200, "not a dict")
|
||
assert result.success is False
|
||
|
||
def test_biz_error_code_35_returns_failure(self):
|
||
result = _parse_boss_response(200, {"code": 35, "message": "IP地址存在异常"})
|
||
assert result.success is False
|
||
assert result.status_code == 35
|
||
assert "IP" in result.error
|
||
|
||
def test_joblist_payload_parsed_correctly(self):
|
||
raw = {
|
||
"code": 0,
|
||
"zpData": {
|
||
"jobList": [{"title": "Python工程师"}],
|
||
"hasMore": True,
|
||
},
|
||
}
|
||
result = _parse_boss_response(200, raw)
|
||
assert result.success is True
|
||
assert len(result.list) == 1
|
||
assert result.list[0]["title"] == "Python工程师"
|
||
assert result.is_end_page is False # hasMore=True → is_end_page=False
|
||
|
||
def test_joblist_no_more_pages(self):
|
||
raw = {
|
||
"code": 0,
|
||
"zpData": {"jobList": [{"title": "测试"}], "hasMore": False},
|
||
}
|
||
result = _parse_boss_response(200, raw)
|
||
assert result.is_end_page is True
|
||
|
||
def test_detail_payload(self):
|
||
raw = {"code": 0, "zpData": {"companyName": "测试公司"}}
|
||
result = _parse_boss_response(200, raw)
|
||
assert result.success is True
|
||
assert result.data == {"companyName": "测试公司"}
|
||
|
||
|
||
# ─────────────────────────────────────────────────────────
|
||
# 2. SearchRecJobs
|
||
# ─────────────────────────────────────────────────────────
|
||
|
||
class TestSearchRecJobs:
|
||
|
||
def _make_mock_client(self, return_value):
|
||
mock_client = MagicMock()
|
||
mock_client.get.return_value = return_value
|
||
return mock_client
|
||
|
||
def test_search_success(self):
|
||
raw = {
|
||
"code": 0,
|
||
"zpData": {
|
||
"jobList": [{"title": "测试职位1"}, {"title": "测试职位2"}],
|
||
"hasMore": False,
|
||
},
|
||
}
|
||
searcher = SearchRecJobs(city_code="101010100", client=self._make_mock_client((200, raw)))
|
||
result = searcher.search(page_index=1)
|
||
|
||
assert result.success is True
|
||
assert len(result.list) == 2
|
||
assert result.is_end_page is True
|
||
|
||
def test_search_http_error(self):
|
||
searcher = SearchRecJobs(client=self._make_mock_client((403, {})))
|
||
result = searcher.search(page_index=1)
|
||
assert result.success is False
|
||
assert result.status_code == 403
|
||
|
||
def test_search_biz_error(self):
|
||
raw = {"code": 35, "message": "IP地址存在异常"}
|
||
searcher = SearchRecJobs(client=self._make_mock_client((200, raw)))
|
||
result = searcher.search(page_index=1)
|
||
assert result.success is False
|
||
|
||
def test_search_builds_correct_params(self):
|
||
mock_client = MagicMock()
|
||
mock_client.get.return_value = (200, {"code": 0, "zpData": {"jobList": [], "hasMore": False}})
|
||
searcher = SearchRecJobs(city_code="101280600", page_size=10, client=mock_client)
|
||
searcher.search(page_index=2)
|
||
|
||
called_args = mock_client.get.call_args
|
||
params = called_args[0][1] if len(called_args[0]) > 1 else called_args[1].get("params", called_args[0][1])
|
||
assert params["cityCode"] == "101280600"
|
||
assert params["page"] == 2
|
||
assert params["pageSize"] == 10
|
||
|
||
|
||
# ─────────────────────────────────────────────────────────
|
||
# 3. GetBrandDetail
|
||
# ─────────────────────────────────────────────────────────
|
||
|
||
class TestGetBrandDetail:
|
||
|
||
def test_fetch_success(self):
|
||
mock_client = MagicMock()
|
||
mock_client.get.return_value = (200, {
|
||
"code": 0,
|
||
"zpData": {"brandName": "测试公司", "brandId": "abc123"},
|
||
})
|
||
fetcher = GetBrandDetail(brand_id="abc123", client=mock_client)
|
||
result = fetcher.fetch()
|
||
|
||
assert result.success is True
|
||
assert result.data["brandName"] == "测试公司"
|
||
|
||
def test_fetch_404(self):
|
||
mock_client = MagicMock()
|
||
mock_client.get.return_value = (404, {})
|
||
fetcher = GetBrandDetail(brand_id="notexist", client=mock_client)
|
||
result = fetcher.fetch()
|
||
|
||
assert result.success is False
|
||
assert result.status_code == 404
|
||
|
||
|
||
# ─────────────────────────────────────────────────────────
|
||
# 4. SearchBrandJobs
|
||
# ─────────────────────────────────────────────────────────
|
||
|
||
class TestSearchBrandJobs:
|
||
|
||
def test_search_success_has_more(self):
|
||
mock_client = MagicMock()
|
||
mock_client.get.return_value = (200, {
|
||
"code": 0,
|
||
"zpData": {"list": [{"jobName": "测试岗位"}], "hasMore": True},
|
||
})
|
||
searcher = SearchBrandJobs(brand_id="abc123", client=mock_client)
|
||
result = searcher.search(page_index=1)
|
||
|
||
assert result.success is True
|
||
assert len(result.list) == 1
|
||
assert result.is_end_page is False
|
||
|
||
def test_search_success_no_more(self):
|
||
mock_client = MagicMock()
|
||
mock_client.get.return_value = (200, {
|
||
"code": 0,
|
||
"zpData": {"list": [], "hasMore": False},
|
||
})
|
||
searcher = SearchBrandJobs(brand_id="abc123", client=mock_client)
|
||
result = searcher.search(page_index=1)
|
||
|
||
assert result.is_end_page is True
|
||
|
||
|
||
# ─────────────────────────────────────────────────────────
|
||
# 5. GetJobDetail(batch 接口路径)
|
||
# ─────────────────────────────────────────────────────────
|
||
|
||
class TestGetJobDetail:
|
||
|
||
def test_fetch_success_merges_sub_requests(self):
|
||
mock_client = MagicMock()
|
||
mock_client.batch.return_value = (200, {
|
||
"code": 0,
|
||
"zpData": {
|
||
"/wapi/zpgeek/miniapp/job/detail.json": {
|
||
"zpData": {"jobName": "数据工程师"}
|
||
},
|
||
"/wapi/zpgeek/miniapp/jobdetail/improvement/query.json": {
|
||
"zpData": {"tags": ["Python", "大数据"]}
|
||
},
|
||
},
|
||
})
|
||
fetcher = GetJobDetail(security_id="sid123", job_id="jid456", client=mock_client)
|
||
result = fetcher.fetch()
|
||
|
||
assert result.success is True
|
||
assert result.data["detail"]["jobName"] == "数据工程师"
|
||
assert "Python" in result.data["improvement"]["tags"]
|
||
|
||
def test_fetch_biz_error(self):
|
||
mock_client = MagicMock()
|
||
mock_client.batch.return_value = (200, {"code": 35, "message": "IP地址存在异常"})
|
||
fetcher = GetJobDetail(security_id="sid", job_id="jid", client=mock_client)
|
||
result = fetcher.fetch()
|
||
assert result.success is False
|
||
|
||
def test_fetch_exception_handled(self):
|
||
mock_client = MagicMock()
|
||
mock_client.batch.side_effect = ConnectionError("连接超时")
|
||
fetcher = GetJobDetail(security_id="sid", job_id="jid", client=mock_client)
|
||
result = fetcher.fetch()
|
||
assert result.success is False
|
||
assert "连接超时" in result.error
|
||
|
||
|
||
# ─────────────────────────────────────────────────────────
|
||
# 6. BossClient — Traceid/mpt/wt2 请求头注入
|
||
# ─────────────────────────────────────────────────────────
|
||
|
||
class TestBossClientHeaders:
|
||
|
||
def test_get_injects_traceid(self):
|
||
"""每次 GET 请求头包含 Traceid"""
|
||
client = BossClient(tunnel_proxy=None)
|
||
headers = client._boss_headers()
|
||
assert "Traceid" in headers
|
||
assert headers["Traceid"].startswith("M-W")
|
||
|
||
def test_traceid_is_unique_per_call(self):
|
||
"""每次调用生成不同的 Traceid"""
|
||
client = BossClient()
|
||
t1 = client._boss_headers()["Traceid"]
|
||
t2 = client._boss_headers()["Traceid"]
|
||
# 大概率不同(极低碰撞概率可忽略)
|
||
assert len(t1) > 10
|
||
assert len(t2) > 10
|
||
|
||
def test_mpt_wt2_in_headers(self):
|
||
"""signer 的 mpt/wt2 注入到请求头"""
|
||
from crawler_core.boss.sign import BossSign
|
||
signer = BossSign(mpt="test_mpt_value", wt2="test_wt2_value")
|
||
client = BossClient(signer=signer)
|
||
headers = client._boss_headers()
|
||
assert headers["mpt"] == "test_mpt_value"
|
||
assert headers["wt2"] == "test_wt2_value"
|
||
```
|
||
</action>
|
||
|
||
<acceptance_criteria>
|
||
- `test -f tests/boss/test_boss_client.py && echo "OK"` 输出 OK
|
||
- `pytest tests/boss/ -v 2>&1 | tail -10` 输出包含 "passed",无 "failed" 或 "error"
|
||
- `pytest tests/boss/ -v 2>&1 | grep "PASSED" | wc -l` 输出 >= 15(至少 15 个测试通过)
|
||
- `pytest tests/boss/ -v 2>&1 | grep "import"` 无输出(无 ImportError)
|
||
</acceptance_criteria>
|
||
|
||
---
|
||
|
||
## Verification
|
||
|
||
```bash
|
||
# 运行所有 Boss mock 测试
|
||
pytest tests/boss/ -v
|
||
|
||
# 同时运行 Phase 1 签名测试,确认没有回归
|
||
pytest tests/crawler_core/ -v
|
||
|
||
# 汇总结果
|
||
pytest tests/ -v --tb=short
|
||
```
|