621 lines
24 KiB
Markdown
621 lines
24 KiB
Markdown
---
|
|
phase: 01-shared-core
|
|
plan: 02
|
|
type: execute
|
|
wave: 2
|
|
depends_on:
|
|
- 01-01
|
|
files_modified:
|
|
- crawler_core/boss/sign.py
|
|
- crawler_core/qcwy/sign.py
|
|
- crawler_core/zhilian/sign.py
|
|
- tests/crawler_core/__init__.py
|
|
- tests/crawler_core/test_boss_sign.py
|
|
- tests/crawler_core/test_qcwy_sign.py
|
|
- tests/crawler_core/test_zhilian_sign.py
|
|
autonomous: true
|
|
requirements:
|
|
- QUAL-01
|
|
|
|
must_haves:
|
|
truths:
|
|
- "Three sign algorithm files exist under crawler_core/{boss,qcwy,zhilian}/sign.py"
|
|
- "`pytest tests/crawler_core/` exits 0 — all sign algorithm tests pass"
|
|
- "Sign tests require NO network access, NO tokens, NO mocks — pure function assertions"
|
|
- "BossSign.generate_traceid() returns a string matching the pattern M-W[0-9a-zA-Z]{22}"
|
|
- "Job51Sign.build_sign_path() returns a tuple of (str, str) — (url_path, sign_hex)"
|
|
- "ZhilianSign.sign_headers() returns a dict with key 'x-zp-device-id'"
|
|
- "Old spiderJobs/platforms/*/sign.py files are NOT deleted or modified"
|
|
artifacts:
|
|
- path: "crawler_core/boss/sign.py"
|
|
provides: "Boss traceid generation — pure functions, no I/O"
|
|
exports: ["BossSign"]
|
|
- path: "crawler_core/qcwy/sign.py"
|
|
provides: "51Job HMAC-SHA256 signing — pure functions, no I/O"
|
|
exports: ["Job51Sign", "SIGN_KEY"]
|
|
- path: "crawler_core/zhilian/sign.py"
|
|
provides: "Zhilian header/param signing — pure functions, no I/O"
|
|
exports: ["ZhilianSign"]
|
|
- path: "tests/crawler_core/test_boss_sign.py"
|
|
provides: "BossSign unit tests — 6+ test cases"
|
|
- path: "tests/crawler_core/test_qcwy_sign.py"
|
|
provides: "Job51Sign unit tests — 5+ test cases"
|
|
- path: "tests/crawler_core/test_zhilian_sign.py"
|
|
provides: "ZhilianSign unit tests — 5+ test cases"
|
|
key_links:
|
|
- from: "tests/crawler_core/test_boss_sign.py"
|
|
to: "crawler_core/boss/sign.py"
|
|
via: "from crawler_core.boss.sign import BossSign, _compute_checksum, _generate_uuid"
|
|
- from: "crawler_core/boss/sign.py"
|
|
to: "stdlib only"
|
|
via: "import random, time — no external deps"
|
|
---
|
|
|
|
<objective>
|
|
Migrate all three platform sign algorithm files into crawler_core/ and write comprehensive unit tests for each. Sign algorithms are pure functions — they are the highest-value, lowest-cost tests in the codebase.
|
|
|
|
Purpose: QUAL-01 requires unit test coverage for core signing algorithms. Tests validate that the ported algorithms match the originals. Pure functions mean no HTTP mocking needed.
|
|
|
|
Output: Three sign.py files in crawler_core/{boss,qcwy,zhilian}/ and three test files in tests/crawler_core/ — all tests pass with `pytest tests/crawler_core/`.
|
|
</objective>
|
|
|
|
<execution_context>
|
|
@~/.claude/get-shit-done/workflows/execute-plan.md
|
|
@~/.claude/get-shit-done/templates/summary.md
|
|
</execution_context>
|
|
|
|
<context>
|
|
@.planning/phases/01-shared-core/01-01-SUMMARY.md
|
|
@.planning/phases/01-shared-core/1-CONTEXT.md
|
|
|
|
<interfaces>
|
|
<!-- Sign algorithms to port. Read source files BEFORE writing crawler_core versions. -->
|
|
|
|
From spiderJobs/platforms/boss/sign.py:
|
|
```python
|
|
# Module-level constants:
|
|
_CHARS = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
|
|
|
|
# Module-level private functions:
|
|
def _to_u32(n: int) -> int: ... # Masks to 32-bit unsigned
|
|
def _compute_checksum(uuid_str: str) -> str: ... # 3-char checksum from 19-char uuid
|
|
def _generate_uuid() -> str: ... # 13-char hex timestamp + 6-char base62 random
|
|
|
|
class BossSign:
|
|
def __init__(self, *, mpt: str = "", wt2: str = ""): ...
|
|
@staticmethod
|
|
def generate_traceid(prefix: str = "M-W") -> str: ...
|
|
# Result format: "{prefix}{19-char uuid}{3-char checksum}" e.g. "M-W0019d0a8af5f32gtVvnD4M"
|
|
```
|
|
|
|
From spiderJobs/platforms/job51/sign.py:
|
|
```python
|
|
SIGN_KEY = "abfc8f9dcf8c3f3d8aa294ac5f2cf2cc7767e5592590f39c3f503271dd68562b"
|
|
|
|
class Job51Sign:
|
|
def __init__(self, *, sign_key: str = SIGN_KEY): ...
|
|
@staticmethod
|
|
def generate_uuid() -> str: ... # 13-char timestamp + 10-char random int
|
|
def build_sign_path(
|
|
self,
|
|
endpoint: str,
|
|
method: str = "GET",
|
|
params: dict | None = None,
|
|
body: dict | None = None,
|
|
) -> tuple[str, str]: ... # Returns (url_path, sign_hex)
|
|
# url_path format: "/{endpoint}?api_key=51job×tamp={ts}[¶m=val...]"
|
|
# sign_hex: HMAC-SHA256 hex string, 64 chars
|
|
```
|
|
|
|
From spiderJobs/platforms/zhilian/sign.py:
|
|
```python
|
|
class ZhilianSign:
|
|
def __init__(self, *, at="", rt="", device_id=None, version="4.1.259",
|
|
channel="wxxiaochengxu", platform="12"): ...
|
|
@staticmethod
|
|
def generate_uuid() -> str: ... # UUID4-format with uppercase hex
|
|
def sign_headers(self, page_code: str = "0") -> dict: ...
|
|
# Returns dict with keys: x-zp-at, x-zp-rt, x-zp-action-id, x-zp-page-code,
|
|
# x-zp-version, x-zp-channel, x-zp-platform, x-zp-device-id, x-zp-business-system
|
|
def sign_params(self) -> dict: ...
|
|
# Returns dict with keys: at, rt, channel, platform, version, d
|
|
```
|
|
</interfaces>
|
|
</context>
|
|
|
|
<tasks>
|
|
|
|
<task type="auto">
|
|
<name>Task 1: Port sign algorithms to crawler_core/ platform directories</name>
|
|
<read_first>
|
|
- /Users/win/2025/AICoding/JobData/spiderJobs/platforms/boss/sign.py (source — read every line before writing)
|
|
- /Users/win/2025/AICoding/JobData/spiderJobs/platforms/job51/sign.py (source — read every line before writing)
|
|
- /Users/win/2025/AICoding/JobData/spiderJobs/platforms/zhilian/sign.py (source — read every line before writing)
|
|
- /Users/win/2025/AICoding/JobData/crawler_core/boss/__init__.py (confirm it exists from Plan 01)
|
|
</read_first>
|
|
<files>
|
|
crawler_core/boss/sign.py
|
|
crawler_core/qcwy/sign.py
|
|
crawler_core/zhilian/sign.py
|
|
</files>
|
|
<action>
|
|
Copy the three sign algorithm files to their new locations under crawler_core/, making only one change per file: update the module docstring to reference crawler_core.
|
|
|
|
**crawler_core/boss/sign.py:**
|
|
Copy `spiderJobs/platforms/boss/sign.py` verbatim. Update the module docstring first line to:
|
|
```
|
|
Boss直聘 Traceid 生成算法 (crawler_core)
|
|
```
|
|
No other changes. ALL private functions (_to_u32, _compute_checksum, _generate_uuid) and the BossSign class stay exactly as-is.
|
|
|
|
**crawler_core/qcwy/sign.py:**
|
|
Copy `spiderJobs/platforms/job51/sign.py` verbatim. Update the module docstring first line to:
|
|
```
|
|
前程无忧 (51Job) 签名算法 (crawler_core)
|
|
```
|
|
No other changes. SIGN_KEY constant and Job51Sign class stay exactly as-is. The `import json` inside `build_sign_path` stays inside the method (do not hoist it).
|
|
|
|
**crawler_core/zhilian/sign.py:**
|
|
Copy `spiderJobs/platforms/zhilian/sign.py` verbatim. Update the module docstring first line to:
|
|
```
|
|
智联招聘签名算法 (crawler_core)
|
|
```
|
|
No other changes. ZhilianSign class stays exactly as-is.
|
|
|
|
**Verification after writing each file:**
|
|
- No file imports from `spiderJobs.*`
|
|
- No file imports from `app.*`
|
|
- No file uses `requests` or any HTTP library
|
|
- Each file is entirely self-contained with only stdlib imports (random, time, math, hmac, hashlib, urllib.parse)
|
|
|
|
**DO NOT delete or modify:**
|
|
- `spiderJobs/platforms/boss/sign.py`
|
|
- `spiderJobs/platforms/job51/sign.py`
|
|
- `spiderJobs/platforms/zhilian/sign.py`
|
|
|
|
These old files remain in place until Phase 4 cleanup.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /Users/win/2025/AICoding/JobData && python -c "
|
|
import sys
|
|
sys.path.insert(0, '.')
|
|
import re
|
|
|
|
from crawler_core.boss.sign import BossSign, _compute_checksum, _generate_uuid
|
|
from crawler_core.qcwy.sign import Job51Sign, SIGN_KEY
|
|
from crawler_core.zhilian.sign import ZhilianSign
|
|
|
|
# Boss
|
|
tid = BossSign.generate_traceid()
|
|
assert re.match(r'^M-W[0-9a-f]{13}[0-9a-zA-Z]{6}[0-9a-zA-Z]{3}$', tid), f'Traceid format wrong: {tid}'
|
|
print(f'BossSign OK: {tid}')
|
|
|
|
# Job51
|
|
signer = Job51Sign()
|
|
path, sign = signer.build_sign_path('open/test', 'GET', params={'key': 'val'})
|
|
assert path.startswith('/open/test?api_key=51job×tamp='), f'Path format wrong: {path}'
|
|
assert len(sign) == 64, f'Sign length wrong: {len(sign)}'
|
|
print(f'Job51Sign OK: sign={sign[:8]}...')
|
|
|
|
# Zhilian
|
|
zs = ZhilianSign(at='token123', rt='refresh456')
|
|
headers = zs.sign_headers()
|
|
assert 'x-zp-device-id' in headers, 'device-id missing'
|
|
assert headers['x-zp-business-system'] == '73', 'business-system wrong'
|
|
assert len(headers) == 9, f'Expected 9 header keys, got {len(headers)}'
|
|
params = zs.sign_params()
|
|
assert params['at'] == 'token123', f'at field wrong: {params[\"at\"]}'
|
|
assert len(params) == 6, f'Expected 6 param keys, got {len(params)}'
|
|
print('ZhilianSign OK')
|
|
print('All sign algorithms imported and validated')
|
|
"</automated>
|
|
</verify>
|
|
<acceptance_criteria>
|
|
- `crawler_core/boss/sign.py` exists: `ls crawler_core/boss/sign.py` exits 0
|
|
- `crawler_core/qcwy/sign.py` exists: `ls crawler_core/qcwy/sign.py` exits 0
|
|
- `crawler_core/zhilian/sign.py` exists: `ls crawler_core/zhilian/sign.py` exits 0
|
|
- `grep -r "from spiderJobs" crawler_core/` returns empty (no cross-imports)
|
|
- `grep -r "import requests" crawler_core/boss/sign.py crawler_core/qcwy/sign.py crawler_core/zhilian/sign.py` returns empty (no HTTP imports in sign files)
|
|
- `BossSign.generate_traceid()` returns a string of length 25 (3-char prefix + 19-char uuid + 3-char checksum)
|
|
- `Job51Sign().build_sign_path("test", "GET")` returns tuple where `[1]` is 64-char string
|
|
- `ZhilianSign().sign_headers()` returns dict with `x-zp-business-system` == `"73"`
|
|
- Old files UNCHANGED: `diff spiderJobs/platforms/boss/sign.py crawler_core/boss/sign.py` shows only docstring difference
|
|
</acceptance_criteria>
|
|
<done>Three sign.py files in crawler_core/ — pure functions, no HTTP, no cross-imports from app or spiderJobs.</done>
|
|
</task>
|
|
|
|
<task type="auto">
|
|
<name>Task 2: Write sign algorithm unit tests</name>
|
|
<read_first>
|
|
- /Users/win/2025/AICoding/JobData/crawler_core/boss/sign.py (just created — read to understand exact exports)
|
|
- /Users/win/2025/AICoding/JobData/crawler_core/qcwy/sign.py (just created)
|
|
- /Users/win/2025/AICoding/JobData/crawler_core/zhilian/sign.py (just created)
|
|
- /Users/win/2025/AICoding/JobData/.planning/research/STACK.md (testing section: "Sign algorithm tests — Pure functions, no HTTP, no mocks needed")
|
|
- /Users/win/2025/AICoding/JobData/tests/ (check what already exists to avoid conflicts)
|
|
</read_first>
|
|
<files>
|
|
tests/crawler_core/__init__.py
|
|
tests/crawler_core/test_boss_sign.py
|
|
tests/crawler_core/test_qcwy_sign.py
|
|
tests/crawler_core/test_zhilian_sign.py
|
|
</files>
|
|
<action>
|
|
Create `tests/crawler_core/__init__.py` (empty file) and three test files.
|
|
|
|
**tests/crawler_core/__init__.py:** Empty file (just creates the package).
|
|
|
|
**Important: Add sys.path setup to each test file** since crawler_core is not yet pip-installed:
|
|
|
|
```python
|
|
import sys
|
|
import os
|
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
|
|
```
|
|
|
|
Place this at the TOP of each test file before any crawler_core imports.
|
|
|
|
---
|
|
|
|
**tests/crawler_core/test_boss_sign.py:**
|
|
|
|
```python
|
|
"""Unit tests for crawler_core.boss.sign — BossSign and helper functions.
|
|
|
|
All tests are pure function assertions: no HTTP, no network, no mocks.
|
|
"""
|
|
import sys
|
|
import os
|
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
|
|
|
|
import re
|
|
import pytest
|
|
from crawler_core.boss.sign import BossSign, _compute_checksum, _generate_uuid, _CHARS
|
|
|
|
|
|
class TestBossSignGenerateTraceid:
|
|
def test_traceid_format(self):
|
|
tid = BossSign.generate_traceid()
|
|
assert re.match(r'^M-W[0-9a-f]{13}[0-9a-zA-Z]{6}[0-9a-zA-Z]{3}$', tid), \
|
|
f"Traceid format wrong: {tid}"
|
|
|
|
def test_traceid_length(self):
|
|
tid = BossSign.generate_traceid()
|
|
assert len(tid) == 25, f"Expected 25 chars, got {len(tid)}: {tid}"
|
|
|
|
def test_traceid_custom_prefix(self):
|
|
tid = BossSign.generate_traceid(prefix="X-Y")
|
|
assert tid.startswith("X-Y"), f"Expected X-Y prefix, got: {tid}"
|
|
|
|
def test_traceid_uniqueness(self):
|
|
t1 = BossSign.generate_traceid()
|
|
t2 = BossSign.generate_traceid()
|
|
assert t1 != t2, "Two calls should return different traceids"
|
|
|
|
def test_bosssign_init_defaults(self):
|
|
sign = BossSign()
|
|
assert sign.mpt == ""
|
|
assert sign.wt2 == ""
|
|
|
|
def test_bosssign_init_with_tokens(self):
|
|
sign = BossSign(mpt="mpt_token", wt2="wt2_token")
|
|
assert sign.mpt == "mpt_token"
|
|
assert sign.wt2 == "wt2_token"
|
|
|
|
|
|
class TestComputeChecksum:
|
|
def test_checksum_length(self):
|
|
checksum = _compute_checksum("1234567890abc456789") # 19 chars
|
|
assert len(checksum) == 3, f"Expected 3 chars, got {len(checksum)}"
|
|
|
|
def test_checksum_chars_in_base62(self):
|
|
checksum = _compute_checksum("1234567890abc456789")
|
|
for ch in checksum:
|
|
assert ch in _CHARS, f"Char {ch!r} not in base62 set"
|
|
|
|
def test_checksum_deterministic(self):
|
|
uuid_str = "1234567890abc456789"
|
|
c1 = _compute_checksum(uuid_str)
|
|
c2 = _compute_checksum(uuid_str)
|
|
assert c1 == c2, "Same input must produce same checksum"
|
|
|
|
def test_checksum_differs_for_different_input(self):
|
|
# Different inputs should (almost always) produce different checksums
|
|
c1 = _compute_checksum("1234567890abc456789")
|
|
c2 = _compute_checksum("9876543210xyz456789")
|
|
# Not guaranteed to differ but extremely likely
|
|
# We test at least that they are valid 3-char strings
|
|
assert len(c1) == 3 and len(c2) == 3
|
|
|
|
|
|
class TestGenerateUuid:
|
|
def test_generate_uuid_length(self):
|
|
uuid = _generate_uuid()
|
|
assert len(uuid) == 19, f"Expected 19 chars, got {len(uuid)}: {uuid}"
|
|
|
|
def test_generate_uuid_hex_prefix(self):
|
|
uuid = _generate_uuid()
|
|
hex_part = uuid[:13]
|
|
assert re.match(r'^[0-9a-f]{13}$', hex_part), \
|
|
f"First 13 chars should be hex: {hex_part}"
|
|
|
|
def test_generate_uuid_base62_suffix(self):
|
|
uuid = _generate_uuid()
|
|
rand_part = uuid[13:]
|
|
for ch in rand_part:
|
|
assert ch in _CHARS, f"Char {ch!r} in random suffix not in base62"
|
|
```
|
|
|
|
---
|
|
|
|
**tests/crawler_core/test_qcwy_sign.py:**
|
|
|
|
```python
|
|
"""Unit tests for crawler_core.qcwy.sign — Job51Sign.
|
|
|
|
All tests are pure function assertions: no HTTP, no network, no mocks.
|
|
"""
|
|
import sys
|
|
import os
|
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
|
|
|
|
import re
|
|
import pytest
|
|
from crawler_core.qcwy.sign import Job51Sign, SIGN_KEY
|
|
|
|
|
|
class TestJob51SignInit:
|
|
def test_default_sign_key(self):
|
|
signer = Job51Sign()
|
|
assert signer.sign_key == SIGN_KEY
|
|
assert len(SIGN_KEY) == 64 # 64-char hex key
|
|
|
|
def test_custom_sign_key(self):
|
|
custom_key = "a" * 64
|
|
signer = Job51Sign(sign_key=custom_key)
|
|
assert signer.sign_key == custom_key
|
|
|
|
|
|
class TestJob51SignBuildSignPath:
|
|
def setup_method(self):
|
|
self.signer = Job51Sign()
|
|
|
|
def test_returns_tuple_of_two_strings(self):
|
|
result = self.signer.build_sign_path("open/test")
|
|
assert isinstance(result, tuple)
|
|
assert len(result) == 2
|
|
assert all(isinstance(s, str) for s in result)
|
|
|
|
def test_get_path_format(self):
|
|
path, sign = self.signer.build_sign_path("open/test", "GET")
|
|
assert path.startswith("/open/test?api_key=51job×tamp="), \
|
|
f"Path format wrong: {path}"
|
|
|
|
def test_sign_hex_length(self):
|
|
_, sign = self.signer.build_sign_path("open/test")
|
|
assert len(sign) == 64, f"Sign should be 64-char hex, got {len(sign)}: {sign}"
|
|
|
|
def test_sign_hex_format(self):
|
|
_, sign = self.signer.build_sign_path("open/test")
|
|
assert re.match(r'^[0-9a-f]{64}$', sign), f"Sign not hex: {sign}"
|
|
|
|
def test_get_vs_post_different_sign(self):
|
|
_, get_sign = self.signer.build_sign_path("open/test", "GET")
|
|
_, post_sign = self.signer.build_sign_path("open/test", "POST", body={"k": "v"})
|
|
assert get_sign != post_sign, "GET and POST should produce different signatures"
|
|
|
|
def test_get_with_params_includes_params_in_path(self):
|
|
path, _ = self.signer.build_sign_path("open/test", "GET", params={"city": "shanghai"})
|
|
assert "city" in path and "shanghai" in path, \
|
|
f"Params should appear in path: {path}"
|
|
|
|
def test_sign_key_in_path(self):
|
|
path, _ = self.signer.build_sign_path("open/jobs")
|
|
assert "api_key=51job" in path, f"api_key=51job missing from path: {path}"
|
|
|
|
|
|
class TestJob51SignGenerateUuid:
|
|
def test_generate_uuid_is_string(self):
|
|
uuid = Job51Sign.generate_uuid()
|
|
assert isinstance(uuid, str)
|
|
|
|
def test_generate_uuid_length(self):
|
|
uuid = Job51Sign.generate_uuid()
|
|
# 13-char ms timestamp + 10-char random int = 23 chars
|
|
assert len(uuid) == 23, f"Expected 23 chars, got {len(uuid)}: {uuid}"
|
|
|
|
def test_generate_uuid_numeric(self):
|
|
uuid = Job51Sign.generate_uuid()
|
|
assert uuid.isdigit(), f"UUID should be all digits: {uuid}"
|
|
```
|
|
|
|
---
|
|
|
|
**tests/crawler_core/test_zhilian_sign.py:**
|
|
|
|
```python
|
|
"""Unit tests for crawler_core.zhilian.sign — ZhilianSign.
|
|
|
|
All tests are pure function assertions: no HTTP, no network, no mocks.
|
|
"""
|
|
import sys
|
|
import os
|
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
|
|
|
|
import re
|
|
import pytest
|
|
from crawler_core.zhilian.sign import ZhilianSign
|
|
|
|
EXPECTED_HEADER_KEYS = {
|
|
"x-zp-at", "x-zp-rt", "x-zp-action-id", "x-zp-page-code",
|
|
"x-zp-version", "x-zp-channel", "x-zp-platform", "x-zp-device-id",
|
|
"x-zp-business-system",
|
|
}
|
|
|
|
EXPECTED_PARAM_KEYS = {"at", "rt", "channel", "platform", "version", "d"}
|
|
|
|
|
|
class TestZhilianSignInit:
|
|
def test_defaults(self):
|
|
sign = ZhilianSign()
|
|
assert sign.at == ""
|
|
assert sign.rt == ""
|
|
assert sign.version == "4.1.259"
|
|
assert sign.channel == "wxxiaochengxu"
|
|
assert sign.platform == "12"
|
|
assert sign.device_id # auto-generated, not empty
|
|
|
|
def test_custom_tokens(self):
|
|
sign = ZhilianSign(at="at_token", rt="rt_token")
|
|
assert sign.at == "at_token"
|
|
assert sign.rt == "rt_token"
|
|
|
|
def test_custom_device_id(self):
|
|
sign = ZhilianSign(device_id="CUSTOM-DEVICE-ID")
|
|
assert sign.device_id == "CUSTOM-DEVICE-ID"
|
|
|
|
def test_auto_device_id_is_uuid4_format(self):
|
|
sign = ZhilianSign()
|
|
uuid_pattern = r'^[0-9A-F]{8}-[0-9A-F]{4}-4[0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$'
|
|
assert re.match(uuid_pattern, sign.device_id), \
|
|
f"device_id not UUID4 format: {sign.device_id}"
|
|
|
|
|
|
class TestZhilianSignHeaders:
|
|
def setup_method(self):
|
|
self.sign = ZhilianSign(at="at123", rt="rt456")
|
|
|
|
def test_keys_exactly_nine(self):
|
|
headers = self.sign.sign_headers()
|
|
assert set(headers.keys()) == EXPECTED_HEADER_KEYS, \
|
|
f"Header keys wrong: {set(headers.keys())}"
|
|
|
|
def test_business_system_is_73(self):
|
|
headers = self.sign.sign_headers()
|
|
assert headers["x-zp-business-system"] == "73"
|
|
|
|
def test_tokens_reflected(self):
|
|
headers = self.sign.sign_headers()
|
|
assert headers["x-zp-at"] == "at123"
|
|
assert headers["x-zp-rt"] == "rt456"
|
|
|
|
def test_action_id_is_uuid4_format(self):
|
|
headers = self.sign.sign_headers()
|
|
action_id = headers["x-zp-action-id"]
|
|
uuid_pattern = r'^[0-9A-F]{8}-[0-9A-F]{4}-4[0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$'
|
|
assert re.match(uuid_pattern, action_id), \
|
|
f"action_id not UUID4 format: {action_id}"
|
|
|
|
def test_action_id_unique_per_call(self):
|
|
h1 = self.sign.sign_headers()
|
|
h2 = self.sign.sign_headers()
|
|
assert h1["x-zp-action-id"] != h2["x-zp-action-id"], \
|
|
"action_id must be freshly generated on each call"
|
|
|
|
def test_device_id_in_headers(self):
|
|
headers = self.sign.sign_headers()
|
|
assert headers["x-zp-device-id"] == self.sign.device_id
|
|
|
|
|
|
class TestZhilianSignParams:
|
|
def setup_method(self):
|
|
self.sign = ZhilianSign(at="at789", rt="rt012", device_id="DEV-ID")
|
|
|
|
def test_keys_exactly_six(self):
|
|
params = self.sign.sign_params()
|
|
assert set(params.keys()) == EXPECTED_PARAM_KEYS, \
|
|
f"Param keys wrong: {set(params.keys())}"
|
|
|
|
def test_device_id_as_d(self):
|
|
params = self.sign.sign_params()
|
|
assert params["d"] == "DEV-ID"
|
|
|
|
def test_tokens_reflected(self):
|
|
params = self.sign.sign_params()
|
|
assert params["at"] == "at789"
|
|
assert params["rt"] == "rt012"
|
|
|
|
|
|
class TestZhilianGenerateUuid:
|
|
def test_uuid4_format(self):
|
|
uuid = ZhilianSign.generate_uuid()
|
|
uuid_pattern = r'^[0-9A-F]{8}-[0-9A-F]{4}-4[0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$'
|
|
assert re.match(uuid_pattern, uuid), \
|
|
f"UUID not UUID4 format: {uuid}"
|
|
|
|
def test_uuid_length(self):
|
|
uuid = ZhilianSign.generate_uuid()
|
|
assert len(uuid) == 36, f"Expected 36 chars, got {len(uuid)}"
|
|
|
|
def test_uuid_version_4(self):
|
|
uuid = ZhilianSign.generate_uuid()
|
|
assert uuid[14] == "4", f"Version digit should be 4, got: {uuid[14]}"
|
|
```
|
|
|
|
After creating all files, run the tests to verify they pass. If any test fails due to a mismatch between the test expectation and the actual sign algorithm behavior, fix the TEST (not the sign algorithm) to match the actual behavior — the sign algorithms are the ground truth.
|
|
</action>
|
|
<verify>
|
|
<automated>cd /Users/win/2025/AICoding/JobData && python -m pytest tests/crawler_core/ -v --tb=short 2>&1 | tail -30</automated>
|
|
</verify>
|
|
<acceptance_criteria>
|
|
- `tests/crawler_core/__init__.py` exists (even if empty)
|
|
- `tests/crawler_core/test_boss_sign.py` exists with at least 8 test functions
|
|
- `tests/crawler_core/test_qcwy_sign.py` exists with at least 7 test functions
|
|
- `tests/crawler_core/test_zhilian_sign.py` exists with at least 9 test functions
|
|
- `python -m pytest tests/crawler_core/ -v` exits 0 — ALL tests pass
|
|
- `python -m pytest tests/crawler_core/ -v` output contains "passed" and zero "failed" or "error"
|
|
- No test in any file makes HTTP requests or reads from files (pure function tests only)
|
|
- `grep -r "requests\|httpx\|mock\|patch" tests/crawler_core/` returns empty (no mocking needed)
|
|
- Test count: `python -m pytest tests/crawler_core/ --collect-only -q` reports at least 24 tests collected
|
|
</acceptance_criteria>
|
|
<done>Three test files cover all sign algorithm edge cases. `pytest tests/crawler_core/` exits 0. No network access required.</done>
|
|
</task>
|
|
|
|
</tasks>
|
|
|
|
<verification>
|
|
Run the complete verification suite to confirm Phase 1 is done:
|
|
|
|
```bash
|
|
cd /Users/win/2025/AICoding/JobData
|
|
|
|
# 1. All sign algorithms importable from crawler_core
|
|
python -c "
|
|
import sys
|
|
sys.path.insert(0, '.')
|
|
from crawler_core.boss.sign import BossSign
|
|
from crawler_core.qcwy.sign import Job51Sign
|
|
from crawler_core.zhilian.sign import ZhilianSign
|
|
print('All sign imports OK')
|
|
"
|
|
|
|
# 2. All tests pass
|
|
python -m pytest tests/crawler_core/ -v
|
|
|
|
# 3. No cross-contamination
|
|
grep -r "from spiderJobs" crawler_core/ && echo "FAIL" || echo "OK: no spiderJobs imports"
|
|
grep -r "import requests" crawler_core/boss/sign.py crawler_core/qcwy/sign.py crawler_core/zhilian/sign.py && echo "FAIL" || echo "OK: sign files have no HTTP imports"
|
|
|
|
# 4. Old files untouched
|
|
git diff --name-only spiderJobs/platforms/ && echo "FAIL: spiderJobs modified" || echo "OK: spiderJobs untouched"
|
|
```
|
|
</verification>
|
|
|
|
<success_criteria>
|
|
1. `python -m pytest tests/crawler_core/ -v` exits 0 with at least 24 tests collected and passing
|
|
2. `from crawler_core.boss.sign import BossSign` succeeds (with repo root on sys.path)
|
|
3. `from crawler_core.qcwy.sign import Job51Sign` succeeds
|
|
4. `from crawler_core.zhilian.sign import ZhilianSign` succeeds
|
|
5. `grep -r "from spiderJobs" crawler_core/` returns empty
|
|
6. `grep -r "import requests" crawler_core/boss/sign.py crawler_core/qcwy/sign.py crawler_core/zhilian/sign.py` returns empty
|
|
7. `git diff --name-only spiderJobs/` returns empty — old files untouched
|
|
8. All three sign.py files in crawler_core/ are under 120 lines each (no bloat)
|
|
</success_criteria>
|
|
|
|
<output>
|
|
After completion, create `.planning/phases/01-shared-core/01-02-SUMMARY.md` with:
|
|
- List of files created (sign.py files + test files with line counts)
|
|
- Test counts per file and total
|
|
- Any behavioral differences discovered between spiderJobs/ originals and crawler_core/ copies
|
|
- Confirmation that old spiderJobs/ files were not modified
|
|
</output>
|