424 lines
20 KiB
Markdown
424 lines
20 KiB
Markdown
# Architecture Patterns: Shared Crawler Core
|
||
|
||
**Domain:** Recruiting data crawler — dual-deployment shared core
|
||
**Researched:** 2026-03-21
|
||
**Overall confidence:** HIGH (based entirely on direct codebase inspection)
|
||
|
||
---
|
||
|
||
## Problem Statement
|
||
|
||
The codebase currently has the same code in two places, copied verbatim:
|
||
|
||
| File | Copy location |
|
||
|------|---------------|
|
||
| `spiderJobs/core/http_client.py` | `app/services/crawler/_http_client.py` |
|
||
| `spiderJobs/core/base.py` | `app/services/crawler/_base.py` |
|
||
| `spiderJobs/platforms/boss/sign.py` | `app/services/crawler/_boss_sign.py` |
|
||
| `spiderJobs/platforms/boss/client.py` | `app/services/crawler/_boss_client.py` |
|
||
| `spiderJobs/platforms/boss/api.py` | `app/services/crawler/_boss_api.py` |
|
||
|
||
The docstrings in `app/services/crawler/_http_client.py` literally say: "复制自 spiderJobs/core/http_client.py — 不要直接 import spiderJobs，避免跨模块依赖" ("copied from — do not import directly to avoid cross-module dependency"). The comment reveals the source of the problem: the team correctly identified that two separate deployment units cannot cross-import, but solved it by copy-paste rather than extraction.
|
||
|
||
There are also two competing external spider frameworks: `jobs_spider/` (older, monolithic, single large file per platform) and `spiderJobs/` (newer, modular, uses `runner/loop.py` + `runner/api_client.py` as a proper execution harness). The refactor must unify these.
|
||
|
||
---
|
||
|
||
## Recommended Architecture
|
||
|
||
### Core Design Decision: Installable Shared Package
|
||
|
||
Extract shared crawler code into a proper Python package at `crawler_core/` (project root level). Both deployment contexts import from it. No copying. No circular imports.
|
||
|
||
```
|
||
crawler_core/ # Shared library — zero app/* or spiderJobs/* imports
|
||
├── __init__.py
|
||
├── http_client.py # HTTPClient (requests-go, TLS fingerprint, proxy rotation)
|
||
├── base.py # ApiResult, BaseFetcher, BaseSearcher, parse_response()
|
||
├── boss/
|
||
│ ├── __init__.py
|
||
│ ├── sign.py # BossSign (Traceid generation algorithm)
|
||
│ ├── client.py # BossClient(HTTPClient) — header injection
|
||
│ └── api.py # SearchRecJobs, GetBrandDetail, etc.
|
||
├── qcwy/
|
||
│ ├── __init__.py
|
||
│ ├── sign.py
|
||
│ ├── client.py
|
||
│ └── api.py
|
||
└── zhilian/
|
||
├── __init__.py
|
||
├── sign.py
|
||
├── client.py
|
||
└── api.py
|
||
```
|
||
|
||
**Why a root-level package and not inside `app/` or `spiderJobs/`:**
|
||
- `app/` is the FastAPI backend. Importing it from external scripts would pull in Tortoise-ORM, APScheduler, ClickHouse, and every other backend dependency — making external scripts impossibly heavy.
|
||
- `spiderJobs/` is a deployment directory, not a library.
|
||
- A root-level `crawler_core/` has no such baggage. It imports only `requests-go` and stdlib. Both `app/` and `spiderJobs/` import from it.
|
||
|
||
**Constraints satisfied:**
|
||
- No circular imports
|
||
- No shared runtime state (crawler_core is stateless pure logic)
|
||
- Installable as local package (editable install via `pip install -e .` or `pipenv install -e .`) for ECS deployment
|
||
- Tests can import `crawler_core` directly without mocking a FastAPI app
|
||
|
||
---
|
||
|
||
## Component Boundaries
|
||
|
||
### What Talks to What
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ crawler_core/ │
|
||
│ HTTPClient · BaseFetcher · BaseSearcher · ApiResult │
|
||
│ boss/ · qcwy/ · zhilian/ (sign + client + api) │
|
||
│ Imports: requests-go, stdlib only │
|
||
└────────────────┬────────────────────────┬───────────────────┘
|
||
│ │
|
||
┌────────────▼──────────┐ ┌─────────▼─────────────────┐
|
||
│ app/services/ │ │ spiderJobs/ │
|
||
│ crawler/ │ │ platforms/ + runner/ │
|
||
│ │ │ │
|
||
│ BossService facade │ │ run_crawl_loop() │
|
||
│ QcwyService facade │ │ RunnerAPIClient │
|
||
│ ZhilianService facade│ │ (HTTP push to backend) │
|
||
│ │ │ │
|
||
│ Called by: │ │ Called by: │
|
||
│ CompanyCleaner │ │ ECS node scripts │
|
||
│ CompanyController │ │ jobs_spider/ (deprecated) │
|
||
└────────────┬──────────┘ └────────────────────────────┘
|
||
│
|
||
┌────────────▼──────────────────────────────────────────┐
|
||
│ app/ (FastAPI backend — async boundary) │
|
||
│ CompanyCleaner (asyncio, calls sync crawler via │
|
||
│ asyncio.to_thread()) │
|
||
│ IngestService (async ClickHouse writes) │
|
||
│ APScheduler (triggers CompanyCleaner async jobs) │
|
||
└───────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### Component Responsibilities
|
||
|
||
| Component | Responsibility | What It Must NOT Do |
|
||
|-----------|---------------|---------------------|
|
||
| `crawler_core/` | Signing, HTTP requests, response parsing — per platform | Import `app.*`, use async, maintain global state |
|
||
| `app/services/crawler/` facades | Wrap `crawler_core` with logging, token loading, proxy mgmt | Contain signing or HTTP logic |
|
||
| `spiderJobs/runner/` | Crawl loop orchestration, keyword polling, data upload | Import `app.*` |
|
||
| `app/services/ingest/` | Registry-driven ClickHouse insertion, dedup | Know anything about platform HTTP clients |
|
||
| `app/core/scheduler.py` | Schedule jobs, manage distributed lock | Run synchronous I/O on the event loop directly |
|
||
|
||
---
|
||
|
||
## Abstract Base Class Hierarchy
|
||
|
||
```python
|
||
# crawler_core/base.py
|
||
|
||
@dataclass
|
||
class ApiResult:
|
||
success: bool
|
||
status_code: int
|
||
data: Any = None
|
||
list: list[dict] = field(default_factory=list)
|
||
count: int = 0
|
||
is_end_page: bool = True
|
||
error: Optional[str] = None
|
||
|
||
|
||
class BaseFetcher:
|
||
"""Single-resource GET (company detail, job detail)."""
|
||
ENDPOINT: str = ""
|
||
|
||
def __init__(self, http_client: HTTPClient): ...
|
||
def _build_params(self) -> dict: raise NotImplementedError
|
||
def _parse(self, http_code: int, raw: Any) -> ApiResult: ...
|
||
def fetch(self) -> ApiResult: ... # sync — wraps _http.get()
|
||
|
||
|
||
class BaseSearcher:
|
||
"""Paginated search (job list, brand job list)."""
|
||
ENDPOINT: str = ""
|
||
|
||
def __init__(self, page_size: int, http_client: HTTPClient): ...
|
||
def _build_params(self, page_index: int) -> dict: raise NotImplementedError
|
||
def _request(self, params: dict) -> tuple[int, Any]: ...
|
||
def _parse(self, http_code: int, raw: Any) -> ApiResult: ...
|
||
def search(self, page_index: int = 1) -> ApiResult: ... # sync
|
||
def load_all(self, max_pages: int, on_page: Callable | None) -> list[dict]: ...
|
||
```
|
||
|
||
Platform clients subclass `HTTPClient` from `crawler_core/http_client.py`:
|
||
|
||
```python
|
||
# crawler_core/boss/client.py
|
||
class BossClient(HTTPClient):
|
||
"""Injects mpt/wt2/Traceid headers on every request."""
|
||
def __init__(self, signer: BossSign, ...): ...
|
||
def post(self, path, body, headers=None) -> tuple[int, Any]: ... # sync
|
||
def get(self, path, params=None, headers=None) -> tuple[int, Any]: ... # sync
|
||
```
|
||
|
||
Platform API classes subclass `BaseFetcher` / `BaseSearcher`:
|
||
|
||
```python
|
||
# crawler_core/boss/api.py
|
||
class SearchRecJobs(BaseSearcher):
|
||
ENDPOINT = "/wapi/zpgeek/miniapp/homepage/recjoblist.json"
|
||
def _build_params(self, page_index: int) -> dict: ...
|
||
def _parse(self, http_code: int, raw: Any) -> ApiResult: ... # Boss-specific parsing
|
||
|
||
class GetBrandDetail(BaseFetcher):
|
||
ENDPOINT = "/wapi/zpgeek/miniapp/brand/get.json"
|
||
def _build_params(self) -> dict: ...
|
||
```
|
||
|
||
**The hierarchy is deliberately flat.** Three levels: `HTTPClient` → `PlatformClient` → `PlatformAPIClass`. No further abstraction.
|
||
|
||
---
|
||
|
||
## Sync vs Async: The Critical Design Point
|
||
|
||
### Why the Core Must Stay Synchronous
|
||
|
||
`crawler_core/` uses `requests-go` (a sync library that wraps a TLS fingerprinting mechanism). There is no async version of `requests-go`. The platform APIs require TLS fingerprint spoofing that `httpx` does not support without significant additional work. Keeping the core synchronous is the correct choice given the constraint.
|
||
|
||
### How the Backend (async) Calls Synchronous Crawlers
|
||
|
||
`CompanyCleaner` runs inside the FastAPI async event loop (called by APScheduler). It cannot block the event loop with synchronous `requests` calls. The correct pattern is `asyncio.to_thread()`:
|
||
|
||
```python
|
||
# app/services/company_cleaner.py
|
||
|
||
async def _fetch_company_detail(self, source: str, company_id: str) -> Optional[dict]:
|
||
"""Run synchronous crawler call off the event loop thread."""
|
||
if source == "boss":
|
||
return await asyncio.to_thread(
|
||
self.boss_service.get_company_detail_by_id, company_id
|
||
)
|
||
elif source == "qcwy":
|
||
return await asyncio.to_thread(
|
||
self.qcwy_service.get_company_detail_by_id, company_id
|
||
)
|
||
elif source == "zhilian":
|
||
return await asyncio.to_thread(
|
||
self.zhilian_service.get_company_detail_by_id, company_id
|
||
)
|
||
```
|
||
|
||
`asyncio.to_thread()` runs the synchronous function in a thread pool, yielding back to the event loop while it executes. This is the standard Python pattern for sync I/O in an async context.
|
||
|
||
**Current state:** The existing `CompanyCleaner.process_pending_companies()` likely calls sync methods directly on the event loop — this is a bug to fix during refactoring. The fix is wrapping each crawler call with `asyncio.to_thread()`.
|
||
|
||
### External Scripts: Pure Synchronous, No Async
|
||
|
||
`spiderJobs/runner/loop.py` (`run_crawl_loop`) and any ECS scripts are pure synchronous `while True` loops. They import from `crawler_core/` directly. No async. This is correct and intentional.
|
||
|
||
```
|
||
External Script (sync) Backend (async)
|
||
────────────────────── ─────────────────────────────
|
||
while True: async def process_company():
|
||
kw = api.fetch_keyword() detail = await asyncio.to_thread(
|
||
searcher = BossSearcher(...) boss_service.get_company_detail
|
||
result = searcher.search() )
|
||
api.upload_data(result.list) await company_storage.save(detail)
|
||
```
|
||
|
||
---
|
||
|
||
## Data Flow Patterns
|
||
|
||
### Flow 1: External Spider → Backend Storage
|
||
|
||
```
|
||
ECS Script (spiderJobs/runner/loop.py)
|
||
│
|
||
├─ GET /api/v1/keyword/available
|
||
│ → KeywordController marks keyword as "crawling"
|
||
│
|
||
├─ crawler_core/boss/api.py SearchRecJobs.search(page)
|
||
│ → returns ApiResult.list (raw job dicts)
|
||
│
|
||
├─ POST /api/v1/universal/data/batch-store-async
|
||
│ body: {platform, channel, data_type, data_list}
|
||
│
|
||
└─ Backend: IngestService.store_batch()
|
||
→ registry lookup → dedup → ClickHouse insert
|
||
```
|
||
|
||
The external script knows nothing about ClickHouse. It only pushes raw dicts over HTTP. This boundary is correct and must stay.
|
||
|
||
### Flow 2: In-Process Company Cleaning (Scheduled)
|
||
|
||
```
|
||
APScheduler → company_cleaning_job() [async]
|
||
│
|
||
├─ CompanyCleaner.collect_pending_companies() [async]
|
||
│ → ClickHouse: query job tables for company IDs
|
||
│ → MySQL: check which company IDs already processed
|
||
│ → MySQL: insert new companies into CompanyCleaningQueue
|
||
│
|
||
└─ CompanyCleaner.process_pending_companies() [async]
|
||
├─ MySQL: fetch N companies from queue
|
||
├─ asyncio.to_thread(boss_service.get_company_detail_by_id)
|
||
│ → crawler_core/boss/api.py GetBrandDetail.fetch() [sync]
|
||
├─ company_storage.save_company() [async, ClickHouse]
|
||
└─ MySQL: update company status to "done"
|
||
```
|
||
|
||
The flow crosses the sync/async boundary exactly once: at `asyncio.to_thread()`. Everything above (ClickHouse queries, MySQL operations) stays async. Everything below (HTTP to Boss API) stays sync.
|
||
|
||
### Flow 3: Company Jobs Sync (new, post-refactor)
|
||
|
||
After a company is cleaned, the system should backfill that company's current job listings:
|
||
|
||
```
|
||
CompanyJobsSyncService.sync_company_jobs(company_id, source) [async]
|
||
├─ asyncio.to_thread(boss_service.get_company_jobs_by_id, company_id)
|
||
│ → crawler_core/boss/api.py SearchBrandJobs.search() [sync]
|
||
└─ IngestService.store_batch(platform="boss", data_type="company_job")
|
||
→ ClickHouse insert into boss_company_job table (new table needed)
|
||
```
|
||
|
||
---
|
||
|
||
## Package Structure: Suggested Build Order
|
||
|
||
Build `crawler_core/` first — everything else depends on it.
|
||
|
||
### Phase 1: Extract `crawler_core/`
|
||
|
||
1. Create `crawler_core/__init__.py`, `crawler_core/http_client.py`, `crawler_core/base.py`
|
||
- These are identical to the current `spiderJobs/core/` files with one import path change
|
||
- Zero functional change; pure extraction
|
||
2. Create `crawler_core/boss/`, `crawler_core/qcwy/`, `crawler_core/zhilian/`
|
||
- Extract from `spiderJobs/platforms/boss/`, qcwy/, zhilian/
|
||
- Update internal imports to `from crawler_core.boss.sign import BossSign`, etc.
|
||
3. Add `crawler_core` to `Pipfile` as an editable local dependency:
|
||
```toml
|
||
[packages]
|
||
crawler_core = {path = ".", editable = true}
|
||
```
|
||
(Or configure `pyproject.toml` with `[tool.setuptools.packages.find]` including `crawler_core`)
|
||
|
||
**Verification gate:** `spiderJobs/` and `app/services/crawler/` both import from `crawler_core` without errors. `spiderJobs/core/` and `spiderJobs/platforms/` can be deleted.
|
||
|
||
### Phase 2: Rewire `app/services/crawler/` Facades
|
||
|
||
The service facades (`boss.py`, `qcwy.py`, `zhilian.py`) stay in `app/services/crawler/` — they belong to the backend. Their job is to:
|
||
- Load tokens from MySQL (`BossToken`)
|
||
- Load proxies from `ProxyController`
|
||
- Instantiate `crawler_core` clients with those credentials
|
||
- Expose typed async-friendly methods (`get_company_detail_by_id`)
|
||
- Wrap `asyncio.to_thread()` for each sync call
|
||
|
||
The private underscore modules (`_boss_client.py`, `_boss_sign.py`, etc.) are deleted. Facades import directly from `crawler_core`.
|
||
|
||
```python
|
||
# app/services/crawler/boss.py (after refactor)
|
||
from crawler_core.boss.client import BossClient, create_client
|
||
from crawler_core.boss.sign import BossSign
|
||
from crawler_core.boss.api import GetBrandDetail, SearchBrandJobs
|
||
```
|
||
|
||
**Verification gate:** `CompanyCleaner` test passes with mocked `crawler_core` calls.
|
||
|
||
### Phase 3: Rewire `spiderJobs/` External Scripts
|
||
|
||
`spiderJobs/runner/` imports `from spiderJobs.core.base import ...` → change to `from crawler_core.base import ...`. Delete `spiderJobs/core/` and `spiderJobs/platforms/`.
|
||
|
||
`spiderJobs/platforms/boss/main.py` becomes a thin entry point:
|
||
```python
|
||
from crawler_core.boss.client import create_client
|
||
from crawler_core.boss.api import SearchRecJobs
|
||
from spiderJobs.runner.loop import run_crawl_loop
|
||
```
|
||
|
||
`jobs_spider/` (the older framework with monolithic files) is deprecated in favor of `spiderJobs/` + `crawler_core/`.
|
||
|
||
---
|
||
|
||
## Anti-Patterns to Avoid
|
||
|
||
### Anti-Pattern 1: Async Crawler Core
|
||
**What:** Making `crawler_core/` async because the backend is async.
|
||
**Why bad:** `requests-go` is sync. The TLS fingerprint mechanism is sync. Wrapping everything in `asyncio.run()` or creating fake async wrappers adds complexity with no benefit. The `asyncio.to_thread()` boundary at the facade layer is the correct and idiomatic solution.
|
||
**Instead:** Keep `crawler_core/` synchronous. Use `asyncio.to_thread()` in the backend facades.
|
||
|
||
### Anti-Pattern 2: Putting `RunnerAPIClient` in `crawler_core/`
|
||
**What:** Moving the HTTP push client (backend API caller) into the shared core.
|
||
**Why bad:** `RunnerAPIClient` is deployment-specific: it knows about `/api/v1/keyword/available`, `/api/v1/universal/data/batch-store-async`, and the backend's URL scheme. That is runner orchestration logic, not platform API logic.
|
||
**Instead:** `RunnerAPIClient` stays in `spiderJobs/runner/api_client.py`. It is not shared.
|
||
|
||
### Anti-Pattern 3: Importing `app.*` from External Scripts
|
||
**What:** Having ECS scripts import from `app.settings.config`, `app.models`, or `app.core.*`.
|
||
**Why bad:** That would require Tortoise-ORM initialization, database connections, and all backend startup logic to run on ECS nodes. This would fail.
|
||
**Instead:** External scripts get all config from environment variables. They communicate with the backend only via HTTP.
|
||
|
||
### Anti-Pattern 4: Blocking the Event Loop with Sync Crawler Calls
|
||
**What:** Calling `boss_service.get_company_detail_by_id()` directly in an `async def` without `asyncio.to_thread()`.
|
||
**Why bad:** Each HTTP call takes 2-10 seconds. With 30 companies to process, that blocks the event loop for up to 5 minutes. Other requests, heartbeats, and scheduler jobs starve.
|
||
**Instead:** Always wrap sync crawler calls in `asyncio.to_thread()` inside async context.
|
||
|
||
### Anti-Pattern 5: Monolithic Platform Files
|
||
**What:** Keeping the existing `jobs_spider/boss/boos_api.py` (2246 lines) pattern in the refactored code.
|
||
**Why bad:** Impossible to test individual components. Signing, HTTP, and business logic are entangled.
|
||
**Instead:** Three files per platform: `sign.py` (pure functions, no I/O), `client.py` (HTTP wrapper, no business logic), `api.py` (endpoint classes).
|
||
|
||
---
|
||
|
||
## Scalability Considerations
|
||
|
||
| Concern | Current | After Refactor |
|
||
|---------|---------|---------------|
|
||
| Adding a 4th platform (e.g., job51) | Copy files into both `spiderJobs/` and `app/services/crawler/` | Add `crawler_core/job51/` once; both consumers get it |
|
||
| Testing signing algorithm | Must run full crawler | Test `crawler_core/boss/sign.py` in isolation — pure functions, no I/O |
|
||
| Updating User-Agent headers | Two files to update | One file: `crawler_core/boss/client.py` |
|
||
| Running multiple concurrent companies | Single-threaded | `asyncio.to_thread()` uses thread pool; default pool size = CPU count × 5 |
|
||
| ECS deployment of crawlers | Must deploy `spiderJobs/` + copy of core | Deploy `crawler_core/` + `spiderJobs/`; single source of truth |
|
||
|
||
---
|
||
|
||
## Suggested Build Order for Roadmap Phases
|
||
|
||
Dependencies flow in one direction:
|
||
|
||
```
|
||
Phase 1: crawler_core/ extraction
|
||
└─ Produces: installable shared package, 3-platform coverage
|
||
|
||
Phase 2: app/services/crawler/ facade rewire
|
||
└─ Depends on: Phase 1 (imports crawler_core)
|
||
└─ Produces: asyncio.to_thread() boundary, deleted underscore copies
|
||
|
||
Phase 3: spiderJobs/ rewire + jobs_spider/ deprecation
|
||
└─ Depends on: Phase 1 (imports crawler_core)
|
||
└─ Produces: single external runner codebase
|
||
|
||
Phase 4: Company cleaning flow optimization
|
||
└─ Depends on: Phase 2 (async-safe facades)
|
||
└─ Produces: CompanyJobsSyncService using new facades
|
||
|
||
Phase 5: Unit tests
|
||
└─ Depends on: Phase 1 (crawler_core is testable in isolation)
|
||
└─ crawler_core tests: no mocking needed for sign.py (pure functions)
|
||
└─ facade tests: mock crawler_core, test async wrapping
|
||
```
|
||
|
||
Phases 2 and 3 can run in parallel (both depend only on Phase 1, not each other).
|
||
|
||
---
|
||
|
||
## Sources
|
||
|
||
- Direct inspection: `/Users/win/2025/AICoding/JobData/app/services/crawler/_http_client.py` (docstring confirms copy from `spiderJobs/core/http_client.py`)
|
||
- Direct inspection: `/Users/win/2025/AICoding/JobData/spiderJobs/core/base.py` vs `app/services/crawler/_base.py` — byte-identical logic
|
||
- Direct inspection: `/Users/win/2025/AICoding/JobData/app/services/company_cleaner.py` — async consumer of sync crawlers
|
||
- Direct inspection: `/Users/win/2025/AICoding/JobData/spiderJobs/runner/loop.py` — sync-only external runner
|
||
- Direct inspection: `/Users/win/2025/AICoding/JobData/.planning/codebase/ARCHITECTURE.md` — existing architecture analysis
|
||
|
||
---
|
||
|
||
*Architecture analysis: 2026-03-21*
|