# Python worker — implementation notes and handoff

Owner: worker agent. Files owned: `worker/`, `tests/worker/`, `requirements-worker.txt`,
`.venv` (shared uv venv at project root, created via `uv venv .venv --python 3.11` +
`uv pip install --python .venv/bin/python -r requirements-worker.txt`),
`artifacts/worker/` evidence, this doc only.

Status: working worker, 27/27 pytest green (network-free unit tests), real AKShare
fetch + real Backtrader run executed on host and inside Docker image
`strategy-lab-worker:local` (nonroot, read-only rootfs, `--network none` backtest).

## Components
- `worker/data.py` — AKShare adapters: eastmoney (`stock_zh_a_hist`,
  `fund_etf_hist_em`, `index_zh_a_hist`) and tencent (`stock_zh_a_hist_tx`,
  `stock_zh_index_daily_tx`). Explicit source selection:
  `source: 'eastmoney' | 'tencent' | 'auto'` (default auto).
  - `auto` = eastmoney first, then TRANSPARENT same-symbol tencent fallback with
    warning `provider_fallback: ...; served by tencent for the SAME symbol`.
    Never a different symbol or silent provider/adjustment substitution.
  - Explicit `source` never silently falls back; failures surface.
  - Tencent adapter returns volume in shares already; no multipliers applied.
  - Index data: adjustment must be `none` (provider tencent labels it 前复权; recorded as warning).
- `worker/normalize.py` — canonical frame (`date,open,high,low,close,volume` +
  preserved raw fields e.g. 成交额/换手率 with units/precision untouched), symbol
  stamping (never substituted), date sort/dedup, NaN-OHLCV rejection (gaps are
  kept, never imputed), content hashes, coverage segments/gaps, manifest entries
  (`immutable: true`, `path` internal-only — backend must rewrite before client exposure).
- `worker/backtest.py` — real Backtrader (1.9.78.123). User source defines class
  `Strategy` subclassing `bt.Strategy`; executed via `exec` after `ast` syntax
  check, in the isolated worker process only (Docker). Recording:
  - feeds added with `name=<canonical instrument symbol>` → user uses
    `self.getdatabyname("<symbol>")`; prices never substituted across feeds.
  - A never-trading `RowsRecorder` strategy records per-bar
    `{date, cash, equity, closes, positions}`.
  - A generated subclass of the user's `Strategy` intercepts `notify_order` /
    `notify_trade` (documented hooks) to record orders, executed fills
    (trades: `{date,symbol,side,quantity,price,commission,value}`) and closed
    round-trip trades.
  - Execution rules (documented, tested): fills at NEXT bar open (never
    signal-bar close), commission % of trade value, slippage % of fill price,
    indicator warmup honored, T+1/lot/suspension/limits NOT simulated,
    `index` feeds are nontradable proxies (server must reject direct index orders).
  - `metrics.trade_count` = number of CLOSED round-trip trades (buy-only runs = 0),
    fills recorded separately in `trades`.
  -   `max_drawdown` = negative loss fraction (e.g. -0.05 = 5% drawdown; **0.0 is a
  valid value for flat equity**, null only when not computable); NaN/Inf never
  emitted (null instead). `peak_rss_kb` = real process max RSS.
- Broker-level cn stock/ETF rules (`CnDailyRulesBroker`, in-process, tested):
  - `asset_type=index` feeds are **rejected at the broker level for any order**
    (warning `nontradable` + `order_state: rejected` record); allowed as benchmark feeds.
  - buys rounded DOWN to whole lot=100; reject when < 100. Requested size kept
    on `order.prereject_size` for audit; fill follows enforced size.
  - sells rejected/trimmed when they would exceed owned shares minus pending
    same-symbol buys (no naked short, no T+0 sale of same-cycle buys).
  - T+1 at daily granularity emerges from next-bar-open fills: a sell submitted
    on a buy's fill day executes next trading day, exactly the A-share rule.
    Position can never go negative in the equity series.
- Result additions: `initial_cash` (from config capital), per-bar `positions`
  dict in every equity bar (symbol→position size), and `benchmark` per equity
  bar **normalized to initial capital** so equity and benchmark share one scale
  (both start at `initial_cash`); raw benchmark closes stay in `equity[].closes`
  under the benchmark symbol. Frontend Chart must NEVER mix `equity[].benchmark`
  (capital scale) with `equity[].closes` (raw prices) on the same axis.
- `worker/main.py` — CLI: `fetch` / `backtest` / `search` / `probe`.
  - fetch validates request (1..5 instruments, daily, none|qfq|hfq, fields
    whitelist amount/turnover extras, 15-year POC bound, source whitelist) and is
    bounded by SIGALRM wall clock, default 180s
    (`STRATEGY_LAB_FETCH_WALL_SECS` env override). No unbounded network calls.
  - raw provider response stored as an immutable content-addressed object at
    `output/objects/raw_<slug>_<hash12>.json` (written BEFORE the normalized
    transform; `raw_object_hash` in the manifest is the sha256 of the raw object)
  - result.json `{status, manifest, preview, cache_key, warnings, elapsed_ms}`;
    manifest per-object: provider/endpoint/params/akshare_version/fetched_at/
    schema_version/normalization_version/adjustment/requested_start/requested_end/
    actual_start/actual_end/row_count/columns/object_hash/raw_object_hash/warnings.
  - cache key = sha256 of manifest objects content (stable content identity,
    independent of user/request ids/fetch time). Same-key fetches of same
    content produce identical hashes; backend copies/hash-verifies into
    `data/objects` (shared physical object reuse; old snapshots never overwritten).

## Docker contract
Image `strategy-lab-worker:local` from `worker/Dockerfile` (python:3.11-slim,
nonroot uid 10001, app at `/app`). Invocation used and verified:

    docker run --rm --user 10001:10001 --cap-drop ALL \
      --security-opt no-new-privileges --read-only --tmpfs /tmp \
      --network none --memory 512m --cpus 1 --pids-limit 128 \
      -v <req.json>:/input/request.json:ro -v <data>:/data:ro \
      -v <out>:/output -e HOME=/tmp --entrypoint python \
      strategy-lab-worker:local -m worker.main backtest --request /input/request.json --output /output

- backtest: `--network none`. fetch: network enabled for adapter only.
- /output must be writable by uid 10001 (backend: pre-chmod or run with same uid).
- no Docker socket, no provider/auth env in container.

## For backend integration
- fetch request: `{instruments[{symbol,market,asset_type,name?}], start_date,
  end_date, frequency:'daily', adjustment, fields[], source?}`; instruments
  symbol may be bare code or canonical `SH#600000`.
- fetch result.json status: ready|failed; failed carries `errors[{code,message}]`.
- backend: strip `path`-host info from client-exposed manifest; rewrite object
  paths into stored immutable objects; mount only the referenced files per run
  into `/data` read-only and put per-run request in `/input`.
- cache: hash result.manifest.objects content; backend should cache both raw and
  normalized objects keyed by (provider, endpoint, params incl. adjust,
  normalization_version); dataset manifest hash stays stable when content unchanged.
- backtest request: `{code, config:{capital,commission,slippage,benchmark_symbol?,parameters},
  dataset_manifest, data_root:'/data'}`. result.json: SPEC.md Result shape +
  `closes` per equity bar, `closed_trades`, `execution_assumptions`.
- data warnings (coverage gaps, provider fallback, synthetic) MUST be surfaced
  to the user and acknowledged per SPEC.

## Real evidence in artifacts/worker (fresh, current source)
- Tests: `.venv/bin/python -m pytest tests/worker -q` → **35 passed** (adds
  tests/worker/test_rules_regression.py: zero-drawdown 0.0, broker index-order
  rejection, index-as-benchmark accepted, lot rounding 250→200, no naked short,
  sell-on-same-cycle trim, cash+position reconciliation).
- `akshare-probe.json` — real probe: stock_zh_a_hist + fund_etf_hist_em OK;
  stock_zh_index_daily_em + stock_info_a_code_name FAILED (network, honest).
- `real-fetch/` — 浦发银行 600000 unadjusted via tencent: 117 bars
  (2024-01-02 6.63/6.60/vol 22,066,700 → 2024-06-28), raw object
  `objects/raw_600000_<hash12>.json` + normalized CSV + manifest.
- `real-backtest/` — host run: 16 fills (900×16, lot-compliant), final equity
  118905.65, MDD -0.0525, Sharpe 2.82, initial_cash 100000 recorded, positions
  per bar, peak RSS ~157-165 MB.
- `docker-run/` — container backtest (`--network none`, uid 10001, read-only,
  cap-drop ALL, tmpfs /tmp) → identical metrics to host; container fetch run
  produced object hash a63c2c71f518 identical to host → same-object sharing.
- `docker-run/request-benchmark.json` + `output/result.json` — honest
  benchmark contract verified in real container run: benchmark starts at
  100000.0 and ends 124696.97 (normalized), raw close 8.23 kept separately in
  `equity[-1].closes['600000']`, warning "benchmark normalized to initial capital".

## Tests
`tests/worker/` — 27 tests, all synthetic fixtures carry `_synthetic`/`synthetic`
markers and are never production fallback: normalization correctness/rejection,
hash stability, deterministic accounting (hand-verified commission/slippage/
next-bar fills), named-feed isolation, benchmark series, error paths (syntax,
missing Strategy class, missing data, lookahead-at-end), catalog concat identity
regression, auto-fallback honest warning, explicit-source no-fallback, CLI
provider/raw-retention/error paths. All providers fully mocked in unit tests;
no network. Run: `.venv/bin/python -m pytest tests/worker -q`.

Limitations (honest): eastmoney currently blocked from this host; tencent lacks
listed-ETF daily adapter; index via tencent labeled 前复权; suspension/price
limits/intraday sequencing/liquidity NOT modeled; ETF sell lot approximated as
lot-100 buy rule; T+1 enforced at daily-bar granularity (no intraday detail);
search CLI uses catalog endpoints that can be slow/blocked (backend should call
with bounded timeout and show failure UI). No fake data or fabricated metrics.