1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
|
# Python worker — implementation notes and handoff
Owner: worker agent. Files owned: `worker/`, `tests/worker/`, `requirements-worker.txt`,
`.venv` (shared uv venv at project root, created via `uv venv .venv --python 3.11` +
`uv pip install --python .venv/bin/python -r requirements-worker.txt`),
`artifacts/worker/` evidence, this doc only.
Status: working worker, 27/27 pytest green (network-free unit tests), real AKShare
fetch + real Backtrader run executed on host and inside Docker image
`strategy-lab-worker:local` (nonroot, read-only rootfs, `--network none` backtest).
## Components
- `worker/data.py` — AKShare adapters: eastmoney (`stock_zh_a_hist`,
`fund_etf_hist_em`, `index_zh_a_hist`) and tencent (`stock_zh_a_hist_tx`,
`stock_zh_index_daily_tx`). Explicit source selection:
`source: 'eastmoney' | 'tencent' | 'auto'` (default auto).
- `auto` = eastmoney first, then TRANSPARENT same-symbol tencent fallback with
warning `provider_fallback: ...; served by tencent for the SAME symbol`.
Never a different symbol or silent provider/adjustment substitution.
- Explicit `source` never silently falls back; failures surface.
- Tencent adapter returns volume in shares already; no multipliers applied.
- Index data: adjustment must be `none` (provider tencent labels it 前复权; recorded as warning).
- `worker/normalize.py` — canonical frame (`date,open,high,low,close,volume` +
preserved raw fields e.g. 成交额/换手率 with units/precision untouched), symbol
stamping (never substituted), date sort/dedup, NaN-OHLCV rejection (gaps are
kept, never imputed), content hashes, coverage segments/gaps, manifest entries
(`immutable: true`, `path` internal-only — backend must rewrite before client exposure).
- `worker/backtest.py` — real Backtrader (1.9.78.123). User source defines class
`Strategy` subclassing `bt.Strategy`; executed via `exec` after `ast` syntax
check, in the isolated worker process only (Docker). Recording:
- feeds added with `name=<canonical instrument symbol>` → user uses
`self.getdatabyname("<symbol>")`; prices never substituted across feeds.
- A never-trading `RowsRecorder` strategy records per-bar
`{date, cash, equity, closes, positions}`.
- A generated subclass of the user's `Strategy` intercepts `notify_order` /
`notify_trade` (documented hooks) to record orders, executed fills
(trades: `{date,symbol,side,quantity,price,commission,value}`) and closed
round-trip trades.
- Execution rules (documented, tested): fills at NEXT bar open (never
signal-bar close), commission % of trade value, slippage % of fill price,
indicator warmup honored, T+1/lot/suspension/limits NOT simulated,
`index` feeds are nontradable proxies (server must reject direct index orders).
- `metrics.trade_count` = number of CLOSED round-trip trades (buy-only runs = 0),
fills recorded separately in `trades`.
- `max_drawdown` = negative loss fraction (e.g. -0.05 = 5% drawdown; **0.0 is a
valid value for flat equity**, null only when not computable); NaN/Inf never
emitted (null instead). `peak_rss_kb` = real process max RSS.
- Broker-level cn stock/ETF rules (`CnDailyRulesBroker`, in-process, tested):
- `asset_type=index` feeds are **rejected at the broker level for any order**
(warning `nontradable` + `order_state: rejected` record); allowed as benchmark feeds.
- buys rounded DOWN to whole lot=100; reject when < 100. Requested size kept
on `order.prereject_size` for audit; fill follows enforced size.
- sells rejected/trimmed when they would exceed owned shares minus pending
same-symbol buys (no naked short, no T+0 sale of same-cycle buys).
- T+1 at daily granularity emerges from next-bar-open fills: a sell submitted
on a buy's fill day executes next trading day, exactly the A-share rule.
Position can never go negative in the equity series.
- Result additions: `initial_cash` (from config capital), per-bar `positions`
dict in every equity bar (symbol→position size), and `benchmark` per equity
bar **normalized to initial capital** so equity and benchmark share one scale
(both start at `initial_cash`); raw benchmark closes stay in `equity[].closes`
under the benchmark symbol. Frontend Chart must NEVER mix `equity[].benchmark`
(capital scale) with `equity[].closes` (raw prices) on the same axis.
- `worker/main.py` — CLI: `fetch` / `backtest` / `search` / `probe`.
- fetch validates request (1..5 instruments, daily, none|qfq|hfq, fields
whitelist amount/turnover extras, 15-year POC bound, source whitelist) and is
bounded by SIGALRM wall clock, default 180s
(`STRATEGY_LAB_FETCH_WALL_SECS` env override). No unbounded network calls.
- raw provider response stored as an immutable content-addressed object at
`output/objects/raw_<slug>_<hash12>.json` (written BEFORE the normalized
transform; `raw_object_hash` in the manifest is the sha256 of the raw object)
- result.json `{status, manifest, preview, cache_key, warnings, elapsed_ms}`;
manifest per-object: provider/endpoint/params/akshare_version/fetched_at/
schema_version/normalization_version/adjustment/requested_start/requested_end/
actual_start/actual_end/row_count/columns/object_hash/raw_object_hash/warnings.
- cache key = sha256 of manifest objects content (stable content identity,
independent of user/request ids/fetch time). Same-key fetches of same
content produce identical hashes; backend copies/hash-verifies into
`data/objects` (shared physical object reuse; old snapshots never overwritten).
## Docker contract
Image `strategy-lab-worker:local` from `worker/Dockerfile` (python:3.11-slim,
nonroot uid 10001, app at `/app`). Invocation used and verified:
docker run --rm --user 10001:10001 --cap-drop ALL \
--security-opt no-new-privileges --read-only --tmpfs /tmp \
--network none --memory 512m --cpus 1 --pids-limit 128 \
-v <req.json>:/input/request.json:ro -v <data>:/data:ro \
-v <out>:/output -e HOME=/tmp --entrypoint python \
strategy-lab-worker:local -m worker.main backtest --request /input/request.json --output /output
- backtest: `--network none`. fetch: network enabled for adapter only.
- /output must be writable by uid 10001 (backend: pre-chmod or run with same uid).
- no Docker socket, no provider/auth env in container.
## For backend integration
- fetch request: `{instruments[{symbol,market,asset_type,name?}], start_date,
end_date, frequency:'daily', adjustment, fields[], source?}`; instruments
symbol may be bare code or canonical `SH#600000`.
- fetch result.json status: ready|failed; failed carries `errors[{code,message}]`.
- backend: strip `path`-host info from client-exposed manifest; rewrite object
paths into stored immutable objects; mount only the referenced files per run
into `/data` read-only and put per-run request in `/input`.
- cache: hash result.manifest.objects content; backend should cache both raw and
normalized objects keyed by (provider, endpoint, params incl. adjust,
normalization_version); dataset manifest hash stays stable when content unchanged.
- backtest request: `{code, config:{capital,commission,slippage,benchmark_symbol?,parameters},
dataset_manifest, data_root:'/data'}`. result.json: SPEC.md Result shape +
`closes` per equity bar, `closed_trades`, `execution_assumptions`.
- data warnings (coverage gaps, provider fallback, synthetic) MUST be surfaced
to the user and acknowledged per SPEC.
## Real evidence in artifacts/worker (fresh, current source)
- Tests: `.venv/bin/python -m pytest tests/worker -q` → **35 passed** (adds
tests/worker/test_rules_regression.py: zero-drawdown 0.0, broker index-order
rejection, index-as-benchmark accepted, lot rounding 250→200, no naked short,
sell-on-same-cycle trim, cash+position reconciliation).
- `akshare-probe.json` — real probe: stock_zh_a_hist + fund_etf_hist_em OK;
stock_zh_index_daily_em + stock_info_a_code_name FAILED (network, honest).
- `real-fetch/` — 浦发银行 600000 unadjusted via tencent: 117 bars
(2024-01-02 6.63/6.60/vol 22,066,700 → 2024-06-28), raw object
`objects/raw_600000_<hash12>.json` + normalized CSV + manifest.
- `real-backtest/` — host run: 16 fills (900×16, lot-compliant), final equity
118905.65, MDD -0.0525, Sharpe 2.82, initial_cash 100000 recorded, positions
per bar, peak RSS ~157-165 MB.
- `docker-run/` — container backtest (`--network none`, uid 10001, read-only,
cap-drop ALL, tmpfs /tmp) → identical metrics to host; container fetch run
produced object hash a63c2c71f518 identical to host → same-object sharing.
- `docker-run/request-benchmark.json` + `output/result.json` — honest
benchmark contract verified in real container run: benchmark starts at
100000.0 and ends 124696.97 (normalized), raw close 8.23 kept separately in
`equity[-1].closes['600000']`, warning "benchmark normalized to initial capital".
## Tests
`tests/worker/` — 27 tests, all synthetic fixtures carry `_synthetic`/`synthetic`
markers and are never production fallback: normalization correctness/rejection,
hash stability, deterministic accounting (hand-verified commission/slippage/
next-bar fills), named-feed isolation, benchmark series, error paths (syntax,
missing Strategy class, missing data, lookahead-at-end), catalog concat identity
regression, auto-fallback honest warning, explicit-source no-fallback, CLI
provider/raw-retention/error paths. All providers fully mocked in unit tests;
no network. Run: `.venv/bin/python -m pytest tests/worker -q`.
Limitations (honest): eastmoney currently blocked from this host; tencent lacks
listed-ETF daily adapter; index via tencent labeled 前复权; suspension/price
limits/intraday sequencing/liquidity NOT modeled; ETF sell lot approximated as
lot-100 buy rule; T+1 enforced at daily-bar granularity (no intraday detail);
search CLI uses catalog endpoints that can be slow/blocked (backend should call
with bounded timeout and show failure UI). No fake data or fabricated metrics.
|
