1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
|
# Recovery 01 results — real listed-ETF ingestion and failure semantics
Task: docs/recovery-task-01.md. Date: 2026-09-17. Scope: this repo. No
deployment, no production image replacement, no commits, no user DB access,
no restarts. The user's failed dataset was not refetched or mutated.
## Skills actually loaded (native skill tool calls, this session)
1. systematic-debugging — loaded before any fix attempt.
2. test-driven-development — loaded; RED→GREEN cycles recorded below.
3. writing-plans — loaded; plan written to docs/recovery-01-plan.md.
4. verification-before-completion — loaded before this handoff.
No fallback to manual SKILL.md reads was needed. Discovery succeeded.
## Root cause (with real upstream evidence)
- Symptom reproduced live first attempt: `ak.fund_etf_hist_em(symbol="159399",
start 20251231, end 20260917)` →
`ConnectionError: RemoteDisconnected('Remote end closed connection without response')`.
- Direct replay of the exact endpoint akshare calls
(`https://push2his.eastmoney.com/api/qt/stock/kline/get`, secid `0.159399`)
with no UA and with a browser UA both failed identically → the eastmoney
kline host is genuinely refusing/silently dropping connections from this
environment. Not a UA/timeout bug, not a code bug in the worker.
- Verified against installed akshare 1.18.94: there is no tencent listed-ETF
daily adapter (`fund_etf_hist_em`, `fund_etf_hist_min_em`,
`fund_etf_hist_sina` are the only ETF end-of-day ETF adapters). The
pre-existing `source_unavailable` message at worker/data.py:124 is accurate.
- Live-verified alternative provider: `ak.fund_etf_hist_sina(symbol="sz159399")`
→ 381 rows (2025-02-27..2026-09-16), columns
`date,open,high,low,close,volume,amount,postVol,postAmt`, unadjusted, no date
parameters (full history; must be sliced locally), volume unit 股 (shares).
- Units cross-verified live on the same session with stock sz000001 2026-09-16:
tencent volume 949,626 (lots, 手) vs sina 94,962,632 (shares, 股) — exact ×100,
identical turnover (1,106,652,900 vs 1,106,652,940). Hence honest units
warnings on sina output rather than silent rescaling.
## Change (smallest correct; worker + trivial frontend label)
- `worker/data.py`:
- new `_fetch_sina`: etf-only, daily-only, adjustment none-only (explicit
`unsupported_*` rejections otherwise), sina symbol `sz159399`, full-history
fetch sliced locally to the requested window, canonical identity
`SZ#159399` preserved, verbatim provider numbers, source_warnings with
slicing/units/unadjusted disclosure.
- `SUPPORTED_SOURCES` now `{eastmoney, tencent, sina, auto}`; auto chain for
ETF: eastmoney → sina (honestly labeled `provider_fallback` warning naming
both providers); stock/index auto chains unchanged (eastmoney → tencent).
- explicit `source=tencent` + etf still raises the unchanged, accurate
"tencent source has no listed-ETF daily adapter".
- `worker/main.py`: `ALLOWED_SOURCES` += `sina`.
- `frontend/src/lib/format.ts`: `sina` → `新浪` source label (display only,
no UI redesign). Backend (server/) unchanged: it never passes or validates
a source field (datasets.rs validates frequency/adjustment/asset_type only).
## TDD evidence
Tests added in `tests/worker/test_data.py` (synthetic fixtures are clearly
labeled synthetic; live-captured provider shapes are quoted in comments):
- test_sina_etf_serves_159399_with_honest_provenance
- test_sina_rejects_adjustment_and_range_semantics
- test_sina_rejects_stock_and_index
- test_auto_etf_falls_back_from_eastmoney_to_sina_honestly
- test_auto_etf_fails_when_no_provider_is_viable
- test_tencent_etf_rejection_message_unchanged
- test_sina_registered_in_explicit_source_surface
RED: 6 of the above failed against the unmodified worker (the tencent
rejection test passed as pre-existing behavior). GREEN after implementation.
## Verification commands and fresh results
- `python -m pytest tests/worker -q` → **56 passed** (0.79s), final run.
- `python -m pytest tests -q` → **56 passed** (0.72s), final run.
- `npm run check` (frontend) → **0 errors, 0 warnings**.
- `npm test` (frontend) → **13 files / 45 tests passed**.
- Live exact-request fetch (real upstreams, single bounded attempts,
SIGALRM 180s wall clock):
`python -m worker.main fetch --request artifacts/recovery-01/req159399.json
--output artifacts/recovery-01/etf_out` → **status: ready**, ~0.9s.
## Exact reported dataset facts (159399, requested 2025-12-31..2026-09-17)
- provider **sina**, endpoint `fund_etf_hist_sina`, params `{"symbol": "sz159399"}`
- row_count **173**, actual_start **2025-12-31**, actual_end **2026-09-16**
- columns: `date,open,high,low,close,volume,amount,symbol` (amount requested)
- adjustment **none (unadjusted)**; prices CNY decimals; **volume in 股
(shares)** — 100× the lot convention of eastmoney/tencent (verified ×100 on
the same session, not silently converted)
- 2026-09-17 has **no bar**: today's session end-of-day not yet published; the
warning "actual coverage 2025-12-31..2026-09-16 differs from request; gaps
kept" is on the manifest. All calendar gaps present-truthful (holidays etc.).
- identity preserved all the way through: normalized symbol `159399` CSV and
canonical `SZ#159399` contract in adapter tests.
- fetch → persisted raw JSON (`objects/raw_159399_*.json`) + normalized CSV
(`objects/159399.csv`) + preview contract in `result.json` (columns,
row_count, coverage, last-20 rows), exercised outside production in
`artifacts/recovery-01/etf_out/`.
## Comparison cases (truthful, live attempts)
- stock 600000 auto (unchanged path): attempt 1 failed (eastmoney down +
tencent proxy.finance.qq.com connect timeout); attempt 2 → **ready**
provider **tencent**, endpoint stock_zh_a_hist_tx, 173 rows,
2025-12-31..2026-09-16.
- index 000300 auto (unchanged path): attempts 1–2 failed — attempt 1 tencent
proxy connect timeout; attempt 2 eastmoney RemoteDisconnected + tencent
returned an empty payload to akshare ("Length mismatch"). Environmental,
pre-existing flakiness; not caused by this change and out of this task's
repair scope.
- 159399 with source=tencent: explicitly, correctly rejected
(source_unavailable). 159399 with qfq: with eastmoney down auto now fails
honestly because sina has no adjustment factors (tested).
## Remaining blockers
1. eastmoney push2his connectivity from this environment is down (affects
stock/index first choice too); recovery is environmental, workarounds live
in fallback chains.
2. tencent proxy.finance.qq.com flaky/intermittent under this environment;
index fetch unverified live in this session.
3. Full release gate (docs/skill-assisted-development.md) is a separate
acceptance workflow; this task only fixes worker ETF ingestion and its
failure semantics. Parent independent verification still required.
## Changed files
- worker/data.py
- worker/main.py
- tests/worker/test_data.py (new tests only)
- frontend/src/lib/format.ts (one-line source label)
- docs/recovery-01-plan.md (new)
- docs/recovery-01-results.md (this file, new)
- artifacts/recovery-01/ (isolated live-fetch artifacts: 3 request files,
etf_out/, stock_out/, index_out/)
|
