1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
|
# Recovery 01 plan: 159399 listed-ETF ingestion and failure semantics
Evidence-based plan. All live evidence below was collected 2026-09-17 with bounded
real calls (single attempts, ≤15s timeouts, no retries). Skill loading is recorded
in the session transcript: systematic-debugging, test-driven-development,
writing-plans were invoked via the native skill tool before this plan; if not
visible, fallback statement: skills were loaded with the skill tool and their
content is authoritative for this task.
## Symptom (reproduced)
User defect: cash-flow ETF 159399, cn, 2025-12-31..2026-09-17, daily, unadjusted
fails with `eastmoney failed (ConnectionError: RemoteDisconnected ...)` then
`tencent fallback failed (tencent source has no listed-ETF daily adapter)`.
## Root cause (Phase 1 evidence)
1. **Eastmoney kline host is genuinely unreachable from this environment.**
Direct replay of exactly what akshare 1.18.94's `fund_etf_hist_em` sends
(`https://push2his.eastmoney.com/api/qt/stock/kline/get`, secid 0.159399,
both with no User-Agent and with a browser UA) →
`ConnectionError: RemoteDisconnected('Remote end closed connection without response')`
in both cases. Hypotheses (wrong UA, missing header) ruled out. No code bug in
our worker path — `_fetch_eastmoney` is correct; the upstream endpoint is
refusing/dropping our connections.
2. **Tencent genuinely has no listed-ETF daily adapter.** Verified against the
installed akshare 1.18.94 surface: only
`fund_etf_hist_em`, `fund_etf_hist_min_em`, `fund_etf_hist_sina` exist.
The existing error at `worker/data.py:124` is accurate, not a bug.
3. **A genuine alternative provider exists: sina.** Live probe
`ak.fund_etf_hist_sina(symbol="sz159399")` → DataFrame, 381 rows,
columns `date,open,high,low,close,volume,amount,postVol,postAmt`,
prices decimal CNY, **volume unit is 股 (shares)** — cross-verified live on the
same session: for stock 000001 2026-09-16 tencent reports volume 949,626 (手)
while sina reports 94,962,632 (股), a consistent ×100, with identical turnover.
Returns full history (no date-range parameter); the requested window must be
sliced locally. Unadjusted only (no adjust parameter).
4. **Identity.** `split_identity("159399")` on an cn ETF already resolves to
SZ#159399 (6-digit code not starting 3/6/9 → SZ). Preserved.
5. **Resulting behavior.** With source=auto, eastmoney fails → tencent ETF
fallback is a designed explicit rejection → `provider_unavailable` failure.
The failure message is truthful but the product is unusable for ETFs while
eastmoney is down even though a genuine provider exists and is live.
## Adjustment behaviors per provider (verified, not assumed)
- eastmoney `fund_etf_hist_em`: adjust none|qfq|hfq — currently unreachable.
- sina `fund_etf_hist_sina`: unadjusted only; no date params; volume in 股.
- tencent: no listed-ETF daily adapter (explicit rejection).
## Chosen repair (smallest correct change, worker only)
Backend inspection: `server/src/datasets.rs` validates frequency/adjustment/
asset_type but does not pass or validate a `source` field; jobs.rs builds the
fetch request without it. So the fix is entirely in the worker:
1. Add `_fetch_sina` (asset_type etf, frequency daily, adjustment none only).
Local slice to requested dates; keep provider numbers verbatim (volume 股);
attach `source_warnings` recording units and that eastmoney-style adjustments
are not available from sina.
2. Register `sina` in `SUPPORTED_SOURCES` and in worker `ALLOWED_SOURCES` so it
is both explicit and honestly reportable. auto chain for ETF becomes
eastmoney → sina (same symbol, honestly labeled provider_fallback + units
warning). qfq/hfq with eastmoney down still fails honestly (sina cannot serve).
3. Explicit-source semantics unchanged: `source=tencent` + etf still raises the
accurate `source_unavailable` message.
## TDD steps
Each step: failing test → run → minimal code → run.
1. Test: sina adapter normalizes a live-captured-shape sina frame
(fixture labeled synthetic) for 159399 → SZ#159399, sliced to requested
range, provider="sina", endpoint="fund_etf_hist_sina", units warning present.
2. Test: sina rejects adjustment qfq/hfq and non-daily frequency with explicit
`unsupported_*` errors.
3. Test: auto ETF chain — eastmoney monkeypatched to fail → served by sina with
`provider_fallback` warning naming eastmoney/sina honestly.
4. Test: explicit sina request for stock/index rejected (out of verified scope).
5. Run full worker suite; then live bounded end-to-end run of the exact
requested dataset via `python -m worker.main fetch` into an isolated output
dir; report exact rows/coverage/source/units; no production touched.
## Constraints honored
- All network calls bounded (≤15s per call, one call per attempt, 180s wall
clock via SIGALRM already in main.py). No retries, no auto-refetch of user
data, no production writes, no deployment, no commits.
- The user's failed dataset is never mutated; coverage/consumer decisions
reported truthfully (2026-09-17 has no EOD bar yet — intraday at report time).
|
