# Recovery 01 results — real listed-ETF ingestion and failure semantics

Task: docs/recovery-task-01.md. Date: 2026-09-17. Scope: this repo. No
deployment, no production image replacement, no commits, no user DB access,
no restarts. The user's failed dataset was not refetched or mutated.

## Skills actually loaded (native skill tool calls, this session)

1. systematic-debugging — loaded before any fix attempt.
2. test-driven-development — loaded; RED→GREEN cycles recorded below.
3. writing-plans — loaded; plan written to docs/recovery-01-plan.md.
4. verification-before-completion — loaded before this handoff.
No fallback to manual SKILL.md reads was needed. Discovery succeeded.

## Root cause (with real upstream evidence)

- Symptom reproduced live first attempt: `ak.fund_etf_hist_em(symbol="159399",
  start 20251231, end 20260917)` →
  `ConnectionError: RemoteDisconnected('Remote end closed connection without response')`.
- Direct replay of the exact endpoint akshare calls
  (`https://push2his.eastmoney.com/api/qt/stock/kline/get`, secid `0.159399`)
  with no UA and with a browser UA both failed identically → the eastmoney
  kline host is genuinely refusing/silently dropping connections from this
  environment. Not a UA/timeout bug, not a code bug in the worker.
- Verified against installed akshare 1.18.94: there is no tencent listed-ETF
  daily adapter (`fund_etf_hist_em`, `fund_etf_hist_min_em`,
  `fund_etf_hist_sina` are the only ETF end-of-day ETF adapters). The
  pre-existing `source_unavailable` message at worker/data.py:124 is accurate.
- Live-verified alternative provider: `ak.fund_etf_hist_sina(symbol="sz159399")`
  → 381 rows (2025-02-27..2026-09-16), columns
  `date,open,high,low,close,volume,amount,postVol,postAmt`, unadjusted, no date
  parameters (full history; must be sliced locally), volume unit 股 (shares).
- Units cross-verified live on the same session with stock sz000001 2026-09-16:
  tencent volume 949,626 (lots, 手) vs sina 94,962,632 (shares, 股) — exact ×100,
  identical turnover (1,106,652,900 vs 1,106,652,940). Hence honest units
  warnings on sina output rather than silent rescaling.

## Change (smallest correct; worker + trivial frontend label)

- `worker/data.py`:
  - new `_fetch_sina`: etf-only, daily-only, adjustment none-only (explicit
    `unsupported_*` rejections otherwise), sina symbol `sz159399`, full-history
    fetch sliced locally to the requested window, canonical identity
    `SZ#159399` preserved, verbatim provider numbers, source_warnings with
    slicing/units/unadjusted disclosure.
  - `SUPPORTED_SOURCES` now `{eastmoney, tencent, sina, auto}`; auto chain for
    ETF: eastmoney → sina (honestly labeled `provider_fallback` warning naming
    both providers); stock/index auto chains unchanged (eastmoney → tencent).
  - explicit `source=tencent` + etf still raises the unchanged, accurate
    "tencent source has no listed-ETF daily adapter".
- `worker/main.py`: `ALLOWED_SOURCES` += `sina`.
- `frontend/src/lib/format.ts`: `sina` → `新浪` source label (display only,
  no UI redesign). Backend (server/) unchanged: it never passes or validates
  a source field (datasets.rs validates frequency/adjustment/asset_type only).

## TDD evidence

Tests added in `tests/worker/test_data.py` (synthetic fixtures are clearly
labeled synthetic; live-captured provider shapes are quoted in comments):

- test_sina_etf_serves_159399_with_honest_provenance
- test_sina_rejects_adjustment_and_range_semantics
- test_sina_rejects_stock_and_index
- test_auto_etf_falls_back_from_eastmoney_to_sina_honestly
- test_auto_etf_fails_when_no_provider_is_viable
- test_tencent_etf_rejection_message_unchanged
- test_sina_registered_in_explicit_source_surface

RED: 6 of the above failed against the unmodified worker (the tencent
rejection test passed as pre-existing behavior). GREEN after implementation.

## Verification commands and fresh results

- `python -m pytest tests/worker -q` → **56 passed** (0.79s), final run.
- `python -m pytest tests -q` → **56 passed** (0.72s), final run.
- `npm run check` (frontend) → **0 errors, 0 warnings**.
- `npm test` (frontend) → **13 files / 45 tests passed**.
- Live exact-request fetch (real upstreams, single bounded attempts,
  SIGALRM 180s wall clock):
  `python -m worker.main fetch --request artifacts/recovery-01/req159399.json
  --output artifacts/recovery-01/etf_out` → **status: ready**, ~0.9s.

## Exact reported dataset facts (159399, requested 2025-12-31..2026-09-17)

- provider **sina**, endpoint `fund_etf_hist_sina`, params `{"symbol": "sz159399"}`
- row_count **173**, actual_start **2025-12-31**, actual_end **2026-09-16**
- columns: `date,open,high,low,close,volume,amount,symbol` (amount requested)
- adjustment **none (unadjusted)**; prices CNY decimals; **volume in 股
  (shares)** — 100× the lot convention of eastmoney/tencent (verified ×100 on
  the same session, not silently converted)
- 2026-09-17 has **no bar**: today's session end-of-day not yet published; the
  warning "actual coverage 2025-12-31..2026-09-16 differs from request; gaps
  kept" is on the manifest. All calendar gaps present-truthful (holidays etc.).
- identity preserved all the way through: normalized symbol `159399` CSV and
  canonical `SZ#159399` contract in adapter tests.
- fetch → persisted raw JSON (`objects/raw_159399_*.json`) + normalized CSV
  (`objects/159399.csv`) + preview contract in `result.json` (columns,
  row_count, coverage, last-20 rows), exercised outside production in
  `artifacts/recovery-01/etf_out/`.

## Comparison cases (truthful, live attempts)

- stock 600000 auto (unchanged path): attempt 1 failed (eastmoney down +
  tencent proxy.finance.qq.com connect timeout); attempt 2 → **ready**
  provider **tencent**, endpoint stock_zh_a_hist_tx, 173 rows,
  2025-12-31..2026-09-16.
- index 000300 auto (unchanged path): attempts 1–2 failed — attempt 1 tencent
  proxy connect timeout; attempt 2 eastmoney RemoteDisconnected + tencent
  returned an empty payload to akshare ("Length mismatch"). Environmental,
  pre-existing flakiness; not caused by this change and out of this task's
  repair scope.
- 159399 with source=tencent: explicitly, correctly rejected
  (source_unavailable). 159399 with qfq: with eastmoney down auto now fails
  honestly because sina has no adjustment factors (tested).

## Remaining blockers

1. eastmoney push2his connectivity from this environment is down (affects
   stock/index first choice too); recovery is environmental, workarounds live
   in fallback chains.
2. tencent proxy.finance.qq.com flaky/intermittent under this environment;
   index fetch unverified live in this session.
3. Full release gate (docs/skill-assisted-development.md) is a separate
   acceptance workflow; this task only fixes worker ETF ingestion and its
   failure semantics. Parent independent verification still required.

## Changed files

- worker/data.py
- worker/main.py
- tests/worker/test_data.py (new tests only)
- frontend/src/lib/format.ts (one-line source label)
- docs/recovery-01-plan.md (new)
- docs/recovery-01-results.md (this file, new)
- artifacts/recovery-01/ (isolated live-fetch artifacts: 3 request files,
  etf_out/, stock_out/, index_out/)