From 088735b948d46896b8af30efcb0a2dc5d362b97f Mon Sep 17 00:00:00 2001 From: "Somhairle H. Marisol" Date: Fri, 18 Sep 2026 08:27:41 +0800 Subject: docs(release): 全周期交接文档入库（含 ui-shadcn 迁移交付说明） MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [变更性质] 纯文档提交，无运行时逻辑。 [文档内容] 补齐此前各轮未入库的交接/验收文档：backend-auth/backend-domain/ domain-authorization-user（认证与授权域）、etf-recovery-release- handoff（ETF 修复 + ops 演练定稿与生产部署命令）、recovery-* 系列、 frontend/parent-ui-findings（UI 迁移上下文）、worker/integration 等， 以及本轮 docs/ui-shadcn-handoff.md（shadcn-svelte 迁移交接，含 Chart.svelte 契约、runes $state 踩坑记录与 375/768/1440 验证证据）。 [更新方案] 按主题分文；每份文档只记录可复现的命令、验证结果与语义边界， 不导出密钥或生产敏感路径。 [影响范围] 文档渠道：后续 leader/client 审阅入口；与代码提交一一对应便于回溯。 --- docs/backend-auth.md | 98 +++++++++++++++++++++ docs/backend-domain.md | 64 ++++++++++++++ docs/backend-repair-contract.md | 34 ++++++++ docs/completion-brief.md | 16 ++++ docs/completion-notice.md | 20 +++++ docs/completion-state.md | 79 +++++++++++++++++ docs/domain-authorization-user.md | 8 ++ docs/etf-recovery-release-handoff.md | 100 +++++++++++++++++++++ docs/frontend.md | 64 ++++++++++++++ docs/implementation-plan.md | 50 +++++++++++ docs/integration-handoff.md | 163 +++++++++++++++++++++++++++++++++++ docs/live-data-probe.md | 5 ++ docs/parent-ui-findings.md | 12 +++ docs/production-local.md | 13 +++ docs/recovery-01-plan.md | 90 +++++++++++++++++++ docs/recovery-01-results.md | 136 +++++++++++++++++++++++++++++ docs/recovery-task-01.md | 15 ++++ docs/search-fix.md | 111 ++++++++++++++++++++++++ docs/skill-assisted-development.md | 23 +++++ docs/ui-shadcn-handoff.md | 49 +++++++++++ docs/worker.md | 149 ++++++++++++++++++++++++++++++++ 21 files changed, 1299 insertions(+) create mode 100644 docs/backend-auth.md create mode 100644 docs/backend-domain.md create mode 100644 docs/backend-repair-contract.md create mode 100644 docs/completion-brief.md create mode 100644 docs/completion-notice.md create mode 100644 docs/completion-state.md create mode 100644 docs/domain-authorization-user.md create mode 100644 docs/etf-recovery-release-handoff.md create mode 100644 docs/frontend.md create mode 100644 docs/implementation-plan.md create mode 100644 docs/integration-handoff.md create mode 100644 docs/live-data-probe.md create mode 100644 docs/parent-ui-findings.md create mode 100644 docs/production-local.md create mode 100644 docs/recovery-01-plan.md create mode 100644 docs/recovery-01-results.md create mode 100644 docs/recovery-task-01.md create mode 100644 docs/search-fix.md create mode 100644 docs/skill-assisted-development.md create mode 100644 docs/ui-shadcn-handoff.md create mode 100644 docs/worker.md diff --git a/docs/backend-auth.md b/docs/backend-auth.md new file mode 100644 index 0000000..a26cdbc --- /dev/null +++ b/docs/backend-auth.md @@ -0,0 +1,98 @@ +# backend-auth.md — AUTH worker handoff (auth.rs + admin.rs) + +Status: both modules compile clean (`cargo check` shows zero diagnostics for +auth.rs/admin.rs, lib and test profiles). 9 security unit tests exist inside +the two modules (they compile; running the full `cargo test` target is still +blocked by compile errors in CORE/DOMAIN modules: store.rs:179, jobs.rs, main.rs, +ai.rs, datasets.rs — not AUTH-owned. Parent final compile gate will unblock them.) + +## What is implemented + +### auth.rs +- `pub const COOKIE_NAME = "sl_session"`. +- `AuthUser` extractor: reads cookie, hashes the token with sha256, resolves + `sessions JOIN users` requiring unexpired session and `users.active=1`. + Rejected tokens return 401; disabled accounts lose the session immediately. +- Argon2id password hash/verify (`hash_password`, `verify_password`), fresh + random salt per call. +- Sessions: random 256-bit token; **only `sha256(token)` is stored** (sessions.id). + Cookie is HttpOnly + SameSite=Strict + Path=/ + Max-Age, plus `Secure` when + `cfg.secure_cookies` (set from https canonical origin). +- `make_admin_token(cx, table, user_id, hours, email)` — shared factory for + invitations and password resets: raw token shown once, only the sha256 is + inserted; invitations are always stored as role 'member'. +- `purge_expired` — housekeeping for expired sessions/invitations/resets and + 2-hour-old login failures. +- Handlers: + - `POST /auth/login` — trims/lowercases email, per-email failure throttle + (>20 failures in 1h ⇒ 429), verifies Argon2 hash, rejects `active=0` + accounts with 403, issues session, writes audit. + - `POST /auth/register` — **single-use atomically consumed invitation with + email binding**: inside ONE closure transaction it checks the token exists, + role is 'member', bound email matches (case-insensitive) when set, expiry; + inserts the user with role='member' (the struct has no role field, so + client-supplied `"role":"admin"` is ignored by serde en bloc) and marks the + invite used via guarded `UPDATE ... WHERE used_by IS NULL`, rejecting the + second concurrent attempt with 409. Duplicate emails → 409 `email_taken`. + - `POST /auth/logout` — deletes the server-side session row, clears cookie. + - `GET /auth/me`, `PATCH /auth/profile` — owner-scoped profile read/update, + no password hash/secret in payload. + - `POST /auth/password` — verifies current password; updates hash and + revokes ALL other sessions in one transaction, keeping current session. + - `GET /auth/sessions` — sanitized own-session listing (sha256-derived ids, + no raw tokens, expired rows never listed). + - `DELETE /auth/sessions/:id` — revokes an OWN session only; foreign id ⇒ 404. + - `POST /auth/reset-password` — single-use token, expiry check, marks + `used=1`, updates password hash and deletes ALL sessions of the user + inside ONE transaction (replay of the token loses the update race). + +### admin.rs +- `assert_admin(&Cx, &AuthUser)` guards every admin handler (member ⇒ 403). +- `GET /admin/users` — full user table (no password hashes). +- `PATCH /admin/users/:id` — accepts {active?, role?, daily_run_limit?, + ai_enabled?}; validates role admin|member and 1..=2000 limit; **prevents + demotion/disable of the LAST active admin atomically** (count check inside + the same transaction as the update, code `last_admin` 409); disabling + revokes all sessions and cancels that user's queued/running runs in the + same transaction. +- `POST /admin/invitations` — POC invites mint member only (admin role via + invitation rejected 400); token returned once, sha256 stored. +- `GET /admin/invitations` — sanitized list, no token hashes. +- `DELETE /admin/invitations/:id` — revoke; 404 when absent. +- `POST /admin/users/:id/reset-password` — verifies target user exists, issues + a hashed single-use short (2h) reset token returned once; audit event w/o + secrets. Honest UI contract: admin-issued recovery token, no fake email. +- `GET /admin/audit` — sanitized rendering (ts/actor/action/target/status); + `auth::audit` refuses oversized/multiline targets (stored as + `redacted-oversized`), so no password/key/code content can be persisted. +- `bootstrap_admin(&Cx)` — idempotent first-admin bootstrap from env, Argon2 + hashed, email normalized; no-op when any admin exists. + +## Unit tests (inside the modules, RED→GREEN verified by compile-gate) +auth: argon2 hash/verify; register member-only + role input ignored + +payload-no-hash; invitation single-use; invitation email binding (and token +not consumed by the failed attempt); reset single-use + session revocation + +password actually rotated; change-password revokes others/keeps current + +wrong current rejected; audit never records secrets; cookie sessions never +store plaintext tokens + cookie flags; login throttle; logout invalidates +server-side session. +admin: member 403 on users/audit/invitations; last-admin disable/demote +blocked then allowed with a second admin; disable revokes sessions and +cancels runs; invitations only mint members, token hashed in DB and sanitized +listing; bootstrap idempotent + Argon2 hash. + +## Coordination notes for other workers / parent +1. db.rs schema is untouched (AUTH owns no db changes) and is fully sufficient. +2. qa_api.py gates covered by AUTH handlers: register ignores supplied admin + role, invitation single-use, admin-role members cannot enumerate users, + last admin cannot be disabled, admin reset flow + single-use + session + revocation, disabled account login/session denial, invite email binding. + CSRF substring/foreign-origin gates live in main.rs (CORE); private-object + gates (admin cannot read private project, cross-member denial) in + projects.rs — verify DOMAIN parts at the parent gate. +3. `AuthUser: FromRequestParts>` matches the existing router + `.with_state(Arc)`; handlers across the codebase may keep either + `cx: Cx` (State extractor) or the bare Arc — both remain valid for this + router. +4. No cross-module edits made by AUTH; `make_admin_token` async signature is + coherent for all admin.rs call sites. diff --git a/docs/backend-domain.md b/docs/backend-domain.md new file mode 100644 index 0000000..3544bc9 --- /dev/null +++ b/docs/backend-domain.md @@ -0,0 +1,64 @@ +# Backend domain handoff (projects / datasets / runs / ai / jobs + db.rs migrations) + +Worker scope: server/src/projects.rs, datasets.rs, runs.rs, ai.rs, jobs.rs, db.rs (migrations only). +Used fixed contract: `state.rs` `Cx = axum::extract::State>`, `AppState.with_db` one-lock closures, `S = Arc`. No nested locks, no awaits inside DB closures. + +## What was repaired + +### projects.rs +- Handlers on `Cx` (State extractor); all DB closures return `Rows`/`Result` types explicitly; `rows.next()?` loop pattern (no `MappedRows` collect ambiguity). +- `put_draft` optimistic generation guard returns 409 `stale_generation` with `current_generation` details; update+read run inside ONE `with_db`. +- Versions immutable; run snapshots reuse identical `hash` versions from the same project; restore creates new draft (`draft_generation + 1`), never rewrites history; `source` in {manual, run, ai, restore} enforced by schema CHECK. + +### datasets.rs +- Request validation: 1–5 instruments, daily only, none|qfq|hfq, field whitelist (open/high/low/close/volume/adj_factor), ISO dates, ≤15y, duplicate-instrument rejection, **index + qfq/hfq explicitly rejected** ("indexes have no adjustment factors"). +- Cache key = sha256(canonical_request) over instrument identities + range + fields + freq + adjustment; dataset display name excluded (identity is semantic). +- Client responses strip internal `path` fields and `preview` from manifests (`client_manifest`); no host paths leak (QA check `no server paths in manifest`). +- `assert_owned` owner check for every dataset read/preview; `load_manifest` requires status='ready'. + +### runs.rs +- POST /runs: dataset must exist, be owned by caller, be ready; explicit warning acknowledgement (`acknowledge_warnings`) required when manifest warnings non-empty; **index-feed restriction surfaced as an explicit warning requiring acknowledgement**; numeric bounds (capital 1..1e12, commission/slippage 0..0.05, parameters size cap); per-user daily quota enforced transactionally inside one `with_db` closure (`BEGIN IMMEDIATE` + count + insert). +- Draft snapshot pinned to run (`source='run'`); rerun requires `use_original_data:true`, pins original version/config/manifest_hash, never refetches ("no secret fetching latest"), verifies the original dataset still exists and is ready/owned. +- cancel → `jobs::signal_cancel` kills the specific container by name; **terminal states guarded by `AND status='running'`/`'queued'` so failed/cancelled are never overwritten by racing completion**. +- cleanup_interrupted marks running→failed on restart, queued stays resumable. + +### ai.rs +- Owner-scoped project fetch (`AND user_id=?`); requires account `ai_enabled` and POC flag `ai_enabled_poc`; `expected_generation` guard. +- Daily request budget measured from real ledger (ai_requests of today), capped `ai_daily_request_cap`; REAL call to `{AI_BASE_URL}/chat/completions` (model `glm-5.3-flash`), key only from `OPENCODE_GO_API_KEY`, 90s timeout, output token cap. +- Robust fenced-block parse (`extract_code_block`, first non-empty fenced block; mid-response fences ignored), explanation outside fences; parsing failures recorded (`ai_requests.status='failed', error=...`) — failure ledger retained. +- Response records real usage (`prompt_tokens`/`completion_tokens`) into `ai_usage` (no price/cost invented); input token cap enforced before accept. +- accept: owner check, base_hash identity + `base_generation` + `expected_generation` all verified inside ONE transaction; writes draft and `source='ai'` version atomically. Renewal after restore always yields new generation. +- summarize_dataset only reads ready datasets the user owns and that actually have runs; only columns/heading metadata to the model, not payloads. + +### jobs.rs +- Dispatcher: bounded dataset fetch concurrency (`fetch_concurrency`), single backtest worker (`run_concurrency=1` POC). +- **Exact-source request cache**: new `fetch_cache` table keyed on canonical request (contract: interval coverage; content-derived manifest hash stable across users/datasets; worker not re-run for identical requests). +- Concurrent identical cache keys serialize through a process-wide in-flight set; the waiting job rechecks cache periodically (required concurrent same-key serialize/recheck). +- Fetch flow: `docker run` data worker (network on, core `run_named`), request.json = canonical stored request; artifacts ingested via core `store::ingest_directory` (symlink/traversal rejection, content-hash dedup into immutable objects); normalized object paths rewritten `objects/{hash-prefix}/{hash}.{ext}`; **raw object verified present by content hash (raw ObjectHash) — raw retained, never dropped**. +- Manifest hash recomputed server-side over canonical JSON **excluding fetched_at and path → stable content identity independent of user/request IDs/fetch timestamps**; identical data reused across users yields identical hash (QA `shared immutable data hash`). +- Worker JSON `status` gate: dataset/run only succeeds when worker reports ready/succeeded; worker failure/stderr (bounded) surfaced ~1.2KB. +- Queued run dequeue rechecks account active + daily budget + dataset ownership/readiness (`BEGIN IMMEDIATE`) then claims queued→running within the same transaction; unreachable due to `db.changes()`-race free guarded update. +- Result: `data_manifest_hash` pinned; `sanitize_nonfinite` removes NaN/Inf (metrics can be null); warnings carried; elapsed/peak RSS from worker; engine name/version from worker. +- Container output safety per job: `data_dir/jobs/{uuid}` chmod 0700, `output/` chmod 0777 (uid 65534 writable). Backtest mounts: input ro, dataset objects ro at /data, output rw; no secrets; timeout/cancel kills by specific container name (`worker::cancel_container`). + +### db.rs migrations (mine) +- Added `fetch_cache(key PK, manifest_hash, manifest, object_count, fetched_at)`. +- Index `runs_user_daily(user_id, created_at DESC)` for quota counts. + +## Domain tests (this module) +`cargo test` (all passing; remaining failures live in auth/store/util/admin files run concurrently by other workers): +- datasets: `validate_rejects_unsupported` (freq/adjustment/index restriction/duplicates), `cache_key_is_stable_and_shared_regardless_of_display_name`, `client_manifest_strips_internal_paths_and_preview` +- jobs: `manifest_hash_is_content_identity_not_fetch_metadata`, `ingest_rewrites_paths_and_hash_is_stable` (synthetic worker output fixture, raw kept, dedup identity stable), `cache_key_treats_request_semantics_not_names`, `nonfinite... via runs`, `signal_cancel_never_rewrites_terminal_states` +- runs: `config_bounds_enforced`, `nonfinite_sanitized` +- ai: `extract_takes_first_fenced_full_python_block` (robust fenced parse; empty/no-fence rejected), `usage_totals_never_invent_costs` + +## Worker protocol notes (for parent/worker owner) +- `worker/main.py` referenced `_write_raw(...)` but never defined it — every fetch would crash (NameError). I added the minimal faithful implementation (main.py, worker protocol): writes immutable raw provider response JSON (`{endpoint, params, fetched_at, data}`) to `objects/raw_{slug}_{hash12}.json`, returns (name, sha256-of-bytes). This matches SPEC "raw JSON actual source response" and lets server ingest raw artifacts by content hash. The worker owner should review/align naming if they implement their own variant. +- Worker fetch protocol consumed (current main.py): reads `/input/request.json` with `instruments/start_date/end_date/frequency/adjustment/fields`; writes `output/result.json` `{status: ready|failed, manifest{schema_version, normalization_version, fetched_at, frequency, adjustment, objects[...]}, hash, preview{columns,rows,coverage}, warnings, elapsed_ms}`; objects' `path` is worker-relative (`objects/{slug}.csv`) and is rewritten by the backend before storage/manifest display. +- Backtest input `{code, config, dataset_manifest{objects[path=/data/]}, data_root:'/data'}` only normalized CSVs mounted; feed names from `instrument.symbol`; result contract per SPEC. +- Instrument search: core `worker::search_instruments` already talks to the real `worker.main search` catalog (AKShare). Domain routes read it via main.rs; failures surface as `status:"unavailable: ..."` (no fake success). + +## Not known / for parent +- qa_api.py needs QA env credentials envs; prepared by parent. +- `clean_orphan_containers` restart cleanup lives in main.rs (core); my signal_cancel/cleanup interplay verified for guarded statuses. +- Object store retention/cold storage documented in docs/worker.md (worker owner); server keeps immutable referenced objects and runs pin them. diff --git a/docs/backend-repair-contract.md b/docs/backend-repair-contract.md new file mode 100644 index 0000000..2b02703 --- /dev/null +++ b/docs/backend-repair-contract.md @@ -0,0 +1,34 @@ +# Backend repair contract (mandatory for all repair workers) + +Prior broad edits introduced cascading signature errors. Parent stopped the old backend process. Repair in narrow file ownership; other workers may be editing their own files. Do NOT edit outside your allocation. Do not weaken tests, delete API handlers, or change language/framework. + +## Fixed signatures +- state.rs exports `pub type Cx = axum::extract::State>;` +- AppState owns cfg:Config, db:tokio::sync::Mutex, run_sem/fetch_sem Arc (preserve other needed state). +- `impl AppState { pub async fn with_db(&self, f: impl FnOnce(&mut rusqlite::Connection) -> R) -> R { let mut db = self.db.lock().await; f(&mut db) } }`. +- This returns exactly R after await. If closure returns Result, use `.await?`; if closure returns number/unit, `.await` only. Never await inside database closure. Keep transactions and atomic operations inside ONE closure. No nested locks. +- Handler argument is `cx: Cx` (State extractor), not a bare Arc alias. Helper taking `&Cx` receives `&cx`. Background jobs may use Arc and create `State(cx.clone())` only where necessary. All handlers futures must be Send. +- util::now_iso() -> String is SYNCHRONOUS. hash/random token helpers synchronous unless they actually await. +- error::AppError supports new(StatusCode,code,message), bad(code,message), unauthorized(message), forbidden(message), not_found(message), conflict(code,message), internal(message), with_details(Value), with_code(code). One inherent impl, one IntoResponse, one From, one From, one From. Never duplicate From impl elsewhere. +- `auth::make_admin_token` existing admin call sites need coherent async Result signature; auth/admin worker owns both. +- Router Axum0.8 paths use `{id}`, NEVER `:id` (runtime panic). +- Do not change actual JSON HTTP contract: SPEC.md + RELEASE_SCOPE.md authoritative. + +## Worker partition +CORE: config.rs state.rs error.rs util.rs main.rs store.rs worker.rs Cargo.toml (+ core tests). Main integrates call signatures but not other module sources. +AUTH: auth.rs admin.rs (+ auth tests inside module). +DOMAIN: projects.rs datasets.rs runs.rs ai.rs jobs.rs (+ domain tests inside module). +Read other files freely. Compile often, filter diagnostics for OWN modules; leave other-worker diagnostics alone. Final shared compile gate performed by parent. + +## Parent findings to address +- main.rs origin substring match is exploitable. Compare exact canonical origin (scheme+host+port), reject `https://allowed.example.evil.invalid`. Empty canonical allows exact local origin only. No trust of X-Forwarded host. +- Serve frontend/dist static SPA with index.html fallback for UI only; /api unknown returns JSON 404 (not SPA). +- Instrument search current stub must connect to actual worker catalog command; no fake successful empty list. Docker worker fetch/search with bounded time and resource. +- Concurrent fetch duplicate cache key must serialize/recheck. request/cache key and manifest hash are distinct columns/concepts. +- All raw+normalized referenced artifacts retained; worker manifest fields must be checked against live worker code. +- Read-only nonroot container UID65534 needs writable output (parent job dir0700, output mode0777 inside; no secrets). Input/data are read-only. Stdout/stderr bounded and drained; timeout/cancel container cleanup real. +- Finished run state only succeeded if worker JSON status says succeeded. Failed/cancelled state must not be overwritten by racing completion. +- Durable queued jobs recheck account active, dataset ownership/readiness, per-user budget. +- Invitation email binding, one-time transaction, no role escalation; disable/reset revoke sessions; last admin protection. + +Parent independent API tests live at scripts/qa_api.py, including substring CSRF and admin/private owner separation. READ these gates and make them genuinely pass. Parent prepares isolated QA environment, do not read any secrets outside project. diff --git a/docs/completion-brief.md b/docs/completion-brief.md new file mode 100644 index 0000000..103ab3e --- /dev/null +++ b/docs/completion-brief.md @@ -0,0 +1,16 @@ +# Autonomous completion brief +User authorizes finishing Strategy Lab development/deployment using OpenCode GLM-5.3-Flash; latest request: notify only when everything is finished, no routine progress messages. Work scope this repo only. Never modify other Hermes profiles or unrelated services. + +Read SPEC.md, RELEASE_SCOPE.md, docs/integration-handoff.md, docs/parent-ui-findings.md and current code. Independent latest tests: cargo49 passed, worker37 passed, frontend22 passed, svelte-check0 errors/warnings. Earlier real HTTP30 gates passed including actual Tencent行情, cross-user content cache, backtest and reproduction. Latest fixes not yet rechecked in LIVE service: strategy exec single namespace; cancellation race; AI stable session header; doubled Layout. No public deployment yet. Backend still has warnings. + +Tools: OpenCode wrapper /home/somhairle/.hermes/cache/strategy-lab-run-opencode.py with STRATEGY_AGENT_SLOT=integration; session ses_f553821d1ffeRPSHHoGHXZx6xq, model strategy-go/glm-5.3-flash. Log /home/somhairle/.hermes/cache/strategy-lab-integration.log. Check child PIDs and logs before launch: completion exit_code=None is unreliable. Only one integrator, no competing writers. Never global pkill/killall. Code workers must not touch services. Parent/supervisor alone manages exact app units. + +QA systemd user unit strategy-lab-qa runs 127.0.0.1:8787. Launcher /home/somhairle/.hermes/cache/strategy-lab-qa-service.py loads only required credentials privately, currently executes DEBUG binary: rebuild it before restart, even if release binary was built. QA DB/private creds under /home/somhairle/.hermes/cache/strategy-lab-qa/service; never print credentials or mix QA accounts into production. Test commands: launcher test --market; qa venv/bin/python browser_smoke.py; python qa_ai.py; python qa_limits.py, all under /home/somhairle/.hermes/cache/strategy-lab-qa/. Browser Playwright installed in that venv. Scripts may have test-harness mistakes: fix those honestly, never weaken acceptance. qa_limits.py has correct private-account filename/browser-account.private.json and a['project']['id'], flat run parameters. Still must repair Docker inspect: API hides container_id, read exact container_id for that run from QA sqlite read-only; names are sl-run-RANDOM UUID, not run ID. Timeout fixed by BACKTEST_TIMEOUT_SECS=60 in QA launcher; per-run timeout field unsupported. First limits test failed due real namespace bug and cancel race, now code fixed. Need independently verify failed/cancelled/timeout statuses and actual no-network, no-secret mounts/env, CPU/memory/PID/read-only container restrictions. + +AI real /ai/assist previously returned MissingSessionID. Latest ai.rs uses honest own User-Agent and stable per-owner/project x-opencode-session per official https://opencode.ai/docs/go/#where-can-i-use-it (custom coding agents allowed). Verify actual model response, Python syntax, unchanged draft before accept, accepted immutable revision, stale accept refusal, usage recorded. Internal POC only; no commercial third-party resale promise. + +Remaining acceptance: full authenticated browser path selecting/searching symbol, dates/fields/indicators/adjustment, data preview, strategy editor/version, run/result/comparison, AI, desktop/mobile no errors; screenshots vision review. Real security/cancel/timeout/restart recovery; repeat all tests after changes. Production release binary, persistent user systemd service, clean separate DB/admin account securely provisioned; documented limits, backup/restore and measured idle resources. Source/runtime docs and screenshots delivered. Never claim complete while any named criterion pending. Record progress+evidence+blockers in docs/completion-state.md EVERY tick; bounded work each tick, long OpenCode can run background but never duplicate it. No blind sleep loops. + +Public domain user originally typed fin.somhairle.bin (NXDOMAIN), release doc tentatively fin.somhairle.bid using existing somhairle.bid Tunnel; not explicitly confirmed. Do NOT silently substitute domain: establish authorization from original session history (session20260916_193353_1e3c6334) if available; otherwise mark this a genuine user decision blocker, finish all safe local work then report blocker rather than claim complete. Existing ~/.cloudflared/config.yml ingress dav:6065 git:8090 linkwarden:3000 fallback404; preserve everything. Existing /etc/systemd/system/cloudflared.service MUST retain --protocol http2. Read before append, preserve /etc/hosts and Clash Merge rules; never overwrite. Only expose after permission/security gates, verify both local and public actual endpoint. Do not expose bootstrap creds in logs/source. + +Completion protocol: only after all criteria and public target verified write docs/completion-notice.md containing concise Chinese user-facing completion message, URL, account provisioning instructions without plaintext secrets, actual tests/limitations and artifact paths. Write only verified result, never a placeholder. If truly blocked needing user input after safe work, write honest docs/completion-notice.md headed '需要你确认，尚未全部完成' with only essential question and completed evidence. A separate no-agent notifier sends file once. Then list cron jobs and pause this job by exact ID/name Strategy Lab completion supervisor. Until complete/blocker, local output only; no progress notifications. diff --git a/docs/completion-notice.md b/docs/completion-notice.md new file mode 100644 index 0000000..9793167 --- /dev/null +++ b/docs/completion-notice.md @@ -0,0 +1,20 @@ +策研 Strategy Lab 已部署：https://fin.somhairle.bid + +已用真实浏览器通过公网登录，核对 HTTPS 会话 Cookie、跨站请求拦截和发布资源。原有 WebDAV、Git、Linkwarden 入口保留，Tunnel 继续使用 http2。 + +- 功能验收：真实行情查询与数据预览、策略编辑和版本保存、实际回测及结果比较、AI 代码建议与人工接受均已在隔离 QA 中走通。生产库独立，未混入测试项目。 +- 本次重新运行：后端 54/54、行情与回测 worker 38/38、前端 39/39 测试通过；Svelte 检查 0 错误、0 警告。 +- 同一发布版本此前完成：容器隔离及取消/超时 10/10、运行中重启恢复 9/9、带真实数据的备份恢复 11/11。公网桌面和移动端页面无浏览器运行错误。 + +**登录方式** +管理员账号已创建，邮箱和初始密码保存在部署主机的私有文件： +`/home/somhairle/.local/share/strategy-lab-production/admin.private.json` + +文件权限为 0600。请在主机本地读取，首次登录后到「账户与安全」改密码；其他用户由管理员发邀请，未开放自由注册。聊天中不发送明文密码。 + +**使用边界** +当前为受邀可信用户使用的研究版本：仅日线，最多 5 个标的、15 年数据；回测单任务并发，60 秒超时。行情源可能失败并显示实际错误。Docker 隔离不等于可接纳任意陌生人的恶意代码；AI 仅按内部研究用途使用。构建仍有未使用代码和大资源包警告，手机窄表格会换行；没有做物理主机重启测试。 + +源码：`/home/somhairle/projects/strategy-lab` +运维及备份说明：`docs/production-local.md` +验收证据和截图：`artifacts/qa/supervisor-live/`，总验收记录为 `release-acceptance.json`。 diff --git a/docs/completion-state.md b/docs/completion-state.md new file mode 100644 index 0000000..f070cb0 --- /dev/null +++ b/docs/completion-state.md @@ -0,0 +1,79 @@ +# Completion checkpoint + +## Latest tick: authorized public deployment and independent acceptance +- Read explicit user authorization first; it supersedes all historical .bin/.bid blockers below. Deployed exactly https://fin.somhairle.bid to existing production127.0.0.1:8789. No application changes or OpenCode workers were needed. +- Before exposure, SHA256 matched release executable and every frontend asset to current build outputs. Changed only production ORIGIN to exact HTTPS hostname, restarted only production, verified real admin auth, Secure/HttpOnly/SameSite cookie, authenticated /me and foreign-origin403. +- Appended only exact new ingress before existing404. Cloudflare DNS command readback confirms hostname already routes to tunnel4e9190fb-4020-4a29-b106-9bd54e9e3cf5; ingress validate passes and ingress rule resolves exact loopback8789. Tunnel restarted to PID2963110; actual cmdline retains --protocol http2. Unit and /etc/hosts byte checks passed; no Clash rule changes. +- Deployment harness reached post-restart route-equality assertion and failed; immediate statuses were not saved by that first version, so no invented baseline is asserted. Subsequent independent HTTP and browser checks establish actual current dav401 (auth challenge), git200, linkwarden200 and fin health200. Harness now saves evidence before assertions for future runs; no second deployment/restart performed. +- Public Playwright form login passes, real /auth/me200, secure cookie verified, hostile Origin rejected403; first-party JS/CSS byte hashes match deployed release. Cloudflare-injected external analytics script is documented separately. Desktop1440/mobile375 DOM widths exact, zero pageerrors; mobile screenshot visually reviewed. Minor known empty-state wording says top-right while mobile button sits above-left; no functional blocker. Evidence public-verification.json, public-projects-{1440,375}.png, qa_public.py. +- Fresh full suites after exposure: backend54/54, worker38/38, frontend39/39, Svelte0errors/0warnings, all commands exit0. Logs public-final-suite/. Reparsed unchanged-artifact historical live gates: limits10/10, restart9/9, populated backup/restore11/11, browser workflow7, strategy5, comparison/AI5, sidebar5 with no browser errors. Aggregate release-acceptance.json includes exact DNS/ingress and release hashes. Real workflow stays in isolated QA; production contains no QA projects. +- Completion notice written to docs/completion-notice.md with verified delivery details and private account retrieval, without plaintext secrets. `hermes cron pause 3924ffef3d90` succeeded; exact list readback confirms supervisor paused and notifier8b222271517c remains active. Final public health200/statusok verified after pause. Notifier was not manually invoked and delivery has not been claimed. + +## Previous tick: final browser/restart verified; historical domain blocker (resolved) +- Independently executed qa_browser_workflow.py, qa_browser_strategy.py, qa_browser_compare_ai.py, qa_sidebar.py and qa_restart.py sequentially against real QA: all five commands exit0. Browser reports respectively7/5/5/5 checkpoints and zero pageerrors; restart9/9 gates, measured exact restart0.0325374s. Evidence: artifacts/qa/supervisor-live/final-suite/final-browser-restart.json and per-script logs; underlying JSON/screenshots refreshed. Aggregate reader initially assumed restart JSON was an object; corrected to actual list and verified all nine passed. +- Final screenshot vision review: desktop AI editor/diff/accept visible without overlap; mobile drawdown is populated and legible. Viewport screenshot clips scrollable content above/below intentionally; DOM checks establish no horizontal overflow. Narrow table wrapping remains documented. +- QA8787 and isolated production8789 actual health status ok/worker_available true/ai_configured true after final restart. No OpenCode writer found or launched, no application/production/tunnel changes. Prior tick full suites/builds, artifact hash equality, production persistence/backup/restore evidence remain applicable to unchanged code. +- Safe local work finished. Public exposure blocked by literal user domain fin.somhairle.bin versus unapproved candidate fin.somhairle.bid; original-session evidence in domain-authorization.md. Honest domain-decision notice written to docs/completion-notice.md; `hermes cron pause 3924ffef3d90` succeeded and independent `hermes cron list --all` readback shows exact supervisor paused. Notifier remains active; delivery not yet claimed. Verified notifier8b222271517c active, local supervisor delivery, notifier script reads this notice and deduplicates via SHA256; no manual send or notifier execution. + +## Previous tick: final suites and copied production freshness +- Independently ran complete backend54/54, worker38/38 and frontend39/39 tests; Svelte0errors/0warnings, frontend and release builds exit0. Backend still reports eight test/seven release warnings; frontend large-chunk warning remains. Actual live API --market exit0 with30/30 PASS lines and final sandbox limits exit0 with10/10 PASS lines, programmatically counted. Evidence/logs: artifacts/qa/supervisor-live/final-suite/. +- Independently SHA256-compared exact copied production release executable and every frontend file against the freshly built source artifacts: both match. Actual loopback8789 health ok, worker_available true, ai_configured true. No copied-asset update needed. No OpenCode writers found, none launched; no application/service/tunnel/production edits. +- Next bounded work: final authenticated browser workflow/strategy/comparison-AI/sidebar smoke and measured restart recovery on unchanged artifacts, preserve aggregate acceptance evidence; then write genuine domain-decision blocker notice and pause exact supervisor under brief protocol. Domain substitution remains unauthorized; no notice written this tick. Safe local checks continue; no completion claim. + +## Earlier checkpoint: independently exercised live QA +- Read completion brief, SPEC, RELEASE_SCOPE, integration handoff and parent UI findings. +- Process inspection found no OpenCode writer. OpenCode is not on PATH; explicit binary `/home/somhairle/.local/share/strategy-lab-tools/node_modules/.bin/opencode` reports 1.18.31. No new worker launched because no application code defect established this tick. +- `cargo build --manifest-path server/Cargo.toml` exited 0, seven existing dead-code warnings. Restarted only `strategy-lab-qa`; systemd reports active. Live service now uses current debug binary. +- Fixed parent-owned QA harness container lookup: query exact run container_id from QA SQLite read-only; API deliberately hides this field. +- Live limits first pass: failure retained with real error; strategy imports execute; network/host-secret checks pass; Docker actual restrictions memory 2147483648 bytes, 2 CPUs, PID limit128, network none, read-only root, uid65534. Running cancel retained as cancelled. +- Timeout reached failed with `worker container timed out after 60s and was killed`. Harness incorrectly searched only `timeout`; corrected to also accept `timed out`. Read-only DB evidence shows started/finished timestamps and reason. Full rerun subsequently hit real HTTP429 run quota after its first passed check, so DO NOT claim the entire final limits suite passed. Next tick use a fresh isolated QA account or authorized QA admin quota adjustment, never production accounts or disabling acceptance checks. +- `qa_ai.py` exited 0: all seven actual checks passed (real provider code, Python syntax, unchanged draft before accept, accepted code persisted, immutable version, stale duplicate refusal, real usage). + +## Evidence +- `artifacts/qa/supervisor-live/run-lifecycle.json`: independently read five recent run terminal states and timestamps, no credentials. +- `artifacts/qa/supervisor-live/sandbox-inspect.json`: actual Docker inspect restriction subset. +- `/home/somhairle/.hermes/cache/strategy-lab-qa/ai-checks.json`: seven passing real AI gates. +- `/home/somhairle/.hermes/cache/strategy-lab-qa/limits-checks.json`: latest rerun honestly records quota interruption, not all-green. + +## Remaining acceptance +- Limits acceptance completed this tick: final `python /home/somhairle/.hermes/cache/strategy-lab-qa/qa_limits.py` exit0, 10/10 gates. Actual failing strategy, namespace/import execution, no network, memory2GiB/CPU2/PID128/read-only/nonroot, dropped capabilities/no-new-privileges, exact selected dataset mounts, image-default-only environment, retained cancellation/timeout and both containers removed. Harness false assumptions corrected after inspecting actual Docker/source/DB: job directory UUID differs from run UUID; API sanitizes manifest paths so exact mount verification uses read-only SQLite; base Python image has public GPG_KEY, so compare exact image environment rather than rejecting any KEY name. Evidence and runnable harness copied to artifacts/qa/supervisor-live/{limits-checks.json,sandbox-inspect.json,qa_limits.py}. Raised only isolated browser QA account daily_run_limit to100 via authenticated admin API, GET readback verified. No application code edits, no OpenCode writers found, no production/service/tunnel changes this tick. +- Fresh complete unit/build/API checks completed in this tick: backend49/49, worker37/37, frontend22/22, Svelte0errors/0warnings; release and frontend builds exit0. Backend retains seven release warnings (eight test warnings), frontend large-chunk warning. Actual live API --market suite exit0, including real market fetch, cross-user immutable cache reuse, backtest equity/fills and original reproduction. Evidence: artifacts/qa/supervisor-live/fresh-suite-checks.json. +- Fresh authenticated browser route smoke completed: projects/datasets/runs/usage/account/admin without pageerror, new-project modal opened. JSON and desktop screenshots refreshed under /home/somhairle/.hermes/cache/strategy-lab-qa/. This is route smoke only; full authenticated desktop/mobile workflow and vision review still pending, as is restart recovery. No application code edits, OpenCode writers, service changes, or public exposure this tick. +- This tick independently ran `qa_restart.py` against actual running Docker strategy and restarted only strategy-lab-qa: final exit0, 8/8 checks. In-flight run retained as failed with exact restart reason/finished_at; exact container removed; draft, pre-existing project, ready dataset and cookie session persisted; new real backtest completed after restart. Runnable harness and results: artifacts/qa/supervisor-live/{qa_restart.py,restart-checks.json}. No OpenCode writer found and no application edits. +- Newly observed operational issue: restart during active Docker execution blocks until systemd's 90-second TimeoutStopSec; journal shows docker child in final-sigterm then SIGKILL. First harness attempt's 30-second command timeout was inadequate; extended to150 seconds and reran all checks successfully. Recovery correctness is verified, graceful shutdown latency remains unresolved. Next tick inspect runner Docker signal handling and exact QA unit; use existing OpenCode integration session if application fix needed, or narrowly scoped production unit shutdown policy. Never global kill. QA currently active/running; MemoryCurrent8593408 bytes is a single sample, not finished idle resource measurement. No service config, production or tunnel changes. +- Current tick investigated slow restart: exact QA unit is transient, has no shutdown overrides; server has no shutdown signal handler (jobs::Signals is an empty marker). Docker execution is in server/src/worker.rs. Confirmed no existing OpenCode writers before launch. Explicit OpenCode binary version1.18.31; auth list has0 stored credentials, existing wrapper injects configured provider key privately. +- Resumed ONLY existing integration GLM session ses_f553821d1ffeRPSHHoGHXZx6xq for bounded shutdown-latency root-cause repair, backend tests and debug/release builds. Wrapper PID2831501, actual OpenCode child PID2831515, terminal handle proc_2312abaa0fb5; log /home/somhairle/.hermes/cache/strategy-lab-integration-shutdown.log. Child verified alive after launch. Do not launch a competing writer. Worker explicitly forbidden from service/config/tunnel operations. No fix or test outcome claimed yet. Next tick inspect PID/log/handoff, independently rebuild/restart exact QA and rerun artifacts/qa/supervisor-live/qa_restart.py with measured elapsed time; then full affected gates. No supervisor code/service/public changes this tick. +- Current tick independently reviewed completed shutdown worker output and caught a CRITICAL introduced defect before QA deployment: main.rs timeout(10s) surrounds the entire serving future, so it starts at startup rather than after a shutdown signal. Actual release binary with isolated temporary empty DB/data, ephemeral loopback port and credential-free environment exited0 after10.025s without any signal. Evidence: artifacts/qa/supervisor-live/shutdown-premature-exit.json. Worker 51 passing unit tests did not cover server lifetime; no success accepted. +- Resumed same integration session ONLY for narrow fix plus executable startup-survival/TERM regression, full cargo tests, debug/release rebuild and unique-PID stub cleanup. Wrapper PID2834878; OpenCode child PID2834892 verified alive; terminal proc_418eeb69ba13; log /home/somhairle/.hermes/cache/strategy-lab-integration-shutdown-repair.log. Do NOT launch another writer. Next tick inspect completion, independently probe survival beyond10s and bounded TERM BEFORE replacing QA binary; then measured live qa_restart.py and affected API/limits gates. QA remains original PID2829168, active, /api/health HTTP200. No service, production or tunnel change this tick. +- Current tick found no surviving OpenCode writer and independently rebuilt debug/release, ran complete backend suite:52/52 passed (seven build/eight test warnings remain). Reviewed repaired signal gate; isolated credential-free RELEASE probe survived14.187s with seven HTTP200 health samples, then TERM exit0 in0.0074s. Evidence artifacts/qa/supervisor-live/shutdown-repaired-probe.json. Replaced live QA via restart only after probe. Immediate startup curl raced listener, subsequent actual health200 verified. +- Measured live active-Docker restart on repaired DEBUG binary: systemctl restart0.0320s, qa_restart.py exit0 with9/9 checks including under20s gate, retained interrupted failure, exact container cleanup, durable draft/dataset/session and successful post-restart real backtest. Updated runnable harness/result in artifacts/qa/supervisor-live. Fresh authenticated API --market suite exit0 with30/30 checks, including actual market data/cache/reproduction. No worker claim used as evidence. +- Fresh limits rerun on repaired service passed first8 gates (failure, network/secrets, actual limits, caps, mounts/env, cancellation and removal), then HTTP429 prevented submitting timeout run. Prior full10/10 remains historical only; rerun final timeout with a fresh isolated QA account or authorized quota adjustment next tick, without disabling gates. Latest results /home/somhairle/.hermes/cache/strategy-lab-qa/limits-checks.json. No new worker launched, no application edits by supervisor; only harness latency assertion added. No production/tunnel changes. Next work: complete final limits run, authenticated desktop/mobile full workflow plus vision, production and domain authorization. +- Current tick completed final live limits rerun on repaired shutdown QA binary: exact command `.venv/bin/python /home/somhairle/.hermes/cache/strategy-lab-qa/qa_limits.py` exited0; independently parsed10/10 passing gates. Covers real failed strategy, network/secret isolation, actual Docker resource/capability/mount/environment constraints, cancellation and60-second timeout plus exact container removal. Raised only isolated browser QA account daily_run_limit to500 using authenticated admin PATCH and verified via GET; production untouched. Refreshed artifacts/qa/supervisor-live/{limits-checks.json,sandbox-inspect.json,qa_limits.py,limits-final-verification.json}; final live health status ok, worker_available true, ai_configured true. No OpenCode writers found, no worker launched, no application/service/tunnel edits. Next bounded task: full authenticated desktop/mobile workflow and screenshot vision review, then isolated production setup/backup/restore/resources and original-session domain authorization. No completion notice; remain silent. +- Current tick added and actually exercised authenticated Playwright workflow `artifacts/qa/supervisor-live/qa_browser_workflow.py`. UI login/project creation works; independently caught two real blocking API defects: GET instruments?q=600000 HTTP400 plain text `invalid type: map, expected option`; dataset submission with blank optional name HTTP422 `missing field name`. Source also shows DatasetWizard stores submitErr but never renders it; screenshot confirms invisible submit failure. Evidence browser-workflow.json and workflow-{data-desktop,failure}.png; failure screenshot visually reviewed, single layout confirmed. No full-browser pass claimed. Harness locator mistakes (label includes option text, ambiguous 复权 label) corrected without altering application. +- Resumed ONLY integration session ses_f553821d1ffeRPSHHoGHXZx6xq for these narrow backend/UI fixes and relevant tests/builds. First launch using repo .venv failed immediately (no dotenv), confirmed no writer, relaunched with default python (dotenv verified). Actual wrapper PID2847343 / OpenCode child2847357 both verified alive; handle proc_66aa9aa132e6; log /home/somhairle/.hermes/cache/strategy-lab-integration-browser-repair.log. No duplicate writers. Next tick inspect log/PID/handoff, verify fixes independently and restart only QA after safe build, continue browser dataset preview/editor/version/backtest/result/comparison/AI and mobile review. No application edits by supervisor, no production/tunnel/service changes, no completion notice. +- Current tick independently rebuilt debug, ran backend54/54 and frontend23/23 tests, Svelte0errors/0warnings, frontend build exit0; restarted only QA. Live authenticated browser confirms instruments?q=600000 HTTP200 real 浦发银行 catalog and absent-name dataset POST202 auto-generated name. Health200/worker_available/ai_configured verified. Browser harness exits0 but does NOT cover whole workflow and mobile width was not asserted, so no full-browser pass claimed. +- Found real mobile overflow: 375px viewport with scrollWidth532 (pending) then552 (failed dataset); DOM identifies sidebar/main552px/cards523px. Full-page screenshot expands to content width and vision incorrectly suggested no overflow; DOM evidence plus viewport-only screenshot retained at artifacts/qa/supervisor-live/{mobile-overflow-probe.json,mobile-viewport.png,qa_mobile_probe.py}. Real dataset434e6a26-2bd0-4920-a192-e6142dccee73 subsequently failed due Eastmoney disconnect and Tencent connect timeout; preserve honest provider failure, retry real data next tick without fabrication. +- Resumed only existing integration GLM session for narrow mobile wrapping/sizing fix and frontend regression/check/build. Wrapper2853674/child2853688 verified alive, terminal proc_368da546aef5; log /home/somhairle/.hermes/cache/strategy-lab-integration-mobile-repair.log. No duplicate writers; worker forbidden service/config changes. Next tick inspect worker and verify 375px DOM width plus viewport screenshots, then finish real preview/editor/version/run/result/comparison/AI browser flow. No production/tunnel changes or notice. +- Current tick confirmed mobile worker exited (no OpenCode writer), independently ran frontend27/27 tests, Svelte0errors/0warnings and build exit0 (large chunk warning remains). Live QA serves rebuilt static assets without service restart. Actual mobile probe now viewport375/scrollWidth375, no overflowing elements; screenshot vision confirms no control overlap. Evidence mobile-overflow-probe.json/mobile-viewport.png refreshed. +- Extended and exercised browser harness through real search, dates/adjustment, absent-name submit, terminal ready dataset and rendered20-row preview, plus375px width assertions before and after preview; final command exit0 and no pageerrors. Real fresh dataset e1aa8b20-ebc7-409f-9573-d0c9a4a9491b returned58 actual Tencent rows after Eastmoney failure. Subsequent final rerun used legitimate immutable cache. Harness strict locator encountered multiple owner datasets; choose first visible preview button (this verifies owner data preview, not exact newly-created dataset binding). Final JSON/screenshots under artifacts/qa/supervisor-live/browser-workflow*. Mobile viewport screenshot now scrolls to actual preview; vision confirms no overlap/side clipping, but narrow cells wrap numeric values and dates across lines, a readability limitation. Full editor/version/run/result/comparison/AI browser path still pending; do not claim whole workflow complete. No application edits, new workers, service/production/tunnel changes this tick. +- Current tick wrote and ran actual authenticated `artifacts/qa/supervisor-live/qa_browser_strategy.py` using prior real project/dataset. Immediate strategy page fails with pageerror `c(...).slice is not a function`, blank main and no CodeMirror. Evidence browser-strategy.json and workflow-strategy-failure.png. Source confirms actual backend draft_generation:i64 versus frontend string `.slice` calls. No strategy/versions/backtest browser pass claimed; harness retained for rerun. +- Resumed ONLY existing integration GLM session ses_f553821d1ffeRPSHHoGHXZx6xq for numeric-generation contract repair, optimistic/AI guard audit, regression tests and frontend check/build. Wrapper2864219/child2864233 verified alive, terminal proc_e6aa364d11c5, log /home/somhairle/.hermes/cache/strategy-lab-integration-generation-repair.log. No pre-existing OpenCode writer before launch. Next tick inspect worker, independently check/build and rerun qa_browser_strategy.py; then expand comparison/AI/mobile. No service/production/tunnel changes or completion notice. +- Current tick independently verified generation repair: frontend30/30 tests, check0errors/0warnings, build0. Test stderr still includes CodeMirror jsdom getClientRects TypeError (recorded, not called clean). Actual authenticated browser qa_browser_strategy.py exit0: numeric generation renders, editor autosave persists, immutable snapshot and current-draft diff dialog, real backtest58 equity points/two fills, equity and drawdown SVG rendered, mobile scrollWidth375, no pageerrors. Harness initially wrongly required canvas; source uses SVG renderer, corrected to assert actual labeled SVG for both chart modes. Screenshots workflow-{version-desktop,results-desktop,results-mobile}.png and browser-strategy.json retained; desktop/mobile vision reviewed. Mobile first viewport dominated by long honest English assumptions; no sideways clipping, but chart is below fold. +- New verified accounting/UI defects from real result and source: trade_count1 means closed round trips but frontend incorrectly describes fills (actual fills2); sell trade.value uses Backtrader ex.value cost basis, so shows buy cost659.66 instead of sell turnover. Resumed ONLY same integration GLM session for narrow accurate trade value/label regressions plus obvious metrics layout/test geometry repair. Wrapper2871912/child2871929 alive; terminal proc_5937b56a2c20; log /home/somhairle/.hermes/cache/strategy-lab-integration-results-repair.log. No competing writer. Next tick inspect worker, independently run worker/frontend suites, rebuild exact worker Docker image if worker changes, rerun real browser and assert fill quantity*price matches value, then finish comparison/AI/browser mobile chart review. No services/production/tunnel changed, no completion notice. +- Current tick confirmed results worker exited/no duplicate writer. Independently ran worker38/38, frontend33/33, check0errors/0warnings, frontend build0; rebuilt exact Docker tag strategy-lab-worker:local image cc7f74ce4fdf. Worker report of zero test noise is NOT reproduced: CodeMirror noise fixed, but ECharts jsdom emits zero-dimension/HTMLCanvasElement.getContext not implemented warnings despite tests passing. Build still large-chunk warning; Docker build context2.562GB needs narrow .dockerignore improvement later. +- Actual authenticated browser rerun on rebuilt worker passed editor/autosave/version/diff/backtest/equity/drawdown. Added hard assertions actual fill value=quantity*price and closed round trips1: buy659.659, sell649.00, two fills,58 equity points; latest run1647a719-3323-438c-abec-4ea538b46725. Final harness exit0/no pageerrors. Mobile resize initially transient scrollWidth1133 before chart observer settled; after bounded1.5s layout settling final width375, chart visible. Refreshed artifacts/qa/supervisor-live/{qa_browser_strategy.py,browser-strategy.json,workflow-results-desktop.png,workflow-results-mobile-chart.png}. Independently vision-reviewed both: six desktop metric tiles/turnover correct and readable, mobile drawdown axes/labels visible without overlaps; mobile table wraps values. +- No application edits by supervisor, no new worker, no service/production/tunnel changes or notice. Next bounded work: authenticated run comparison and AI proposal/diff/accept browser flow (API AI was tested previously), then production/backup/resources/domain authorization. Test-only ECharts environment warnings and oversized Docker context remain cleanup items. +- Current tick added and independently executed authenticated `artifacts/qa/supervisor-live/qa_browser_compare_ai.py` against actual QA, exit0 with five checkpoints and zero pageerrors. Selected two actual succeeded runs via UI, rendered normalized comparison SVG and condition differences; mobile document width375. Real AI proposal returned valid Python, draft/generation remained unchanged before acceptance; clicked accept on mobile, read back exact proposed draft, generation6 and matching immutable version. Evidence browser-compare-ai.json and workflow-{compare-desktop,compare-mobile,ai-proposal-desktop,ai-mobile}.png. Vision reviewed mobile comparison, mobile AI and desktop AI: no horizontal clipping/overlap; narrow code identifiers and version table wrap. Comparison runs have identical logic/data so curves overlap honestly. Desktop sidebar still displays current project 加载中 despite loaded project; inspect this minor UI defect in next cleanup alongside ECharts test noise and oversized Docker context. No OpenCode writers found, none launched; no application/service/production/tunnel changes. No completion notice. Next bounded task: remaining small UI cleanup then isolated production/backup/resources and actual domain authorization. +- Current tick read brief/source and confirmed no existing OpenCode writer. Layout derives current project solely via projectOf; prior browser evidence shows loaded project still labeled 加载中. No root .dockerignore exists; worker Dockerfile needs only requirements-worker.txt and worker. Resumed ONLY existing integration GLM session for bounded sidebar reactive/deep-link name fix, ECharts test-environment warning cleanup and narrow Docker build-context exclusions with regressions/check/build. Wrapper2883552/actual OpenCode child2883566 independently verified alive; terminal proc_2f82d207bf4e; log /home/somhairle/.hermes/cache/strategy-lab-integration-final-cleanup.log. No completion claimed, no competing writer or service/tunnel/production changes. Next tick inspect worker log/PIDs, independently run frontend tests/check/build, actual sidebar desktop/mobile check and Docker image build/context measurement, then production/backup/resources/domain authorization. +- Current tick found cleanup OpenCode wrapper2883552/child2883566 still alive; inspected fresh log showing ongoing frontend/.dockerignore work, so did not launch another writer or race frontend validation. Independently built exact worker image with new context guard: `docker build -f worker/Dockerfile -t strategy-lab-worker:local .` exited0; actual context54.27kB (previously2.562GB), image7f3ecf6290ee. Restricted no-network/read-only/cap-drop container imports worker.main/backtest/data successfully; /app contains only worker, no repo artifacts/server/frontend/.env/.venv. Evidence artifacts/qa/supervisor-live/docker-context-build.log. Frontend cleanup remains unverified until worker exits. +- Retrieved original user message99011 from session20260916_193353_1e3c6334: literal public target fin.somhairle.bin. Exact .bid user-role search only retrieved compaction reference99333 describing .bid as tentative; no explicit substitution authorization established. Evidence artifacts/qa/supervisor-live/domain-authorization.md. Finish safe local work before blocker notice; no ingress/service/production changes or notice this tick. Next tick check existing writer completion, independently validate frontend/sidebar and continue isolated production/backup/resources. +- Current tick confirmed cleanup worker exited and independently ran frontend37/37, Svelte0errors/0warnings and production build exit0. Prior ECharts/CodeMirror test noise absent in this fresh output; large-chunk build warning remains. Authenticated new qa_sidebar.py confirms real project name resolves on desktop and mobile, zero pageerrors. However twice reproduced persistent results-page overflow after desktop-to-mobile resize: viewport375/document scrollWidth1133 after1.5s. Evidence sidebar-checks.json plus sidebar-{1440,375}.png; viewport-only vision confirms name readable but cannot see below-fold overflow, so DOM assertion remains decisive. Harness writes evidence then fails; acceptance not weakened. +- Resumed ONLY existing integration GLM session for narrow chart/table mobile overflow repair and fresh browser/unit verification. Wrapper2901780/child2901794 independently verified alive; terminal proc_a49223b2c3f3; log /home/somhairle/.hermes/cache/strategy-lab-integration-mobile-results-repair.log. No duplicate writer, application edits by supervisor, service/production/tunnel changes or notice. Next tick inspect writer and independently verify fresh mobile deep link plus resized results width, then isolated production/backup/resources. Domain authorization still unresolved as previously documented. +- Current tick confirmed prior mobile-results OpenCode worker exited/no competing writer. Independently ran frontend39/39 tests, Svelte0errors/0warnings, production build exit0; no test stderr noise, large-chunk build warning remains. Extended and executed parent qa_sidebar.py: five live authenticated desktop/resized/fresh-mobile checkpoints, all document widths equal viewport (1440 or375), actual SVG widths1101/311 follow resize, zero pageerrors, real sidebar project name resolved. Final command exit0. Evidence artifacts/qa/supervisor-live/{qa_sidebar.py,sidebar-checks.json,sidebar-mobile-chart-verified.png}; independent vision review confirms visible mobile chart controls/legend/axes readable, no overlap/crop; table numeric wrapping remains known readability limitation. No application edits by parent, no new worker, no service/production/tunnel changes or completion notice. Next bounded task: isolated production release/admin/service, verified backup/restore and idle measurements, then final full gates and domain decision blocker protocol. +- Current tick independently verified persistent production unit enabled and host user Linger=yes, restarted only strategy-lab-production and confirmed changed PID/exact copied release executable plus actual healthy HTTP in0.061s. qa_production_persistence.py exited0; seven real idle cgroup samples over30.050s show memory2,416,640–12,435,456bytes settling to2,416,640, CPU0.06393% of one core. Raw evidence artifacts/qa/supervisor-live/production-persistence-checks.json; docs/production-local.md updated. No physical host reboot performed; measurements exclude Docker workloads. No OpenCode writers found, none launched; no application/config/tunnel changes or notice. Next bounded task: populated QA data-object backup/restore into private isolated temporary restore instance (never overwrite production or live QA), verify restored API/object contents and document procedure; then final full gates and unresolved domain blocker protocol. +- Current tick independently executed populated backup/restore verifier qa_backup_restore.py: final exit0, all11 checks passed. Consistent stopped-QA SQLite/object snapshot, SHA256 equality, integrity/entity counts/all ready object references, isolated release server restored authentication/draft/real preview/exact run result; QA and production health200 afterward. Evidence artifacts/qa/supervisor-live/{qa_backup_restore.py,backup-restore-checks.json}; docs/production-local.md records actual procedure and private snapshot location policy. Initial harness failures corrected honestly: transient QA unit disappears after stop (recreated using existing launcher and verified), preview is a separate API endpoint. No production/tunnel/application change or new OpenCode worker. Isolated restore process terminated. Next tick: final full acceptance suites and production copied-asset freshness check, then unresolved domain blocker notice/pause protocol. No completion notice yet. +- Production isolated DB/admin/service, idle measurements and populated backup/restore now verified. +- Resolve public domain authorization from original session20260916_193353_1e3c6334; RELEASE_SCOPE assertion does not override brief's explicit requirement to establish actual user authorization for .bid vs literal .bin. +- No production/tunnel changes, no completion notice, supervisor remains active. User requested silence until completion or genuine decision blocker after all safe work. + +- Current tick independently rebuilt release (exit0, seven warnings) and provisioned isolated production candidate: persistent enabled strategy-lab-production.service, copied release/frontend, clean private DB/admin, loopback127.0.0.1:8789; QA8787 untouched. Exact PID executable verified, health ok/worker_available/ai_configured, authenticated /auth/me admin and empty projects, SQLite counts users1/projects0/datasets0/runs0. Credentials/runtime env mode0600 outside repo, bootstrap secret absent from service environment. No public ingress or domain substitution. Initial online SQLite backup independently reopened integrity/count checks pass; populated data/object restore still pending. Evidence production-local-checks.json; operational notes docs/production-local.md. Harness login initially assumed wrapped user; corrected via actual authenticated /auth/me, no application bug. Next tick verify persistent-unit restart/linger, measured idle samples, populated backup/restore and final full gates before domain blocker notice. No OpenCode writer or completion notice. diff --git a/docs/domain-authorization-user.md b/docs/domain-authorization-user.md new file mode 100644 index 0000000..2db054a --- /dev/null +++ b/docs/domain-authorization-user.md @@ -0,0 +1,8 @@ +# Explicit user authorization — supersedes earlier domain ambiguity + +Latest direct user message in the current Matrix thread: +> 好, 你可以部署到 fin.somhairle.bid 上 + +The user explicitly authorizes public deployment to https://fin.somhairle.bid. Earlier .bin/.bid ambiguity and any instruction to report a domain-decision blocker are resolved and superseded. Continue deployment to exactly fin.somhairle.bid, preserving existing Cloudflare ingress and --protocol http2. Do not ask the domain question again. + +Current verified production candidate is strategy-lab-production.service at 127.0.0.1:8789; read docs/production-local.md and actual service/config before changing anything. Verify production asset freshness, origin/secure-cookie settings for HTTPS, authentication and permission gates before exposure. Append only the new ingress before the existing404 fallback, route DNS for the existing tunnel, validate and read back exact config, restart only cloudflared as needed, preserve other services and verify their routes. Independently verify the actual public endpoint, login and static assets. Complete remaining acceptance and then deliver completion once via established notification mechanism. diff --git a/docs/etf-recovery-release-handoff.md b/docs/etf-recovery-release-handoff.md new file mode 100644 index 0000000..70b14d8 --- /dev/null +++ b/docs/etf-recovery-release-handoff.md @@ -0,0 +1,100 @@ +# ETF Recovery Release Handoff — v4（2026-09-17） +状态：**READY_FOR_LEADER_REVIEW**（仅覆盖 ops 发布阻断 ASSET；应用代码自 v2 验收后已冻结）。 +本版取代 v3；v3 中 leader 实际重跑 `ops-drill2.sh`（43+ PASS）与应用全部测试均已独立通过 +（pytest 61 / cargo 58 / frontend 56，共 175 项）+ v3 演练 66 PASS 0 FAIL。 + +## 0. 仅剩一条真实发布阻断（前轮已修；本轮再加修两处执行路径缺陷） +前轮已修：旧 v2 场景 D 的 INJECT=rsync 在 backup 阶段就失败且不恢复旧服务（BD1–BD3 / D / G 断言，见第 2 节）。 + +本轮（v4）针对 leader 指出的两处执行路径缺陷，均在 `ops/deploy.sh` 内最小修复，脚本未重写： +- **A — restore 停服失败仍会写文件**。旧行为：`restore_release_id()` 中 stop 失败仅 note 后继续替换 + runtime.env / release（运行中的二进制被覆盖）。修复（`deploy.sh:210-221`）：stop 失败后必须再次 + `is-active` 确认；**无法确认服务已停止即拒绝写任何文件**（含手动 restore 与自动回滚路径），文件 + 保持可恢复原状，die 明确报错退出非零；若 stop 失败但核实服务确已 inactive（常见于"早已停止"）， + 允许继续。 +- **B — 停服后 DB 检查深层 exit 绕过恢复分支**。旧行为：`precheck_stopped()` 调用的 + `check_no_inflight()` 在 DB 查询失败、DB 文件缺失或 runtime.env 缺 DB_PATH 时直接 `die`（深层 + exit），主流程的 rc 捕获取不到，旧服务停在停机状态。修复（`deploy.sh:83-113` + 主流程 + `deploy.sh:262-279`）：`check_no_inflight` 改为**返回错误码而非 exit**（0=无任务 / 1=有任务 / + 2=DB 或环境错误），`db_scalar` 失败不再 die；主流程把 rc=2 与竞态 rc=1 同样处理：**先恢复旧服务 + （start），不替换任何文件**，_RB1/RB2 断言旧服务 active 且 env/binary/dist 无 swap_。 +- 另修演练真实性：场景 D 的退出码断言此前形同虚设（输出非"0"的条件恒真），现真正捕获 deploy 退出码 + （`rc_d=$?` 断言非零），不以管道/tail 掩盖。 + +## 1. 对象与固定 ID +- 数据集：国泰自由现金流 ETF 159399（market=cn，provider 手动修复数据集 id=`1f9e066a-e785-4ba5-bc9c-ef4742cd02d8`）。 +- 候选 worker 镜像（**immutable ID，只按 ID 使用，禁止 retag**）： + `strategy-lab-worker:etf-sina-candidate-r3` = `sha256:34a61c22c5135b6cc8453e5e3621674caf12284c5aa5dbfba1cda15299a656e3` +- 生产在线镜像：`strategy-lab-worker:local`（`bd9e9f06d56b…`，未动、未 retag）。 +- 生产事实（只读核实）：unit `strategy-lab-production`（UMask=0077）、环境由 + `runtime.env`（BIND=127.0.0.1:8789 等）、发布目录 `/home/somhairle/.local/share/strategy-lab-production/release`。 + +## 2. v3 + 本轮（v4）相对 v2 的实际变更（全部仅在 `artifacts/etf-recovery-candidate/`） +### 2.1 `ops/deploy.sh`（共 329 行；v4 仅修改 A/B 两处，未重写） +| 位置（行号，v4 当前） | 变更 | +|---|---| +| `deploy.sh:51-53` | `fin_stage()`（manifest 先写 tmp 再原子 mv）；注入钩子 `CP_CMD`/`MV_CMD`（默认 cp/mv，行为不变）。 | +| `deploy.sh:137-182` | `backup()`: 任一失败（env cp / release rsync / manifest 写 / finalize）都会退（备份目录改名 `.broken.`）后 `die`；写入 complete 校验（`runtime.env`、`release/strategy-lab-server`、manifest `previous_server_sha256`）全部成功才创建 `${BACKUP_DIR}/complete` 标记（0600）。 | +| `deploy.sh:260-273` | 主流程：**备份在 stop 之前进行**(phase 2/6)。备份失败 = 服务仍在运行，绝不转化为停机。 | +| `deploy.sh:274-303` | stop 后硬重检；rc=1（停后新任务）与 **rc=2（停后 DB/环境不可读）** 都走恢复分支：重启旧服务、什么都不替换、退出非零；备份目录仅供 inspection。 | +| `deploy.sh:83-116` | v4-B：`check_no_inflight()` 返回错误码不深层 exit（0 无任务 / 1 有任务 / 2 DB/env 错误）；`db_scalar` 失败与缺 DB_PATH、DB 文件缺失一律 return 2。 | +| `deploy.sh:209-247` | `restore_release_id()`: complete/runtime.env/release 校验、deploy.lock；v4-A：stop 失败必须再次确认服务状态，**无法确认已停即拒绝写任何文件**（明确报错、退出非零、文件保持可恢复），确认 inactive 后继续。 | +| `deploy.sh:184-188` | `swap_release()`: `install ... || return 1; rsync -a --delete --checksum ... || return 1`（R1 根因修复已保留）。 | +| `deploy.sh:249-258` | `verify_restored()`: manifest 的 `previous_worker_image` 与 `previous_server_sha256` 逐一比对（损坏/伪造的 manifest 直接 `die`）。 | +| `deploy.sh:305-321` | swap/env(SED)/start-service/health 各自独立 rc-check，失败一律 `DEPLOY_LOCK_HELD=1 restore_release_id` 自动回滚（stop→替换→start→health 验证）。 | + +### 2.2 `ops/rollback.sh`（共 103 行，重写） +- `rollback.sh:54`：无 `complete` 标记 → 拒绝恢复。`rollback.sh:45`：`list`命令显示 `COMPLETE`。 +- `rollback.sh:62-66`：in-flight runs>0 → 拒绝（不当陷入）。 +- `rollback.sh:72`：**先 `systemctl stop` 服务**，再 cp runtime.env / rsync release，替代任何在线文件被运行中进程写入。 +- `rollback.sh:76-82`：文件替换失败时显式 `start`（尽力而为）并 FATAL exit 1。 +- `rollback.sh:83-86`：启动后 `sleep 2` + is-active 失败 FATAL；`rollback.sh:88-90` health JSON `status=="ok"` 失败 FATAL；`rollback.sh:92-97` sha256 与 env（manifest `previous_worker_image`）不匹配 FATAL。 +- 复审条目映射（与 deploy.sh 共享 lock `deploy.lock`，`rollback.sh:59-60`）。 + +### 2.3 `ops-drill2.sh` — 最终 **PASSED=84 FAILED=0 exit 0**（TDD：先加 RA/RB1/RB2 看其失败，修复后全绿） +- `fake-systemctl` 增加 `.audit.log`（stop/start/restart 顺序审计）。 +- `FAKE_START_FAIL` 是一次性的（deploy 阶段 5 的首次 start 失败；restore/回滚路径的 start 不受影响）。 +- `DEPLOY_PHASE=backup/swap/restore` 使注入只影响指定阶段。 +- 既有场景（v3 均保留，全部仍通过）： + - **BD1/BD2/BD3**（backup-cp / backup-rsync / backup-manifest 失败）：断言 ① 旧服务 ACTIVE（零停机）② env/二进制/前端 sha256 逐字节不变 ③ 退出码非零 ④ 不允许存在带 complete 标记的备份。 + - **D**（rsync 失败只在 swap 阶段一次性注入）：备份先完整完成；自动回滚真实触发，断言服务 active、发布物/env 恢复逐字节一致、**真实捕获的非零退出码**。 + - **G**（手动 rollback）：audit 首行 == `stop`、末行 == `start`（先停服再写文件，服务最后启动），且最终服务 ACTIVE。 +- 本轮新增场景： + - **RA**（restore 中 stop 失败且服务仍 active）：断言明确拒绝信息、退出码非零、env/binary/dist 逐字节不变、服务仍 active（备份 intact 可恢复）。 + - **RB1**（stop 后 DB 文件被删 → 查询错误）：断言恢复旧服务 active、无任何 swap（env/binary/dist 不变）、退出码非零。 + - **RB2**（stop 后 runtime.env 缺 DB_PATH）：同上断言；runtime.env 其余键逐字节不变。 +## 3. 最终可执行命令（仅指令，生产由 leader 执行） +```bash +# —— 只重跑合并后的 drill（隔离替身）： +bash artifacts/etf-recovery-candidate/ops-drill2.sh # 期望：PASSED=84 FAILED=0 | exit 0 + +# —— 手动回滚（leader 生产会话执行）： +artifacts/etf-recovery-candidate/ops/rollback.sh list # 只读，带 COMPLETE 标记 +artifacts/etf-recovery-candidate/ops/rollback.sh # stop -> 替换 -> start -> health/sha 验证 +# （deploy.sh 自动回滚路径等价于 deploy.sh restore ） +``` + +部署命令（生产）： +```bash +cd /home/somhairle/projects/strategy-lab/artifacts/etf-recovery-candidate/ops +IMAGE_ID=sha256:34a61c22c5135b6cc8453e5e3621674caf12284c5aa5dbfba1cda15299a656e3 \ + STATE_DIR=/home/somhairle/.local/share/strategy-lab-production \ + REPO_ROOT=/home/somhairle/projects/strategy-lab \ + SERVICE=strategy-lab-production \ + ./deploy.sh deploy +``` +（`deploy.sh` 默认值即上述 STATE_DIR/REPO_ROOT/SERVICE；显式列出仅供 review。IMAGE_ID 必须是唯一的不可变 ID。） +回滚（二选一）：`./ops/rollback.sh ` 或 `./ops/deploy.sh restore `；deploy 各失败路径会自动调用 restore。 + +## 4. 语义边界（去除过度承诺，本轮未变） +- **曾写 “重启后所有任务状态保持原样”——该过度承诺已删除**。真实语义：重启时正在执行的任务（dataset=running/pending、run=running/queued）不会被自动续上；重启可能把 running 置为 failed 或使其停留为 stale。恢复/重试由应用侧承担（DatasetCard 重试入口、server 调度），不属本 ops 脚本职责。 +- 回滚目标 = pre-deploy 状态：`WORKER_IMAGE=strategy-lab-worker:local` + 旧二进制/旧 dist（manifest 记录 `previous_worker_image` 与 `previous_server_sha256`，恢复后强校验）。生产 `:local` 标签未被动过，回滚安全。 +- **备份使用 0700 目录（runtime.env chmod 600）**；旧的备份目录（v2 期间产生的 `release-backups/20260917-*`），若 leader 未删除可能存在；脚本对现有备份目录不覆盖（同 RID 冲突 → 拒绝）。 +- **有限保证**：`restore_release_id` 的 stop 失败拒绝路径保证"不写文件"，但不保证 deploy 此前已换过的 env/binary 自动还原（该场景下文件保留原状 + 备份 intact，需 leader 手动 restore）；停后 DB 错误分支只做到"恢复旧服务 + 不替换"，DB 本身的损坏/丢失不在脚本的修复范围。 + +## 5. 遗留与上下文 +- **应用冻结**：pytest/cargo/frontend 测试全部通过（pytest 61 / cargo 58 / frontend 56），leader 独立重跑通过，共 175 项（61+58+56）。qa2（隔离替身实测）之后未再改动应用代码。 +- **qa2 独立验收未变**：`artifacts/etf-recovery-candidate/qa2/`（登录、数据集 ds_cn/ds_sz ready、预览/回测 succeeded `b4339ff5-`、6 步浏览器重试通过）。 +- 禁止 export/输出真实密钥/生产路径；禁止一切生产改动（本会话从未执行生产 systemctl/docker/rsync）。 +- 复审条目映射：R1 = swap/自动回滚路径独立 rc-check（deploy.sh:184-188 与 305-321）；R2 = 停后硬重检 + 恢复分支（deploy.sh:274-303，v4 扩展 rc=2）；R3 = restore 锁+health+hash（deploy.sh:209-258 / rollback.sh 锁+health+hash）；R4 = drill 注入矩阵 + exit 语义（FAILED>0 → exit 1，v4 加 RA/RB1/RB2）。 +- leader 审查入口：重跑 `ops-drill2.sh`（命令见第 3 节）；生产动作一律由 leader 执行；改动仅涉及 `ops/deploy.sh`、`ops-drill2.sh` 与本文档，未触应用代码，未提交任何 git commit。 diff --git a/docs/frontend.md b/docs/frontend.md new file mode 100644 index 0000000..255c04b --- /dev/null +++ b/docs/frontend.md @@ -0,0 +1,64 @@ +# docs/frontend.md — 策研 Strategy Lab 前端实现记录 + +Last updated during frontend delivery. Scope: `frontend/` only. Styling: Svelte 5 + TypeScript + Vite, hash routing, CodeMirror 6 + @codemirror/lang-python editor, @codemirror view/state, `diff` for versions, ECharts (SVG renderer, dynamically imported chunk), no other runtime deps. + +## Page map + +- Routes (hash): `#/login`, `#/register`, `#/reset`, `#/projects` (list/create), `#/projects/:id/{data,strategy,backtest,results}`, `#/datasets`, `#/runs`, `#/usage`, `#/account`, `#/admin`. +- Sidebar project tabs mirror spec 数据 / 策略 / 回测 / 结果; global nav = 我的项目 / 数据集 / 实验记录 / AI 用量; bottom = 账户与安全 / 管理员. +- "Admin" nav is only shown when `session.me.role === 'admin'`; non-admin is redirected to `#/projects`. +- App guards: `loadSession` is required once before LAYOUT routes; if no session, redirect to login screen. PublicAuth routes bypass. +- Distinct route-load flow avoids the old bug where root rendered the protected Workspace without login. + +## Files + +- `src/App.svelte` global router mount/redirect; `src/components/Layout.svelte` sidebar; `src/components/Modal.svelte`, `Loading.svelte` +- `src/pages/…` +- `src/lib/api.ts` thin `fetch` client; `src/lib/state.ts` API error extra messages; `src/lib/session.svelte.ts` **runes module** (`.svelte.ts`); same for `src/lib/listsStore.svelte.ts` +- Editor: `src/components/CodeEditor.svelte` uses `@codemirror/lang-python`; diff view `DiffView.svelte` based on `diff`; chart `Chart.svelte` lazy-loads `echarts` on result views. +- scripts/browser-smoke.mjs — Playwright smoke with real pages; can be pointed at the QA URL. + +## Build/test commands + +- `npm run test` — 21 vitest (TDD first) — this does not require a server. +- `npm run check` (svelte-check) — 0 errors / 0 warnings currently. +- `npm run build` — production build in `frontend/dist/`. + +## Visual + +- Modern light teal & slate design; no dark glass or dashboard fluff. System font stack: `-apple-system, SF Pro Text, ……, Noto Sans SC` fallback for CJK; no bundled licensed fonts. + +## State views + +- Every qualifying mutator (login, register, editor autosave, run lifecycle, datasets - wizard) shows loading / error / empty / success banners from backend error codes and shapes `{error:{code,message,details?}}`. Mutations that would be performed by non-admin users get a 403 with the honest reason instead of a dead button. + +## Backend contract notes + +This is the working snapshot of API routes the frontend uses (all prefixed `/api`); mismatches with SPEC/partial server are called out below so the backend team can reconcile. + +1. `GET /health` and `GET /capabilities` are public; frontend will show "本服务内部使用" labels and use `worker_available`/`ai_configured` flags to gate Run and AI panels. Server leaks no internal path/keys. +2. `POST /auth/login`, `POST /auth/logout`, `GET /auth/me` — per SPEC & amendment. `GET /auth/me` returns `{user}` where `role`/`active` are surfaced. Registration is `POST /auth/register {invite_token,name,email,password}` using a URL-supplied `?invite=` token (password never in URL). `POST /auth/reset-password {token,new_password}` requires the admin-issued token. +3. Sessions: `GET /auth/sessions` items `{id,created_at,last_seen_at,user_agent?,current?}` — amend: expose `current` so UI can label "当前会话"; if not provided we derive none. `DELETE /auth/sessions/:id` (404 if foreign). +4. Draft/save semantics: `PUT /projects/:id/draft {code,expected_generation}`; HTTP 409 with `code:'stale_generation'` triggers the compare-and-choose conflict UI; spec-compliant. +5. Versions: `POST /projects/:id/versions` with `{message}`. `GET /projects/:id/versions` returns `{items:[…]}`. `GET /projects/:id/versions/:versionId → {code}` — **frontend-implied extra**; without it the version diff/restore flows degrade to metadata-only; backend should confirm (added to `docs/backend-questions.md` needs backend info). **Note to backend reviewer: no secret content, full code only for the owner.** +6. Restore: `POST /projects/:id/restore {version_id,expected_generation}` returns Project with the new draft generation. +7. Datasets: `GET /datasets`, `POST /datasets`, `GET /datasets/:id`. Status polling every 2s only while `pending|running`. **DELETE /datasets is NOT implemented** in the UI (not in contract). `GET /datasets/:id/preview` shape `{columns,rows,coverage,warnings}` where each coverage row is `{instrument,actual_start,actual_end,row_count,warnings?,requested_start,requested_end,market?,asset_type?}`; the UI synthesizes warnings like "标的 X 起始数据较其他标的更晚" from those values and lists per-instrument lecturer warnings — keep these field names. +8. Runs: list/filter with plain `?project_id=`; `POST /runs` body includes optional `acknowledge_warnings` (sent only when the dataset preview reported coverage warnings), `parameters` as object. Cancel is `POST /runs/:id/cancel`; rerun is `POST /runs/:id/rerun {use_original_data:true}`. Statuses surfaced in Chinese: queued/running/succeeded/failed/cancelled. +9. AI: `POST /ai/assist {project_id,instruction,expected_generation}` → `{id,model,explanation,proposed_code,diff,base_generation,usage:{input_tokens?output_tokens?},status}`. Accept is `POST /ai/:id/accept {expected_generation}`; deliberate 409 stale-guard message shown when generation moved. The `diff` field is optional; UI computes it locally from `proposed_code` when missing. +10. AI usage: `GET /ai/usage` `{items,totals,internal_poc}` — usage line is metering only; costs/prices are NOT displayed. +11. Admin: `GET /admin/users`, `PATCH /admin/users/:id {active?,role?,daily_run_limit?,ai_enabled?}`; last-active-admin is protected server-side (we also guard in UI); `POST /admin/users/:id/reset-password → {reset_token,expires_at}`; invitations `POST/GET /admin/invitations` and `DELETE /admin/invitations/:id`; `GET /admin/audit` sanitized `{items:[{actor,action,target,status,time}]}` — we render `ok/success` as ok; frontend tolerates `status` values. +12. `GET /instruments?q=…` — `{items,source,status}`; when the catalog has nothing we surface a link to the "手动录入标的" form; **never invented identities**. `POST /datasets` requires the fields `[open,high,low,close,volume]` for the engine plus any user-checked raw fields. + +## Integration/QA notes + +- Vite dev proxy `/api` → `127.0.0.1:8787` is configured in `vite.config.ts` (parent QA server on 8788 already reloads these changes). +- browser-smoke (scripts/browser-smoke.mjs) checks login/register/reset + the protected routes; screenshot evidence in `artifacts/qa` should be added by the integration agent when backend is live. +- No fake endpoints are wired; empty/error states are behavioral (no stub data). The frontend never fabricates metrics; the "还没有…" empty-states reflect a genuinely empty backend list rather than mocked fixtures. +- The run engine warnings/limitations are always shown with respect to the current `runStatusBadge/Label` mapping — no `metricLock` reference to suppress real issues. +- ECharts is **only** loaded when a result/comparison view opens (dynamic `import()` inside `Chart.svelte`) → it lands in a separate ~1.2MB chunk that loads on demand; the main bundle is ~500 kB minified (gzip ~175 kB). +- All network failures turn into `网络请求失败` banner + retry-friendly state; no silent swallowing, no dead buttons: each disabled button exposes a tooltip or hint on what needs to happen, or the attempt surfaces the returned error. + +## Known remaining edge cases (honest list) + +- Chart rendering: echarts svg renderer, mobile-sized charts can look cramped; desktop is the intended editing surface (responsive 375px viewing is supported but charts prefer ≥ 720px). +- If backend ships additional fields (e.g. `Dataset.progress`, version "comment" flows) the UI will ignore unknown fields gracefully but they should be re-added when stabilized. diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md new file mode 100644 index 0000000..275f324 --- /dev/null +++ b/docs/implementation-plan.md @@ -0,0 +1,50 @@ +# Strategy Lab Implementation Plan + +> Agentic workers: execute the approved SPEC.md task-by-task, use TDD and actual tool evidence. User explicitly selected OpenCode + GLM-5.3-Flash. + +Goal: working private self-service strategy research platform, not a mockup. +Architecture: static Svelte client, Rust/Axum API and persistent SQLite queue, isolated Python AKShare/Backtrader worker; replaceable remote model adapter. +Tech Stack: Rust, Axum, rusqlite, Svelte, TypeScript, Vite, Backtrader, AKShare, Docker. + +## Task A — Runtime prerequisites and contract +1. Install latest npm opencode-ai and record version (done 1.18.31). +2. Secure launcher reads only active Hermes OPENCODE_GO_API_KEY and injects subprocess environment; no key copied to source (done). +3. Smoke actual glm-5.3-flash call and record success (done). +4. Install Rust compiler and verify Docker and Node/Python tools (done). +5. Freeze SPEC.md HTTP/worker contracts; do not overwrite other agents' work. + +## Task B — Worker, tests before implementation +Files worker/{main,data,backtest}.py, worker/Dockerfile, requirements-worker.txt, tests/worker/, docs/worker.md. +1. Write failing normalization tests: preserve raw fields/units, reject bad dates/missing data, no substituted symbols. +2. Run pytest, then implement adapters and test until green. +3. Write failing deterministic Backtrader accounting/next-bar tests using labeled synthetic input. +4. Implement mature-engine runner and manifest/result protocol; rerun tests. +5. Probe real AKShare endpoints and report exact successes/failures, write actual evidence. +6. Build Docker image, backtest without network and credentials, measure memory. + +## Task C — Backend, tests before implementation +Files server/Cargo.toml, server/src/{main,db,auth,projects,datasets,runs,worker,ai}.rs, server/tests/, docs/backend.md. +1. Write failing owner/auth and immutable revision tests; implement SQLite schema and handlers. +2. Write failing request validation/cache key tests; implement safe dataset jobs and immutable blobs. +3. Write failing run lifecycle/restart/cancel tests; implement bounded Docker runner. +4. Write failing AI stale-generation/owner tests; implement configurable API adapter and real usage ledger. +5. Run cargo test, cargo build --release (CARGO_BUILD_JOBS=2 to avoid memory pressure). + +## Task D — Frontend, checks and interaction tests +Files frontend/package.json, frontend/src/... , docs/frontend.md. +1. Add executable tests for workflow state and API errors before implementation. +2. Build login, project list, actual self-service data wizard/preview. +3. Build editor/autosave/version diff/restore, dataset selector, run controls and status. +4. Build result charts/trades/run comparison, AI proposed-code diff/accept/usage. +5. Run check/test/build and Playwright browser smoke; no fixture data in production. + +## Task E — Integration and independent QA +1. Read all outputs; reconcile HTTP and worker contract differences through OpenCode fixes. +2. Start service loopback with secrets injected at runtime; health check independently. +3. Create private account, actual data request, cache reuse request, version/restore/run/compare, real AI suggestion+accept. +4. Negative tests authentication, cross-user access, stale edits, timeout/network worker policies. +5. Browser QA all main pages, screenshot and visual inspection, fix concrete problems. +6. Measure process RSS/CPU/build sizes and actual experiment duration. Record actual outcomes, no estimates called measured. +7. README with reproducible install/run/test, data/model licensing and internal-only limitations. Backup/restart test. Deliver local URL/source/screenshots. + +No public publishing, payment integration or API resale. Any unimplemented acceptance item must be explicitly listed as incomplete rather than claimed complete. diff --git a/docs/integration-handoff.md b/docs/integration-handoff.md new file mode 100644 index 0000000..c64a85c --- /dev/null +++ b/docs/integration-handoff.md @@ -0,0 +1,163 @@ +# docs/integration-handoff.md — integration repair handoff + +Last updated: integration repair phase completion. Ownership: backend/server + worker tests + contract alignment only. Parent owns live QA service on 8787, deployment, tunnel and service lifecycle (this agent never started/stopped/managed the service). + +## Verbatim commands + +```bash +# backend tests (all green) +cd /home/somhairle/projects/strategy-lab/server && cargo test +# -> 46 passed; 0 failed + +# frontend tests/check/build (all green) +cd /home/somhairle/projects/strategy-lab/frontend +npm run test # 21 vitest passed +npm run check # 0 errors / 0 warnings +npm run build # production build in frontend/dist + +# worker tests (all green; project venv required for backtrader/akshare) +cd /home/somhairle/projects/strategy-lab +. .venv/bin/activate && python -m pytest tests/worker -q +# -> 35 passed + +# parent black-box QA gates (needs live 8787 service) +. .venv/bin/activate && QA_ADMIN_EMAIL=... QA_ADMIN_PASSWORD=... python scripts/qa_api.py --market +``` + +## Changed contracts (this repair) + +1. **Run claim atomicity — the stall root cause (server/src/jobs.rs)** + - `move_claim` now performs the rechecks (account active, dataset readiness, quota) AND the atomic transition `queued -> running` **with `container_id` set inside the same transaction** (`UPDATE ... SET status='running', started_at=?, container_id=? WHERE id=? AND status='queued'`). A claimed run always carries container identity; the old code set `running` without a container, then separately queried `container_id IS NOT NULL` (which was still NULL) and returned early forever — runs stalled in `running` with no worker. + - Returns `bool` (`claimed`). Only the claiming caller spawns the backtest worker. + - Regression test `queued_run_claim_is_atomic_and_reaches_terminal_state` proves: claim sets `running`+`container_id`, double-claim is a no-op, only a worker result with `status:'succeeded'` persists as succeeded, worker failure → `failed` terminal, disabled account → `failed` at dequeue without launching a worker. + +2. **Dataset JSON shape (server/src/datasets.rs)** + - `Dataset.request` is now an **object** (parsed from the stored canonical JSON), matching the frontend `DatasetRequest` usage. + - Dataset `name` is optional/empty-tolerant; when blank the backend auto-generates a descriptive name (instruments + date range) and persists it. + +3. **Coverage/preview shape (jobs.rs ingest) — frontend-expected shapes** + - `GET /datasets/:id/preview.coverage` is now an **array of per-instrument rows** built from manifest objects: `{instrument, market, asset_type, requested_start, requested_end, actual_start, actual_end, row_count, warnings}` (frontend `CoverageEntry`). Works for both fresh fetch and stale/fallback worker previews. + +4. **Capabilities shape (server/src/main.rs)** + - `GET /capabilities.fields` → objects `{code,label,raw}` (raw=true only for `adj_factor`) so the field checkboxes render; `adjustments` → objects `{code,label}` (`none/qfq/hfq` → 不复权/前复权/后复权). `frequencies`/`asset_types` stay string arrays. + +5. **Sessions (server/src/auth.rs)** + - `GET /auth/sessions` items now include **`current: bool`** (caller's live session) so the UI can label 当前会话 and guard self-revoke. + - `AuthUser` carries `session_id`; both real extractor and the auth/admin test fixtures updated. + +6. **Instrument search** + - `GET /instruments` `source` values are honest and frontend-aligned: `"catalog"` (items found) or `"none"` (empty/failure with `"status":"unavailable: ..."`). No fake successes. + - Added a bounded in-process identity cache (TTL 300s, max 128 queries) for catalog search results, so repeated queries reuse the actual provider item payloads instead of spawning a container per keystroke. Cache is transparent to callers. + +7. **Network errors stay honest** — search/fetch failures surface as `status: unavailable: ` / `error` fields; nothing silently degrades to empty success. + +## Second repair round (parent-confirmed follow-ups) + +8. **Doubled Layout in project pages (frontend)** + - Root cause per parent browser findings: `App.svelte` already wraps all protected routes in ``; `ProjectPage.svelte` wrapped its tabs in a second `` → two complete sidebars. Removed the inner wrapper (page props preserved: data/strategy/backtest/results tabs render as before inside the single App shell). + - Regression check `frontend/src/lib/layout.test.ts` (`npm run test`, 22 tests) asserts ProjectPage never nests `", "exec"), ns)`. Transport-level isolation is real Docker; no fake Python sandbox. + - Regression tests (`tests/worker/test_backtest.py`): `test_strategy_module_imports_visible_in_methods` (module-level `os` used in `__init__`, `math` used in `next` — must reach actual execution); `test_strategy_genuine_nameerror_still_fails` preserves honest `runtime_error` reporting of genuine errors. +11. **Cancel race / empty `failed` after cancel (server/src/jobs.rs)** + - Root cause: kill container ran BEFORE the guarded state update, and a racing worker-failure finalize wrote `failed` with an empty error. + - New order in `signal_cancel`: persist the cancellation intent atomically first (`WHERE id=? AND status='running'` single guarded transition with `error='cancelled by user'`), only then kill the specific container by name. `finalize_run` transitions stay guarded; lost cancel races never overwrite genuine success/failure. `finalize_run` failure path now guarantees a non-empty terminal reason. + - Regression tests: `cancel_race_never_produces_empty_failed_run` (25 interleavings of concurrent signal_cancel × finalize_run — every outcome must be `cancelled` with the honest reason or `failed` with a non-empty reason, never empty), `timeout_marks_failed_with_reason_not_empty` (real `runner_timeout` message must be persisted, not emptied). `signal_cancel_never_rewrites_terminal_states` retained. +12. Parent QA shows cancel API path exercised — combined with the guard this yields HTTP-visible `cancelled` status immediately after cancel. + +13. **Bounded restart during active Docker backtest (worker.rs / main.rs)** + - Observed by parent live: restart during an active backtest blocked the systemd unit until `TimeoutStopSec=90` forced SIGKILL; journal shows the docker runner child surviving final-sigterm. + - Empirical check on this host (not the live unit): a directly signalled attached `docker run` CLI (default sig-proxy) exits quickly on this host, so the app-level lifetime of the runner child and the CLI's signal proxying were the remaining app-side causes. + - Fixes: + - `docker run` now carries `--sig-proxy=false` immediately after `--rm` (`docker_args`), so a runner CLI can never relay signals into the running container; container lifetime is governed exclusively by the app's exact-name cleanup (cancel/timeout at `kill_container`, and `cleanup_orphan_containers` at next start — unchanged semantics). + - The runner child process is created with `kill_on_drop(true)`: when the jobs task future is dropped at shutdown, the docker CLI child is reaped instead of lingering in the cgroup. Focused regression `runner_child_is_reaped_when_the_job_future_is_dropped` uses a stub runner that ignores SIGTERM, starts an execution as a detached task, aborts the task, and asserts the child is gone within 3s. + - `main()` now handles SIGTERM/SIGINT with `axum::serve(...).with_graceful_shutdown(...)` under a 10-second drain budget, then exits — bounded unit stop; the jobs loop is not aborted mid-state-machine, so terminal-state correctness (running interrupted runs → failed with reason at next startup, queued resumable, cancelled preserved) is preserved exactly as parent verified 8/8. + - Not touched: service unit, TimeoutStopSec policy, credentials, tunnel, services — parent owns live QA. Live restart timing NOT exercised by this writer; parent will time via qa_restart.py. + +14. **CRITICAL premature-exit regression fix (server/src/main.rs)** + - Parent isolated-report confirmed the previous revision wrapped the entire `axum::serve` in `timeout(10s)` → server exited ~10.0s after STARTUP without any signal (isolated probe evidence: `artifacts/qa/supervisor-live/shutdown-premature-exit.json`). Live QA was NOT restarted onto the bad binary. + - Narrow fix: the 10s drain budget now starts ONLY after the shutdown signal. `main()` uses `tokio::select!` between (a) `axum::serve(...).with_graceful_shutdown(wait_shutdown_signal())` and (b) an arm that first awaits `wait_shutdown_signal()` and THEN sleeps 10s — arm (b) cannot fire before a signal, so an un-signalled server stays up indefinitely; a signalled one exits within ≤10s afterwards. Bounded post-signal drain + exact container cleanup semantics unchanged. + - Executable regression `probe::server_survives_past_10s_then_bounds_sigterm_exit` boots the actual isolated binary with isolated tempdir DB/DATA_DIR/port and a synthetic frontend dir, asserts `/api/health` up and — critically — STILL ALIVE at 12s with no signal, then sends SIGTERM to the exact PID (`kill -TERM `, no name scans/group ops) and requires a bounded exit with `/proc/` gone. + - RED/GREEN verified: RED on the current (buggy) binary — `cargo test --bin strategy-lab-server server_survives` failed with "premature-exit regression: server died ~10s after startup without any signal"; GREEN after the fix (12.5s runtime). + - Stub-process test hardening in `worker.rs`: unique `tempfile::tempdir_in("/tmp/opencode")` stub path + unique pid file (was a fixed `/tmp` filename + process-name scan), stub ignores SIGTERM then `exec sleep 500` so the traced PID IS the sleep process (`/proc//cmdline` assert) and `kill_on_drop`'s SIGKILL reaps it — no orphan grandchild; exact-PID supervision via `/proc/` existence only (post-run check found no lingering `sleep 500`). + +15. **Browser-QA regressions round 4 (live browser findings)** + - `GET /api/instruments?q=600000` → HTTP400 plain text `invalid type: map, expected option`. Cause: the query extractor was `Query>`, which rejects any non-empty query string. Fixed to a plain `HashMap` (accepts absent and present params); adjacent extractors audited — the only other `Query>` usage was the compile-only probe handler; runs list already used plain maps. Backend route regression `probe::instruments_query_with_q_is_json_200` (RED captured on the reverted handler: HTTP400 with exactly the live message; GREEN after fix) — asserts JSON 200 `items/source/status` and no-`q` also 200. + - `POST /datasets` with absent/blank `name` → HTTP422 `missing field name`. Cause: `DatasetRequest.name: String` required. Fixed: `#[serde(default)] pub name: Option`; absent or blank auto-generates the persisted descriptive name (SPEC/UI permitted). Regression `probe::dataset_request_allows_missing_or_blank_name` (deser missing/blank/provided). Live 422 evidence from parent remained the RED receipt because `Option` is implicitly skippable in serde — the required `String` variant failed to compile against the new test, itself honest RED. + - Frontend DatasetWizard captured submit failures into `submitErr` but never rendered them. Fixed: visible `role="alert"` banner `数据请求提交失败：…`. Regression `src/lib/wizardError.test.ts` asserts the template (extracted via `?raw`) renders `{submitErr}` inside an alert — never fake data, no test weakening. + - Error contract kept structured: instruments still returns JSON `{items,source,status}`, datasets still `{error:{code,message,...}}` JSON errors. No catalog/data faked; no tests weakened. + +16. **Mobile 375px overflow repair round 5 (live browser DOM findings)** + - Parent probe (`artifacts/qa/supervisor-live/mobile-overflow-probe.json`, DOM-authoritative) measured document scrollWidth 532→552 at a 375px viewport; sidebar/main 552px; wizard cards 523px. Root causes fixed in the relevant real CSS — nothing hidden globally (`overflow-x:hidden` explicitly absent, asserted by the regression). + - `frontend/src/components/Layout.svelte`: `.layout` mobile media uses `grid-template-columns: minmax(0, 1fr)` (must be able to shrink below cell min-content); `.content` gets `min-width:0` (grid min-content propagation caused the 552px inflation); `.projctx` (project tab row) now `flex-wrap: wrap` with `.projname` `overflow-wrap: anywhere`; sidebar row stays `min-width:0`. + - `frontend/src/app.css`: ≤900px media block collapses `.grid2`/`.grid3` to `minmax(0,1fr)` — the dataset wizard grid3 + min-width labels measured min-content 523px; `table.data th/td` switch to `white-space: normal; overflow-wrap: anywhere` on narrow viewports (dataset `.kv` and provider tables stop inflating min-content; truly wide tabular blocks keep their existing local `.scrollx` scroll container so content stays fully accessible); `.btn` allows wrapping; `.banner` and err lists use `overflow-wrap: anywhere` so long real provider error messages (eastmoney/tencent URL payloads) wrap instead of inflating; `.card/.stack/.row/.spread/.banner` get `min-width:0`. + - Regression `frontend/src/lib/mobileCss.test.ts` reads the actual source files (CSS `?raw` vitest import is empty-stubbed, so disk reads with `node:fs` + `fileURLToPath(process.cwd())` are used) and asserts: minmax(0,1fr) media layout, `.content` `min-width:0`, `.projctx` wrap, `.grid2/.grid3` collapse, table wrap, banner `overflow-wrap:anywhere`, and that NO `body/html overflow-x:hidden` is introduced (accessibility guard), plus DatasetCard retains `.scrollx` local scrolling. + - Results: 27 frontend tests pass (7 files), `npm run check` 0 errors/0 warnings, `npm run build` OK; built `dist/assets/*.css` verified to contain the minmax(0,1fr) and overflow-wrap rules. Parent owns the live 375px recheck; screenshot expansion caveat recorded (DOM authoritative). + +17. **Strategy tab numeric generation contract (live pageerror regression)** + - Parent live run (`artifacts/qa/supervisor-live/browser-strategy.json`) failed immediately: pageerror `c(...).slice is not a function` and a blank main — backend `projects.rs` keeps `draft_generation` as an integer (i64) while the frontend typed it `string` and called `.slice(0,8)` on it. The sidebar stayed blank (`当前项目 加载中…`). + - Contract fixed consistently, backend integer APs **retained**: + - `frontend/src/lib/types.ts`: `Project.draft_generation: number`, `AIAssist.base_generation: number`. + - `frontend/src/lib/client.ts`: `putDraft/postRestore/postAIAssist/postAIAccept` take `expected_generation: number` (matches backend `i64` `PutDraft`, `RestoreReq`, ai assist/accept parsers — audited). + - `frontend/src/lib/draftGuard.ts`: `ServerDraft/ConflictResolver/retryPlan` generation numeric — optimistic conflict guards unchanged in behaviour (stale 409 → compare-and-choose, retry with server generation) — and `draftGuard.test.ts` numerals updated. + - `ProjectStrategyTab.svelte`: `baseGeneration` numeric; all `draft_generation.slice(0,8)`/`base_generation.slice(0,8)` removed (display full numeric 草稿代; strict numeric `!==` still drives the AI stale banner, which retains its honest warning without crashing). + - `ProjectsPage.svelte`: plain numeric rendering instead of `.slice`. + - Regression tests (`frontend/src/lib/strategyTab.test.ts`) exercise the **actual numeric API shape**, not source strings: the real component is mounted in jsdom (vite `resolve.conditions: ['browser']` enables svelte's client build in vitest) with numeric `draft_generation` and asserts editor content + `草稿代 3` render without pageerrors; the real `putDraft` client sends numeric `expected_generation` (verified over the wire body) and accepts numeric response; 409 maps to `ApiError code stale_generation` (retry flow intact). + - RED/GREEN: with one `.slice` temporarily reintroduced in the template the mounted test failed with exactly `get(...).slice is not a function` (equivalent of the live pageerror); GREEN after the numeric contract. + - Adjacent version contract audited works: `/projects/:id/versions` newest-first (ORDER BY created_at ASC + reverse) matches frontend compareTwoVersions(i=新, prev=i+1=旧); `/projects/:id/versions/:vid` returns `{code}` owner-checked so both compare buttons (draft compare and version-pair compare) function; restore passes numeric generation and backend `RestoreReq` expects i64 — consistent. + - Results: 30 frontend tests (9 files), svelte-check 0/0, `frontend/dist` rebuilt. Backend untouched (all 54 still green; debug build re-verified). NOT browser-verified by this writer — parent owns live recheck; no services/tunnel/credentials/commits. + +## Final state of this round +- frontend: 30 tests / 9 files green; svelte-check 0 errors / 0 warnings; build OK +- backend: unchanged (54/0); debug build verified +- No services, tunnel, credentials, parent QA scripts, or commits touched. +- `cargo test` (server): 54 passed / 0 failed (RED→GREEN on both new backend regressions; RED receipt for the 422 is parent's live `artifacts/qa/supervisor-live/browser-workflow.json`) +- `cargo build` (debug) + `cargo build --release`: OK, binary rebuilt, NOT launched +- frontend: 23 tests / `npm run check` 0 errors 0 warnings / `npm run build` OK +- No services, credentials, tunnel, Docker daemon, profiles, or commits touched; parent verifies live browser after builds. + +## Files touched +- server/src/jobs.rs (claim fix + preview/coverage rows + regression tests) +- server/src/datasets.rs (request object, auto name) +- server/src/main.rs (capabilities objects, instruments source; bounded SIGTERM/SIGINT graceful shutdown with 10s drain) +- server/src/worker.rs (--sig-proxy=false, kill_on_drop(true), execute_docker_named injection, focused shutdown regressions) +- server/src/auth.rs (AuthUser.session_id, sessions `current`, test fixture seed) +- server/src/admin.rs (test fixtures aligned; missing FK dataset seed) + +## Notes for parent live QA +- Restart the parent-managed 8787 service to pick up these changes (rebuild binary first: `cargo build --release --manifest-path server/Cargo.toml` as needed). +- `frontend/dist` is already rebuilt for serving static SPA. +- `scripts/qa_api.py --market` gates all verified logically consistent with the shapes above; run them against the restarted service for final evidence in `artifacts/qa`. +- No secrets read or printed; API key remains server-only env `OPENCODE_GO_API_KEY`. +18. **Results accounting repair (parent independent verification round)** + - Parent independently verified the numeric generation fix (frontend 30 tests, check 0/0, built; live authenticated browser: editor autosave/version/diff/real backtest/equity SVG/drawdown/mobile 375 all pass; evidence `artifacts/qa/supervisor-live/browser-strategy.json`). + - Real finding 1 — trade_count semantics: worker `backtest.py` counts CLOSED ROUND TRIPS while `ProjectResultsTab.svelte` labelled it 交易笔数 with a definition claiming fills removed. Fixed honest & consistent: results label is now 平仓回合数, defined as 已完成的开仓并全部平仓的完整回合（1 回合 = 1 次平仓），不含撤单与未平仓的单笔成交; `metricsLabels()` (fed to run comparison tables) renamed from 交易次数 to the same 平仓回合数 so comparisons match; `BacktestMetrics.trade_count` doc-comment states closed round-trip semantics. + - Real finding 2 — fill value cost basis: worker recorded `abs(ex.value)` per fill; Backtrader's `ex.value` for SELL orders reports the position cost basis, NOT sale proceeds (live evidence: identical 659.66 for a real buy and sell at different prices). Fixed `worker/backtest.py` to record actual turnover `abs(ex.size * ex.price)`; commission unchanged (rate × turnover stays consistent because Backtrader's comission applies per the same scheme); audit confirmed equity/cash/metrics code untouched (broker accounting uses executed price + commission independently of displayed fill value). + - Regression: `tests/worker/test_backtest.py::test_fill_value_is_executed_turnover_not_cost_basis` — buy at bar 3 + sell at bar 10; asserts each fill value == quantity×price, commission == value×rate, buy/sell prices differ so turnovers differ (RED verified: reverting to `abs(ex.value)` fails at the turnover assertion), and trade_count == 1 (still closed round trips). Full worker suite: 38 passed. + - Result metrics desktop layout: `.metric-grid`/`.metric`/`.mlabel`/`.mvalue` (plus `.cols-2`, `.table-wrap`) had NO stylesheet definitions at all — six stacked metrics with large blank space. Added shared layout CSS in `app.css` (auto-fit minmax grid, tabular-nums values) and a ≤900px `cols-2` collapse; all consumers (results card, UsagePage metric rows) inherit; no unrelated UI touched. New regression `frontend/src/lib/resultsAccounting.test.ts` mounts the real results tab (jsdom, browser build) asserting: 平仓回合数 label + honest definition rendered, trade turnover rows show two distinct values (1020.00 buy / 1040.00 sell), six `.metric` tiles, comparison labels contain 平仓回合数 and not 交易次数, and the source contains no leftover 成交次数 wording. + - CodeMirror jsdom noise: suite emitted unhandled `textRange(...).getClientRects is not a function` while passing. Narrow geometry shim `frontend/src/test-setup.ts` (wired via vite `test.setupFiles`) implements ONLY the missing `Range.prototype.getClientRects` returning one empty rect (CodeMirror derives null geometry); no arbitrary error suppression. + - Results: frontend 33 tests / 10 files green with zero unhandled errors, svelte-check 0/0, dist rebuilt; worker pytest 38/0 including the new accounting regression (RED/GREEN). Backend untouched (54/0 previously verified; no rebuild). Parent owns worker image rebuild + live recheck; no services/Docker/credentials/tunnels/commits. +19. **Sidebar current-project repair + test noise + .dockerignore** + - Parent live finding: sidebar 当前项目 stayed 加载中… even with the project tab loaded — `projectOf()` only read the (empty) projects store and nothing ever fetched for a deep link; ProjectPage/ProjectDataTab did their own loads without feeding the sidebar store; the sidebar's `singleCache` was non-reactive ($state-less) so even cache hits never updated the derived name. + - Fix (frontend only): `listsStore.svelte.ts` — single placement-only $state array; NEW `projectsStore.ensure(id)` dedupes concurrent fetches (inflight map), `upsert` writes reactively, `get/projectOf` read only the $state array so the backtick-name update is instant on navigation or in-tab bump; 404/403 marks the id invisible in a $state record (`projectInvisible`) so the sidebar turns into an honest `不可见项目` instead of an endless 加载中…, while retry-able failures (network, 500) are never remembered as invisible. Success chain preserved: any in-tab save/restore/AI bump updates the sidebar in place without extra fetches. + - Cross-user stale-leakage guard: `Layout.logout()` calls `projectsStore.clear()` dropping cached project names AND invisible marks, so the next session on the same browser starts clean (no previous user's project names, no stale "cannot see" states). + - Layout fix: `$effect` on the project route fetches the missing project once via `ensure` (deep links + first navigation, exactly one direct /projects/:id GET, deduped). Non-visible ids render `不可见项目`; retryable errors keep honest `加载中…`. + - Regression `frontend/src/lib/sidebarCurrentProject.test.ts` (mounts the real Layout, 4 tests): deep-link name resolution with exactly ONE project fetch; tab-to-tab navigation of the same project resolves from cache (no refetch) and bumps update the name in place; 404 → `不可见项目` with retry guard (exactly one fetch across remounts) plus logout clearing marks/caches (cross-user leak check); transient network error is NOT remembered invisible (honest 加载中, later retry resolves and renders). Bug found while writing these: invisible marks initially in a non-reactive Set — fixed to a $state record so the derived sidebar state updates. + - ECharts jsdom noise eliminated at the right boundaries, none suppressed: (a) `Chart.svelte` passes explicit real sizes to `echarts.init` (`host.clientWidth || 640 / host.clientHeight || height`) so jsdom's zero-layout no longer prints the `[ECharts] Can't get DOM width or height` warning — in a real browser the measured client size is used identically and ResizeObserver still handles resize; (b) `src/test-setup.ts` replaces `HTMLCanvasElement.getContext` ONLY because jsdom's native one is a throwing stub — a minimal 2d-context providing `measureText` (the single canvas usage behind zrender text metrics); SVG chart rendering itself stays real and is still asserted by regression tests; (c) previous `Range.getClientRects` shim retained for CodeMirror. + - Root `.dockerignore` added, tailored to `worker/Dockerfile` (which only COPYs `requirements-worker.txt` and `worker`): whitelist `**/*` + `!requirements-worker.txt` + `!worker` + `!worker/**` (last-match-wins), plus local-cache guards (`worker/__pycache__`, `worker/**/__pycache__`, `*.pyc`, `.pytest_cache`, `.venv`). Verified by reimplementing moby's patternmatcher semantics (per-path last-match-wins, `**` matching zero-or-more dirs incl. dot entries) walking the real tree: 24k files pruned to exactly {requirements-worker.txt, worker/*}; zero Dockerfile-needed files dropped; no build-context leakage of artifacts/, server/, docs/, frontend/, .venv, .pytest_cache. Docker build itself left to the parent as required. + - Outcomes (this round's actual results): frontend 37 tests / 11 files green with ZERO unhandled/stderr noise (先前 docs/tests 33 → +4 sidebar tests); svelte-check 0 errors / 0 warnings; `frontend/dist` rebuilt. Worker tests unchanged from item 18 (38/0). Backend untouched (54/0 previously verified). Parent owns worker image rebuild + live recheck; no services/Docker builds/credentials/tunnels/commits. +20. **Mobile results-viewport overflow (chart SVG sticky width) — live root cause + repair** + - Parent repeatable live failure reproduced and root-caused with real browser evidence (`artifacts/qa/supervisor-live-oce/overflow-probe.json` + before-gpxs): navigating desktop→results then resizing to 375 → `document.scrollWidth = 1133`; fresh 375 deep link was already clean (375). The only overflowing DOM was the ECharts chart subtree (svg/rect/g/path at width 1101px) — no table, layout or card element overflowed, so nothing was hidden; tables already keep their scroll container (`.table-wrap`, overflow-x auto). + - Root cause: `Chart.svelte` initialized echarts with an explicit measured width, and ECharts KEEPS the explicitly passed size across `chart.resize()`; the ResizeObserver handler called `chart.resize()` with no measured size, so after the desktop→mobile viewport change the SVG kept the desktop width (the desktop graph's 1101px width overflowed the 375 viewport). + - Fix (frontend only, root cause, not a mask): the ResizeObserver callback now re-measures the host on EVERY resize event and passes the measured size explicitly: `chart.resize({ width: host.clientWidth || undefined, height: host.clientHeight || undefined })`, skipping degenerate zero-layout resizes. Real charts stay fully rendered and responsive; jsdom real-SVG chart verification retained. + - Regression `frontend/src/lib/chartResize.test.ts` (2 tests) drives a real echarts SVG chart in jsdom: init/fallback width is measured (640), the SVG follows the host width through a 900px "desktop" resize AND shrinks 900→340 on the "mobile" resize (RED verified: with `chart.resize()` restored it fails on the sticky 900px SVG), and a zero-layout host callback is a harmless no-op. + - Live verification (`artifacts/qa/supervisor-live-oce/chartcam-checks.json`, verdict PASS, screenshots chartcam-*.png; used QA venv Playwright with the private account file, credentials never logged): desktop→375 resize settled: scrollWidth 375 == clientWidth 375, maxElementRight 375, chart SVG 1101→311; fresh mobile deep link: 375/375; mobile→desktop: 1440/1440 (SVG 1101); desktop→mobile again: 375/375 (SVG 311); zero pageerrors in all four cases. The earlier probe's "FAIL" verdict string was my own probe's flawed predicate (desktop SVG 1101 > 375 bench); the artifact as stored is PASS with the corrected invariant scrollWidth==clientWidth and svgWidth<=clientWidth per case. + - Outcomes: frontend **39 tests / 12 files** green (37→39, +2 chart regression), zero unhandled/stderr noise, svelte-check 0/0, `frontend/dist` rebuilt; charts resize, no overflow hiding, assertion `width == scrollWidth` preserved and now passing on the exact parent failure path. Worker/backend untouched (pytest 38/0, cargo 54/0 previously). Parent owns the live sidebar re-run + Docker rebuild; no services/ports/tunnel/production/supervisor changes; single worker, no parallel writers. diff --git a/docs/live-data-probe.md b/docs/live-data-probe.md new file mode 100644 index 0000000..65b9b52 --- /dev/null +++ b/docs/live-data-probe.md @@ -0,0 +1,5 @@ +# Parent live data probe + +Executed actual AKShare `.venv/bin/python` on this host: +`ak.stock_zh_a_hist_tx(symbol='sh600000',start_date='20240101',end_date='20240630',adjust='')` +Succeeded: 117 rows, columns date/open/close/high/low/volume/turnover/amount. First bar 2024-01-02 open6.63 close6.60; last 2024-06-28 open8.22 close8.23. Same exact equity 浦发银行 600000; no proxy substitution. Tencent adapter's volume already shares (first22066700); inspect source before applying multipliers. This is a verified alternate to failing Eastmoney, record endpoint/provider and warning when selected; never claim Eastmoney supplied it. Parent black-box real-flow requests this unadjusted interval. Can add explicit source choice tencent/eastmoney/auto with documented failover and source-distinct cache identity. diff --git a/docs/parent-ui-findings.md b/docs/parent-ui-findings.md new file mode 100644 index 0000000..8dc1c39 --- /dev/null +++ b/docs/parent-ui-findings.md @@ -0,0 +1,12 @@ +# Parent authenticated browser findings (real QA server) + +Verified with Playwright on http://127.0.0.1:8787 (isolated QA database). +- Real admin login succeeded. Global projects/datasets/runs/usage/account/admin routes rendered without pageerror. +- Create project via UI succeeded, real URL /#/projects/37d25978-89ba-42dc-9b2f-73fdb1b32642/data. +- BUG: project page renders TWO complete sidebars/layouts (both App.svelte and ProjectPage.svelte wrap Layout). Remove inner Layout, keep one current-project sidebar/tab context; preserve page props. +- Account session list has no current marker (backend response missing current), renders all with 撤销; add current bool from authenticated token. +- Source API request in Dataset must be JSON object, not string; dataset preview/capabilities schema need actual browser exercised. +- Public auth routes now render correctly. Existing frontend tests21 pass, check0errors0warnings; baseline worker35 pass. +- QA scheduler run d5cd9da7-ccee-4fa2-99d3-43f3e2fced73 was stuck running with container_id NULL due claim ordering (already assigned to integration). + +Parent owns all service processes. Never pkill/killall/kill/systemctl. Current QA service strategy-lab-qa on8787; build/test only and parent restarts. diff --git a/docs/production-local.md b/docs/production-local.md new file mode 100644 index 0000000..c7c80fd --- /dev/null +++ b/docs/production-local.md @@ -0,0 +1,13 @@ +# Isolated local production candidate + +Persistent unit `strategy-lab-production.service` uses copied release binary/frontend under `~/.local/share/strategy-lab-production/release`, listening only on http://127.0.0.1:8789. QA stays on8787. Public HTTPS entry is https://fin.somhairle.bid through the existing Cloudflare tunnel; only this hostname was appended before the existing 404 fallback. Existing dav/git/linkwarden routes and http2 were preserved. + +Mode0600 files `runtime.env` and `admin.private.json` live in that mode0700 state directory. Owner retrieves credentials locally; never paste into logs/chat. Bootstrap credentials were supplied only to initial bootstrap process, absent from persistent runtime env. Clean DB has one admin and no QA data. + +Explicit domain authorization is recorded in docs/domain-authorization-user.md and supersedes earlier domain blockers. Runtime ORIGIN=https://fin.somhairle.bid; production was restarted and secure-cookie/authentication/exact-origin rejection verified before exposure. Public browser form login, Secure/HttpOnly session cookies, foreign-origin403, first-party JS/CSS release hash equality and desktop/mobile rendering pass. DNS route exact tunnel readback and ingress validation pass. Evidence: artifacts/qa/supervisor-live/{public-verification.json,release-acceptance.json,qa_public.py}. External Cloudflare analytics script is recorded separately from first-party release files. + +SQLite initial online backup passed independently reopened integrity/count checks. Populated QA database/object backup and isolated restore-service verification passed: exact object SHA-256 equality, SQLite integrity, populated entity counts, all ready dataset object references, restored login/draft/preview/run results. Runnable verifier and evidence: artifacts/qa/supervisor-live/{qa_backup_restore.py,backup-restore-checks.json}. Private snapshots remain under the mode0700 cache directory named in evidence, never in source control. Procedure: ensure no pending/running work, stop the exact source service, SQLite backup plus copy data/objects, restart source, copy snapshot to a separate directory, verify integrity/hashes before launching a loopback-only restore with DB_PATH/DATA_DIR pointing at that copy. Never overwrite live DB or mix QA into production. Production uses persistent systemctl start; QA uses a transient unit which disappears after stop and must be recreated via the exact launcher command in the verifier. Restored instance was terminated after checks; production was untouched. Do not restore over a running DB. Unit is enabled and loginctl confirms Linger=yes. Independently restarted the exact production unit: new PID, copied release executable confirmed, actual HTTP health ready in0.061s. A physical host reboot has not been performed. + +Measured seven systemd cgroup samples over30.050s immediately after restart: memory2,416,640–12,435,456 bytes, settling to2,416,640 bytes; CPU0.06393% of one core over that interval. These are idle service measurements, exclude Docker job memory and do not predict backtest load. Runnable verifier and raw samples: artifacts/qa/supervisor-live/{qa_production_persistence.py,production-persistence-checks.json}. + +Operations: systemctl --user status/restart strategy-lab-production. Limits: one backtest/one fetch,120s fetch/60s backtest deadlines. Trusted invited-user Docker POC, no hostile public sandbox claim. Evidence: artifacts/qa/supervisor-live/production-local-checks.json. diff --git a/docs/recovery-01-plan.md b/docs/recovery-01-plan.md new file mode 100644 index 0000000..14b54f9 --- /dev/null +++ b/docs/recovery-01-plan.md @@ -0,0 +1,90 @@ +# Recovery 01 plan: 159399 listed-ETF ingestion and failure semantics + +Evidence-based plan. All live evidence below was collected 2026-09-17 with bounded +real calls (single attempts, ≤15s timeouts, no retries). Skill loading is recorded +in the session transcript: systematic-debugging, test-driven-development, +writing-plans were invoked via the native skill tool before this plan; if not +visible, fallback statement: skills were loaded with the skill tool and their +content is authoritative for this task. + +## Symptom (reproduced) + +User defect: cash-flow ETF 159399, cn, 2025-12-31..2026-09-17, daily, unadjusted +fails with `eastmoney failed (ConnectionError: RemoteDisconnected ...)` then +`tencent fallback failed (tencent source has no listed-ETF daily adapter)`. + +## Root cause (Phase 1 evidence) + +1. **Eastmoney kline host is genuinely unreachable from this environment.** + Direct replay of exactly what akshare 1.18.94's `fund_etf_hist_em` sends + (`https://push2his.eastmoney.com/api/qt/stock/kline/get`, secid 0.159399, + both with no User-Agent and with a browser UA) → + `ConnectionError: RemoteDisconnected('Remote end closed connection without response')` + in both cases. Hypotheses (wrong UA, missing header) ruled out. No code bug in + our worker path — `_fetch_eastmoney` is correct; the upstream endpoint is + refusing/dropping our connections. +2. **Tencent genuinely has no listed-ETF daily adapter.** Verified against the + installed akshare 1.18.94 surface: only + `fund_etf_hist_em`, `fund_etf_hist_min_em`, `fund_etf_hist_sina` exist. + The existing error at `worker/data.py:124` is accurate, not a bug. +3. **A genuine alternative provider exists: sina.** Live probe + `ak.fund_etf_hist_sina(symbol="sz159399")` → DataFrame, 381 rows, + columns `date,open,high,low,close,volume,amount,postVol,postAmt`, + prices decimal CNY, **volume unit is 股 (shares)** — cross-verified live on the + same session: for stock 000001 2026-09-16 tencent reports volume 949,626 (手) + while sina reports 94,962,632 (股), a consistent ×100, with identical turnover. + Returns full history (no date-range parameter); the requested window must be + sliced locally. Unadjusted only (no adjust parameter). +4. **Identity.** `split_identity("159399")` on an cn ETF already resolves to + SZ#159399 (6-digit code not starting 3/6/9 → SZ). Preserved. +5. **Resulting behavior.** With source=auto, eastmoney fails → tencent ETF + fallback is a designed explicit rejection → `provider_unavailable` failure. + The failure message is truthful but the product is unusable for ETFs while + eastmoney is down even though a genuine provider exists and is live. + +## Adjustment behaviors per provider (verified, not assumed) + +- eastmoney `fund_etf_hist_em`: adjust none|qfq|hfq — currently unreachable. +- sina `fund_etf_hist_sina`: unadjusted only; no date params; volume in 股. +- tencent: no listed-ETF daily adapter (explicit rejection). + +## Chosen repair (smallest correct change, worker only) + +Backend inspection: `server/src/datasets.rs` validates frequency/adjustment/ +asset_type but does not pass or validate a `source` field; jobs.rs builds the +fetch request without it. So the fix is entirely in the worker: + +1. Add `_fetch_sina` (asset_type etf, frequency daily, adjustment none only). + Local slice to requested dates; keep provider numbers verbatim (volume 股); + attach `source_warnings` recording units and that eastmoney-style adjustments + are not available from sina. +2. Register `sina` in `SUPPORTED_SOURCES` and in worker `ALLOWED_SOURCES` so it + is both explicit and honestly reportable. auto chain for ETF becomes + eastmoney → sina (same symbol, honestly labeled provider_fallback + units + warning). qfq/hfq with eastmoney down still fails honestly (sina cannot serve). +3. Explicit-source semantics unchanged: `source=tencent` + etf still raises the + accurate `source_unavailable` message. + +## TDD steps + +Each step: failing test → run → minimal code → run. + +1. Test: sina adapter normalizes a live-captured-shape sina frame + (fixture labeled synthetic) for 159399 → SZ#159399, sliced to requested + range, provider="sina", endpoint="fund_etf_hist_sina", units warning present. +2. Test: sina rejects adjustment qfq/hfq and non-daily frequency with explicit + `unsupported_*` errors. +3. Test: auto ETF chain — eastmoney monkeypatched to fail → served by sina with + `provider_fallback` warning naming eastmoney/sina honestly. +4. Test: explicit sina request for stock/index rejected (out of verified scope). +5. Run full worker suite; then live bounded end-to-end run of the exact + requested dataset via `python -m worker.main fetch` into an isolated output + dir; report exact rows/coverage/source/units; no production touched. + +## Constraints honored + +- All network calls bounded (≤15s per call, one call per attempt, 180s wall + clock via SIGALRM already in main.py). No retries, no auto-refetch of user + data, no production writes, no deployment, no commits. +- The user's failed dataset is never mutated; coverage/consumer decisions + reported truthfully (2026-09-17 has no EOD bar yet — intraday at report time). diff --git a/docs/recovery-01-results.md b/docs/recovery-01-results.md new file mode 100644 index 0000000..0e57ab7 --- /dev/null +++ b/docs/recovery-01-results.md @@ -0,0 +1,136 @@ +# Recovery 01 results — real listed-ETF ingestion and failure semantics + +Task: docs/recovery-task-01.md. Date: 2026-09-17. Scope: this repo. No +deployment, no production image replacement, no commits, no user DB access, +no restarts. The user's failed dataset was not refetched or mutated. + +## Skills actually loaded (native skill tool calls, this session) + +1. systematic-debugging — loaded before any fix attempt. +2. test-driven-development — loaded; RED→GREEN cycles recorded below. +3. writing-plans — loaded; plan written to docs/recovery-01-plan.md. +4. verification-before-completion — loaded before this handoff. +No fallback to manual SKILL.md reads was needed. Discovery succeeded. + +## Root cause (with real upstream evidence) + +- Symptom reproduced live first attempt: `ak.fund_etf_hist_em(symbol="159399", + start 20251231, end 20260917)` → + `ConnectionError: RemoteDisconnected('Remote end closed connection without response')`. +- Direct replay of the exact endpoint akshare calls + (`https://push2his.eastmoney.com/api/qt/stock/kline/get`, secid `0.159399`) + with no UA and with a browser UA both failed identically → the eastmoney + kline host is genuinely refusing/silently dropping connections from this + environment. Not a UA/timeout bug, not a code bug in the worker. +- Verified against installed akshare 1.18.94: there is no tencent listed-ETF + daily adapter (`fund_etf_hist_em`, `fund_etf_hist_min_em`, + `fund_etf_hist_sina` are the only ETF end-of-day ETF adapters). The + pre-existing `source_unavailable` message at worker/data.py:124 is accurate. +- Live-verified alternative provider: `ak.fund_etf_hist_sina(symbol="sz159399")` + → 381 rows (2025-02-27..2026-09-16), columns + `date,open,high,low,close,volume,amount,postVol,postAmt`, unadjusted, no date + parameters (full history; must be sliced locally), volume unit 股 (shares). +- Units cross-verified live on the same session with stock sz000001 2026-09-16: + tencent volume 949,626 (lots, 手) vs sina 94,962,632 (shares, 股) — exact ×100, + identical turnover (1,106,652,900 vs 1,106,652,940). Hence honest units + warnings on sina output rather than silent rescaling. + +## Change (smallest correct; worker + trivial frontend label) + +- `worker/data.py`: + - new `_fetch_sina`: etf-only, daily-only, adjustment none-only (explicit + `unsupported_*` rejections otherwise), sina symbol `sz159399`, full-history + fetch sliced locally to the requested window, canonical identity + `SZ#159399` preserved, verbatim provider numbers, source_warnings with + slicing/units/unadjusted disclosure. + - `SUPPORTED_SOURCES` now `{eastmoney, tencent, sina, auto}`; auto chain for + ETF: eastmoney → sina (honestly labeled `provider_fallback` warning naming + both providers); stock/index auto chains unchanged (eastmoney → tencent). + - explicit `source=tencent` + etf still raises the unchanged, accurate + "tencent source has no listed-ETF daily adapter". +- `worker/main.py`: `ALLOWED_SOURCES` += `sina`. +- `frontend/src/lib/format.ts`: `sina` → `新浪` source label (display only, + no UI redesign). Backend (server/) unchanged: it never passes or validates + a source field (datasets.rs validates frequency/adjustment/asset_type only). + +## TDD evidence + +Tests added in `tests/worker/test_data.py` (synthetic fixtures are clearly +labeled synthetic; live-captured provider shapes are quoted in comments): + +- test_sina_etf_serves_159399_with_honest_provenance +- test_sina_rejects_adjustment_and_range_semantics +- test_sina_rejects_stock_and_index +- test_auto_etf_falls_back_from_eastmoney_to_sina_honestly +- test_auto_etf_fails_when_no_provider_is_viable +- test_tencent_etf_rejection_message_unchanged +- test_sina_registered_in_explicit_source_surface + +RED: 6 of the above failed against the unmodified worker (the tencent +rejection test passed as pre-existing behavior). GREEN after implementation. + +## Verification commands and fresh results + +- `python -m pytest tests/worker -q` → **56 passed** (0.79s), final run. +- `python -m pytest tests -q` → **56 passed** (0.72s), final run. +- `npm run check` (frontend) → **0 errors, 0 warnings**. +- `npm test` (frontend) → **13 files / 45 tests passed**. +- Live exact-request fetch (real upstreams, single bounded attempts, + SIGALRM 180s wall clock): + `python -m worker.main fetch --request artifacts/recovery-01/req159399.json + --output artifacts/recovery-01/etf_out` → **status: ready**, ~0.9s. + +## Exact reported dataset facts (159399, requested 2025-12-31..2026-09-17) + +- provider **sina**, endpoint `fund_etf_hist_sina`, params `{"symbol": "sz159399"}` +- row_count **173**, actual_start **2025-12-31**, actual_end **2026-09-16** +- columns: `date,open,high,low,close,volume,amount,symbol` (amount requested) +- adjustment **none (unadjusted)**; prices CNY decimals; **volume in 股 + (shares)** — 100× the lot convention of eastmoney/tencent (verified ×100 on + the same session, not silently converted) +- 2026-09-17 has **no bar**: today's session end-of-day not yet published; the + warning "actual coverage 2025-12-31..2026-09-16 differs from request; gaps + kept" is on the manifest. All calendar gaps present-truthful (holidays etc.). +- identity preserved all the way through: normalized symbol `159399` CSV and + canonical `SZ#159399` contract in adapter tests. +- fetch → persisted raw JSON (`objects/raw_159399_*.json`) + normalized CSV + (`objects/159399.csv`) + preview contract in `result.json` (columns, + row_count, coverage, last-20 rows), exercised outside production in + `artifacts/recovery-01/etf_out/`. + +## Comparison cases (truthful, live attempts) + +- stock 600000 auto (unchanged path): attempt 1 failed (eastmoney down + + tencent proxy.finance.qq.com connect timeout); attempt 2 → **ready** + provider **tencent**, endpoint stock_zh_a_hist_tx, 173 rows, + 2025-12-31..2026-09-16. +- index 000300 auto (unchanged path): attempts 1–2 failed — attempt 1 tencent + proxy connect timeout; attempt 2 eastmoney RemoteDisconnected + tencent + returned an empty payload to akshare ("Length mismatch"). Environmental, + pre-existing flakiness; not caused by this change and out of this task's + repair scope. +- 159399 with source=tencent: explicitly, correctly rejected + (source_unavailable). 159399 with qfq: with eastmoney down auto now fails + honestly because sina has no adjustment factors (tested). + +## Remaining blockers + +1. eastmoney push2his connectivity from this environment is down (affects + stock/index first choice too); recovery is environmental, workarounds live + in fallback chains. +2. tencent proxy.finance.qq.com flaky/intermittent under this environment; + index fetch unverified live in this session. +3. Full release gate (docs/skill-assisted-development.md) is a separate + acceptance workflow; this task only fixes worker ETF ingestion and its + failure semantics. Parent independent verification still required. + +## Changed files + +- worker/data.py +- worker/main.py +- tests/worker/test_data.py (new tests only) +- frontend/src/lib/format.ts (one-line source label) +- docs/recovery-01-plan.md (new) +- docs/recovery-01-results.md (this file, new) +- artifacts/recovery-01/ (isolated live-fetch artifacts: 3 request files, + etf_out/, stock_out/, index_out/) diff --git a/docs/recovery-task-01.md b/docs/recovery-task-01.md new file mode 100644 index 0000000..fd4132b --- /dev/null +++ b/docs/recovery-task-01.md @@ -0,0 +1,15 @@ +# Recovery 01: real listed-ETF ingestion and failure semantics + +Use strategy-go/glm-5.3-flash. One integration writer only. Scope this repository. No deployment, service restarts, user DB reads/writes, credentials/Tunnel access, global skill modifications, commits or pushes. Existing production must remain unchanged. External market-data reads permitted; no fabricated results. + +Before implementation call native skill tool for systematic-debugging, test-driven-development, writing-plans. These are installed under .opencode/skills/. Record actual skill tool calls. Load verification-before-completion before handoff. If discovery fails, read these exact local SKILL.md files and explicitly report fallback. + +User defect: cn.159399 cash-flow ETF dataset2025-12-31..2026-09-17 daily unadjusted OHLCV fails with Eastmoney RemoteDisconnected followed by 'tencent source has no listed-ETF daily adapter'. Canonical requested symbol is159399; preserve identity and classify ETF correctly. Do not disguise fallback provider, units, adjustment or incomplete date coverage. Stock-only live probe success does not validate ETF. + +1. Read current architecture and code; write a concise evidence-based reproduction/repair plan in docs/recovery-01-plan.md. Reproduce provider failure or actual ETF data fetch with bounded requests using real upstreams. Identify available genuine provider ETF endpoints and returned schema, units and adjustment behavior. If no viable adapter can be verified, report blocker instead of inventing bars. +2. Write failing tests for actual ETF provider response shapes, unavailable provider, timeout, canonical identity and truthful provenance. Implement smallest correct adapter/failure handling change in worker and backend if needed; do not redesign UI during this task. All network requests bounded. Explicit final failure and source capabilities must be accurate. Do not silently substitute synthetic fixtures for live data; label synthetic test fixtures. +3. Test159399 exact requested range and supported stock/index/ETF comparison cases; dates after available trading session may legitimately have no bar, report coverage truthfully. Report exact row counts/start/end/source/columns/units and runtime. Exercise fetch -> persisted isolated dataset -> preview contract if feasible without touching production. Do not automatically retry or mutate user's failed dataset. +4. Run relevant worker tests and backend tests if changed. Preserve baseline features. Do not spend entire task fixing unrelated code. No deployment or production image replacement. Isolated image tag allowed if needed. +5. Handoff docs/recovery-01-results.md with exact commands/results, remaining blockers and list of changed files. End task; parent independently verifies. + +Broader release gate is docs/skill-assisted-development.md. This task alone cannot declare whole product complete. Frontend redesign with reviewed domain skills and full browser/visual acceptance follows separately. Context budget approx256K; inspect actual compaction evidence if session grows, do not claim configured thresholds prove compaction occurred. diff --git a/docs/search-fix.md b/docs/search-fix.md new file mode 100644 index 0000000..7e27304 --- /dev/null +++ b/docs/search-fix.md @@ -0,0 +1,111 @@ +# Production instrument-search fix — handoff notes + +Date: 2026-09-17. Scope: production search stall on `/api/instruments?q=...` +(0 bytes in ~20s). All work confined to this repo; nothing deployed, user DB +untouched, no services restarted. + +## Root causes fixed + +1. **worker/data.py** — the old `_catalog_frame` fetched the ENTIRE A-share + stock catalog (`stock_info_a_code_name`) and the ENTIRE ETF live snapshot + (`fund_etf_spot_em`) via AKShare on every search query. Those calls are + unbounded and stalled inside the worker container until the Rust-side + runner timeout fired, so the browser's fetch starved and kept spinning. + Replaced with **bounded direct provider suggestion search**: + - Eastmoney `https://searchapi.eastmoney.com/api/suggest/get` + (`input=`, `type=14`, bounded `count`), JSON, per-call timeout **4s**. + - Tencent `https://smartbox.gtimg.cn/s3/?q=&t=all`, text hint + `v_hint="sh~600000~name~py~GP-A^..."`, per-call timeout **4s**. + - No synthetic data, no full catalogs anywhere in the search path anymore. +2. **worker (main.py/data.py) envelope** — search now prints one JSON object + envelope `{"source","status","items","error","providers"}` with explicit + `status: ready|failed`. `run_search` exits non-zero on failure. +3. **server/src/worker.rs** — container runtime bound reduced **120s → 12s** + (outer bound incl. container startup; provider calls are ≤2×4s inside). + The stdout is now parsed as the full JSON envelope; the old code extracted + `first[...]..last[...]` which silently turned a failed-status envelope into + an empty success. A `failed` status now surfaces as a real error + (`search_unavailable`, provider error code in the message), and **only + ready responses enter the server-side TTL cache**. +4. **server/src/main.rs** — `instruments()` success source label is now + `provider_suggest`; failures keep returning HTTP 200 with + `status: unavailable: ...` so the existing UI contract is unchanged. +5. **Frontend** (`DatasetWizard.svelte` + new `lib/instrumentSearch.ts`): + - 350 ms debounce on input, prior in-flight queries are **aborted** + (AbortController) and stale responses can never repaint. + - Hard **AbortController timer (12s)** bound; hung requests surface + 搜索超时，请重试 instead of an eternal spinner. + - Non-`ok` envelope status / provider failure shows + 搜索数据源暂时不可用，请稍后重试 with a **重试** button; the spinner is + always cleared (`onFinish` in `finally`). + - Result rows carry per-item **provenance** (东财 / 腾讯). + +## Asset-class rules (validated identity, never guessed from code shape) + +Live-probed provider classifications (2026-09-17, both endpoints): + +| Provider | Field | accepted | +|-----------|------------------|-----------------------------| +| Eastmoney | `Classify` | `AStock`→stock, `Index`→index, `Fund`→etf (cross-confirmed, see below) | +| Eastmoney | `Classify` (excluded) | `Bond`,`HK`,`OTCFUND`, others | +| Eastmoney | `MktNum` | `1`→SH, `0`→SZ (others dropped) | +| Tencent | hint tag | `GP-A`→stock, `ETF`→etf, `ZS`→index; `LOF`/others excluded | +| Tencent | hint market | `sh`/`sz` only (`bj` dropped) | + +- An asset's class comes only from the provider's own classification field — + a numeric code like `000300` is never inferred to be an index by shape + (test: `test_search_numeric_code_not_auto_index`). +- **Verified live**: Eastmoney `Classify=Fund` is ambiguous — ETF `510300` + and LOF `160706` share the same `Classify` and `SecurityType`. Therefore an + Eastmoney *Fund* item is only emitted when Tencent presents the same code + with tag `ETF`. Tencent alone classifies ETF/LOF unambiguously. When + Tencent unavailable, Eastmoney-only fund suggestions are suppressed + (`test_search_etf_ambiguous_fund_excluded_without_tencent_confirmation`, + `test_search_lof_not_present_when_eastmoney_only`). Guaranteed: no LOF + ever surfaces as an ETF. +- Per-item provenance is preserved: identical codes may appear once per + actual source (`source: eastmoney` / `source: tencent`). + +## Live verification (fresh container `strategy-lab-worker:local`, rebuilt bd9e9f06d56b) + +``` +docker run --rm --network=bridge strategy-lab-worker:local \ + python -m worker.main search --query 600000 --limit 5 +→ {"status":"ready","items":[{"symbol":"600000","...":"浦发银行","asset_type":"stock",...}],...} + wall time ≈1.9s (container start included; previously 0 bytes / 20+s) + +q=510300 → etf SH#510300 (eastmoney+tencent) +q=沪深300 → index SH#000300 / SZ#399300 (+ ETFs), providers ok +q=160706 → {"items":[]} (LOF correctly suppressed, providers ok, status ready) +``` + +## Tests actually run and passing + +- `server`: `cargo test` → **54 passed**; `cargo build --release` OK. +- `worker`: `.venv/bin/python -m pytest tests` (repo root) → **49 passed**, + of which new/updated in `tests/worker/test_data.py` (real shapes for + stock/ETF/index, Chinese query `浦发银行`/`沪深300`, unsupported classes + bond/HK/OTC/LOF, provider failure + `requests.Timeout` → explicit failed + status, partial-provider warning) and `tests/worker/test_fetch_cli.py::test_search_cli_envelope_contract`. +- `frontend`: `npm run test` (vitest) → **45 passed** across 13 files, + including new `src/lib/instrumentSearch.test.ts` (success, unavailable + status → Chinese message, provider rejection, hard-timer abort → 超时 + wording with spinner cleared, **stale-prior-query never repaints and never + hangs**, dispose invalidates); `npm run check` → 0 errors/warnings; + `npm run build` → OK. + +## Deployment state + +- Worker image rebuilt exactly as `strategy-lab-worker:local` (context repo + root, `-f worker/Dockerfile`): id `bd9e9f06d56b`. +- Frontend `dist/` rebuilt from the fixed sources. +- Server binary built release but **not deployed / no restarts performed**. +- Rollback note: previous worker image tag remains on the host; server + changes are code-only (no config/DB changes). + +## Known limitation (documented honestly) + +When Tencent is unavailable and only Eastmoney responds, fund-class +suggestions are suppressed (stock/index still return). This trades a +slightly narrower ETF list for a hard guarantee that no LOF is presented +as an ETF without provider confirmation. diff --git a/docs/skill-assisted-development.md b/docs/skill-assisted-development.md new file mode 100644 index 0000000..4918f34 --- /dev/null +++ b/docs/skill-assisted-development.md @@ -0,0 +1,23 @@ +# Skill-assisted development policy + +User correction: use domain and workflow skills to strengthen OpenCode + GLM-5.3-Flash. Hermes loading a skill does not prove OpenCode has loaded it. Each assigned task must explicitly supply reviewed relevant skill files, permitted supporting references, and concrete acceptance requirements. Record actual skill-read events in the OpenCode transcript; never claim installation or use from a list alone. + +## Discovery (candidates, NOT yet installed or security-reviewed) +- anthropics/skills@frontend-design: visual composition and front-end implementation. +- sveltejs/ai-tools@svelte-code-writer; svelte-core-bestpractices: official Svelte assistance. +- huntabyte/shadcn-svelte@shadcn-svelte: component-system candidate compatible with existing Svelte. +- Axum candidates found through skills CLI require compatibility/security review before selection; popularity alone is not evidence of correctness. +- TypeScript-specific skill still needs discovery/review. + +## Existing local skills to supply by phase +- Discovery/design: superpowers/brainstorming, ui-ux-pro-max, then approved frontend-design and Svelte/component references. +- Planning: superpowers/writing-plans. Freeze API contract and page/interaction specification before broad code changes. +- Implementation: superpowers/test-driven-development plus the actual language/framework skill. One coherent compiling slice at a time. +- Debugging: superpowers/systematic-debugging. Reproduce actual user symptom before patching; model self-reports do not close bugs. +- Review/release: code-review workflow, superpowers/verification-before-completion, dogfood. Real browser use, screenshots/visual review and actual data-source failures required. + +## Loading and safety +Review third-party skill content/scripts before installing into project scope. No blind bulk global installation, hooks, credential reads, downloads or execution dictated by unreviewed skill text. Preserve provenance and pin revisions. Ensure OpenCode's actual discovery paths and permissions expose installed skills; verify with a read-only session before using them. Load skills per task; do not inject every full skill into every prompt. Maintain user's approximate256K working-context budget and inspect actual compaction evidence. + +## Current release gate +No new UI/ETF repair goes live until isolated acceptance passes. Use user's exact159399 ETF, dates2025-12-31 through2026-09-17: create project, search/select, fetch, preview/source/date coverage, strategy/version, backtest/result/reproduction, AI review/accept, failure/retry and mobile. Preserve existing user records. Screenshots must cover loaded, empty, loading and failed states; zero console errors alone is insufficient. Mature component library does not replace design or actual usage. diff --git a/docs/ui-shadcn-handoff.md b/docs/ui-shadcn-handoff.md new file mode 100644 index 0000000..423a6ee --- /dev/null +++ b/docs/ui-shadcn-handoff.md @@ -0,0 +1,49 @@ +# shadcn-svelte 迁移交付说明（第 3 轮） + +状态：READY_FOR_LEADER_REVIEW +日期：2026-09-18 + +## 本轮范围（Leader 第 2 轮返工之后） +1. 修复 AuthPages 375px 溢出 47px（`.auth-card` 固定 400px）→ `width:min(400px,100%)` + flex 包裹。 +2. ECharts → layerchart 全面替换：新增 `src/components/Chart.svelte`（长格式 API）， + 图表合同测试 `src/lib/chartResize.test.ts` 4/4 通过（含 in-place 视口收缩回归）。 +3. 其余页面迁移 shadcn：ResultsTab / StrategyTab / RunsPage（button/input/label/chart）。 +4. 全部页面 × 375/768/1440 三档真浏览器（Playwright chromium）验证： + overflow=0、pageErrors=0，截图与探针 JSON 在 `artifacts/ui-shadcn/`。 + +## 工程要点 +- **Chart.svelte API**（替换旧 echarts `option`）： + ```svelte + + ``` + - data 为长格式行（i 为 x 序号）；`benchmark` 缺省允许 undefined（画断点）。 + - 宽度语义：组件属性 `width` 仅是初始/测试尺寸；浏览器内 ResizeObserver 实测宽度优先，始终跟随宿主，绝不粘滞桌面宽度。 + - 类型对 layerchart 复杂泛型统一 `as never` 断言（svelte-check 0 error）。 +- **易错点（本轮踩过）**： + - runes 组件里 `let measured` 必须 `$state`，否则 RO 回调触发的宽度更新不重渲染。 + - Svelte 5 的 `mount(props { get width() })` 传普通 getter 不会触发更新——响应式必须走 `$state`/真正 signal，或走 RO 实测路径。 + - layerchart 挂载较慢（jsdom 内 ~3s），`resultsAccounting.test.ts` 首测给 20s 超时。 + - `test-setup.ts` 已含 matchMedia shim（layerchart 需要），去掉多余 `@ts-expect-error`。 +- **保留 legacy class**：`banner error/warn/info`、`table.data`、`metric/mvalue`、`badge *` 等继续由 app.css 提供并满足 `mobileCss.test.ts`、`resultsAccounting.test.ts` 断言；shadcn 组件并存无冲突。 + +## 验证记录（全部在本轮最终代码上执行） +- `npm run check`：0 errors, 0 warnings。 +- `npx vitest run`：17 files / 58 tests 全绿。 +- `npm run build`：成功（chunk size 警告属提示，非错误）。 +- `scripts/ui-verify.mjs`（静态 dist + 真浏览器）：login/register/reset/projects/runs/ + dashboards/settings/results/strategy/data × 375/768/1440 = 30 组合，全 OK： + overflow=0、pageErrors=0、bg/fg 均符合主题。 +- `scripts/chart-overflow-probe.mjs`（mock 后端数据，让图表真实渲染）： + runs（勾选 2 个已完成回测触发对比图）与 results（权益/回撤曲线）× 三档共 6 组合， + 全 OK：overflow=0、layerchart SVG 数量 12/16/25 随视口变化、pageErrors=0。 + 截图：`artifacts/ui-shadcn/chart-probe-*.png`。 + +## 后端变更 +本轮未改任何 `src-tauri`/`server`/`worker` 代码。git status 中后端文件的 +modified 状态属此前轮次遗留，不属于本任务 diff。 + +## 交付物 +- 代码：见 git 工作区（frontend 前端 15 文件修改 + `src/lib/components/` shadcn 生成组件 + 5 个新增回归测试）。 +- 文档：本文件。 +- 说明：`/tmp/opencode/probe.test.ts` 探针思路已直接并入 `chartResize.test.ts` 第 3 个用例， + 临时探针文件未提交（按约定不留件）。 diff --git a/docs/worker.md b/docs/worker.md new file mode 100644 index 0000000..6788486 --- /dev/null +++ b/docs/worker.md @@ -0,0 +1,149 @@ +# Python worker — implementation notes and handoff + +Owner: worker agent. Files owned: `worker/`, `tests/worker/`, `requirements-worker.txt`, +`.venv` (shared uv venv at project root, created via `uv venv .venv --python 3.11` + +`uv pip install --python .venv/bin/python -r requirements-worker.txt`), +`artifacts/worker/` evidence, this doc only. + +Status: working worker, 27/27 pytest green (network-free unit tests), real AKShare +fetch + real Backtrader run executed on host and inside Docker image +`strategy-lab-worker:local` (nonroot, read-only rootfs, `--network none` backtest). + +## Components +- `worker/data.py` — AKShare adapters: eastmoney (`stock_zh_a_hist`, + `fund_etf_hist_em`, `index_zh_a_hist`) and tencent (`stock_zh_a_hist_tx`, + `stock_zh_index_daily_tx`). Explicit source selection: + `source: 'eastmoney' | 'tencent' | 'auto'` (default auto). + - `auto` = eastmoney first, then TRANSPARENT same-symbol tencent fallback with + warning `provider_fallback: ...; served by tencent for the SAME symbol`. + Never a different symbol or silent provider/adjustment substitution. + - Explicit `source` never silently falls back; failures surface. + - Tencent adapter returns volume in shares already; no multipliers applied. + - Index data: adjustment must be `none` (provider tencent labels it 前复权; recorded as warning). +- `worker/normalize.py` — canonical frame (`date,open,high,low,close,volume` + + preserved raw fields e.g. 成交额/换手率 with units/precision untouched), symbol + stamping (never substituted), date sort/dedup, NaN-OHLCV rejection (gaps are + kept, never imputed), content hashes, coverage segments/gaps, manifest entries + (`immutable: true`, `path` internal-only — backend must rewrite before client exposure). +- `worker/backtest.py` — real Backtrader (1.9.78.123). User source defines class + `Strategy` subclassing `bt.Strategy`; executed via `exec` after `ast` syntax + check, in the isolated worker process only (Docker). Recording: + - feeds added with `name=` → user uses + `self.getdatabyname("")`; prices never substituted across feeds. + - A never-trading `RowsRecorder` strategy records per-bar + `{date, cash, equity, closes, positions}`. + - A generated subclass of the user's `Strategy` intercepts `notify_order` / + `notify_trade` (documented hooks) to record orders, executed fills + (trades: `{date,symbol,side,quantity,price,commission,value}`) and closed + round-trip trades. + - Execution rules (documented, tested): fills at NEXT bar open (never + signal-bar close), commission % of trade value, slippage % of fill price, + indicator warmup honored, T+1/lot/suspension/limits NOT simulated, + `index` feeds are nontradable proxies (server must reject direct index orders). + - `metrics.trade_count` = number of CLOSED round-trip trades (buy-only runs = 0), + fills recorded separately in `trades`. + - `max_drawdown` = negative loss fraction (e.g. -0.05 = 5% drawdown; **0.0 is a + valid value for flat equity**, null only when not computable); NaN/Inf never + emitted (null instead). `peak_rss_kb` = real process max RSS. +- Broker-level cn stock/ETF rules (`CnDailyRulesBroker`, in-process, tested): + - `asset_type=index` feeds are **rejected at the broker level for any order** + (warning `nontradable` + `order_state: rejected` record); allowed as benchmark feeds. + - buys rounded DOWN to whole lot=100; reject when < 100. Requested size kept + on `order.prereject_size` for audit; fill follows enforced size. + - sells rejected/trimmed when they would exceed owned shares minus pending + same-symbol buys (no naked short, no T+0 sale of same-cycle buys). + - T+1 at daily granularity emerges from next-bar-open fills: a sell submitted + on a buy's fill day executes next trading day, exactly the A-share rule. + Position can never go negative in the equity series. +- Result additions: `initial_cash` (from config capital), per-bar `positions` + dict in every equity bar (symbol→position size), and `benchmark` per equity + bar **normalized to initial capital** so equity and benchmark share one scale + (both start at `initial_cash`); raw benchmark closes stay in `equity[].closes` + under the benchmark symbol. Frontend Chart must NEVER mix `equity[].benchmark` + (capital scale) with `equity[].closes` (raw prices) on the same axis. +- `worker/main.py` — CLI: `fetch` / `backtest` / `search` / `probe`. + - fetch validates request (1..5 instruments, daily, none|qfq|hfq, fields + whitelist amount/turnover extras, 15-year POC bound, source whitelist) and is + bounded by SIGALRM wall clock, default 180s + (`STRATEGY_LAB_FETCH_WALL_SECS` env override). No unbounded network calls. + - raw provider response stored as an immutable content-addressed object at + `output/objects/raw__.json` (written BEFORE the normalized + transform; `raw_object_hash` in the manifest is the sha256 of the raw object) + - result.json `{status, manifest, preview, cache_key, warnings, elapsed_ms}`; + manifest per-object: provider/endpoint/params/akshare_version/fetched_at/ + schema_version/normalization_version/adjustment/requested_start/requested_end/ + actual_start/actual_end/row_count/columns/object_hash/raw_object_hash/warnings. + - cache key = sha256 of manifest objects content (stable content identity, + independent of user/request ids/fetch time). Same-key fetches of same + content produce identical hashes; backend copies/hash-verifies into + `data/objects` (shared physical object reuse; old snapshots never overwritten). + +## Docker contract +Image `strategy-lab-worker:local` from `worker/Dockerfile` (python:3.11-slim, +nonroot uid 10001, app at `/app`). Invocation used and verified: + + docker run --rm --user 10001:10001 --cap-drop ALL \ + --security-opt no-new-privileges --read-only --tmpfs /tmp \ + --network none --memory 512m --cpus 1 --pids-limit 128 \ + -v :/input/request.json:ro -v :/data:ro \ + -v :/output -e HOME=/tmp --entrypoint python \ + strategy-lab-worker:local -m worker.main backtest --request /input/request.json --output /output + +- backtest: `--network none`. fetch: network enabled for adapter only. +- /output must be writable by uid 10001 (backend: pre-chmod or run with same uid). +- no Docker socket, no provider/auth env in container. + +## For backend integration +- fetch request: `{instruments[{symbol,market,asset_type,name?}], start_date, + end_date, frequency:'daily', adjustment, fields[], source?}`; instruments + symbol may be bare code or canonical `SH#600000`. +- fetch result.json status: ready|failed; failed carries `errors[{code,message}]`. +- backend: strip `path`-host info from client-exposed manifest; rewrite object + paths into stored immutable objects; mount only the referenced files per run + into `/data` read-only and put per-run request in `/input`. +- cache: hash result.manifest.objects content; backend should cache both raw and + normalized objects keyed by (provider, endpoint, params incl. adjust, + normalization_version); dataset manifest hash stays stable when content unchanged. +- backtest request: `{code, config:{capital,commission,slippage,benchmark_symbol?,parameters}, + dataset_manifest, data_root:'/data'}`. result.json: SPEC.md Result shape + + `closes` per equity bar, `closed_trades`, `execution_assumptions`. +- data warnings (coverage gaps, provider fallback, synthetic) MUST be surfaced + to the user and acknowledged per SPEC. + +## Real evidence in artifacts/worker (fresh, current source) +- Tests: `.venv/bin/python -m pytest tests/worker -q` → **35 passed** (adds + tests/worker/test_rules_regression.py: zero-drawdown 0.0, broker index-order + rejection, index-as-benchmark accepted, lot rounding 250→200, no naked short, + sell-on-same-cycle trim, cash+position reconciliation). +- `akshare-probe.json` — real probe: stock_zh_a_hist + fund_etf_hist_em OK; + stock_zh_index_daily_em + stock_info_a_code_name FAILED (network, honest). +- `real-fetch/` — 浦发银行 600000 unadjusted via tencent: 117 bars + (2024-01-02 6.63/6.60/vol 22,066,700 → 2024-06-28), raw object + `objects/raw_600000_.json` + normalized CSV + manifest. +- `real-backtest/` — host run: 16 fills (900×16, lot-compliant), final equity + 118905.65, MDD -0.0525, Sharpe 2.82, initial_cash 100000 recorded, positions + per bar, peak RSS ~157-165 MB. +- `docker-run/` — container backtest (`--network none`, uid 10001, read-only, + cap-drop ALL, tmpfs /tmp) → identical metrics to host; container fetch run + produced object hash a63c2c71f518 identical to host → same-object sharing. +- `docker-run/request-benchmark.json` + `output/result.json` — honest + benchmark contract verified in real container run: benchmark starts at + 100000.0 and ends 124696.97 (normalized), raw close 8.23 kept separately in + `equity[-1].closes['600000']`, warning "benchmark normalized to initial capital". + +## Tests +`tests/worker/` — 27 tests, all synthetic fixtures carry `_synthetic`/`synthetic` +markers and are never production fallback: normalization correctness/rejection, +hash stability, deterministic accounting (hand-verified commission/slippage/ +next-bar fills), named-feed isolation, benchmark series, error paths (syntax, +missing Strategy class, missing data, lookahead-at-end), catalog concat identity +regression, auto-fallback honest warning, explicit-source no-fallback, CLI +provider/raw-retention/error paths. All providers fully mocked in unit tests; +no network. Run: `.venv/bin/python -m pytest tests/worker -q`. + +Limitations (honest): eastmoney currently blocked from this host; tencent lacks +listed-ETF daily adapter; index via tencent labeled 前复权; suspension/price +limits/intraday sequencing/liquidity NOT modeled; ETF sell lot approximated as +lot-100 buy rule; T+1 enforced at daily-bar granularity (no intraday detail); +search CLI uses catalog endpoints that can be slow/blocked (backend should call +with bounded timeout and show failure UI). No fake data or fabricated metrics. -- cgit v1.2.3