ECS Platform Convergence Plan

Status: active plan, started 2026-06-12. Current decision: use the ECS Docker PostgreSQL container as the production central database. Aliyun managed RDS is not a near-term target because of cost.

North Star

Public product control-plane traffic should terminate on ECS platform:

/api/platform/* -> ECS platform

Dell should be demoted to a compute node. It remains important, but it should no longer be the public platform API gateway.

Current Production Shape

ECS
  - static site
  - nginx
  - platform-api
  - Docker PostgreSQL container: biof3-pg

Dell
  - R container
  - Python container
  - biof3-r-server in api mode, biof3-r-worker in worker mode
  - biof3-py-server in api mode, biof3-py-worker in worker mode
  - large result files, demo assets, cache, tmp

OSS
  - uploads
  - archives / backups

Database reality:

• ECS apps connect to PostgreSQL through 127.0.0.1:5432.
• Dell connects through an autossh local forward 127.0.0.1:5433 -> ECS 127.0.0.1:5432.
• In project docs, prefer ECS Docker PostgreSQL over RDS unless talking about the old plan or a future optional migration.

Runtime check, 2026-06-13:

• Dell biof3-r-server: BIOF3_RUN_MODE=api, R_SERVER_PORT=3001.
• Dell biof3-r-worker: BIOF3_RUN_MODE=worker, BIOF3_WORKER_ID=r-dell.
• Dell biof3-py-server: BIOF3_RUN_MODE=api, PY_SERVER_PORT=3002.
• Dell biof3-py-worker: BIOF3_RUN_MODE=worker, BIOF3_WORKER_ID=py-dell.
• Python server and worker actual /proc/<pid>/environ both use DATABASE_URL=...@localhost:5433/biof3; if pm2 env shows an older direct managed-RDS URL, treat it as stale display until the process environment is checked directly.
• Startup recovery must not run on platform/api HTTP nodes. In shared PostgreSQL queue mode, only a real worker or legacy all process may mark its own interrupted running tasks failed; restarting ECS platform or Dell api server does not imply compute work died.

Engineering Principles

1. ECS Docker PostgreSQL is the only business truth source: users, permissions, projects, datasets, analyses, tasks, tool_jobs, Copilot state, AI usage, audit, analytics, feedback, and site settings.

2. Dell local disk is for compute artifacts, not business truth: result directories, public demo assets, cache, tmp, and large intermediate files.

3. Dell may expose internal ops endpoints, but user-facing control-plane requests should go through ECS platform first.

4. ECS enqueues tasks; Dell workers claim and execute tasks.

browser -> ECS platform -> PostgreSQL task/tool_job row
Dell worker -> PostgreSQL claim -> R/Python container -> PostgreSQL status update

5. nginx should become boring.

Final target:

/api/platform/* -> ECS platform
/api/r/*        -> Dell R compute compatibility surface, api mode only
/api/py/*       -> Dell Python compute compatibility surface, api mode only
/api/*          -> Waline / non-BioF3 platform services

Removed public Dell control-plane exception:

/api/platform/tools/* -> explicit ECS cuts or 404

Current tools split:

/api/platform/tools/history               -> ECS platform
/api/platform/tools/status/*              -> ECS platform
/api/platform/tools/run                   -> ECS platform -> Dell internal
/api/platform/tools/job/:id[/(name|public)] -> ECS platform
/api/platform/tools/public-demo/*         -> ECS platform -> Dell internal
/api/platform/tools/public-file/*         -> ECS platform -> Dell internal
/api/platform/tools/demo-data/*           -> ECS platform -> Dell internal
/api/platform/tools/demo/*                -> ECS platform -> Dell internal
/api/platform/tools/file/*                -> ECS platform -> Dell internal
/api/platform/tools/cancel/*              -> ECS platform -> Dell internal
/api/platform/tools/job/:id               -> ECS platform -> Dell internal
/api/platform/tools/*                     -> 404, no Dell public gateway

Phase 0: Protect PostgreSQL First

Goal: make ECS Docker PostgreSQL operable like a production database.

• Identify the biof3-pg data volume / bind mount. Result: bind mount /opt/biof3-platform/pgdata -> /var/lib/postgresql/data.
• Document commands to inspect, stop, start, dump, and restore biof3-pg. See docs/ops/ECS_DOCKER_POSTGRES_BACKUP.md.
• Add a pg_dump backup script. Script: scripts/backup-ecs-pg.sh.
• Upload backups to OSS or copy them to Dell. Current path: OSS backups/postgresql/.
• Define retention, e.g. 7 daily + 4 weekly + 3 monthly. Current script keeps 14 daily, 35-day weekly, and 120-day monthly backups.
• Run one restore drill into a disposable PostgreSQL container. 2026-06-12 drill restored latest.sql.gz into biof3-pg-restore-drill and verified users=8, projects=3, tool_jobs=1.
• Confirm deploy scripts never run docker compose down -v or delete the PG volume. 2026-06-12 grep found no destructive PG volume operation in deploy scripts.

Done when: a broken PostgreSQL container or accidental container recreation can be recovered from a recent dump without guessing.

Phase 1: Fix Documentation And Inventory

Goal: make future work follow the real architecture, not stale RDS/Dell-SQLite assumptions.

• Update steering docs to say ECS Docker PostgreSQL is the current center.
• Mark Aliyun managed RDS as optional future work, not current target.
• Inventory all nginx /api/platform/* locations that still point to Dell. See docs/ops/PLATFORM_ROUTE_INVENTORY_2026-06-12.md.
• Inventory Dell control-plane endpoints still reachable from public nginx. Remaining platform exceptions: /api/platform/admin/cache/ and /api/platform/tools/.
• Classify each endpoint as: move to ECS platform directly; keep as Dell internal ops called by ECS; or leave as /api/r or /api/py compute surface.

Done when: one checklist shows exactly what remains before Dell can stop acting as a public control-plane backend.

Phase 2: Move Pure Control Plane Back To ECS

Goal: endpoints that only need PostgreSQL / OSS / AI APIs should run on ECS platform.

Verified ECS-owned endpoints:

• /api/platform/admin/users
• /api/platform/admin/guest-permissions
• /api/platform/analytics/*
• /api/platform/public/*
• /api/platform/ai/admin/*
• /api/platform/feedback/*

For each endpoint:

• Verify no Dell local disk dependency.
• Verify no sync deasync PG hot path.
• Curl ECS platform directly.
• Curl public nginx.
• Check logs for 30s after cutover.

Status:

• 2026-06-13: Phase 2 routes were re-checked against production nginx loaded state and ECS direct upstream 127.0.0.1:3011.
• Public nginx and direct ECS probes passed for admin users, guest permissions, public guest permissions, analytics POST /heartbeat and POST /page-view, AI admin users overview / usage stats, feedback admin list, realtime stats, OSS stats, and PostgreSQL backup list.
• GET /api/platform/analytics/heartbeat is expected to be 404 because analytics heartbeat is a POST endpoint.
• During log review, the tools zombie scan was found still referencing the old dbAsync variable name. It was fixed and deployed to both ECS platform and Dell r-server because the tools surface is still split across both nodes.

Done when: these endpoints work through /api/platform/* without Dell nginx exceptions.

Phase 3: Turn Dell Local Resources Into Internal Ops

Goal: keep Dell-only facts available, but hide Dell behind ECS platform.

Examples:

• cache stats / cleanup
• local disk stats
• worker health / queue health
• result asset and demo asset inspection

Target request path:

browser -> /api/platform/admin/cache/stats -> ECS platform
ECS platform -> internal token -> Dell ops endpoint

Rules:

• Dell internal endpoints require BIOF3_INTERNAL_TOKEN or equivalent.
• ECS platform applies user/admin auth before calling Dell.
• Public nginx does not directly route /api/platform/admin/cache/* to Dell.

Status:

• 2026-06-12: /api/platform/admin/cache/* now goes to ECS platform, then to Dell via BIOF3_INTERNAL_TOKEN; Dell cache ops require internal token.
• 2026-06-12: /api/platform/admin/backup/* now manages ECS Docker PostgreSQL dumps under OSS backups/postgresql/; it no longer snapshots the legacy Dell SQLite biof3.db in platform mode.
• 2026-06-12: /api/platform/admin/compute/health now runs on ECS platform, probes Dell R/Python compute health, and summarizes the PostgreSQL task queue by engine, worker, recent tasks, and stale running tasks.
• 2026-06-13: /api/platform/admin/disk/stats now runs through ECS platform, proxies to Dell via the internal token, and reports compute-volume waterline plus R/Python temp/cache/tool-job directory sizes in the admin local storage view.
• 2026-06-13: /api/platform/admin/assets/inspect now runs through ECS platform, proxies to Dell via the internal token, verifies published demo asset mappings on the Dell tool-job volume, and lists recent result directories for admin inspection.

Done when: all Dell local state is visible in admin UI without direct public control-plane proxying to Dell.

Phase 4: Simplify nginx

Goal: make routing match service ownership.

Keep:

/api/platform/* -> ECS platform
/api/r/*        -> Dell R compute, with legacy control-plane prefixes denied
/api/py/*       -> Dell Python compute
/api/*          -> Waline / other non-platform services

Already removed or avoided:

/api/platform/admin/*     -> Dell
/api/platform/analytics/* -> Dell
/api/platform/public/*    -> Dell
/api/platform/cache/*     -> Dell

Current expected exception:

none

Tools are no longer a broad public Dell exception. Public demo assets, demo input files, private result file read/write, cancellation, deletion, and run submission all flow through ECS platform first. ECS reaches Dell only through internal-token routes. Unknown /api/platform/tools/* paths return 404.

R compute no longer exposes the old all-in-one control plane through the /api/r/ prefix. Public nginx denies /api/r/(auth|admin|public|analytics|ai|data|feedback|pageview|error-report) before the broad R compute catch-all, so these requests must use /api/platform/* or return 404.

Done when: nginx has no platform control-plane exceptions to Dell.

Phase 5: Demote Dell

Goal: Dell is a compute node, not a product backend gateway.

Dell keeps:

• R container
• Python container
• biof3-r-server api mode for R compute compatibility and internal ops
• biof3-py-server api mode for Python compute compatibility
• biof3-r-worker / biof3-py-worker worker mode for PostgreSQL task claims
• large result files
• cache / tmp
• internal ops endpoints

Dell stops owning:

• login / registration
• users / permissions
• projects / datasets / analyses CRUD
• analytics
• AI admin
• guest permissions
• full platform API gateway behavior

Done when: if Dell is down, users can still open the site, log in, view projects, view permissions, and use non-compute admin pages. New compute tasks and Dell hosted result files may degrade, which is expected.

Status:

• 2026-06-13: Public probes found legacy Dell control-plane routes reachable through /api/r/* (for example /api/r/admin/users). nginx now blocks auth, admin, public, analytics, ai, data, feedback, pageview, and error-report under /api/r/ before proxying compute traffic to Dell. Verification passed: platform admin/public/AI routes still return 200, /api/r/health and /api/py/health still return 200, and legacy /api/r/* control-plane probes return 404.
• 2026-06-13: Symmetric /api/py/* audit passed without a new nginx deny rule. py-server exposes compute/tool routes (health, queue-status, upload, run, task, tools/*) and public probes for legacy control-plane paths (auth, admin, public, analytics, ai, data, feedback, pageview, error-report) returned 404. Python tool routes behaved as expected: authenticated history/status returned 401 without login, public demo assets returned 200, and unknown tool paths returned 404.
• 2026-06-13: Final non-destructive Phase 5 acceptance passed. ECS platform, nginx, and biof3-pg were online; Dell R/Python servers and workers were online as compute nodes. Public probes passed for platform health, auth, guest permissions, admin users, AI admin overview/usage, feedback admin, realtime stats, OSS stats, PostgreSQL backup list, compute/cache/disk/assets admin views, and tools history. /api/r/health and /api/py/health stayed 200, while legacy R/Python control-plane probes stayed 404. PostgreSQL data volume was /opt/biof3-platform/pgdata, pg_isready passed, table counts were users=8, projects=3, tasks=0, tool_jobs=2, and latest PG dump pointed to the 2026-06-13 03:10 daily backup.
• 2026-06-13: Added scripts/smoke-platform-convergence.mjs and npm entry npm run ops:smoke-platform to make this acceptance repeatable. Run it after any nginx, platform, route, auth, backup, or compute-boundary change.
• 2026-06-13: Added ./deploy.sh --smoke-platform as the same smoke entry for deploy-oriented workflows.

Session Start Checklist

Before doing platform/ops work, read:

1. .kiro/steering/progress.md
2. .kiro/steering/deploy-ops.md
3. docs/ops/ECS-PLATFORM-CONVERGENCE-PLAN.md
4. docs/ops/DEPLOYMENT_LAYOUT_2026-06-13.md

Then answer:

• Am I changing control plane, compute plane, storage, or nginx?
• Does this move /api/platform/* toward ECS?
• Does it make Dell more compute-only?
• Does it protect ECS Docker PostgreSQL?
• Does this touch a path marked active or pending in the deployment layout inventory?
• Did I run npm run ops:smoke-platform if I touched nginx/platform/compute routing?