Backend Structure Convergence Plan

Date: 2026-06-14. Goal: make the backend structure clear enough for the next development cycle without destabilizing the now-working ECS platform / Dell compute split.

Current Diagnosis

The runtime architecture is now clear:

/api/platform/* -> ECS biof3-platform
/api/r/*        -> Dell R compute
/api/py/*       -> Dell Python compute
Database        -> ECS Docker PostgreSQL biof3-pg
Large artifacts -> Dell local storage + OSS

The source tree is less clear. platform-server/server.js is a thin wrapper that sets BIOF3_RUN_MODE=platform and loads ../r-server/server.js.

This means r-server is no longer just an R server. It currently contains:

• platform control plane routes: auth, users, permissions, admin, analytics, AI/Copilot, feedback, tool metadata, backup/OSS views
• platform-to-Dell internal proxy routes for cache, disk, assets, and tools
• R compute routes: /api/run, /api/replot/*, /api/task
• R worker/task execution and R tool scripts
• shared database, auth, permission, OSS, notification, and helper modules

This shape was useful during migration because it minimized code duplication. It is now a maintenance risk because directory names no longer match ownership.

Principles

1. Do not start with a big-bang move.
2. Keep public behavior stable while splitting internals.
3. Move routes by ownership, not by current filename.
4. New platform-only features should not deepen r-server/server.js.
5. Shared helpers should become explicitly shared modules instead of accidental imports from r-server.
6. Every extraction step must pass npm run ops:smoke-platform.

Target Source Ownership

platform-server/
  server.js
  routes/
    auth, admin, analytics, ai, feedback, platform tools metadata
  proxy/
    Dell internal-token proxy for cache/disk/assets/tools files

r-server/
  server.js
  routes/
    R compute compatibility: run, replot, task health/status as needed
  worker.js
  task-manager.js
  task-queue.js
  task-scripts-gen/
  tool-scripts/
  tool-demo-data/
  tool-demo-results/

py-server/
  server.js
  worker.js
  task-queue.js
  task-scripts.js
  tools-router.js

shared-backend/ or backend-core/
  config/env helpers
  database adapters and PG shims
  auth/JWT helpers
  permissions/module gate
  OSS helpers
  logging/notification helpers
  common path/security helpers

The exact shared directory name can be chosen later. shared-backend is clearer than backend-core for a mixed site/API repository, but either is acceptable.

Phased Plan

Phase 0: Freeze The Current Truth

Status: done 2026-06-14.

• Document that platform-server is currently a wrapper.
• Document that r-server is a historical name for the shared backend tree.
• Update steering so new work respects platform/R/Python ownership.
• Do not change runtime behavior in this phase.

Done when:

• A future session can answer "why does platform-server require r-server?" from local docs without rediscovering it.

Phase 1: Local Repository Hygiene

Status: partially done 2026-06-14.

Risk: low.

Actions:

• Classify local generated/runtime directories:
  • exports/
  • static/exports/
  • build/
  • .docusaurus/
  • .waline-data/
  • ..r-server-remote/
  • _archive/
  • r-server/.tmp
• Add or update README/DEPRECATED markers where a directory must remain but is not source of truth.
• Keep generated artifacts ignored and out of architectural decisions.

Done when:

• Local source-of-truth directories are visually distinguishable from generated exports, mirrors, and archives.

Progress:

• Added docs/ops/LOCAL_WORKSPACE_HYGIENE_2026-06-14.md.
• Documented local source-of-truth directories, generated artifacts, remote mirrors, and archives.
• Added r-server/.tmp/ to .gitignore.
• Removed one tracked local upload scratch CSV from git tracking without reading contents.

Phase 2: Route Inventory By Ownership

Status: done 2026-06-14.

Risk: low to medium.

Actions:

• Create a route ownership table from r-server/server.js and r-server/routes/*.
• Mark each endpoint as:
  • platform-only
  • R compute-only
  • Dell internal-only
  • shared/compatibility
• Identify platform routes still living in server.js legacy blocks.

Done when:

• Every route has a target owner before any code is moved.

Result:

• Added docs/ops/BACKEND_ROUTE_OWNERSHIP_2026-06-14.md.
• Classified routes from r-server/server.js and r-server/routes/* into:
  • platform
  • r-compute
  • dell-internal
  • shared-compat
  • needs-decision
• Identified the safest Phase 3 extraction order:
  1. admin backup / OSS routes
  2. admin users / guest permissions / invite / impersonation
  3. remaining legacy AI support routes
  4. upload/sign/proxy-file after ownership decision
  5. R task enqueue/status after /api/r/task compatibility decision
  6. tools submission/metadata split from Dell local execution/file serving

Phase 3: Extract Platform Route Modules

Status: in progress.

Risk: medium.

Actions:

• Move platform-only legacy endpoints from r-server/server.js into route modules that can later be mounted by platform-server.
• Prefer small, reversible extractions:
  • admin/users and guest permissions
  • backup/OSS admin
  • AI interpret/act endpoints if still in legacy server
  • upload/oss-sign only if confirmed platform-owned
• Keep current r-server/server.js requiring these modules until the split is complete, so behavior does not change.

Done when:

• server.js stops being the place for new platform routes.
• Route registry failures are covered by targeted smoke probes, not only /api/health.

Progress:

• 2026-06-14 first extraction: moved admin backup and OSS admin routes from r-server/server.js into r-server/routes/admin-ops.js, registered from r-server/routes/index.js, and deployed to ECS platform plus Dell R server. Public platform smoke passed after deployment.
• 2026-06-14 second extraction: moved admin user management, guest permissions, public guest permissions, invite code, and impersonation routes from r-server/server.js into r-server/routes/admin-users.js, registered from r-server/routes/index.js, and deployed to ECS platform plus Dell R server. Public platform smoke passed after deployment.
• 2026-06-14 third extraction: moved AI support compatibility routes (/api/ai/interpret, /api/ai/act-ack/:token, /api/ai/act-registry) into r-server/routes/ai-legacy.js; moved the remaining conversation PATCH compatibility handler into r-server/routes/ai-conversations.js; unified act registry writes with the existing helpers/runtime.js registry used by the AI agent. Deployed to ECS platform plus Dell R server; public platform smoke passed after deployment.
• 2026-06-14 fourth extraction: moved file upload/sign/proxy support (/api/upload, /api/oss/sign, /api/proxy-file) into r-server/routes/files.js; updated FigCode production OSS signing to use /api/platform/oss/sign; added platform file route probes to the smoke test. Public /api/r/{upload,oss/sign,proxy-file} is blocked at nginx.
• 2026-06-14 fifth extraction: moved observability routes (/api/error-report, /api/pageview, /api/admin/errors) into r-server/routes/observability.js; moved process-level error log append/read logic into r-server/helpers/error-log.js; unified the duplicate /api/error-report handlers while preserving the frontend fire-and-forget 204 response. Added platform telemetry/admin-errors smoke probes.
• 2026-06-14 sixth extraction: moved R analysis task enqueue/status/list routes (/api/task, /api/task/:id, /api/tasks) into r-server/routes/tasks.js. Public product traffic uses /api/platform/task*; Dell continues to execute tasks as a worker from the PostgreSQL queue, and public /api/r/task* remains blocked by nginx. Added platform task validation/status/list smoke probes.
• 2026-06-14 tools pre-split map: added docs/ops/TOOLS_INTERNAL_MAP_2026-06-14.md after classifying r-server/routes/tools.js into platform metadata, platform-to-Dell proxy, Dell internal execution/files, and legacy public/demo compatibility routes. Do not block /api/r/tools/* until legacy docs/static report URLs are migrated.
• 2026-06-14 tools URL cleanup: source machine-learning docs and DESeq2 report templates now use /api/platform/tools/... public demo URLs. Runtime report regeneration accepts both legacy /api/r/tools/demo/... and platform /api/platform/tools/demo/... so old templates/reports remain compatible.
• 2026-06-14 tools R-prefix收口: nginx keeps legacy /api/r/tools/{public-demo,public-file,demo,demo-data}/... public asset URLs compatible through ECS platform, while old /api/r/tools/{history,run,status, file,job,cancel} control-plane paths return 404. Added smoke probes for both compatibility and blocking.
• 2026-06-14 tools asset helper extraction: moved public demo asset mapping, path traversal guard, public-file, demo-data, and legacy static demo handlers from r-server/routes/tools.js into r-server/helpers/tools-assets.js. Route behavior is unchanged; this prepares the remaining tools split.
• 2026-06-14 tools Dell proxy helper extraction: moved ECS-to-Dell internal-token proxy helpers for tool run, assets, private files, cancel, and delete into r-server/helpers/tools-dell-proxy.js. Route ownership and URL behavior are unchanged.
• 2026-06-14 tools executor helper extraction: moved Dell-local execution, local cancel/delete, private file read/write, report regeneration, upload validation, R error translation, and zombie recovery into r-server/helpers/tools-executor.js. routes/tools.js now keeps route registration/auth orchestration only.
• 2026-06-14 tools metadata route extraction: moved metadata-only tool routes (status, history, job rename, public demo flag) into r-server/routes/tools-metadata.js. These routes read/write PostgreSQL only.
• 2026-06-15 tools internal route extraction: moved Dell-only /api/internal/tools/* handlers and the internal smoke result endpoint into r-server/routes/tools-internal.js. ECS still reaches these only through internal-token proxy paths.
• 2026-06-24 tools platform route extraction: moved public tool routes (run, file GET/PUT, cancel, delete, public-demo, public-file, demo, demo-data) from r-server/routes/tools.js into r-server/routes/tools-platform.js. tools.js is now a registration shell plus shared helpers (requireInternal, userId, canAccessToolJob, isCancellableJob, isSmokeTestRequest) consumed by all three tool submodules. Also fixed a duplicate db:pg-migrate key in package.json that crashed bun run, and reverted a JOBS_DIR path typo (omichhub → omichub).
• 2026-06-24 admin compute route extraction: moved compute infrastructure admin routes (compute/health, cache/stats, disk/stats, assets/inspect, cache/cleanup, queue/status) and their helpers (proxyDellCache, proxyDellAdminGet, fetchJsonWithTimeout, getComputeQueueHealth, normalizeCounts) from r-server/server.js into r-server/routes/admin-compute.js. Removed now-unused REQUIRE_INTERNAL_CACHE_OPS, DELL_INTERNAL_API_BASE, and DELL_PY_INTERNAL_API_BASE constants from server.js. server.js dropped from 1262 to 974 lines.
• 2026-06-24 data router move: moved app.use('/api/data', dataRouter) from r-server/server.js into r-server/routes/index.js so all product routes are registered through the central registry. server.js no longer imports api-data.js.

Phase 4: Introduce Shared Backend Modules

Risk: medium.

Actions:

• Move stable helpers that are already cross-role into a shared directory.
• Start with low-side-effect modules:
  • path safety helpers
  • logging wrappers
  • OSS helper wrappers
  • auth token parsing helpers
• Delay database adapter moves until route ownership is clearer.

Done when:

• platform-server can import shared helpers without importing from r-server by name.

Progress:

• 2026-06-24 shared-backend introduced: created shared-backend/ with logger.js (canonical structured logger; replaces the near-duplicate r-server/logger.js and py-server/logger.js — py-server now also gains the child() method it was missing), error-log.js (JSONL error log, moved from r-server/helpers/error-log.js), and path-safe.js (safeJoinUnder path-traversal guard, moved out of r-server/helpers/tools-assets.js). shared-backend/index.js is a barrel re-export. Original locations are now shims that re-export the shared module, so existing require('./logger') / require('../helpers/error-log') paths keep working. Verified shared-backend.logger === r-server/logger and shared-backend.safeJoinUnder === tools-assets.safeJoinUnder are referentially identical.
• 2026-06-24 JWT Bearer verification deduplicated: extracted tryParseBearerUser and verifyBearerUser into r-server/helpers/auth-token.js, removing 4 copies of jwt.verify + require('jsonwebtoken') + require('./auth') from combinedAuth in server.js. Placed in r-server/helpers (not shared-backend) because it depends on jsonwebtoken, which is installed under r-server/node_modules; moving to shared-backend would require giving it a dependency manifest and changing deploy scripts' npm install paths — deferred to Phase 5 when platform-server owns its own app and dependency tree.

Phase 5: Make Platform Server Real

Risk: high; only after prior phases.

Actions:

• Make platform-server/server.js construct its own Express app.
• Mount platform-owned routes directly.
• Keep Dell internal proxy routes explicit.
• Remove dependence on R compute constants, R Docker paths, and R task local execution from platform startup.
• Keep r-server/server.js focused on /api/r/* compatibility and R compute.

Done when:

• ECS platform can start without loading R compute execution modules.
• Dell R compute can start without loading platform admin/AI control-plane routes except explicit compatibility/internal routes.

Progress:

• 2026-06-25 shared middleware bootstrap extracted: moved CORS, JSON body, request-id, setupAuthRoutes, combinedAuth, moduleGate, and the four rate limiters (login/register/taskSubmit/general) from inline definitions in r-server/server.js into r-server/bootstrap.js. bootstrap.js exports setupBootstrap(app, {db, apiToken, internalToken}) which mounts the full middleware stack and returns {rateLimiters}. server.js dropped from 954 to 824 lines. This is the key prerequisite for Phase 5: a future standalone platform-server/server.js can call setupBootstrap() to mount the same auth/middleware stack without copying inline code. bootstrap.js lives in r-server/ (not shared-backend) because it depends on express, cors, express-rate-limit, and uuid.
• 2026-06-25 platform-server dependency manifest: gave platform-server/ its own package.json with the same runtime deps as r-server minus better-sqlite3 (platform uses PG, not local SQLite). Updated scripts/deploy-ecs-platform.sh to run npm install in the platform-server directory as well as r-server. platform-server/ server.js is still the 7-line wrapper — this step only prepares the dependency tree and deploy path, with zero runtime risk.
• 2026-06-25 AI Copilot helper bridge extracted: moved 14 AI helper requires (chat-summary, user-facts, memory-vectors, memory-privacy, site-snapshot, docs-index, act-tool, plans, conversations, admin-gov, ai-usage, system-prompt, llm-client, methods-export), plan table initialization, and 17 globalThis._biof3.* bridge assignments from server.js into r-server/ai-bridge.js. Exports setupAiBridge({tempDir, ossClient}). Also removed dead duplicate code from server.js (AI concurrency guard, Dashscope rate limit, agent-tools require, safeJSON, _quickPing, _requireAdmin, archiveOldPlans timer — all already in helpers/runtime.js or modular routes). server.js dropped from 824 to 410 lines.
• 2026-06-25 platform-server standalone app (Phase 5 done): rewrote platform-server/server.js from the 7-line wrapper into a standalone Express app that no longer require('../r-server/server.js'). It calls setupBootstrap() + setupAiBridge() + routes/index.js and listens on its own. It does NOT load R compute modules (task-queue script generator, cache/backup schedulers, R Docker pre-warm, /api/run, /api/replot/*). Deployed to ECS :3011 via scripts/deploy-ecs-platform.sh; npm run ops:smoke-platform passes all 50 checks. Direct probes confirm /api/run and /api/replot/* are absent on the platform port (404/401). Phase 5 "done when" criteria met: ECS platform starts without loading R compute execution modules.

Current No-Go Items

Do not do these until the route/module split is ready:

• Rename r-server/ in one step.
• Move db.js / PG shims wholesale.
• Split server.js by manual copy/paste without smoke coverage.
• Delete legacy route blocks without first extracting the canonical behavior into a registered route module and smoke probe.
• Change production paths on ECS or Dell as part of local cleanup.

Verification Gate

After every phase that changes code or deploy behavior:

node --check platform-server/server.js
node --check r-server/server.js
npm run ops:smoke-platform

For route moves, also probe at least one endpoint from each moved file. Health alone is insufficient because legacy /api/health can pass while modular route registration is broken.

Immediate Next Step

Complete Phase 0 and then do Phase 1 local repository hygiene. The first code extraction should wait until local clutter and route ownership are documented.