chelsea/balanceboard
0
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6cf35ca03414cc76bc835a52eddda9a58c680d12
balanceboard/REFACTOR_GOAL.md
Chelsea 6cf35ca034 Phase 1-3 + 6: pluggable filter system, app factory, test harness 
Implements the parallel.md workstreams (Agents A-E) toward REFACTOR_GOAL.md.

Phase 1 — app factory + blueprints + /api/v1:
- app.py -> create_app() factory (no module-level app); entrypoints updated
- routes/ (auth, pages, settings, admin, assets) + blueprints/api.py at /api/v1
- config.py / extensions.py / security.py extracted; services/ layer added
- endpoint names preserved so template url_for() calls keep resolving
  (static check: all 27 template url_for endpoints are defined routes)

Phase 2 — one pluggable filter system:
- filter_pipeline/registry.py: @register_stage / @register_plugin + discover_modules
- engine._init_stages() instantiates registered stages (no hardcoded dict);
  process_batch is AI-aware: only short-circuits to the AI-disabled path when
  a filterset's stages declare requires_ai, so offline filtersets run with AI off
- BaseFilterPlugin gets a consumer (stages/plugins.py); Keyword/Quality
  re-enabled via filter_config.json plugins config
- comment tree modes ported to stages/comment_filter.py + shared rules.py;
  wired into /api/v1/posts/<uuid> and /api/v1/comments/<uuid> via
  FilterEngine.filter_comments() (fails open)
- offline quality_filter filterset exercises plugins+ranker without AI
- legacy filter_lib / comment_lib / html_generation_lib / generate_html /
  active_html path deleted

Phase 3 prep — pluggable fetchers + Postgres models:
- Post / Comment SQLAlchemy models added to models.py
- migrate_content_to_db.py backfill (idempotent by uuid, batched, --dry-run)
- platforms/ fetcher registry (extension point)
- live reads/writes still go through PostService (disk JSON); cutover deferred

Phase 6 — test harness:
- pytest.ini + tests/ (conftest with in-memory SQLite fixture, no Postgres;
  stubbed polling/filter singletons)
- test_app_factory.py (route registration, no module-level app),
  test_api_contracts.py (posts/post_detail/comments/filters shape with
  monkeypatched post_service + get_filter_engine),
  test_filter_pipeline.py + test_plugin_contract.py (Flask-free; validated
  locally 12/12 incl. drop-in stage/plugin discovered with zero core edits)

Other: .gitignore added (__pycache__, data/, secrets); pytest in requirements.

Verification: py_compile clean across the project; the Flask-free filter-pipeline
and plugin-contract tests pass locally. App-factory / API-contract tests need
deps+docker to run; runtime flask routes / Auth0-repeated-create_app also gated
on docker.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-03 02:29:46 -05:00

6.3 KiB
Raw Blame History

Goal: Refactor BalanceBoard into a client/server architecture

Objective

Transform BalanceBoard from a Flask monolith with mixed server-rendered Jinja pages and inline-JS API consumption into a clean two-tier system:

  • a backend that is a pure JSON API, and
  • a separate frontend client that consumes it.

Preserve all existing working logic (auth, polling, data collection, filter pipeline, models) and fix the known bugs along the way.

Target end state

  • Backend: Flask serving only JSON under /api/*. No render_template, no Jinja page routes, no static-HTML generation. Split into blueprints (auth, posts, comments, bookmarks, settings, filters, admin). Posts/comments stored in PostgreSQL, not JSON files. One filter system (filter_pipeline/), not two. No import-time side effects.
  • Frontend: a Vite-built SPA served as static files by Flask at / (same origin), calling /api/* with session-cookie credentials. Replaces dashboard.html, post_detail.html, settings/admin/bookmark templates, and the dead generate_html.py path.
  • Auth: Flask-Login session cookies retained (same-origin), exposed via /api/auth/* endpoints. No JWT unless a cross-origin/mobile client is later required.

Non-goals (this round)

  • Do not switch to FastAPI (defer; revisit after the split is stable).
  • Do not introduce JWT/CORS (same-origin only).
  • Do not rewrite the filter pipeline logic or data-collection fetchers; only consolidate and wire them.

Phases (each independently shippable)

Phase 0 - Stop the bleeding (bugfixes, no architecture change)

  • Fix UTF-8 mojibake across .py/.html/.json (icons, checkmarks, separators).
  • Fix migrate_bookmarks.py (it treats init_db's None return as a db object).
  • Make password min-length consistent (8 everywhere; reset route uses 6).
  • Fix post_detail.html undefined moment filter (comment times always render "Recently").
  • Close the comment.content | safe XSS hole in post_detail.html.
  • Make non-no_filter filtersets not silently no-op when AI is disabled (reject clearly or document; do not silently pass everything).
  • Remove import-time side effects from app.py (polling_service.start() and filter_engine init must not run on import).

Phase 1 - Complete the API surface in Flask

  • Audit existing /api/* endpoints; fill gaps so every Jinja page's data need has an API endpoint:
    • /api/auth/login, /api/auth/register, /api/auth/me, /api/auth/logout, /api/auth/password-reset/*
    • /api/posts, /api/posts/ (post + comment tree), /api/comments/<post_uuid>
    • /api/bookmarks, /api/bookmark (toggle), /api/bookmark-status/
    • /api/settings (GET/PUT profile, communities, filters, experience), /api/filtersets, /api/platforms
    • /api/admin/users/, /api/admin/polling/, /api/admin/cache, /api/admin/backup
  • Convert server-rendered form POST routes (settings, profile, avatar upload, admin) to JSON endpoints; keep flash-message behavior as API status codes/messages.
  • Split app.py into blueprints + a services/ layer.

Phase 2 - Collapse the filter stack to one system

  • Adopt filter_pipeline/ as the single filter engine.
  • Port filter_lib's rule operators and comment_lib's tree modes (tree-pruning, individual, score/time/length modes) into filter_pipeline as a comment-filtering stage.
  • Wire comment filtering into the live /api/posts/ path (today only the dead static path filters comments).
  • Delete filter_lib.py, comment_lib.py, html_generation_lib.py, generate_html.py, the active_html/ route, and the theme template path. Keep themes/ only as CSS/JS asset bundles.

Phase 3 - Move posts/comments into PostgreSQL

  • Add Post and Comment SQLAlchemy models; map the existing JSON schema (uuid, platform, id, title, author, timestamp, score, replies, url, content, source, tags, meta, moderation_uuid, parent_comment_uuid).
  • Write a one-shot backfill migration that ingests data/posts/.json and data/comments/.json into the DB.
  • Replace _load_posts_cache() and the directory scans in /api/posts, /api/platforms, /api/content-timestamp with DB queries + a real TTL cache (Flask-Caching now; Redis layer later if needed).
  • Keep data/ as an archive/export only, not the source of truth.

Phase 4 - Build the SPA client (feature-by-feature parity)

  • Scaffold a Vite SPA (framework TBD), served by Flask as static files at / with a catch-all fallback to index.html.
  • Build in order: feed (list + pagination + filters + communities) -> post detail (comment tree) -> auth (login/register/password-reset) -> bookmarks -> settings (profile/communities/filters/experience) -> admin (users/polling/logs).
  • Dev: Vite proxy -> Flask. Prod: Flask serves built dist/.
  • Use credentials: 'include' on all fetch calls; session cookies do the rest.

Phase 5 - Cut over and delete the old render path

  • Once the SPA reaches parity, remove the Jinja templates, render_template calls, the templates/ folder, and the static-HTML generation.
  • Remove the now-dead _nav.html, base.html, page templates, and the serve_theme/serve_logo page-serving routes (replace with plain static asset routes).
  • Update README/DEPLOYMENT/Dockerfile for the new single-container deploy.

Phase 6 - Hardening

  • Add a test suite (pytest for services + API; component tests for the SPA).
  • Add lint/format (ruff + the SPA's linter) and a CI check.
  • Add a charset/encoding CI gate so the mojibake never returns.
  • Lock the API with a versioned prefix (/api/v1/*) so future changes do not break shipped clients.

Decisions baked in (defaults)

  • Backend framework: Flask (kept) - not FastAPI this round.
  • Auth: Flask-Login session cookies, same-origin - not JWT.
  • Deployment: single container, Flask serves the built SPA - not split hosts.
  • Storage: PostgreSQL for posts/comments - not JSON files.

Open decisions (pick before Phase 4 starts)

  • Frontend framework: React / Vue / Svelte / Solid. No wrong answer at this app's size; pick what you enjoy maintaining.
  • Whether to version the API as /api/v1 from day one (recommended - cheap insurance).

Success criteria

  • app.py is gone or reduced to app-factory + blueprint registration.
  • No render_template exists in the codebase.
  • One filter system; one render path.
  • Posts/comments queried from Postgres; no full-disk-scan cache.
  • Importing the app module has zero side effects (no scheduler, no polling).
  • The SPA, served by Flask, reproduces all current user-facing features.