Chelsea
6cf35ca034
Implements the parallel.md workstreams (Agents A-E) toward REFACTOR_GOAL.md. Phase 1 — app factory + blueprints + /api/v1: - app.py -> create_app() factory (no module-level app); entrypoints updated - routes/ (auth, pages, settings, admin, assets) + blueprints/api.py at /api/v1 - config.py / extensions.py / security.py extracted; services/ layer added - endpoint names preserved so template url_for() calls keep resolving (static check: all 27 template url_for endpoints are defined routes) Phase 2 — one pluggable filter system: - filter_pipeline/registry.py: @register_stage / @register_plugin + discover_modules - engine._init_stages() instantiates registered stages (no hardcoded dict); process_batch is AI-aware: only short-circuits to the AI-disabled path when a filterset's stages declare requires_ai, so offline filtersets run with AI off - BaseFilterPlugin gets a consumer (stages/plugins.py); Keyword/Quality re-enabled via filter_config.json plugins config - comment tree modes ported to stages/comment_filter.py + shared rules.py; wired into /api/v1/posts/<uuid> and /api/v1/comments/<uuid> via FilterEngine.filter_comments() (fails open) - offline quality_filter filterset exercises plugins+ranker without AI - legacy filter_lib / comment_lib / html_generation_lib / generate_html / active_html path deleted Phase 3 prep — pluggable fetchers + Postgres models: - Post / Comment SQLAlchemy models added to models.py - migrate_content_to_db.py backfill (idempotent by uuid, batched, --dry-run) - platforms/ fetcher registry (extension point) - live reads/writes still go through PostService (disk JSON); cutover deferred Phase 6 — test harness: - pytest.ini + tests/ (conftest with in-memory SQLite fixture, no Postgres; stubbed polling/filter singletons) - test_app_factory.py (route registration, no module-level app), test_api_contracts.py (posts/post_detail/comments/filters shape with monkeypatched post_service + get_filter_engine), test_filter_pipeline.py + test_plugin_contract.py (Flask-free; validated locally 12/12 incl. drop-in stage/plugin discovered with zero core edits) Other: .gitignore added (__pycache__, data/, secrets); pytest in requirements. Verification: py_compile clean across the project; the Flask-free filter-pipeline and plugin-contract tests pass locally. App-factory / API-contract tests need deps+docker to run; runtime flask routes / Auth0-repeated-create_app also gated on docker. Co-Authored-By: Claude <noreply@anthropic.com>
127 lines
6.3 KiB
Markdown
127 lines
6.3 KiB
Markdown
# Goal: Refactor BalanceBoard into a client/server architecture
|
|
|
|
## Objective
|
|
Transform BalanceBoard from a Flask monolith with mixed server-rendered
|
|
Jinja pages and inline-JS API consumption into a clean two-tier system:
|
|
|
|
- a backend that is a pure JSON API, and
|
|
- a separate frontend client that consumes it.
|
|
|
|
Preserve all existing working logic (auth, polling, data collection, filter
|
|
pipeline, models) and fix the known bugs along the way.
|
|
|
|
## Target end state
|
|
- Backend: Flask serving only JSON under /api/*. No render_template, no
|
|
Jinja page routes, no static-HTML generation. Split into blueprints
|
|
(auth, posts, comments, bookmarks, settings, filters, admin).
|
|
Posts/comments stored in PostgreSQL, not JSON files. One filter system
|
|
(filter_pipeline/), not two. No import-time side effects.
|
|
- Frontend: a Vite-built SPA served as static files by Flask at /
|
|
(same origin), calling /api/* with session-cookie credentials. Replaces
|
|
dashboard.html, post_detail.html, settings/admin/bookmark templates,
|
|
and the dead generate_html.py path.
|
|
- Auth: Flask-Login session cookies retained (same-origin), exposed via
|
|
/api/auth/* endpoints. No JWT unless a cross-origin/mobile client is
|
|
later required.
|
|
|
|
## Non-goals (this round)
|
|
- Do not switch to FastAPI (defer; revisit after the split is stable).
|
|
- Do not introduce JWT/CORS (same-origin only).
|
|
- Do not rewrite the filter pipeline logic or data-collection fetchers;
|
|
only consolidate and wire them.
|
|
|
|
## Phases (each independently shippable)
|
|
|
|
### Phase 0 - Stop the bleeding (bugfixes, no architecture change)
|
|
- Fix UTF-8 mojibake across .py/.html/.json (icons, checkmarks, separators).
|
|
- Fix migrate_bookmarks.py (it treats init_db's None return as a db object).
|
|
- Make password min-length consistent (8 everywhere; reset route uses 6).
|
|
- Fix post_detail.html undefined moment filter (comment times always
|
|
render "Recently").
|
|
- Close the comment.content | safe XSS hole in post_detail.html.
|
|
- Make non-no_filter filtersets not silently no-op when AI is disabled
|
|
(reject clearly or document; do not silently pass everything).
|
|
- Remove import-time side effects from app.py (polling_service.start()
|
|
and filter_engine init must not run on import).
|
|
|
|
### Phase 1 - Complete the API surface in Flask
|
|
- Audit existing /api/* endpoints; fill gaps so every Jinja page's data
|
|
need has an API endpoint:
|
|
- /api/auth/login, /api/auth/register, /api/auth/me, /api/auth/logout,
|
|
/api/auth/password-reset/*
|
|
- /api/posts, /api/posts/<uuid> (post + comment tree), /api/comments/<post_uuid>
|
|
- /api/bookmarks, /api/bookmark (toggle), /api/bookmark-status/<uuid>
|
|
- /api/settings (GET/PUT profile, communities, filters, experience),
|
|
/api/filtersets, /api/platforms
|
|
- /api/admin/users/*, /api/admin/polling/*, /api/admin/cache, /api/admin/backup
|
|
- Convert server-rendered form POST routes (settings, profile, avatar
|
|
upload, admin) to JSON endpoints; keep flash-message behavior as API
|
|
status codes/messages.
|
|
- Split app.py into blueprints + a services/ layer.
|
|
|
|
### Phase 2 - Collapse the filter stack to one system
|
|
- Adopt filter_pipeline/ as the single filter engine.
|
|
- Port filter_lib's rule operators and comment_lib's tree modes
|
|
(tree-pruning, individual, score/time/length modes) into filter_pipeline
|
|
as a comment-filtering stage.
|
|
- Wire comment filtering into the live /api/posts/<uuid> path (today only
|
|
the dead static path filters comments).
|
|
- Delete filter_lib.py, comment_lib.py, html_generation_lib.py,
|
|
generate_html.py, the active_html/ route, and the theme template path.
|
|
Keep themes/ only as CSS/JS asset bundles.
|
|
|
|
### Phase 3 - Move posts/comments into PostgreSQL
|
|
- Add Post and Comment SQLAlchemy models; map the existing JSON schema
|
|
(uuid, platform, id, title, author, timestamp, score, replies, url,
|
|
content, source, tags, meta, moderation_uuid, parent_comment_uuid).
|
|
- Write a one-shot backfill migration that ingests data/posts/*.json and
|
|
data/comments/*.json into the DB.
|
|
- Replace _load_posts_cache() and the directory scans in /api/posts,
|
|
/api/platforms, /api/content-timestamp with DB queries + a real TTL cache
|
|
(Flask-Caching now; Redis layer later if needed).
|
|
- Keep data/ as an archive/export only, not the source of truth.
|
|
|
|
### Phase 4 - Build the SPA client (feature-by-feature parity)
|
|
- Scaffold a Vite SPA (framework TBD), served by Flask as static files at /
|
|
with a catch-all fallback to index.html.
|
|
- Build in order: feed (list + pagination + filters + communities) ->
|
|
post detail (comment tree) -> auth (login/register/password-reset) ->
|
|
bookmarks -> settings (profile/communities/filters/experience) ->
|
|
admin (users/polling/logs).
|
|
- Dev: Vite proxy -> Flask. Prod: Flask serves built dist/.
|
|
- Use credentials: 'include' on all fetch calls; session cookies do the rest.
|
|
|
|
### Phase 5 - Cut over and delete the old render path
|
|
- Once the SPA reaches parity, remove the Jinja templates, render_template
|
|
calls, the templates/ folder, and the static-HTML generation.
|
|
- Remove the now-dead _nav.html, base.html, page templates, and the
|
|
serve_theme/serve_logo page-serving routes (replace with plain static
|
|
asset routes).
|
|
- Update README/DEPLOYMENT/Dockerfile for the new single-container deploy.
|
|
|
|
### Phase 6 - Hardening
|
|
- Add a test suite (pytest for services + API; component tests for the SPA).
|
|
- Add lint/format (ruff + the SPA's linter) and a CI check.
|
|
- Add a charset/encoding CI gate so the mojibake never returns.
|
|
- Lock the API with a versioned prefix (/api/v1/*) so future changes do not
|
|
break shipped clients.
|
|
|
|
## Decisions baked in (defaults)
|
|
- Backend framework: Flask (kept) - not FastAPI this round.
|
|
- Auth: Flask-Login session cookies, same-origin - not JWT.
|
|
- Deployment: single container, Flask serves the built SPA - not split hosts.
|
|
- Storage: PostgreSQL for posts/comments - not JSON files.
|
|
|
|
## Open decisions (pick before Phase 4 starts)
|
|
- Frontend framework: React / Vue / Svelte / Solid. No wrong answer at this
|
|
app's size; pick what you enjoy maintaining.
|
|
- Whether to version the API as /api/v1 from day one (recommended - cheap
|
|
insurance).
|
|
|
|
## Success criteria
|
|
- app.py is gone or reduced to app-factory + blueprint registration.
|
|
- No render_template exists in the codebase.
|
|
- One filter system; one render path.
|
|
- Posts/comments queried from Postgres; no full-disk-scan cache.
|
|
- Importing the app module has zero side effects (no scheduler, no polling).
|
|
- The SPA, served by Flask, reproduces all current user-facing features. |