balanceboard/progress.md

BalanceBoard Refactor — Progress

Tracking work toward REFACTOR_GOAL.md (with the pluginability thread from .claude/plans/nested-stirring-dusk.md). Updated continuously as work proceeds.

Status legend

• done [ ] todo [~] in progress [-] deferred / blocked

Phase 0 — Stop the bleeding ✅ COMMITTED

Branch: refactor/phase-0-bugfixes · Commit: cdba720

• Remove import-time side effects from app.py (polling/filter_engine deferred to a one-shot before_request)
• Fix migrate_bookmarks.py None bug
• Password min-length consistent (8) in reset route
• post_detail.html: timeago filter + data-timestamp; escape-then-Markup nl2br; drop | safe from comment/post content (XSS)
• AI-disabled filtersets now status=FAILED + explicit error (not silent)
• Verified no source-file mojibake (ingest encoding deferred to Phase 3)

Phase 1 — App factory + blueprints + /api/v1 [~]

• config.py (Config class + constants)
• extensions.py (login_manager, oauth, lazy get_filter_engine/get_polling_service)
• security.py (is_safe_filterset/is_safe_path/is_allowed_file)
• services/__init__.py
• services/post_service.py (cache, stats, comment tree, platform config; app.py wrappers now delegate here)
• services/settings_service.py (settings parse/validate/defaults; app.py wrapper delegates here)
• blueprints/api.py (Flask Blueprint mounted at /api/v1; includes posts, post detail, comments, platforms, bookmarks, filters)
• routes/ modules (auth, pages, settings, admin, assets extracted with endpoint names preserved)
• app.py to create_app() factory (no module-level app; __main__ creates a local app)
• Update entrypoints (run_app.py, start_server.py, Dockerfile) to use create_app() / app:create_app
• Update template/theme fetch strings /api/* to /api/v1/* (8 fetches found)
• py_compile verification across the whole project (app, entrypoints, blueprints, routes, services, config/extensions/security, models, and filter_pipeline) — passes
• Static endpoint check: all 27 url_for(...) endpoint names used by templates resolve to a defined route function (no missing endpoints after the route split)
• [-] Runtime verification (flask --app app:create_app routes, Auth0 under repeated create_app(), startup) — needs docker compose up; static review shows the authlib oauth.register path is idempotent (dict-based), so repeated factory calls should not collide, but this is unconfirmed without a deps-installed env

Latest continuation 4: extracted settings/profile/avatar routes to routes/settings.py and admin/polling routes to routes/admin.py; removed duplicate legacy /api/* routes from app.py; converted app.py to create_app() with no module-level Flask app; updated run_app.py, start_server.py, Dockerfile, and README init command to use the factory. Latest continuation 3: extracted legacy Jinja page routes to routes/pages.py, asset/static routes to routes/assets.py, and auth/signup/password-reset/Auth0 routes to routes/auth.py; endpoint function names are preserved for existing url_for calls. app.py now mainly retains legacy /api/*, settings, and admin routes. Latest continuation 2: added blueprints/api.py and mounted it at /api/v1; updated dashboard/bookmarks/theme fetch calls to /api/v1/* while leaving legacy /api/* app routes in place for compatibility during the split. Latest continuation: removed duplicate post/comment cache, platform-config helpers, security helpers, and comment-tree builder from app.py; compatibility wrappers now call PostService, SettingsService, and security.py. services/__init__.py import is now unblocked by the new services/settings_service.py. Design decision: API routes use a real Blueprint at /api/v1 (templates call them via fetch strings, not url_for). Jinja page/auth/settings/admin routes use modular register_*_routes(app) that preserve original endpoint names, so ~60 template url_for calls need no changes (can't runtime-verify, minimizing risk). Promoting page routes to full Blueprints is deferred until Phase 6 adds a test harness to catch endpoint regressions.

Phase 2 — One pluggable filter system [x]

• filter_pipeline/registry.py (@register_stage/@register_plugin + module discovery)
• Replace hardcoded engine.py stage dict with registry
• Port filter_lib operators + comment_lib tree modes into pipeline (filter_pipeline/rules.py, stages/comment_filter.py)
• New stages/comment_filter.py wired into live /api/v1/posts/<uuid> and /api/v1/comments/<uuid>
• Bridge BaseFilterPlugin → FilterResult (stages/plugins.py consumer stage)
• plugins/stages config wiring in filter_config.json (stage_modules/plugins.modules discovery hooks plus default plugins stage)
• Re-enable Keyword/Quality plugins via registry and plugin consumer stage
• Delete filter_lib/comment_lib/html_generation_lib/generate_html/active_html route and theme template prompt path

Latest Phase 2 continuation 3: removed the dead generated-static path. The /feed/<filterset> active_html route is gone, the admin regenerate-content route/form is gone, Docker no longer creates or mounts /app/active_html, and filter_lib.py, comment_lib.py, html_generation_lib.py, generate_html.py, and themes/template_prompt.txt were deleted. Focused rg found only planning references afterward, and python -m py_compile passed for the touched app, route, service, and filter pipeline modules. Latest Phase 2 continuation 2: added shared rule evaluation, moved comment filter tree modes into filter_pipeline/stages/comment_filter.py, wired live /api/v1 post-detail/comment endpoints through FilterEngine.filter_comments(), and added filter_pipeline/stages/plugins.py so registered Keyword/Quality plugins have a pipeline consumer. python -m py_compile passed for the touched app, route, service, and filter pipeline modules after these changes. Latest Phase 2 continuation: added a stage/plugin registry, decorated built-in categorizer/moderator/filter/ranker stages and keyword/quality plugins, changed FilterEngine._init_stages() to instantiate registered stages, and added pipeline.stage_modules / plugins.modules discovery hooks in filter_config.json.

Phase 3 — Pluggable platform fetchers + Postgres [x]

• PlatformFetcher protocol + @register_platform registry
• Convert if/elif dispatch → platforms/ fetcher classes (thin adapters over existing fetch functions)
• Post/Comment SQLAlchemy models + indexes (added to models.py; mirror the on-disk data/{posts,comments}/*.json schema — uuid, platform, source, title, external_id, author, url, content, score, timestamp, tags(JSON), moderation_uuid for posts; uuid, post_uuid (FK→posts), platform, parent_comment_uuid (self-ref), comment_id, author, content, score, timestamp, depth, moderation_uuid for comments)
• migrate_content_to_db.py backfill from data/*.json (idempotent by uuid, batched bulk insert, --dry-run supported; requires the docker/Postgres env)
• Replace _load_posts_cache + directory scans with DB queries + TTL cache (PostService is DB-first, disk fallback only)
• Fetchers write to DB; data/ becomes archive-only (data_collection.py upserts Post/Comment after archive JSON writes)

Latest Phase 3 continuation 2: cut live content access over to the DB path. PostService now refreshes its TTL cache from Post/Comment queries first, with the legacy JSON reader only as a local/dev fallback when DB content is empty or unavailable. latest_content_mtime() and source_counts() also query Postgres first. data_collection.py now upserts collected posts/comments into Postgres after writing archive JSON, and the model/backfill mapping now preserves post external_id, comment platform, and longer source strings. Focused py_compile passed for app, routes, services, models, migration, collection, platforms, and filter pipeline modules. Latest Phase 3 continuation: confirmed existing parallel-agent DB prep (Post/Comment models and migrate_content_to_db.py) and added the platform fetcher extension point. New platforms/ modules define the PlatformFetcher protocol, @register_platform registry, and built-in fetcher classes for reddit, pushshift, hackernews, lobsters, stackexchange, and rss. data_methods.getData() now resolves platforms through the registry instead of an if/elif chain while keeping the legacy network fetch functions as implementation details. Verified with py_compile and a registry smoke test.

Phase 4 — Vite SPA [ ]

• Vite project scaffold (package.json, vite.config, index.html)
• API client w/ credentials:'include'
• Feed → detail → auth → bookmarks → settings → admin
• Flask serves built dist/ with catch-all fallback; dev proxy → Flask

Phase 5 — Cut over (gated on SPA parity) [-]

Deferred: deleting the Jinja render path is destructive and only safe once the SPA reaches parity AND can be runtime-verified. Will not delete templates this pass. Phase 5 also switches Dockerfile off flask run to a real WSGI server.

Phase 6 — Hardening [~]

• pytest scaffold (services + /api/v1 contracts) — pytest.ini, tests/conftest.py (in-memory SQLite app fixture, no Postgres; stubbed polling/filter singletons), tests/test_app_factory.py (endpoint registration + no module-level app), tests/test_api_contracts.py (posts/post_detail/comments/filters JSON shape with monkeypatched post_service + get_filter_engine), tests/test_filter_pipeline.py (offline: registry discovery, offline plugin filterset, AI-disabled fail-open, comment tree modes), tests/test_plugin_contract.py (drop-in stage/plugin)
• ruff config + CI workflow
• charset/encoding CI gate
• plugin contract test (drop-in stage/plugin discovered with zero core edits) — tests/test_plugin_contract.py registers a stage + plugin only in the test module via the public decorators and asserts the engine instantiates and runs it, plus a throwaway on-disk config selecting it

Verification note: tests/test_filter_pipeline.py and tests/test_plugin_contract.py are Flask-free and were exercised locally with a plain-python harness (12/12 pass) since pytest is not installed locally. The app-factory / API-contract tests py_compile clean and run in CI/docker where Flask/SQLAlchemy/bcrypt are present.

Verification constraints

No venv/Flask deps or Postgres available locally (see memory env-no-local-runtime). All Python verified via python -m py_compile only; runtime/endpoint checks require docker compose up or a deps-installed venv. SPA build needs npm.