balanceboard/progress.md

# BalanceBoard Refactor — Progress

Tracking work toward `REFACTOR_GOAL.md` (with the pluginability thread from
`.claude/plans/nested-stirring-dusk.md`). Updated continuously as work proceeds.

## Status legend
- [x] done  [ ] todo  [~] in progress  [-] deferred / blocked

## Phase 0 — Stop the bleeding  ✅ COMMITTED
Branch: `refactor/phase-0-bugfixes` · Commit: `cdba720`
- [x] Remove import-time side effects from `app.py` (polling/filter_engine
      deferred to a one-shot `before_request`)
- [x] Fix `migrate_bookmarks.py` None bug
- [x] Password min-length consistent (8) in reset route
- [x] `post_detail.html`: `timeago` filter + `data-timestamp`; escape-then-Markup
      `nl2br`; drop `| safe` from comment/post content (XSS)
- [x] AI-disabled filtersets now `status=FAILED` + explicit error (not silent)
- [x] Verified no source-file mojibake (ingest encoding deferred to Phase 3)

## Phase 1 — App factory + blueprints + `/api/v1`  [~]
- [x] `config.py` (Config class + constants)
- [x] `extensions.py` (login_manager, oauth, lazy get_filter_engine/get_polling_service)
- [x] `security.py` (is_safe_filterset/is_safe_path/is_allowed_file)
- [x] `services/__init__.py`
- [x] `services/post_service.py` (cache, stats, comment tree, platform config; `app.py` wrappers now delegate here)
- [x] `services/settings_service.py` (settings parse/validate/defaults; `app.py` wrapper delegates here)
- [x] `blueprints/api.py` (Flask Blueprint mounted at `/api/v1`; includes posts, post detail, comments, platforms, bookmarks, filters)
- [x] `routes/` modules (auth, pages, settings, admin, assets extracted with endpoint names preserved)
- [x] `app.py` to `create_app()` factory (no module-level `app`; `__main__` creates a local app)
- [x] Update entrypoints (`run_app.py`, `start_server.py`, Dockerfile) to use `create_app()` / `app:create_app`
- [x] Update template/theme fetch strings `/api/*` to `/api/v1/*` (8 fetches found)
- [x] py_compile verification across the whole project (app, entrypoints, blueprints,
      routes, services, config/extensions/security, models, and filter_pipeline) — passes
- [x] Static endpoint check: all 27 `url_for(...)` endpoint names used by templates
      resolve to a defined route function (no missing endpoints after the route split)
- [-] Runtime verification (`flask --app app:create_app routes`, Auth0 under repeated
      `create_app()`, startup) — needs `docker compose up`; static review shows the
      `authlib` `oauth.register` path is idempotent (dict-based), so repeated factory
      calls should not collide, but this is unconfirmed without a deps-installed env

Latest continuation 4: extracted settings/profile/avatar routes to `routes/settings.py`
and admin/polling routes to `routes/admin.py`; removed duplicate legacy `/api/*`
routes from `app.py`; converted `app.py` to `create_app()` with no module-level
Flask app; updated `run_app.py`, `start_server.py`, Dockerfile, and README init
command to use the factory.
Latest continuation 3: extracted legacy Jinja page routes to `routes/pages.py`,
asset/static routes to `routes/assets.py`, and auth/signup/password-reset/Auth0
routes to `routes/auth.py`; endpoint function names are preserved for existing
`url_for` calls. `app.py` now mainly retains legacy `/api/*`, settings, and admin routes.
Latest continuation 2: added `blueprints/api.py` and mounted it at `/api/v1`;
updated dashboard/bookmarks/theme fetch calls to `/api/v1/*` while leaving legacy
`/api/*` app routes in place for compatibility during the split.
Latest continuation: removed duplicate post/comment cache, platform-config helpers,
security helpers, and comment-tree builder from `app.py`; compatibility wrappers now
call `PostService`, `SettingsService`, and `security.py`. `services/__init__.py`
import is now unblocked by the new `services/settings_service.py`.
Design decision: API routes use a real Blueprint at `/api/v1` (templates call
them via fetch strings, not `url_for`). Jinja page/auth/settings/admin routes
use modular `register_*_routes(app)` that preserve original endpoint names, so
~60 template `url_for` calls need no changes (can't runtime-verify, minimizing
risk). Promoting page routes to full Blueprints is deferred until Phase 6 adds
a test harness to catch endpoint regressions.

## Phase 2 — One pluggable filter system  [x]
- [x] `filter_pipeline/registry.py` (`@register_stage`/`@register_plugin` + module discovery)
- [x] Replace hardcoded `engine.py` stage dict with registry
- [x] Port `filter_lib` operators + `comment_lib` tree modes into pipeline (`filter_pipeline/rules.py`, `stages/comment_filter.py`)
- [x] New `stages/comment_filter.py` wired into live `/api/v1/posts/<uuid>` and `/api/v1/comments/<uuid>`
- [x] Bridge `BaseFilterPlugin` → `FilterResult` (`stages/plugins.py` consumer stage)
- [x] `plugins`/`stages` config wiring in filter_config.json (`stage_modules`/`plugins.modules` discovery hooks plus default `plugins` stage)
- [x] Re-enable Keyword/Quality plugins via registry and plugin consumer stage
- [x] Delete filter_lib/comment_lib/html_generation_lib/generate_html/active_html route and theme template prompt path


Latest Phase 2 continuation 3: removed the dead generated-static path. The
`/feed/<filterset>` active_html route is gone, the admin regenerate-content
route/form is gone, Docker no longer creates or mounts `/app/active_html`, and
`filter_lib.py`, `comment_lib.py`, `html_generation_lib.py`, `generate_html.py`,
and `themes/template_prompt.txt` were deleted. Focused `rg` found only planning
references afterward, and `python -m py_compile` passed for the touched app,
route, service, and filter pipeline modules.
Latest Phase 2 continuation 2: added shared rule evaluation, moved comment
filter tree modes into `filter_pipeline/stages/comment_filter.py`, wired live
`/api/v1` post-detail/comment endpoints through `FilterEngine.filter_comments()`,
and added `filter_pipeline/stages/plugins.py` so registered Keyword/Quality
plugins have a pipeline consumer. `python -m py_compile` passed for the touched
app, route, service, and filter pipeline modules after these changes.
Latest Phase 2 continuation: added a stage/plugin registry, decorated built-in
categorizer/moderator/filter/ranker stages and keyword/quality plugins, changed
`FilterEngine._init_stages()` to instantiate registered stages, and added
`pipeline.stage_modules` / `plugins.modules` discovery hooks in `filter_config.json`.
## Phase 3 — Pluggable platform fetchers + Postgres  [x]
- [x] `PlatformFetcher` protocol + `@register_platform` registry
- [x] Convert if/elif dispatch → `platforms/` fetcher classes (thin adapters over existing fetch functions)
- [x] `Post`/`Comment` SQLAlchemy models + indexes (added to `models.py`; mirror the
      on-disk `data/{posts,comments}/*.json` schema — uuid, platform, source, title,
      external_id, author, url, content, score, timestamp, tags(JSON), moderation_uuid for posts;
      uuid, post_uuid (FK→posts), platform, parent_comment_uuid (self-ref), comment_id, author,
      content, score, timestamp, depth, moderation_uuid for comments)
- [x] `migrate_content_to_db.py` backfill from data/*.json (idempotent by uuid,
      batched bulk insert, `--dry-run` supported; requires the docker/Postgres env)
- [x] Replace `_load_posts_cache` + directory scans with DB queries + TTL cache (`PostService` is DB-first, disk fallback only)
- [x] Fetchers write to DB; data/ becomes archive-only (`data_collection.py` upserts Post/Comment after archive JSON writes)


Latest Phase 3 continuation 2: cut live content access over to the DB path.
`PostService` now refreshes its TTL cache from `Post`/`Comment` queries first,
with the legacy JSON reader only as a local/dev fallback when DB content is
empty or unavailable. `latest_content_mtime()` and `source_counts()` also query
Postgres first. `data_collection.py` now upserts collected posts/comments into
Postgres after writing archive JSON, and the model/backfill mapping now preserves
post `external_id`, comment `platform`, and longer source strings. Focused
py_compile passed for app, routes, services, models, migration, collection,
platforms, and filter pipeline modules.
Latest Phase 3 continuation: confirmed existing parallel-agent DB prep
(`Post`/`Comment` models and `migrate_content_to_db.py`) and added the
platform fetcher extension point. New `platforms/` modules define the
`PlatformFetcher` protocol, `@register_platform` registry, and built-in
fetcher classes for reddit, pushshift, hackernews, lobsters, stackexchange,
and rss. `data_methods.getData()` now resolves platforms through the registry
instead of an if/elif chain while keeping the legacy network fetch functions as
implementation details. Verified with py_compile and a registry smoke test.
## Phase 4 — Vite SPA  [ ]
- [ ] Vite project scaffold (package.json, vite.config, index.html)
- [ ] API client w/ credentials:'include'
- [ ] Feed → detail → auth → bookmarks → settings → admin
- [ ] Flask serves built dist/ with catch-all fallback; dev proxy → Flask

## Phase 5 — Cut over (gated on SPA parity)  [-]
Deferred: deleting the Jinja render path is destructive and only safe once the
SPA reaches parity AND can be runtime-verified. Will not delete templates this
pass. Phase 5 also switches Dockerfile off `flask run` to a real WSGI server.

## Phase 6 — Hardening  [~]
- [x] pytest scaffold (services + /api/v1 contracts) — `pytest.ini`, `tests/conftest.py`
      (in-memory SQLite app fixture, no Postgres; stubbed polling/filter singletons),
      `tests/test_app_factory.py` (endpoint registration + no module-level `app`),
      `tests/test_api_contracts.py` (posts/post_detail/comments/filters JSON shape with
      monkeypatched `post_service` + `get_filter_engine`), `tests/test_filter_pipeline.py`
      (offline: registry discovery, offline plugin filterset, AI-disabled fail-open,
      comment tree modes), `tests/test_plugin_contract.py` (drop-in stage/plugin)
- [ ] ruff config + CI workflow
- [ ] charset/encoding CI gate
- [x] plugin contract test (drop-in stage/plugin discovered with zero core edits) —
      `tests/test_plugin_contract.py` registers a stage + plugin only in the test
      module via the public decorators and asserts the engine instantiates and runs
      it, plus a throwaway on-disk config selecting it

Verification note: `tests/test_filter_pipeline.py` and `tests/test_plugin_contract.py`
are Flask-free and were exercised locally with a plain-python harness (12/12 pass)
since `pytest` is not installed locally. The app-factory / API-contract tests
`py_compile` clean and run in CI/docker where Flask/SQLAlchemy/bcrypt are present.

## Verification constraints
No venv/Flask deps or Postgres available locally (see memory `env-no-local-runtime`).
All Python verified via `python -m py_compile` only; runtime/endpoint checks
require `docker compose up` or a deps-installed venv. SPA build needs npm.