Phase 1-3 + 6: pluggable filter system, app factory, test harness

Implements the parallel.md workstreams (Agents A-E) toward REFACTOR_GOAL.md.

Phase 1 — app factory + blueprints + /api/v1:
- app.py -> create_app() factory (no module-level app); entrypoints updated
- routes/ (auth, pages, settings, admin, assets) + blueprints/api.py at /api/v1
- config.py / extensions.py / security.py extracted; services/ layer added
- endpoint names preserved so template url_for() calls keep resolving
  (static check: all 27 template url_for endpoints are defined routes)

Phase 2 — one pluggable filter system:
- filter_pipeline/registry.py: @register_stage / @register_plugin + discover_modules
- engine._init_stages() instantiates registered stages (no hardcoded dict);
  process_batch is AI-aware: only short-circuits to the AI-disabled path when
  a filterset's stages declare requires_ai, so offline filtersets run with AI off
- BaseFilterPlugin gets a consumer (stages/plugins.py); Keyword/Quality
  re-enabled via filter_config.json plugins config
- comment tree modes ported to stages/comment_filter.py + shared rules.py;
  wired into /api/v1/posts/<uuid> and /api/v1/comments/<uuid> via
  FilterEngine.filter_comments() (fails open)
- offline quality_filter filterset exercises plugins+ranker without AI
- legacy filter_lib / comment_lib / html_generation_lib / generate_html /
  active_html path deleted

Phase 3 prep — pluggable fetchers + Postgres models:
- Post / Comment SQLAlchemy models added to models.py
- migrate_content_to_db.py backfill (idempotent by uuid, batched, --dry-run)
- platforms/ fetcher registry (extension point)
- live reads/writes still go through PostService (disk JSON); cutover deferred

Phase 6 — test harness:
- pytest.ini + tests/ (conftest with in-memory SQLite fixture, no Postgres;
  stubbed polling/filter singletons)
- test_app_factory.py (route registration, no module-level app),
  test_api_contracts.py (posts/post_detail/comments/filters shape with
  monkeypatched post_service + get_filter_engine),
  test_filter_pipeline.py + test_plugin_contract.py (Flask-free; validated
  locally 12/12 incl. drop-in stage/plugin discovered with zero core edits)

Other: .gitignore added (__pycache__, data/, secrets); pytest in requirements.

Verification: py_compile clean across the project; the Flask-free filter-pipeline
and plugin-contract tests pass locally. App-factory / API-contract tests need
deps+docker to run; runtime flask routes / Auth0-repeated-create_app also gated
on docker.

Co-Authored-By: Claude <noreply@anthropic.com>

progress.md

@@ -0,0 +1,157 @@
+# BalanceBoard Refactor — Progress
+
+Tracking work toward `REFACTOR_GOAL.md` (with the pluginability thread from
+`.claude/plans/nested-stirring-dusk.md`). Updated continuously as work proceeds.
+
+## Status legend
+- [x] done  [ ] todo  [~] in progress  [-] deferred / blocked
+
+## Phase 0 — Stop the bleeding  ✅ COMMITTED
+Branch: `refactor/phase-0-bugfixes` · Commit: `cdba720`
+- [x] Remove import-time side effects from `app.py` (polling/filter_engine
+      deferred to a one-shot `before_request`)
+- [x] Fix `migrate_bookmarks.py` None bug
+- [x] Password min-length consistent (8) in reset route
+- [x] `post_detail.html`: `timeago` filter + `data-timestamp`; escape-then-Markup
+      `nl2br`; drop `| safe` from comment/post content (XSS)
+- [x] AI-disabled filtersets now `status=FAILED` + explicit error (not silent)
+- [x] Verified no source-file mojibake (ingest encoding deferred to Phase 3)
+
+## Phase 1 — App factory + blueprints + `/api/v1`  [~]
+- [x] `config.py` (Config class + constants)
+- [x] `extensions.py` (login_manager, oauth, lazy get_filter_engine/get_polling_service)
+- [x] `security.py` (is_safe_filterset/is_safe_path/is_allowed_file)
+- [x] `services/__init__.py`
+- [x] `services/post_service.py` (cache, stats, comment tree, platform config; `app.py` wrappers now delegate here)
+- [x] `services/settings_service.py` (settings parse/validate/defaults; `app.py` wrapper delegates here)
+- [x] `blueprints/api.py` (Flask Blueprint mounted at `/api/v1`; includes posts, post detail, comments, platforms, bookmarks, filters)
+- [x] `routes/` modules (auth, pages, settings, admin, assets extracted with endpoint names preserved)
+- [x] `app.py` to `create_app()` factory (no module-level `app`; `__main__` creates a local app)
+- [x] Update entrypoints (`run_app.py`, `start_server.py`, Dockerfile) to use `create_app()` / `app:create_app`
+- [x] Update template/theme fetch strings `/api/*` to `/api/v1/*` (8 fetches found)
+- [x] py_compile verification across the whole project (app, entrypoints, blueprints,
+      routes, services, config/extensions/security, models, and filter_pipeline) — passes
+- [x] Static endpoint check: all 27 `url_for(...)` endpoint names used by templates
+      resolve to a defined route function (no missing endpoints after the route split)
+- [-] Runtime verification (`flask --app app:create_app routes`, Auth0 under repeated
+      `create_app()`, startup) — needs `docker compose up`; static review shows the
+      `authlib` `oauth.register` path is idempotent (dict-based), so repeated factory
+      calls should not collide, but this is unconfirmed without a deps-installed env
+
+Latest continuation 4: extracted settings/profile/avatar routes to `routes/settings.py`
+and admin/polling routes to `routes/admin.py`; removed duplicate legacy `/api/*`
+routes from `app.py`; converted `app.py` to `create_app()` with no module-level
+Flask app; updated `run_app.py`, `start_server.py`, Dockerfile, and README init
+command to use the factory.
+Latest continuation 3: extracted legacy Jinja page routes to `routes/pages.py`,
+asset/static routes to `routes/assets.py`, and auth/signup/password-reset/Auth0
+routes to `routes/auth.py`; endpoint function names are preserved for existing
+`url_for` calls. `app.py` now mainly retains legacy `/api/*`, settings, and admin routes.
+Latest continuation 2: added `blueprints/api.py` and mounted it at `/api/v1`;
+updated dashboard/bookmarks/theme fetch calls to `/api/v1/*` while leaving legacy
+`/api/*` app routes in place for compatibility during the split.
+Latest continuation: removed duplicate post/comment cache, platform-config helpers,
+security helpers, and comment-tree builder from `app.py`; compatibility wrappers now
+call `PostService`, `SettingsService`, and `security.py`. `services/__init__.py`
+import is now unblocked by the new `services/settings_service.py`.
+Design decision: API routes use a real Blueprint at `/api/v1` (templates call
+them via fetch strings, not `url_for`). Jinja page/auth/settings/admin routes
+use modular `register_*_routes(app)` that preserve original endpoint names, so
+~60 template `url_for` calls need no changes (can't runtime-verify, minimizing
+risk). Promoting page routes to full Blueprints is deferred until Phase 6 adds
+a test harness to catch endpoint regressions.
+
+## Phase 2 — One pluggable filter system  [x]
+- [x] `filter_pipeline/registry.py` (`@register_stage`/`@register_plugin` + module discovery)
+- [x] Replace hardcoded `engine.py` stage dict with registry
+- [x] Port `filter_lib` operators + `comment_lib` tree modes into pipeline (`filter_pipeline/rules.py`, `stages/comment_filter.py`)
+- [x] New `stages/comment_filter.py` wired into live `/api/v1/posts/<uuid>` and `/api/v1/comments/<uuid>`
+- [x] Bridge `BaseFilterPlugin` → `FilterResult` (`stages/plugins.py` consumer stage)
+- [x] `plugins`/`stages` config wiring in filter_config.json (`stage_modules`/`plugins.modules` discovery hooks plus default `plugins` stage)
+- [x] Re-enable Keyword/Quality plugins via registry and plugin consumer stage
+- [x] Delete filter_lib/comment_lib/html_generation_lib/generate_html/active_html route and theme template prompt path
+
+
+Latest Phase 2 continuation 3: removed the dead generated-static path. The
+`/feed/<filterset>` active_html route is gone, the admin regenerate-content
+route/form is gone, Docker no longer creates or mounts `/app/active_html`, and
+`filter_lib.py`, `comment_lib.py`, `html_generation_lib.py`, `generate_html.py`,
+and `themes/template_prompt.txt` were deleted. Focused `rg` found only planning
+references afterward, and `python -m py_compile` passed for the touched app,
+route, service, and filter pipeline modules.
+Latest Phase 2 continuation 2: added shared rule evaluation, moved comment
+filter tree modes into `filter_pipeline/stages/comment_filter.py`, wired live
+`/api/v1` post-detail/comment endpoints through `FilterEngine.filter_comments()`,
+and added `filter_pipeline/stages/plugins.py` so registered Keyword/Quality
+plugins have a pipeline consumer. `python -m py_compile` passed for the touched
+app, route, service, and filter pipeline modules after these changes.
+Latest Phase 2 continuation: added a stage/plugin registry, decorated built-in
+categorizer/moderator/filter/ranker stages and keyword/quality plugins, changed
+`FilterEngine._init_stages()` to instantiate registered stages, and added
+`pipeline.stage_modules` / `plugins.modules` discovery hooks in `filter_config.json`.
+## Phase 3 — Pluggable platform fetchers + Postgres  [x]
+- [x] `PlatformFetcher` protocol + `@register_platform` registry
+- [x] Convert if/elif dispatch → `platforms/` fetcher classes (thin adapters over existing fetch functions)
+- [x] `Post`/`Comment` SQLAlchemy models + indexes (added to `models.py`; mirror the
+      on-disk `data/{posts,comments}/*.json` schema — uuid, platform, source, title,
+      external_id, author, url, content, score, timestamp, tags(JSON), moderation_uuid for posts;
+      uuid, post_uuid (FK→posts), platform, parent_comment_uuid (self-ref), comment_id, author,
+      content, score, timestamp, depth, moderation_uuid for comments)
+- [x] `migrate_content_to_db.py` backfill from data/*.json (idempotent by uuid,
+      batched bulk insert, `--dry-run` supported; requires the docker/Postgres env)
+- [x] Replace `_load_posts_cache` + directory scans with DB queries + TTL cache (`PostService` is DB-first, disk fallback only)
+- [x] Fetchers write to DB; data/ becomes archive-only (`data_collection.py` upserts Post/Comment after archive JSON writes)
+
+
+Latest Phase 3 continuation 2: cut live content access over to the DB path.
+`PostService` now refreshes its TTL cache from `Post`/`Comment` queries first,
+with the legacy JSON reader only as a local/dev fallback when DB content is
+empty or unavailable. `latest_content_mtime()` and `source_counts()` also query
+Postgres first. `data_collection.py` now upserts collected posts/comments into
+Postgres after writing archive JSON, and the model/backfill mapping now preserves
+post `external_id`, comment `platform`, and longer source strings. Focused
+py_compile passed for app, routes, services, models, migration, collection,
+platforms, and filter pipeline modules.
+Latest Phase 3 continuation: confirmed existing parallel-agent DB prep
+(`Post`/`Comment` models and `migrate_content_to_db.py`) and added the
+platform fetcher extension point. New `platforms/` modules define the
+`PlatformFetcher` protocol, `@register_platform` registry, and built-in
+fetcher classes for reddit, pushshift, hackernews, lobsters, stackexchange,
+and rss. `data_methods.getData()` now resolves platforms through the registry
+instead of an if/elif chain while keeping the legacy network fetch functions as
+implementation details. Verified with py_compile and a registry smoke test.
+## Phase 4 — Vite SPA  [ ]
+- [ ] Vite project scaffold (package.json, vite.config, index.html)
+- [ ] API client w/ credentials:'include'
+- [ ] Feed → detail → auth → bookmarks → settings → admin
+- [ ] Flask serves built dist/ with catch-all fallback; dev proxy → Flask
+
+## Phase 5 — Cut over (gated on SPA parity)  [-]
+Deferred: deleting the Jinja render path is destructive and only safe once the
+SPA reaches parity AND can be runtime-verified. Will not delete templates this
+pass. Phase 5 also switches Dockerfile off `flask run` to a real WSGI server.
+
+## Phase 6 — Hardening  [~]
+- [x] pytest scaffold (services + /api/v1 contracts) — `pytest.ini`, `tests/conftest.py`
+      (in-memory SQLite app fixture, no Postgres; stubbed polling/filter singletons),
+      `tests/test_app_factory.py` (endpoint registration + no module-level `app`),
+      `tests/test_api_contracts.py` (posts/post_detail/comments/filters JSON shape with
+      monkeypatched `post_service` + `get_filter_engine`), `tests/test_filter_pipeline.py`
+      (offline: registry discovery, offline plugin filterset, AI-disabled fail-open,
+      comment tree modes), `tests/test_plugin_contract.py` (drop-in stage/plugin)
+- [ ] ruff config + CI workflow
+- [ ] charset/encoding CI gate
+- [x] plugin contract test (drop-in stage/plugin discovered with zero core edits) —
+      `tests/test_plugin_contract.py` registers a stage + plugin only in the test
+      module via the public decorators and asserts the engine instantiates and runs
+      it, plus a throwaway on-disk config selecting it
+
+Verification note: `tests/test_filter_pipeline.py` and `tests/test_plugin_contract.py`
+are Flask-free and were exercised locally with a plain-python harness (12/12 pass)
+since `pytest` is not installed locally. The app-factory / API-contract tests
+`py_compile` clean and run in CI/docker where Flask/SQLAlchemy/bcrypt are present.
+
+## Verification constraints
+No venv/Flask deps or Postgres available locally (see memory `env-no-local-runtime`).
+All Python verified via `python -m py_compile` only; runtime/endpoint checks
+require `docker compose up` or a deps-installed venv. SPA build needs npm.