Phase 1-3 + 6: pluggable filter system, app factory, test harness

Implements the parallel.md workstreams (Agents A-E) toward REFACTOR_GOAL.md.

Phase 1 — app factory + blueprints + /api/v1:
- app.py -> create_app() factory (no module-level app); entrypoints updated
- routes/ (auth, pages, settings, admin, assets) + blueprints/api.py at /api/v1
- config.py / extensions.py / security.py extracted; services/ layer added
- endpoint names preserved so template url_for() calls keep resolving
  (static check: all 27 template url_for endpoints are defined routes)

Phase 2 — one pluggable filter system:
- filter_pipeline/registry.py: @register_stage / @register_plugin + discover_modules
- engine._init_stages() instantiates registered stages (no hardcoded dict);
  process_batch is AI-aware: only short-circuits to the AI-disabled path when
  a filterset's stages declare requires_ai, so offline filtersets run with AI off
- BaseFilterPlugin gets a consumer (stages/plugins.py); Keyword/Quality
  re-enabled via filter_config.json plugins config
- comment tree modes ported to stages/comment_filter.py + shared rules.py;
  wired into /api/v1/posts/<uuid> and /api/v1/comments/<uuid> via
  FilterEngine.filter_comments() (fails open)
- offline quality_filter filterset exercises plugins+ranker without AI
- legacy filter_lib / comment_lib / html_generation_lib / generate_html /
  active_html path deleted

Phase 3 prep — pluggable fetchers + Postgres models:
- Post / Comment SQLAlchemy models added to models.py
- migrate_content_to_db.py backfill (idempotent by uuid, batched, --dry-run)
- platforms/ fetcher registry (extension point)
- live reads/writes still go through PostService (disk JSON); cutover deferred

Phase 6 — test harness:
- pytest.ini + tests/ (conftest with in-memory SQLite fixture, no Postgres;
  stubbed polling/filter singletons)
- test_app_factory.py (route registration, no module-level app),
  test_api_contracts.py (posts/post_detail/comments/filters shape with
  monkeypatched post_service + get_filter_engine),
  test_filter_pipeline.py + test_plugin_contract.py (Flask-free; validated
  locally 12/12 incl. drop-in stage/plugin discovered with zero core edits)

Other: .gitignore added (__pycache__, data/, secrets); pytest in requirements.

Verification: py_compile clean across the project; the Flask-free filter-pipeline
and plugin-contract tests pass locally. App-factory / API-contract tests need
deps+docker to run; runtime flask routes / Auth0-repeated-create_app also gated
on docker.

Co-Authored-By: Claude <noreply@anthropic.com>

data_collection.py

@@ -11,6 +11,8 @@ from datetime import datetime, timedelta
 from pathlib import Path
 from typing import List, Dict, Tuple
 from data_collection_lib import data_methods
+from database import db
+from models import Comment, Post
 
 
 # ===== STORAGE FUNCTIONS =====
@@ -103,6 +105,52 @@ def create_moderation_stub(target_id: str, target_type: str, dirs: Dict) -> str:
     return mod_uuid
 
 
+
+
+def upsert_post_record(post: Dict):
+    """Best-effort DB upsert; JSON files remain an archive/export artifact."""
+    try:
+        db.session.merge(Post(
+            uuid=post["uuid"],
+            external_id=post.get("id"),
+            platform=post.get("platform", "") or "",
+            source=post.get("source", "") or "",
+            title=(post.get("title") or "")[:500],
+            author=post.get("author"),
+            url=post.get("url"),
+            content=post.get("content"),
+            score=int(post.get("score", 0) or 0),
+            timestamp=int(post.get("timestamp", 0) or 0),
+            tags=post.get("tags"),
+            moderation_uuid=post.get("moderation_uuid"),
+        ))
+        db.session.commit()
+    except Exception as e:
+        db.session.rollback()
+        print(f"Warning: could not persist post {post.get('uuid')} to DB: {e}")
+
+
+def upsert_comment_record(comment: Dict):
+    """Best-effort DB upsert for collected comments."""
+    try:
+        db.session.merge(Comment(
+            uuid=comment["uuid"],
+            post_uuid=comment.get("post_uuid") or "",
+            platform=comment.get("platform"),
+            parent_comment_uuid=comment.get("parent_comment_uuid"),
+            comment_id=comment.get("comment_id") or comment.get("id"),
+            author=comment.get("author"),
+            content=comment.get("content"),
+            score=int(comment.get("score", 0) or 0),
+            timestamp=int(comment.get("timestamp", 0) or 0),
+            depth=int(comment.get("depth", 0) or 0),
+            moderation_uuid=comment.get("moderation_uuid"),
+        ))
+        db.session.commit()
+    except Exception as e:
+        db.session.rollback()
+        print(f"Warning: could not persist comment {comment.get('uuid')} to DB: {e}")
+
 # ===== POST FUNCTIONS =====
 
 def save_post(post: Dict, platform: str, index: Dict, dirs: Dict) -> str:
@@ -122,6 +170,8 @@ def save_post(post: Dict, platform: str, index: Dict, dirs: Dict) -> str:
     with open(post_file, 'w') as f:
         json.dump(post, f, indent=2)
 
+    upsert_post_record(post)
+
     # Update index
     index[post_id] = post_uuid
 
@@ -147,6 +197,8 @@ def save_comment(comment: Dict, post_uuid: str, platform: str, dirs: Dict) -> st
     with open(comment_file, 'w') as f:
         json.dump(comment, f, indent=2)
 
+    upsert_comment_record(comment)
+
     return comment_uuid