Phase 1-3 + 6: pluggable filter system, app factory, test harness

Implements the parallel.md workstreams (Agents A-E) toward REFACTOR_GOAL.md.

Phase 1 — app factory + blueprints + /api/v1:
- app.py -> create_app() factory (no module-level app); entrypoints updated
- routes/ (auth, pages, settings, admin, assets) + blueprints/api.py at /api/v1
- config.py / extensions.py / security.py extracted; services/ layer added
- endpoint names preserved so template url_for() calls keep resolving
  (static check: all 27 template url_for endpoints are defined routes)

Phase 2 — one pluggable filter system:
- filter_pipeline/registry.py: @register_stage / @register_plugin + discover_modules
- engine._init_stages() instantiates registered stages (no hardcoded dict);
  process_batch is AI-aware: only short-circuits to the AI-disabled path when
  a filterset's stages declare requires_ai, so offline filtersets run with AI off
- BaseFilterPlugin gets a consumer (stages/plugins.py); Keyword/Quality
  re-enabled via filter_config.json plugins config
- comment tree modes ported to stages/comment_filter.py + shared rules.py;
  wired into /api/v1/posts/<uuid> and /api/v1/comments/<uuid> via
  FilterEngine.filter_comments() (fails open)
- offline quality_filter filterset exercises plugins+ranker without AI
- legacy filter_lib / comment_lib / html_generation_lib / generate_html /
  active_html path deleted

Phase 3 prep — pluggable fetchers + Postgres models:
- Post / Comment SQLAlchemy models added to models.py
- migrate_content_to_db.py backfill (idempotent by uuid, batched, --dry-run)
- platforms/ fetcher registry (extension point)
- live reads/writes still go through PostService (disk JSON); cutover deferred

Phase 6 — test harness:
- pytest.ini + tests/ (conftest with in-memory SQLite fixture, no Postgres;
  stubbed polling/filter singletons)
- test_app_factory.py (route registration, no module-level app),
  test_api_contracts.py (posts/post_detail/comments/filters shape with
  monkeypatched post_service + get_filter_engine),
  test_filter_pipeline.py + test_plugin_contract.py (Flask-free; validated
  locally 12/12 incl. drop-in stage/plugin discovered with zero core edits)

Other: .gitignore added (__pycache__, data/, secrets); pytest in requirements.

Verification: py_compile clean across the project; the Flask-free filter-pipeline
and plugin-contract tests pass locally. App-factory / API-contract tests need
deps+docker to run; runtime flask routes / Auth0-repeated-create_app also gated
on docker.

Co-Authored-By: Claude <noreply@anthropic.com>

platforms/__init__.py

@@ -0,0 +1,19 @@
+"""Platform fetcher extension points."""
+
+from .base import PlatformFetcher
+from .registry import (
+    discover_modules,
+    get_platform_class,
+    get_platform_fetcher,
+    get_registered_platforms,
+    register_platform,
+)
+
+__all__ = [
+    "PlatformFetcher",
+    "discover_modules",
+    "get_platform_class",
+    "get_platform_fetcher",
+    "get_registered_platforms",
+    "register_platform",
+]

platforms/base.py

@@ -0,0 +1,18 @@
+"""Base protocol for platform fetchers."""
+
+from typing import Dict, List, Protocol
+
+
+class PlatformFetcher(Protocol):
+    """Fetch posts for one configured platform/community."""
+
+    name: str
+
+    def fetch_posts(
+        self,
+        start_date: str,
+        end_date: str,
+        community: str,
+        max_posts: int,
+    ) -> List[Dict]:
+        """Return posts normalized to the existing collection schema."""

platforms/builtins.py

@@ -0,0 +1,50 @@
+"""Built-in platform fetchers backed by the legacy fetch functions."""
+
+from .registry import register_platform
+
+
+class _LegacyMethodFetcher:
+    method_name = ""
+
+    def fetch_posts(self, start_date, end_date, community, max_posts):
+        # Import lazily so data_collection_lib can import this module while defining data_methods.
+        from data_collection_lib import data_methods
+
+        method = getattr(data_methods.fetchers, self.method_name)
+        return method(start_date, end_date, community, max_posts)
+
+
+@register_platform("reddit")
+class RedditFetcher(_LegacyMethodFetcher):
+    name = "reddit"
+    method_name = "getRedditData"
+
+
+@register_platform("pushshift")
+class PushshiftFetcher(_LegacyMethodFetcher):
+    name = "pushshift"
+    method_name = "getPushshiftData"
+
+
+@register_platform("hackernews")
+class HackerNewsFetcher(_LegacyMethodFetcher):
+    name = "hackernews"
+    method_name = "getHackerNewsData"
+
+
+@register_platform("lobsters")
+class LobstersFetcher(_LegacyMethodFetcher):
+    name = "lobsters"
+    method_name = "getLobstersData"
+
+
+@register_platform("stackexchange")
+class StackExchangeFetcher(_LegacyMethodFetcher):
+    name = "stackexchange"
+    method_name = "getStackExchangeData"
+
+
+@register_platform("rss")
+class RSSFetcher(_LegacyMethodFetcher):
+    name = "rss"
+    method_name = "getRSSData"

platforms/registry.py

@@ -0,0 +1,40 @@
+"""Platform fetcher registry for data collection."""
+
+from importlib import import_module
+from typing import Dict, Iterable, Optional, Type
+
+from .base import PlatformFetcher
+
+_PLATFORM_FETCHERS: Dict[str, Type[PlatformFetcher]] = {}
+
+
+def register_platform(name: str):
+    """Register a platform fetcher class by config/platform name."""
+    normalized = name.strip().lower()
+    if not normalized:
+        raise ValueError("Platform name must not be empty")
+
+    def decorator(cls: Type[PlatformFetcher]) -> Type[PlatformFetcher]:
+        _PLATFORM_FETCHERS[normalized] = cls
+        return cls
+
+    return decorator
+
+
+def get_platform_class(name: str) -> Optional[Type[PlatformFetcher]]:
+    return _PLATFORM_FETCHERS.get((name or "").strip().lower())
+
+
+def get_platform_fetcher(name: str) -> Optional[PlatformFetcher]:
+    cls = get_platform_class(name)
+    return cls() if cls else None
+
+
+def get_registered_platforms() -> Dict[str, Type[PlatformFetcher]]:
+    return dict(_PLATFORM_FETCHERS)
+
+
+def discover_modules(module_names: Iterable[str]) -> None:
+    for module_name in module_names:
+        if module_name:
+            import_module(module_name)