6cf35ca034
Phase 1-3 + 6: pluggable filter system, app factory, test harness
...
Implements the parallel.md workstreams (Agents A-E) toward REFACTOR_GOAL.md.
Phase 1 — app factory + blueprints + /api/v1:
- app.py -> create_app() factory (no module-level app); entrypoints updated
- routes/ (auth, pages, settings, admin, assets) + blueprints/api.py at /api/v1
- config.py / extensions.py / security.py extracted; services/ layer added
- endpoint names preserved so template url_for() calls keep resolving
(static check: all 27 template url_for endpoints are defined routes)
Phase 2 — one pluggable filter system:
- filter_pipeline/registry.py: @register_stage / @register_plugin + discover_modules
- engine._init_stages() instantiates registered stages (no hardcoded dict);
process_batch is AI-aware: only short-circuits to the AI-disabled path when
a filterset's stages declare requires_ai, so offline filtersets run with AI off
- BaseFilterPlugin gets a consumer (stages/plugins.py); Keyword/Quality
re-enabled via filter_config.json plugins config
- comment tree modes ported to stages/comment_filter.py + shared rules.py;
wired into /api/v1/posts/<uuid> and /api/v1/comments/<uuid> via
FilterEngine.filter_comments() (fails open)
- offline quality_filter filterset exercises plugins+ranker without AI
- legacy filter_lib / comment_lib / html_generation_lib / generate_html /
active_html path deleted
Phase 3 prep — pluggable fetchers + Postgres models:
- Post / Comment SQLAlchemy models added to models.py
- migrate_content_to_db.py backfill (idempotent by uuid, batched, --dry-run)
- platforms/ fetcher registry (extension point)
- live reads/writes still go through PostService (disk JSON); cutover deferred
Phase 6 — test harness:
- pytest.ini + tests/ (conftest with in-memory SQLite fixture, no Postgres;
stubbed polling/filter singletons)
- test_app_factory.py (route registration, no module-level app),
test_api_contracts.py (posts/post_detail/comments/filters shape with
monkeypatched post_service + get_filter_engine),
test_filter_pipeline.py + test_plugin_contract.py (Flask-free; validated
locally 12/12 incl. drop-in stage/plugin discovered with zero core edits)
Other: .gitignore added (__pycache__, data/, secrets); pytest in requirements.
Verification: py_compile clean across the project; the Flask-free filter-pipeline
and plugin-contract tests pass locally. App-factory / API-contract tests need
deps+docker to run; runtime flask routes / Auth0-repeated-create_app also gated
on docker.
Co-Authored-By: Claude <noreply@anthropic.com >
2026-07-03 02:29:46 -05:00
a3ea1e9bdb
Add filter pipeline stages and plugins (Phase 2 & 3)
...
Implements complete content filtering pipeline with AI-powered analysis:
Phase 2 - Pipeline Stages:
- CategorizerStage: AI topic detection with content-hash caching
- ModeratorStage: Safety/quality analysis (violence, hate speech, quality scores)
- FilterStage: Fast rule-based filtering from filtersets.json
- RankerStage: Multi-factor scoring (quality, recency, source tier, engagement)
Phase 3 - Filter Plugins:
- KeywordFilterPlugin: Blocklist/allowlist keyword filtering
- QualityFilterPlugin: Quality metrics (length, caps, clickbait detection)
AI Client:
- OpenRouterClient: Llama 70B integration with retry logic
- Methods: categorize(), moderate(), score_quality(), analyze_sentiment()
- Content-hash based caching for cost efficiency
Pipeline Flow:
Raw Post → Categorizer → Moderator → Filter → Ranker → Scored Post
Key Features:
- All AI results cached permanently by content hash
- Parallel processing support (10 workers)
- Fallback modes when AI disabled
- Comprehensive scoring breakdown
- Plugin architecture for extensibility
Related to filtering engine implementation
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-10-11 22:54:38 -05:00
94e12041ec
Add filter pipeline core infrastructure (Phase 1)
...
Implements plugin-based content filtering system with multi-level caching:
Core Components:
- FilterEngine: Main orchestrator for content filtering
- FilterCache: 3-level caching (memory, AI results, filterset results)
- FilterConfig: Configuration loader for filter_config.json & filtersets.json
- FilterResult & AIAnalysisResult: Data models for filter results
Architecture:
- BaseStage: Abstract class for pipeline stages
- BaseFilterPlugin: Abstract class for filter plugins
- Multi-threaded parallel processing support
- Content-hash based AI result caching (cost savings)
- Filterset result caching (fast filterset switching)
Configuration:
- filter_config.json: AI models, caching, parallel workers
- Using only Llama 70B for cost efficiency
- Compatible with existing filtersets.json
Integration:
- apply_filterset() API compatible with user preferences
- process_batch() for batch post processing
- Lazy-loaded stages to avoid import errors when AI disabled
Related to issue #8 (filtering engine implementation)
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-10-11 22:46:10 -05:00
