commit 7ecc1107b2fc996051da48a577955bd45df724e9 Author: Chelsea Date: Sun Jul 19 21:46:40 2026 -0500 first commit diff --git a/README.md b/README.md new file mode 100644 index 0000000..0b16780 --- /dev/null +++ b/README.md @@ -0,0 +1,383 @@ +# LLM Bot Framework + +A small, cloneable Python 3.11 framework for Discord bots that turn natural +language into validated feature commands. Flask is the application boundary, +PostgreSQL stores users and durable work, and feature packages keep their bot +handler, API routes, prompt, migrations, and scheduled jobs together. + +The included reminders package is both usable and a reference implementation. + +## What is included + +- Discord DM adapter with passwordless provider identities +- Two-stage, OpenAI-compatible command routing and parsing +- Deterministic feature discovery through `modules/*/register(registry)` +- Flask API with short-lived JWTs, user API keys, and scoped service keys +- Transactional PostgreSQL migrations and a small parameterized SQL layer +- Lease-based scheduled jobs and a durable outbound-message queue +- One-time, daily, and weekly reminders with IANA timezone support +- Docker Compose, focused Ruff checks, pytest coverage, and CI + +## Configuration and credential safety + +On a fresh checkout, copy the root template only if `.env` does not already +exist. All processes load this root file. + +```powershell +if (-not (Test-Path .env)) { Copy-Item .env.example .env } +``` + +For an existing installation, do not overwrite either environment file. +Manually merge the needed values from a legacy `config/.env` into the root +`.env`, verify the application with the root file, rotate any live or +previously shared credentials, and only then retire the legacy copy. + +Replace every placeholder before exposing the service. The real `.env` is +ignored by Git and Docker and must stay local. Never commit it or bake it into +an image. + +If a populated environment file or image has ever been shared, assume its +values are exposed. Rotate the Discord token, database password, JWT secret, +bot service key, and model-provider key before deployment. Updating `.gitignore` +does not remove secrets from existing history. + +Important settings: + +| Variable | Purpose | +| --- | --- | +| `DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USER`, `DB_PASS` | PostgreSQL connection | +| `JWT_SECRET` | Signs short-lived user sessions | +| `BOT_API_KEY` | Discord adapter service key; use at least 32 random characters | +| `BOT_API_KEY_SCOPES` | Normally `discord:session,outbox:claim,outbox:deliver` | +| `DISCORD_BOT_TOKEN` | Discord application token | +| `DISCORD_ENROLLMENT_MODE` | `allowlist` by default, or explicit `open` enrollment | +| `DISCORD_ALLOWLIST` | Comma- or whitespace-separated Discord user IDs | +| `API_URL` | API URL used by the bot; Compose default is `http://app:5000` | +| `OPENROUTER_API_KEY`, `OPENROUTER_BASE_URL` | OpenAI-compatible provider access | +| `AI_MODEL`, `AI_CONFIG_PATH` | Optional model and parser-config overrides | +| `DEFAULT_TIMEZONE` | IANA timezone for new users; defaults to `UTC` | +| `JOB_*`, `OUTBOX_*` | Worker polling, batch, and lease settings | + +Generate secrets with a password manager or a cryptographic random generator. +The API registers `BOT_API_KEY` as a hashed service key on its first non-live +request; the Discord process must use the same raw value. + +## Run with Docker + +After configuring `.env`, start the complete stack: + +```bash +docker compose up --build +``` + +The API is available at `http://localhost:8080`. The `migrate` one-shot service +must finish before the API, scheduler, and bot start. To work without Discord or +a live model provider, start only the database, migrations, API, and scheduler: + +```bash +docker compose up --build db migrate app scheduler +``` + +Useful checks: + +```bash +curl http://localhost:8080/health/live +curl http://localhost:8080/health/ready +docker compose logs -f app scheduler bot +``` + +`/health/live` only confirms that the API process is responding. +`/health/ready` also checks PostgreSQL. + +## Run locally + +Create a Python 3.11 virtual environment and install development dependencies: + +```powershell +python -m venv .venv +.\.venv\Scripts\Activate.ps1 +python -m pip install -r requirements-dev.txt +``` + +Install or connect to PostgreSQL 16, create the database/user named in `.env`, +and make sure it accepts connections at `DB_HOST` and `DB_PORT`. The example +uses `DB_HOST=127.0.0.1` for local development; Compose overrides that value +inside its containers. On PowerShell, check that the configured port responds: + +```powershell +Test-NetConnection 127.0.0.1 -Port 5432 +``` + +The Windows PostgreSQL installer includes pgAdmin, which can create the `app` +login role and `app` database from the example configuration. If `psql` is on +PATH, the equivalent commands are: + +```powershell +psql -U postgres -c "CREATE ROLE app LOGIN PASSWORD 'replace-with-a-database-password';" +psql -U postgres -c "CREATE DATABASE app OWNER app;" +``` + +Use a different password in both `.env` and the role command. Then run each +process in its own terminal: + +```bash +python -m core.migrations upgrade +python -m api.main +python -m scheduler.daemon +python -m bot.bot +``` + +The local API command uses Flask's development server. The Docker image uses +Gunicorn. + +## Database migrations + +Core migrations live in `config/migrations`; each feature may add numbered SQL +files under `modules//migrations`. Core migrations run first, followed by +feature namespaces in alphabetical and numeric order. + +```bash +python -m core.migrations status +python -m core.migrations upgrade +``` + +Upgrades run transactionally under a PostgreSQL advisory lock. Applied +migrations are recorded with checksums. Do not edit an applied migration: a +changed checksum stops startup. Add the next numbered migration instead. + +Back up an existing database before the first upgrade. Legacy password users +remain supported. To link one safely to Discord after migrating: + +```bash +python -m core.manage link-discord --discord-id 123456789012345678 --username alice +python -m core.manage link-discord --discord-id 123456789012345678 --user-uuid 2d36fc2b-5145-4fca-a677-858ca6a36e2d +``` + +Choose exactly one of `--username` or `--user-uuid`. The command refuses to +move a Discord identity that is already linked to a different user. + +## Authentication + +The HTTP examples below use Compose's `http://localhost:8080`. For a locally +started Flask process, use `http://localhost:5000` instead. + +There are three authentication paths: + +- A password user registers and logs in for a short-lived JWT. +- That JWT can create, list, and revoke user API keys. The full key is returned + only at creation; only its prefix and hash are stored. +- The Discord adapter uses the scoped `BOT_API_KEY` to exchange a stable + Discord ID for a short-lived user JWT. It never receives a user's password. + +Register and log in: + +```bash +curl -X POST http://localhost:8080/api/register \ + -H "Content-Type: application/json" \ + -d '{"username":"alice","password":"replace-this-password","timezone":"America/Chicago"}' + +curl -X POST http://localhost:8080/api/login \ + -H "Content-Type: application/json" \ + -d '{"username":"alice","password":"replace-this-password"}' +``` + +Use the returned JWT as `JWT_TOKEN` to create a long-lived user key: + +```bash +curl -X POST http://localhost:8080/api/keys \ + -H "Authorization: Bearer JWT_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"name":"local-cli"}' +``` + +Save the response's `key` immediately; it is shown once. A user API key acts as +its owner for user routes, but cannot manage API keys or delete the account. + +```bash +curl http://localhost:8080/api/reminders \ + -H "Authorization: Bearer USER_API_KEY" + +curl http://localhost:8080/api/keys \ + -H "Authorization: Bearer JWT_TOKEN" + +curl -X DELETE http://localhost:8080/api/keys/KEY_UUID \ + -H "Authorization: Bearer JWT_TOKEN" +``` + +Revocation takes effect on the next request. Optional `expires_at` values use +ISO-8601 timestamps. + +### Discord enrollment + +Enrollment is closed by default. Add stable Discord user IDs to the allowlist: + +```dotenv +DISCORD_ENROLLMENT_MODE=allowlist +DISCORD_ALLOWLIST=123456789012345678,234567890123456789 +``` + +To let any Discord user create a provider-backed account explicitly set: + +```dotenv +DISCORD_ENROLLMENT_MODE=open +``` + +Display names are metadata only. Existing accounts are never matched +automatically by a mutable Discord display name; use `core.manage link-discord` +when joining a legacy password account. + +## Feature modules + +Every immediate, non-private package under `modules/` must export +`register(registry)`. Packages load alphabetically. Startup fails on malformed +registration, duplicate module names, duplicate command types, or duplicate job +types, so configuration errors surface early. + +A typical package looks like this: + +```text +modules/tasks/ + __init__.py + commands.py + prompts.py + routes.py + service.py + migrations/ + 0001_tasks.sql +``` + +Its `__init__.py` owns registration: + +```python +from modules.tasks.commands import handleTask, validateTask +from modules.tasks.prompts import TASK_PROMPT +from modules.tasks.routes import registerRoutes +from modules.tasks.service import runFollowUp + + +def register(registry): + registry.describe("Create and manage tasks") + registry.register_command( + "task", + handleTask, + prompt=TASK_PROMPT, + validator=validateTask, + description="Create, list, and complete tasks", + help_text=["add buy milk to my tasks", "list my tasks"], + ) + registry.register_routes(registerRoutes) + registry.register_job("tasks.follow_up", runFollowUp) +``` + +`register_routes` and `register_job` are optional. A command handler keeps a +small async interface: + +```python +async def handleTask(context, parsed): + result, status = await context.api.request("get", "/api/tasks") + if status == 200: + await context.reply(f"You have {len(result['tasks'])} tasks.") +``` + +`CommandContext` exposes `user_uuid`, `discord_user_id`, `timezone`, an async +authenticated `api` client, and `await reply(content)`. The API client's +`request(method, endpoint, data=None, params=None)` returns `(JSON body, HTTP +status)` and refreshes the Discord-issued JWT once after a 401. This keeps +feature handlers independent of discord.py message objects. + +The parser first routes against registered command descriptions, then invokes +the selected module's focused prompt and Python validator. Prompt templates may +use `{user_input}`, `{history_context}`, `{current_time}`, and `{timezone}`; +literal JSON braces are preserved. Validators return a list of human-readable +errors, allowing the parser to retry malformed model output. + +### Extension walkthrough + +1. Create a package under `modules/`; no central import list is needed. +2. Add the feature's tables as `migrations/0001_.sql`. Use the next number + for later schema changes. +3. Put business rules and owner-scoped database operations in `service.py`. +4. Add authenticated Flask routes with `api.security.requireUser()` and derive + ownership from `flask.g.user_uuid`, never request JSON. +5. Add a focused prompt, validator, and `async handler(context, parsed)`. +6. Register the command, optional routes, and optional job types in + `register(registry)`. +7. Run migration status, tests, and Ruff before starting the services. + +Use the reminders package as the concrete example for transactions, ownership, +recurrence, jobs, and outbound delivery. + +## Reminders + +In a Discord DM, users can speak naturally: + +```text +remind me tomorrow at 9 AM to call the dentist +remind me every day at 8 AM to take my medication +remind me every Friday at 5 PM to submit my timesheet +list my reminders +cancel reminder +set my timezone to America/Chicago +``` + +The REST API supports the same data: + +```bash +curl -X POST http://localhost:8080/api/reminders \ + -H "Authorization: Bearer USER_TOKEN_OR_KEY" \ + -H "Content-Type: application/json" \ + -d '{"message":"check the oven","run_at":"2099-01-02T18:00:00-06:00","recurrence":{"frequency":"daily","interval":1}}' + +curl http://localhost:8080/api/reminders \ + -H "Authorization: Bearer USER_TOKEN_OR_KEY" + +curl -X DELETE http://localhost:8080/api/reminders/REMINDER_UUID \ + -H "Authorization: Bearer USER_TOKEN_OR_KEY" +``` + +Omit `recurrence` for a one-time reminder. Supported frequencies are `daily` +and `weekly`, with an optional positive `interval`. Times are stored in UTC, +while recurring reminders preserve the user's local wall-clock time across +daylight-saving changes. User timezones must be IANA names such as `UTC` or +`America/Chicago`. + +After downtime, a due occurrence is delivered once by the scheduler's normal +processing path and the next future occurrence is scheduled; missed intervals +are not replayed. Cancelling marks the reminder and its pending job/outbound +records together where possible. + +## Jobs and outbound delivery + +The scheduler atomically claims due jobs with PostgreSQL row locks and +`SKIP LOCKED`. Jobs and outbound messages use leases so another worker can +recover work after a crash. Defaults are a five-second poll, five-minute lease, +three attempts, and exponential retries beginning at 30 seconds and capped at +15 minutes. + +Delivery is **at least once**. Idempotency keys prevent ordinary duplicate job +and outbox creation, but a process can crash after Discord accepts a message and +before the API records success. Consumers and message wording should tolerate a +rare duplicate. + +Discord claims only `discord_dm` outbox records through scoped internal routes, +sends them, and reports `sent` or `retry`. The existing webhook and ntfy helpers +remain available for custom features but are not the reminder delivery path. + +## Tests and lint + +```bash +python -m pip install -r requirements-dev.txt +ruff check . +pytest +``` + +Local pytest reports coverage without failing when PostgreSQL-dependent tests +are skipped. CI enforces at least 80% coverage across runtime framework code +with those integration tests enabled; only the thin administrative CLI wrappers +are excluded from that metric. Tests mock Discord and model-provider calls; +GitHub Actions runs both the Python 3.11 suite against PostgreSQL 16 and a +Compose startup smoke test. + +## License + +MIT