Troubleshooting

The failures worth recognising by name, and what each one actually means.

Start here in every case:

docker compose psdocker compose logs --tail 200 servercurl -i https://api.example.com/health

The server will not start

It exits naming a variable. Configuration is validated at boot, so a missing or too-short value fails immediately rather than on a Player's first Message. BETTER_AUTH_SECRET needs 32 characters, SIGNUP_INVITE_CODE needs 16, and BETTER_AUTH_URL and CORS_ORIGIN must parse as URLs.

Messages are refused: "has no API key or address configured". The server boots without any Provider key, because a Village may run only local models, but a Message needs a Provider that can answer it. Set ANTHROPIC_API_KEY or OPENAI_API_KEY, or add a local Provider in Settings → Providers.

It complains about VALKEY_URL. Valkey is required, not optional. If you have just upgraded, see Upgrading.

It complains about an unknown LOG_DRAIN adapter. The error lists every accepted name. See Logs and observability.

It complains about TRUSTED_PROXIES. The value must be true, false, or a comma-separated list of addresses and CIDRs. Anything else throws, rather than silently widening back to trusting everyone.

It restarts over and over after an upgrade. Migrations run from the container's command before the server listens, and a failed migration crashloops the container deliberately — a server on a stale schema is worse than a server that is down. Read docker compose logs server for the migration error.

/health returns 503 DATABASE_UNREACHABLE

The server is up and MySQL is not answering. The endpoint runs SELECT 1, so this is a real connection failure rather than a slow query.

Check, in order:

1. docker compose ps mysql — is it running and healthy?
2. docker compose logs mysql — MySQL takes some seconds to become healthy after a start, and reports its own failures clearly.
3. MYSQL_PASSWORD in .env against what MySQL actually believes. Changing the variable without changing the account locks the server out of its own database. See Rotating secrets.

Note that GET / still answers OK in this state, because it touches nothing. That is why it is not the health check.

Registration fails with LIMITER_UNAVAILABLE

A 503 on sign-up, saying registration is temporarily unavailable. It means the server could not reach Valkey.

Everywhere else in the API, a Valkey outage makes rate limiting fail open: a brief spell of uncounted requests is much better than 500-ing every route at once. Registration is the one exception. The limit in front of the Invite Code is what keeps a shared secret from being guessed at HTTP speed, so with no limiter the route refuses rather than admitting anyone.

Fix the cause — docker compose ps valkey, then docker compose logs valkey — and registration returns on its own. Nothing needs restarting.

Registration fails with INVITE_REQUIRED

A 403. The code presented did not match SIGNUP_INVITE_CODE, or none was sent. Confirm the value in .env matches what you handed out, remembering that the server compares against the value it booted with — an edit without docker compose up -d server changes nothing.

Sign-up is limited to 5 attempts per minute, so a run of failures while someone pastes the code around will start being refused.

Replies arrive all at once at the end

The edge is buffering or compressing the API origin. Nothing errors and nothing logs it; the transcript simply stops feeling live.

Measure it rather than guessing:

pnpm sse:check https://api.example.compnpm sse:check http://127.0.0.1:8080

If the first fails and the second passes, the problem is your proxy. The usual causes are gzip on applied to the API server block, buffering left on, or a read timeout short enough to cut long generations off mid-sentence. See Reverse proxy.

Nobody can sign in

Both origins must be HTTPS. The session cookie is SameSite=None; Secure, and a browser will not store a Secure cookie from a plain-HTTP origin, so a Village served over HTTP cannot sign anyone in — on a LAN included.

Check also that CORS_ORIGIN is the web app's public origin and BETTER_AUTH_URL is the API's.

Everything points at localhost in production

PUBLIC_APP_HOST, PUBLIC_API_HOST and PUBLIC_API_URL fall back to localhost defaults when unset, because Compose interpolates the whole file even for pnpm db:start. Nothing fails loudly if you forget them: Caddy simply never gets a certificate, and the browser bundle sends API requests to localhost.

docker compose config | grep -E "PUBLIC_|VITE_"

PUBLIC_API_URL is baked into the browser bundle at build time, so fixing it needs docker compose up -d --build web, not a restart.

One caller is consuming everyone's rate limit

Rate limits are keyed on the client address, which the server takes from X-Forwarded-For according to TRUSTED_PROXIES.

• If the header is not being set at all, every caller shares one bucket and one busy Player locks out the rest. Check that your proxy sets it.
• If TRUSTED_PROXIES is left at true while the application port is reachable by anything other than your own proxy, the opposite happens: a caller can present a new address per request and walk straight through every limit, including the one protecting the Invite Code. Narrow it to the addresses your proxy actually uses.

Sandbox Tools say the feature is not enabled

The API server treats Sandboxes as available only when both SANDBOX_BROKER_URL and SANDBOX_BROKER_TOKEN are non-empty. That is the normal state of a Village that never enabled them, and the Sandbox pane says so rather than showing a broken machine.

If you did enable them and still see this, check that both variables are set and that you started with --profile sandbox. If the broker itself will not start, the token is shorter than 16 characters — it refuses to run unauthenticated, because it can create Docker containers on the host.

pnpm sandbox:check asserts the isolation properties from outside, and is the right thing to run after any change to the Sandbox image or the driver.

A Connector or MCP Server says its credential cannot be decrypted

BETTER_AUTH_SECRET changed. MCP Server headers and Connector tokens are encrypted with a key derived from it, and there is no recovery — the credential has to be entered again by the Player who owns it. See Rotating secrets.

A Connector says it needs an Operator

No OAuth client is configured for that service. AgentsVille cannot ship credentials for somebody else's service, so each one reads its own CONNECTOR_<KIND>_CLIENT_ID and CONNECTOR_<KIND>_CLIENT_SECRET pair. Register a client with the provider, set its redirect URI to https://api.example.com/api/connectors/<kind>/callback, and put the pair in .env.

These names are read by convention rather than declared in the schema, so a misspelled one is not caught at boot — it just looks like a Connector nobody configured.

An NPC fails with PROVIDER_NOT_CONFIGURED

That NPC is pointed at a Provider this Village holds no key for. Either set the key and restart the server, or change the NPC's Provider. Models are filtered to configured Providers in the picker, so this usually appears on an NPC created while a key existed.

The database is empty after a restart

docker compose down -v deletes the named volumes, including the database. It is one character away from docker compose down, which is safe, and there is no confirmation. Restore from a backup — and if you have not tested that path yet, see Backup and restore.

Known limits worth knowing about

Migrations are written by hand and have no down step. There is no generated schema and no rollback: reversing a schema change means restoring a dump taken before it ran. Plan upgrades around that.

Next

• Logs and observability
• Configuration
• Reverse proxy