Operator guide for Valet Dedicated 0.10.1
Troubleshoot
This page covers the failures an operator meets most: a check that fails, an upgrade that fails, a sign-in that returns 401, a certificate that stays pending, a full certificate quota, a stuck upgrade lock, and a task you must open by hand. Each section names the message you see, the cause, and the fix. If a fix does not work, build a support bundle.
Commands use $VD, the valet-dedicated binary of the running release, as
in Install. Replace <name> with the module's name input.
A check fails
check prints one line per item. A cross line names the problem and its fix.
Apply the fix, then run check again.
$VD checkA standby pool that is still booting passes after a wait. Use
$VD check --wait 15m to rerun every 15 seconds until every item passes or
the time runs out.
These are the cross lines you meet most:
installation has no successful upgrade yet
- Cause and fix
- The installation has never finished an upgrade. Run
$VD upgrade <tag>.
module <x> is older than the running release <y>
- Cause and fix
- You applied an older module than the running release. Apply the module at
<y>, then upgrade.
secret <name> is empty or has no value
- Cause and fix
upgradefills secrets. Run$VD upgrade <tag>.
service <name>: desired N and running M
- Cause and fix
- A task failed to start or stopped. Read the service's events:
aws ecs describe-services --cluster <name>-platform --services <name>-<service>. If the events show nothing, runupgradeagain.
runs a ... revision, not an upgrade revision
- Cause and fix
- Terraform changed a task definition and
upgradehas not replaced it. Run$VD upgrade <tag>.
GET <url> returned 503
- Cause and fix
- The service is up but unhealthy. Read its log group
/ecs/<name>/<service>.
GET <url> failed
- Cause and fix
- The load balancer or DNS is down, or your machine has no route to the VPC. See Connect your people.
certificates: <host> is pending
- Cause and fix
- See A certificate stays pending.
certificates: <host> is issued but not attached
- Cause and fix
- See The certificate quota.
impersonation is on
- Cause and fix
- The running dashboard still lets Valet staff start sessions. Re-apply the module, then run
$VD upgrade <tag>. See How WorkOS works.
setting inference.anthropic_api_key is not set
- Cause and fix
- Set the key. See Reference.
standby pool holds N of M
- Cause and fix
- The reconciler is still starting runtimes. Wait. If the count does not rise, read the api service's log.
A line saying a call needs a permission, such as grant the deployment role ecs:DescribeServices
- Cause and fix
- The deployment role lacks the permission. Re-apply the module.
check exits 1 when an item fails and 2 when it cannot run: a bad flag, no
usable AWS credentials, or an unreadable descriptor. Use the AWS credentials
of the installation's account and region.
An upgrade fails
upgrade writes a deployment record whether it succeeds or fails, and
prints the record's key. Read the error first. A rerun at the same tag
changes nothing it has already done, so fix the cause and run it again.
$VD upgrade "$TO"another upgrade holds the lock: <owner>, until <time>. Another operator, or an earlier run that you stopped, holds the lock. See A stuck upgrade lock.migrate task <arn> is still running. A migration from an earlier upgrade has not stopped. Wait for it to finish, then runupgradeagain. If it never stops, stop the task in ECS, then runupgradeagain.migrate failed,settings bootstrap failed,service rollout failed, orhealth check failed. The step named in the error failed. Read the log group of the task that ran it:/ecs/<name>/migratefor a migration, and/ecs/<name>/<service>for a service. Fix the cause, then runupgradeagain.upgrade.unfinished. Migration started but the upgrade did not finish. Retry the named version, or choose a newer release that lists it as a predecessor. Failures before migration impose no new target restriction. Each upgrade takes its own snapshot,<name>-pre-<version>with hyphens for dots. The snapshot from before the failed attempt holds the database as it was before any migration ran. Restore from that one, not from the newer target's, which holds the partly migrated database. See Restore before a failed upgrade.snapshot <id> is failed. Open the AWS RDS console in the installation's account and region. Under Snapshots, verify that the named snapshot belongs to the installation database and has statusfailed. Delete that snapshot using credentials withrds:DeleteDBSnapshot, then rerun the upgrade. The deployment role cannot delete snapshots. Upgrade never deletes one automatically.- An error with an AWS permission in it. Your credentials, or the deployment role, lack the action it names. Re-apply the module to restore the role. Use credentials from the installation's account.
- Any other error. It names the step that failed. Fix what it names and
run
upgradeagain at the same tag.
After a successful rerun, run $VD check --wait 15m.
An upgrade only moves forward. If a release misbehaves, ask for help. Valet fixes it in a newer release.
Sign-in returns 401
- Every request returns 401 right after a successful sign-in. The installation accepts a sign-in token only when it names your WorkOS environment. This is a setting of the environment, which Valet administers. Ask for help and say that every signed-in request returns 401.
- One person cannot sign in. Your WorkOS connection admits only people your identity provider admits. Check that the person can sign in to your identity provider, and that their email domain is verified in the Admin Portal. Until a domain is verified, nobody on it can sign in. Ask Valet for a new Admin Portal link if yours has expired.
check does not read WorkOS, so it passes when sign-in is broken. See
How WorkOS works.
A certificate stays pending
Each org gets a wildcard certificate, *.<org>.<apex>. The installation's
domain worker requests it from AWS Certificate Manager (ACM). ACM
validates it through the public Route 53 zone for the apex. check reports certificates: <host> is pending until ACM issues it.
A certificate pends for a few minutes. If it stays pending:
Confirm that the public hosted zone for the apex exists in the installation's account, and that the parent domain delegates to it with the zone's name servers. ACM cannot validate until the delegation works. See Prepare.
List the certificate in ACM and read its status and validation records:
aws acm list-certificates --region <region> \ --query "CertificateSummaryList[?contains(DomainName, '<org>')]"Read the worker's log group
/ecs/<name>/workerfor the textdomain provision. It names the step that stalled.
If the certificate is ISSUED but check still says it is not attached,
see the next section.
The certificate quota
A listener holds 25 certificates by default, and the apex takes one, so an
installation serves about 24 orgs. When the listener is full, ACM issues an
org's certificate, but the installation cannot attach it. check reports
certificates: <host> is issued but not attached and the org's hosts do not
answer over HTTPS.
Read the listener's ARN from the descriptor, then count its certificates:
aws ssm get-parameter --name /valet-dedicated/descriptor \ --query Parameter.Value --output text aws elbv2 describe-listener-certificates --listener-arn <listener_arn> \ --query 'length(Certificates)'The descriptor's
listener_arnfield holds the ARN. A count of 25 means the listener is full.In the AWS console, open Service Quotas, then Elastic Load Balancing, and request an increase of Certificates per Application Load Balancer. The request takes days. Make it before you near the limit.
When AWS raises the quota, the domain worker retries the attach. Run
checkagain.
A stuck upgrade lock
upgrade holds a lease on the installation so two operators cannot upgrade
at once. The lease is an object in the records bucket, locks/upgrade.json.
The running upgrade renews it every 30 seconds, and it expires 2 minutes
after the last renewal.
If upgrade says another upgrade holds the lock, it names the holder and
the time the lease ends.
- Ask the named holder whether an upgrade is running. If so, wait for it.
- If nobody is upgrading, wait until the time the message names. A lease that no one renews expires.
- Run
upgradeagain. It takes over an expired lease and printsTook over a stale lock from <owner>.
Do not delete the lock object while an upgrade runs. It is the only guard against two upgrades at once.
Break-glass: open a task with ECS Exec
Use ECS Exec to run a command inside a running task when logs and check
do not explain a failure. The module turns on ECS Exec for every platform
service. Sessions are logged to /ecs/<name>/platform/exec, and to
/ecs/<name>/workloads/exec for the workload cluster.
You need the AWS CLI, the Session Manager plugin, and an IAM permission to
call ecs:ExecuteCommand in the installation's account. Replace <service>
with api, worker, inference, dashboard, or build.
task=$(aws ecs list-tasks --cluster <name>-platform \
--service-name <name>-<service> --query 'taskArns[0]' --output text)
aws ecs execute-command --cluster <name>-platform --task "$task" \
--container <service> --interactive --command "/bin/sh"Treat a session as a change to production:
- Read; do not edit. A change you make lives only until the task is replaced.
- Do not print environment variables or secret values. The session log records what you type and what the shell prints.
- Write down why you opened the session. Your auditors can read the exec log group.
If execute-command fails, check that the task is running, that the Session
Manager plugin is installed, and that your role may call ecs:ExecuteCommand.
Ask for help
Build a support bundle and send it to Valet. The command uploads nothing; you review the directory first. See Get help.
$VD support bundle --out ./valet-support