Operator guide for Valet Dedicated 0.10.1
Restore
A restore puts the installation's data back as it was at an earlier time, the restore point. It replaces the database, and optionally the file system that holds every source repository, with copies from that time. Everything written after the restore point is lost.
Database only
- Data loss (RPO)
- About 5 minutes
- Downtime (RTO)
- About 45 minutes for a small database; about 10 minutes of downtime
Database and file system
- Data loss (RPO)
- Database about 5 minutes; file system up to 24 hours
- Downtime (RTO)
- About 45 minutes for a small database; about 10 minutes of downtime
RDS ships its logs about every 5 minutes, so the database loses about 5 minutes of writes. The module's backup plan takes a file system recovery point daily at 05:00 UTC, so the file system loses up to a day. Take an on-demand backup before risky work to bring that near zero. The copy time grows with database size. The installation keeps serving while the copy runs; only steps 4 to 8 stop it. A drill on a sandbox on 14 November 2026 measured these numbers.
The procedure keeps every resource name. The restored database takes the
original instance's identifier, so its address, the database_url
secret, the backup plan, and Terraform keep working without a change.
The restored file system tree takes the original directory's place on the
same file system.
When to restore
Restore when data is lost or damaged and nothing else brings it back:
- Someone destroyed an agent, a site, an app, or an org by mistake.
- A migration failed partway and left the database unusable. See Restore before a failed upgrade.
- The database or the file system is damaged.
Do not restore to undo a bad release. An upgrade only moves forward; Valet fixes a bad release in a newer one. Do not restore to undo one person's change that they can make again, because a restore also undoes every other change since the restore point.
What a restore covers
Database
- Where it lives
- RDS instance
<name>-db - How far back
- Any second in the last 7 days; each
<name>-pre-<version>snapshot; each daily AWS Backup copy - Restored by
- This page
App database
- Where it lives
- RDS instance
<name>-appdb, when enabled - How far back
- The same as the database
- Restored by
- This page, with
DB=<name>-appdb
Source repositories
- Where it lives
- EFS file system
<name>-code-storage - How far back
- Each daily AWS Backup copy, taken at 05:00 UTC
- Restored by
- This page
Site, app, and agent bundles; uploads
- Where it lives
- S3 buckets
<name>-<account>-<region>-releasesand-uploads - How far back
- Every object version; both buckets are versioned
- Restored by
- This page brings back objects deleted after the restore point
Upgrade records and lock
- Where it lives
- S3 bucket
-records - How far back
- Not restored
- Restored by
- Never roll it back: it records what the installation runs
Catalog
- Where it lives
- S3 bucket
-catalog - How far back
- Not restored
- Restored by
upgradeloads it from the release
Secrets
- Where it lives
- Secrets Manager
<name>/* - How far back
- Not restored
- Restored by
- The key ring check
Audit log and service logs
- Where it lives
- CloudWatch Logs
- How far back
- Not restored
- Restored by
- They keep every event, including those after the restore point
AWS Backup copies exist only when the module's protection and
backup.enabled are both true, the default. RDS keeps 7 days of
point-in-time history when protection is true, and 1 day otherwise.
Choose what to restore
Restore the database alone when only database rows are wrong and no agent, site, app, or skill was destroyed after the restore point. You can pick any second in the last 7 days. The file system keeps every repository's newest source. The worker then rebuilds any agent whose source is newer than its restored release, so agent source changes made after the restore point come back.
Restore the database and the file system together when something was destroyed or the file system is damaged. Choose a file system recovery point, and restore the database to that recovery point's creation time. The two then agree on every repository.
Restore the app database with the database when apps use it. Restore both to the same time, because the database records which app databases and roles exist.
The restore point must be after the last successful upgrade. $VD status
prints it as Release: <version> (upgraded <time>). An older restore
point holds a database that the running release cannot use. The one
exception is a failed upgrade, below. For any other older point, ask
Valet.
Restore before a failed upgrade
When an upgrade fails with upgrade.unfinished, restore the database
from the snapshot <name>-pre-<version> of the failed attempt. Use the
procedure below with a snapshot source in step 2. In step 8, run $VD upgrade at the failed release, not at the running one. That upgrade
migrates the restored database again.
Before you start
Run every command from a machine that reaches the installation, as in
Install, with AWS credentials of an
administrator in the installation's account. The steps write to RDS, EFS,
ECS, S3, and Secrets Manager. You need the AWS CLI and jq.
Set these values in one shell. Every command on this page uses them.
NAME=<name> REGION=<region> ACCOUNT=$(aws sts get-caller-identity --query Account --output text) export AWS_REGION=$REGION VD=./vendor/valet-dedicated-<running release> TAG=<running release> VAULT=$NAME BACKUP_ROLE=arn:aws:iam::$ACCOUNT:role/$NAME-backup DB=$NAME-dbTAGis the version on the first line of$VD check.Write down the time you start. You measure the downtime from it.
Run
$VD status, and keep its output. It names the running release, each service's count, and the last upgrade's time.Tell your people that the installation goes down, and that changes after the restore point will be lost.
1. Check the key ring
The encryption_keys secret decrypts the credentials the database
stores. A restore does not touch it. The restored rows open only if the
current key ring still holds every key that sealed them.
List the secret's versions.
aws secretsmanager list-secret-version-ids \ --secret-id "$NAME/encryption_keys" --include-deprecated \ --query 'Versions[].[CreatedDate,VersionId,join(`,`,VersionStages || `[]`)]' \ --output text | sortIf every version was created before the restore point, the check passes. Go to step 2.
Otherwise, set
OLDto the newest version created before the restore point. Compare its keys with the current ring.OLD=<version id> old=$(aws secretsmanager get-secret-value --secret-id "$NAME/encryption_keys" \ --version-id "$OLD" --query SecretString --output text) cur=$(aws secretsmanager get-secret-value --secret-id "$NAME/encryption_keys" \ --query SecretString --output text) jq -n --argjson old "$old" --argjson cur "$cur" \ '[$old.keys | to_entries[] | select($cur.keys[.key] != .value) | .key]'It prints only key versions, never keys.
[]means the current ring holds every key, and the check passes.If it prints a version, a key was retired after the restore point. Add the missing keys back, and keep
currentas it is.jq -n --argjson old "$old" --argjson cur "$cur" \ '$cur | .keys = ($old.keys + $cur.keys)' \ | aws secretsmanager put-secret-value --secret-id "$NAME/encryption_keys" \ --secret-string file:///dev/stdinThe services read the ring when they start in step 8. Run the
cur=andjqcommands of step 2 again; they print[].
If Secrets Manager no longer has the old version, use the copy you made when you backed up the key ring. If you have neither, the credentials sealed with the missing key cannot be decrypted. Stop and ask Valet before you restore.
2. Start the database restore
This step creates a new instance, $DB-restore. It changes nothing the
installation uses.
Read the settings the copy must match.
read -r CLASS SUBNETS PARAMS SG RETENTION PROTECT COPYTAGS ADDRESS <<<"$( aws rds describe-db-instances --db-instance-identifier "$DB" \ --query 'DBInstances[0].[DBInstanceClass,DBSubnetGroup.DBSubnetGroupName,DBParameterGroups[0].DBParameterGroupName,VpcSecurityGroups[0].VpcSecurityGroupId,BackupRetentionPeriod,DeletionProtection,CopyTagsToSnapshot,Endpoint.Address]' \ --output text)" flag() { if [ "$1" = True ]; then echo "--$2"; else echo "--no-$2"; fi; } echo "$ADDRESS"It prints the database's address. Keep it for step 5.
Start the restore from one of three sources.
A point in time. Set
Tto the restore point in UTC, such as2026-11-14T17:00:00Z. It must be within the history RDS keeps.T=<restore point> aws rds restore-db-instance-to-point-in-time \ --source-db-instance-identifier "$DB" \ --target-db-instance-identifier "$DB-restore" \ --restore-time "$T" \ --db-instance-class "$CLASS" --db-subnet-group-name "$SUBNETS" \ --db-parameter-group-name "$PARAMS" --vpc-security-group-ids "$SG" \ --backup-retention-period "$RETENTION" --no-multi-az --no-publicly-accessible \ "$(flag "$PROTECT" deletion-protection)" "$(flag "$COPYTAGS" copy-tags-to-snapshot)" \ --tags Key=Name,Value="$DB" \ --query 'DBInstance.DBInstanceStatus' --output textA snapshot, such as
<name>-pre-<version>, or an AWS Backup copy. List them, and setTto the snapshot's creation time.aws rds describe-db-snapshots --db-instance-identifier "$DB" \ --query 'DBSnapshots[].[SnapshotCreateTime,SnapshotType,DBSnapshotArn]' \ --output text | sort SNAPSHOT=<snapshot ARN> T=<its creation time> aws rds restore-db-instance-from-db-snapshot \ --db-snapshot-identifier "$SNAPSHOT" \ --db-instance-identifier "$DB-restore" \ --db-instance-class "$CLASS" --db-subnet-group-name "$SUBNETS" \ --db-parameter-group-name "$PARAMS" --vpc-security-group-ids "$SG" \ --backup-retention-period "$RETENTION" --no-multi-az --no-publicly-accessible \ "$(flag "$PROTECT" deletion-protection)" "$(flag "$COPYTAGS" copy-tags-to-snapshot)" \ --tags Key=Name,Value="$DB" \ --query 'DBInstance.DBInstanceStatus' --output textThe time of a file system recovery point, when you restore both. Do step 3 first, then use the point-in-time command with
Tset to the recovery point's creation time.
It prints
creating.Wait for the copy.
aws rds wait db-instance-available --db-instance-identifier "$DB-restore"It prints nothing when the copy is ready. It gives up after 30 minutes; run it again until it returns. A copy takes about as long as a snapshot of the same size.
The services still run. Users can work, but every change they make from now on is lost.
3. Start the file system restore
Skip this step when you restore the database alone.
This step restores the repositories into a new directory,
/aws-backup-restore_<time>/code-storage-root, on the same file system.
The installation does not read it yet.
Find the file system and its recovery points.
FS=$(aws efs describe-file-systems \ --query "FileSystems[?Name=='$NAME-code-storage'].FileSystemId" --output text) aws backup list-recovery-points-by-resource \ --resource-arn "arn:aws:elasticfilesystem:$REGION:$ACCOUNT:file-system/$FS" \ --query 'RecoveryPoints[].[CreationDate,Status,RecoveryPointArn]' --output text | sortPick a
COMPLETEDrecovery point. SetTto its creation time, and restore the database to that time in step 2.RP=<recovery point ARN> T=<its creation time>Write the restore's settings to a file, and start the restore.
aws backup get-recovery-point-restore-metadata --backup-vault-name "$VAULT" \ --recovery-point-arn "$RP" --query RestoreMetadata --output json \ | jq --arg fs "$FS" '. + {"file-system-id": $fs, "newFileSystem": "false", "ItemsToRestore": "[\"/code-storage-root\"]"}' >efs-restore.json JOB=$(aws backup start-restore-job --recovery-point-arn "$RP" \ --iam-role-arn "$BACKUP_ROLE" --resource-type EFS \ --metadata file://efs-restore.json --query RestoreJobId --output text)Wait for the restore to finish.
until s=$(aws backup describe-restore-job --restore-job-id "$JOB" \ --query Status --output text) && [ "$s" != PENDING ] && [ "$s" != RUNNING ]; do sleep 30 done; echo "$s"It prints
COMPLETED. Any other status stops the restore. Read the job'sStatusMessagewithaws backup describe-restore-job.Register a task that mounts the file system's root. Step 6 and the undo use it to move directories. It runs the api service's image and role.
API_TD=$(aws ecs describe-services --cluster "$NAME-platform" \ --services "$NAME-api" --query 'services[0].taskDefinition' --output text) aws ecs describe-task-definition --task-definition "$API_TD" \ --query taskDefinition --output json \ | jq --arg family "$NAME-efs-restore" '{family: $family, taskRoleArn, executionRoleArn, networkMode, requiresCompatibilities, cpu, memory, runtimePlatform, containerDefinitions: [.containerDefinitions[0] | {name, image, essential: true, logConfiguration, mountPoints: [{sourceVolume: "code-storage", containerPath: "/mnt/efs"}]}], volumes: [.volumes[] | select(.name == "code-storage") | del(.efsVolumeConfiguration.authorizationConfig.accessPointId)]}' \ >efs-restore-task.json aws ecs register-task-definition --cli-input-json file://efs-restore-task.json \ --query taskDefinition.taskDefinitionArn --output textIt prints the task definition's ARN.
Define the function that runs a shell script in that task. It prints the script's output and its exit code.
efs_task() { net=$(aws ecs describe-services --cluster "$NAME-platform" \ --services "$NAME-api" --query 'services[0].networkConfiguration' --output json) task=$(aws ecs run-task --cluster "$NAME-platform" --launch-type FARGATE \ --platform-version 1.4.0 --task-definition "$NAME-efs-restore" \ --network-configuration "$net" \ --overrides "$(jq -n --arg s "$1" '{containerOverrides: [{name: "api", command: ["sh", "-c", $s]}]}')" \ --query 'tasks[0].taskArn' --output text) || return aws ecs wait tasks-stopped --cluster "$NAME-platform" --tasks "$task" aws logs get-log-events --log-group-name "/ecs/$NAME/api" \ --log-stream-name "api/api/${task##*/}" --query 'events[].message' --output text aws ecs describe-tasks --cluster "$NAME-platform" --tasks "$task" \ --query 'tasks[0].containers[0].exitCode' --output text } efs_task 'ls -la /mnt/efs'It lists the root:
code-storage-root, oneaws-backup-restore_<time>directory, and the exit code0. If an earlier restore left anotheraws-backup-restore_directory, ask Valet before you go on.
4. Stop the services
Destructive: the installation is down from here until step 8.
for svc in api worker inference dashboard build; do
aws ecs update-service --cluster "$NAME-platform" --service "$NAME-$svc" \
--desired-count 0 --query service.serviceName --output text
done
aws ecs wait services-stable --cluster "$NAME-platform" \
--services $(for svc in api worker inference dashboard build; do printf '%s ' "$NAME-$svc"; done)It prints each service's name, then returns when no task runs. Runtimes and apps keep running, but nothing can reach the database.
5. Swap the database
Destructive: this renames the running database. Keep $DB-old until
step 10; it is how you undo the restore.
Rename the original instance to
$DB-old.NEW=$(aws rds describe-db-instances --db-instance-identifier "$DB-restore" \ --query 'DBInstances[0].DbiResourceId' --output text) aws rds modify-db-instance --db-instance-identifier "$DB" \ --new-db-instance-identifier "$DB-old" --apply-immediately \ --query DBInstance.DBInstanceStatus --output text until aws rds describe-db-instances --db-instance-identifier "$DB-old" >/dev/null 2>&1; do sleep 15 done aws rds wait db-instance-available --db-instance-identifier "$DB-old"Rename the copy to the original identifier.
aws rds modify-db-instance --db-instance-identifier "$DB-restore" \ --new-db-instance-identifier "$DB" --apply-immediately \ --query DBInstance.DBInstanceStatus --output text until [ "$(aws rds describe-db-instances --db-instance-identifier "$DB" \ --query 'DBInstances[0].DbiResourceId' --output text 2>/dev/null)" = "$NEW" ]; do sleep 15 done aws rds wait db-instance-available --db-instance-identifier "$DB" aws rds describe-db-instances --db-instance-identifier "$DB" \ --query 'DBInstances[0].Endpoint.Address' --output textIt prints the same address as step 2. The
database_urlsecret names that address, so the services find the copy.
If you restore the app database too, repeat steps 2 and 5 with
DB=$NAME-appdb and the same T.
6. Swap the file system
Skip this step when you restore the database alone.
Destructive: this moves the repositories the installation uses. The
old tree stays on the file system as code-storage-root-before-restore
until step 10.
efs_task 'set -eu
cd /mnt/efs
set -- aws-backup-restore_*
[ "$#" -eq 1 ] && [ -d "$1/code-storage-root" ] || { echo "want one restore directory"; exit 1; }
[ ! -e code-storage-root-before-restore ] || { echo "code-storage-root-before-restore exists"; exit 1; }
mv code-storage-root code-storage-root-before-restore
mv "$1/code-storage-root" code-storage-root
rmdir "$1"/aws-backup-lost+found_* "$1" || true
ls -la'AWS Backup leaves an empty aws-backup-lost+found_<time> directory
inside the restore directory. The script removes it and the restore
directory. It lists code-storage-root and
code-storage-root-before-restore, then the exit code 0. Any other exit code moved nothing; read the message.
7. Bring back deleted objects
The restored database may name bundles and uploads that were deleted after the restore point. Both buckets are versioned, so a deleted object has a delete marker over its last version. Removing the marker brings the object back.
List the markers made after the restore point.
for purpose in releases uploads; do aws s3api list-object-versions --bucket "$NAME-$ACCOUNT-$REGION-$purpose" \ --output json \ | jq -r --arg t "$T" '.DeleteMarkers[]? | select(.IsLatest and .LastModified[0:19] >= $t[0:19]) | [.Key, .VersionId] | @tsv' >"undelete-$purpose.tsv" wc -l <"undelete-$purpose.tsv" doneIt prints how many objects each bucket would get back.
Remove those markers.
for purpose in releases uploads; do while IFS=#x27;\t' read -r key version; do aws s3api delete-object --bucket "$NAME-$ACCOUNT-$REGION-$purpose" \ --key "$key" --version-id "$version" >/dev/null done <"undelete-$purpose.tsv" doneIt prints nothing. It deletes only delete markers, never data.
8. Start the services
Run upgrade at the running release. It sets each service's count and
waits until every service answers. For a
failed upgrade, use the failed
release instead.
$VD upgrade "$TAG"
$VD check --wait 15mYou see a check mark on every line. If the standby pool does not refill,
replace the runtimes with $VD roll --runtimes --yes.
Write down the time. The downtime is the time since step 4.
9. Verify
Search the logs for data the key ring could not open.
for svc in api worker; do aws logs filter-log-events --log-group-name "/ecs/$NAME/$svc" \ --start-time $(( ($(date +%s) - 900) * 1000 )) \ --filter-pattern '"decryption failed"' --query 'length(events)' doneEach prints
0. Any other count means a key is missing; see step 1.Send an agent one message, and open a site and an app. Each shows its state at the restore point.
Restart each app whose environment changed after the restore point, with
valet ps restart --app <app>. A running app keeps the variables it started with until it restarts.Set again any setting changed after the restore point.
$VD settings history <name>lists the changes, but the restored database has only those before the restore point.
10. Remove the old copies
Destructive: after this step you cannot undo the restore. Wait until your people confirm the installation works.
Delete the old database. It keeps a final snapshot,
$DB-old-final, and its automated backups.aws rds modify-db-instance --db-instance-identifier "$DB-old" \ --no-deletion-protection --apply-immediately \ --query DBInstance.DBInstanceStatus --output text aws rds delete-db-instance --db-instance-identifier "$DB-old" \ --final-db-snapshot-identifier "$DB-old-final" --no-delete-automated-backups \ --query DBInstance.DBInstanceStatus --output text aws rds wait db-instance-deleted --db-instance-identifier "$DB-old"Delete the old repositories and any leftover restore directory.
efs_task 'rm -rf /mnt/efs/code-storage-root-before-restore /mnt/efs/aws-backup-restore_* && ls -la /mnt/efs'Remove the helper task and the local files.
aws ecs deregister-task-definition --task-definition "$(aws ecs describe-task-definition \ --task-definition "$NAME-efs-restore" --query taskDefinition.taskDefinitionArn --output text)" \ --query taskDefinition.status --output text rm -f efs-restore.json efs-restore-task.json undelete-*.tsvRun
terraform plan. It must show no change to the database or the file system. If it does, stop and ask Valet.
Terraform during a restore
Do not run terraform plan or apply from step 5 until step 10 deletes
$DB-old. Terraform tracks the original instance by its resource ID, not
its name. While $DB-old exists, an apply would rename it back and fail.
Once it is gone, Terraform finds the restored instance by its identifier
and adopts it. No import or state change is needed.
Undo a restore
Before step 10, you can put the original data back.
Destructive: the installation is down until the last command.
Stop the services, as in step 4.
Swap the database back.
aws rds modify-db-instance --db-instance-identifier "$DB" \ --new-db-instance-identifier "$DB-failed" --apply-immediately \ --query DBInstance.DBInstanceStatus --output text until aws rds describe-db-instances --db-instance-identifier "$DB-failed" >/dev/null 2>&1; do sleep 15 done aws rds modify-db-instance --db-instance-identifier "$DB-old" \ --new-db-instance-identifier "$DB" --apply-immediately \ --query DBInstance.DBInstanceStatus --output text until aws rds describe-db-instances --db-instance-identifier "$DB" >/dev/null 2>&1; do sleep 15 done aws rds wait db-instance-available --db-instance-identifier "$DB"Swap the file system back, if you swapped it.
efs_task 'set -eu cd /mnt/efs mv code-storage-root code-storage-root-failed mv code-storage-root-before-restore code-storage-root ls -la'Start the services, as in step 8.
The undo does not delete objects that step 7 brought back. They are
harmless; nothing names them. Delete $DB-failed and
code-storage-root-failed once the installation works, as in
step 10.