Operator guide for Valet Dedicated 0.10.1

Restore

A restore puts the installation's data back as it was at an earlier time, the restore point. It replaces the database, and optionally the file system that holds every source repository, with copies from that time. Everything written after the restore point is lost.

Database only

Data loss (RPO)
About 5 minutes
Downtime (RTO)
About 45 minutes for a small database; about 10 minutes of downtime

Database and file system

Data loss (RPO)
Database about 5 minutes; file system up to 24 hours
Downtime (RTO)
About 45 minutes for a small database; about 10 minutes of downtime

RDS ships its logs about every 5 minutes, so the database loses about 5 minutes of writes. The module's backup plan takes a file system recovery point daily at 05:00 UTC, so the file system loses up to a day. Take an on-demand backup before risky work to bring that near zero. The copy time grows with database size. The installation keeps serving while the copy runs; only steps 4 to 8 stop it. A drill on a sandbox on 14 November 2026 measured these numbers.

The procedure keeps every resource name. The restored database takes the original instance's identifier, so its address, the database_url secret, the backup plan, and Terraform keep working without a change. The restored file system tree takes the original directory's place on the same file system.

When to restore

Restore when data is lost or damaged and nothing else brings it back:

  • Someone destroyed an agent, a site, an app, or an org by mistake.
  • A migration failed partway and left the database unusable. See Restore before a failed upgrade.
  • The database or the file system is damaged.

Do not restore to undo a bad release. An upgrade only moves forward; Valet fixes a bad release in a newer one. Do not restore to undo one person's change that they can make again, because a restore also undoes every other change since the restore point.

What a restore covers

Database

Where it lives
RDS instance <name>-db
How far back
Any second in the last 7 days; each <name>-pre-<version> snapshot; each daily AWS Backup copy
Restored by
This page

App database

Where it lives
RDS instance <name>-appdb, when enabled
How far back
The same as the database
Restored by
This page, with DB=<name>-appdb

Source repositories

Where it lives
EFS file system <name>-code-storage
How far back
Each daily AWS Backup copy, taken at 05:00 UTC
Restored by
This page

Site, app, and agent bundles; uploads

Where it lives
S3 buckets <name>-<account>-<region>-releases and -uploads
How far back
Every object version; both buckets are versioned
Restored by
This page brings back objects deleted after the restore point

Upgrade records and lock

Where it lives
S3 bucket -records
How far back
Not restored
Restored by
Never roll it back: it records what the installation runs

Catalog

Where it lives
S3 bucket -catalog
How far back
Not restored
Restored by
upgrade loads it from the release

Secrets

Where it lives
Secrets Manager <name>/*
How far back
Not restored
Restored by
The key ring check

Audit log and service logs

Where it lives
CloudWatch Logs
How far back
Not restored
Restored by
They keep every event, including those after the restore point

AWS Backup copies exist only when the module's protection and backup.enabled are both true, the default. RDS keeps 7 days of point-in-time history when protection is true, and 1 day otherwise.

Choose what to restore

Restore the database alone when only database rows are wrong and no agent, site, app, or skill was destroyed after the restore point. You can pick any second in the last 7 days. The file system keeps every repository's newest source. The worker then rebuilds any agent whose source is newer than its restored release, so agent source changes made after the restore point come back.

Restore the database and the file system together when something was destroyed or the file system is damaged. Choose a file system recovery point, and restore the database to that recovery point's creation time. The two then agree on every repository.

Restore the app database with the database when apps use it. Restore both to the same time, because the database records which app databases and roles exist.

The restore point must be after the last successful upgrade. $VD status prints it as Release: <version> (upgraded <time>). An older restore point holds a database that the running release cannot use. The one exception is a failed upgrade, below. For any other older point, ask Valet.

Restore before a failed upgrade

When an upgrade fails with upgrade.unfinished, restore the database from the snapshot <name>-pre-<version> of the failed attempt. Use the procedure below with a snapshot source in step 2. In step 8, run $VD upgrade at the failed release, not at the running one. That upgrade migrates the restored database again.

Before you start

Run every command from a machine that reaches the installation, as in Install, with AWS credentials of an administrator in the installation's account. The steps write to RDS, EFS, ECS, S3, and Secrets Manager. You need the AWS CLI and jq.

  1. Set these values in one shell. Every command on this page uses them.

    NAME=<name>
    REGION=<region>
    ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
    export AWS_REGION=$REGION
    VD=./vendor/valet-dedicated-<running release>
    TAG=<running release>
    VAULT=$NAME
    BACKUP_ROLE=arn:aws:iam::$ACCOUNT:role/$NAME-backup
    DB=$NAME-db

    TAG is the version on the first line of $VD check.

  2. Write down the time you start. You measure the downtime from it.

  3. Run $VD status, and keep its output. It names the running release, each service's count, and the last upgrade's time.

  4. Tell your people that the installation goes down, and that changes after the restore point will be lost.

1. Check the key ring

The encryption_keys secret decrypts the credentials the database stores. A restore does not touch it. The restored rows open only if the current key ring still holds every key that sealed them.

  1. List the secret's versions.

    aws secretsmanager list-secret-version-ids \
      --secret-id "$NAME/encryption_keys" --include-deprecated \
      --query 'Versions[].[CreatedDate,VersionId,join(`,`,VersionStages || `[]`)]' \
      --output text | sort

    If every version was created before the restore point, the check passes. Go to step 2.

  2. Otherwise, set OLD to the newest version created before the restore point. Compare its keys with the current ring.

    OLD=<version id>
    old=$(aws secretsmanager get-secret-value --secret-id "$NAME/encryption_keys" \
      --version-id "$OLD" --query SecretString --output text)
    cur=$(aws secretsmanager get-secret-value --secret-id "$NAME/encryption_keys" \
      --query SecretString --output text)
    jq -n --argjson old "$old" --argjson cur "$cur" \
      '[$old.keys | to_entries[] | select($cur.keys[.key] != .value) | .key]'

    It prints only key versions, never keys. [] means the current ring holds every key, and the check passes.

  3. If it prints a version, a key was retired after the restore point. Add the missing keys back, and keep current as it is.

    jq -n --argjson old "$old" --argjson cur "$cur" \
      '$cur | .keys = ($old.keys + $cur.keys)' \
    | aws secretsmanager put-secret-value --secret-id "$NAME/encryption_keys" \
      --secret-string file:///dev/stdin

    The services read the ring when they start in step 8. Run the cur= and jq commands of step 2 again; they print [].

If Secrets Manager no longer has the old version, use the copy you made when you backed up the key ring. If you have neither, the credentials sealed with the missing key cannot be decrypted. Stop and ask Valet before you restore.

2. Start the database restore

This step creates a new instance, $DB-restore. It changes nothing the installation uses.

  1. Read the settings the copy must match.

    read -r CLASS SUBNETS PARAMS SG RETENTION PROTECT COPYTAGS ADDRESS <<<"$(
      aws rds describe-db-instances --db-instance-identifier "$DB" \
        --query 'DBInstances[0].[DBInstanceClass,DBSubnetGroup.DBSubnetGroupName,DBParameterGroups[0].DBParameterGroupName,VpcSecurityGroups[0].VpcSecurityGroupId,BackupRetentionPeriod,DeletionProtection,CopyTagsToSnapshot,Endpoint.Address]' \
        --output text)"
    flag() { if [ "$1" = True ]; then echo "--$2"; else echo "--no-$2"; fi; }
    echo "$ADDRESS"

    It prints the database's address. Keep it for step 5.

  2. Start the restore from one of three sources.

    • A point in time. Set T to the restore point in UTC, such as 2026-11-14T17:00:00Z. It must be within the history RDS keeps.

      T=<restore point>
      aws rds restore-db-instance-to-point-in-time \
        --source-db-instance-identifier "$DB" \
        --target-db-instance-identifier "$DB-restore" \
        --restore-time "$T" \
        --db-instance-class "$CLASS" --db-subnet-group-name "$SUBNETS" \
        --db-parameter-group-name "$PARAMS" --vpc-security-group-ids "$SG" \
        --backup-retention-period "$RETENTION" --no-multi-az --no-publicly-accessible \
        "$(flag "$PROTECT" deletion-protection)" "$(flag "$COPYTAGS" copy-tags-to-snapshot)" \
        --tags Key=Name,Value="$DB" \
        --query 'DBInstance.DBInstanceStatus' --output text
    • A snapshot, such as <name>-pre-<version>, or an AWS Backup copy. List them, and set T to the snapshot's creation time.

      aws rds describe-db-snapshots --db-instance-identifier "$DB" \
        --query 'DBSnapshots[].[SnapshotCreateTime,SnapshotType,DBSnapshotArn]' \
        --output text | sort
      SNAPSHOT=<snapshot ARN>
      T=<its creation time>
      aws rds restore-db-instance-from-db-snapshot \
        --db-snapshot-identifier "$SNAPSHOT" \
        --db-instance-identifier "$DB-restore" \
        --db-instance-class "$CLASS" --db-subnet-group-name "$SUBNETS" \
        --db-parameter-group-name "$PARAMS" --vpc-security-group-ids "$SG" \
        --backup-retention-period "$RETENTION" --no-multi-az --no-publicly-accessible \
        "$(flag "$PROTECT" deletion-protection)" "$(flag "$COPYTAGS" copy-tags-to-snapshot)" \
        --tags Key=Name,Value="$DB" \
        --query 'DBInstance.DBInstanceStatus' --output text
    • The time of a file system recovery point, when you restore both. Do step 3 first, then use the point-in-time command with T set to the recovery point's creation time.

    It prints creating.

  3. Wait for the copy.

    aws rds wait db-instance-available --db-instance-identifier "$DB-restore"

    It prints nothing when the copy is ready. It gives up after 30 minutes; run it again until it returns. A copy takes about as long as a snapshot of the same size.

The services still run. Users can work, but every change they make from now on is lost.

3. Start the file system restore

Skip this step when you restore the database alone.

This step restores the repositories into a new directory, /aws-backup-restore_<time>/code-storage-root, on the same file system. The installation does not read it yet.

  1. Find the file system and its recovery points.

    FS=$(aws efs describe-file-systems \
      --query "FileSystems[?Name=='$NAME-code-storage'].FileSystemId" --output text)
    aws backup list-recovery-points-by-resource \
      --resource-arn "arn:aws:elasticfilesystem:$REGION:$ACCOUNT:file-system/$FS" \
      --query 'RecoveryPoints[].[CreationDate,Status,RecoveryPointArn]' --output text | sort

    Pick a COMPLETED recovery point. Set T to its creation time, and restore the database to that time in step 2.

    RP=<recovery point ARN>
    T=<its creation time>
  2. Write the restore's settings to a file, and start the restore.

    aws backup get-recovery-point-restore-metadata --backup-vault-name "$VAULT" \
      --recovery-point-arn "$RP" --query RestoreMetadata --output json \
    | jq --arg fs "$FS" '. + {"file-system-id": $fs, "newFileSystem": "false",
        "ItemsToRestore": "[\"/code-storage-root\"]"}' >efs-restore.json
    JOB=$(aws backup start-restore-job --recovery-point-arn "$RP" \
      --iam-role-arn "$BACKUP_ROLE" --resource-type EFS \
      --metadata file://efs-restore.json --query RestoreJobId --output text)
  3. Wait for the restore to finish.

    until s=$(aws backup describe-restore-job --restore-job-id "$JOB" \
        --query Status --output text) && [ "$s" != PENDING ] && [ "$s" != RUNNING ]; do
      sleep 30
    done; echo "$s"

    It prints COMPLETED. Any other status stops the restore. Read the job's StatusMessage with aws backup describe-restore-job.

  4. Register a task that mounts the file system's root. Step 6 and the undo use it to move directories. It runs the api service's image and role.

    API_TD=$(aws ecs describe-services --cluster "$NAME-platform" \
      --services "$NAME-api" --query 'services[0].taskDefinition' --output text)
    aws ecs describe-task-definition --task-definition "$API_TD" \
      --query taskDefinition --output json \
    | jq --arg family "$NAME-efs-restore" '{family: $family, taskRoleArn,
        executionRoleArn, networkMode, requiresCompatibilities, cpu, memory,
        runtimePlatform,
        containerDefinitions: [.containerDefinitions[0] | {name, image,
          essential: true, logConfiguration,
          mountPoints: [{sourceVolume: "code-storage", containerPath: "/mnt/efs"}]}],
        volumes: [.volumes[] | select(.name == "code-storage")
          | del(.efsVolumeConfiguration.authorizationConfig.accessPointId)]}' \
      >efs-restore-task.json
    aws ecs register-task-definition --cli-input-json file://efs-restore-task.json \
      --query taskDefinition.taskDefinitionArn --output text

    It prints the task definition's ARN.

  5. Define the function that runs a shell script in that task. It prints the script's output and its exit code.

    efs_task() {
      net=$(aws ecs describe-services --cluster "$NAME-platform" \
        --services "$NAME-api" --query 'services[0].networkConfiguration' --output json)
      task=$(aws ecs run-task --cluster "$NAME-platform" --launch-type FARGATE \
        --platform-version 1.4.0 --task-definition "$NAME-efs-restore" \
        --network-configuration "$net" \
        --overrides "$(jq -n --arg s "$1" '{containerOverrides: [{name: "api", command: ["sh", "-c", $s]}]}')" \
        --query 'tasks[0].taskArn' --output text) || return
      aws ecs wait tasks-stopped --cluster "$NAME-platform" --tasks "$task"
      aws logs get-log-events --log-group-name "/ecs/$NAME/api" \
        --log-stream-name "api/api/${task##*/}" --query 'events[].message' --output text
      aws ecs describe-tasks --cluster "$NAME-platform" --tasks "$task" \
        --query 'tasks[0].containers[0].exitCode' --output text
    }
    efs_task 'ls -la /mnt/efs'

    It lists the root: code-storage-root, one aws-backup-restore_<time> directory, and the exit code 0. If an earlier restore left another aws-backup-restore_ directory, ask Valet before you go on.

4. Stop the services

Destructive: the installation is down from here until step 8.

for svc in api worker inference dashboard build; do
  aws ecs update-service --cluster "$NAME-platform" --service "$NAME-$svc" \
    --desired-count 0 --query service.serviceName --output text
done
aws ecs wait services-stable --cluster "$NAME-platform" \
  --services $(for svc in api worker inference dashboard build; do printf '%s ' "$NAME-$svc"; done)

It prints each service's name, then returns when no task runs. Runtimes and apps keep running, but nothing can reach the database.

5. Swap the database

Destructive: this renames the running database. Keep $DB-old until step 10; it is how you undo the restore.

  1. Rename the original instance to $DB-old.

    NEW=$(aws rds describe-db-instances --db-instance-identifier "$DB-restore" \
      --query 'DBInstances[0].DbiResourceId' --output text)
    aws rds modify-db-instance --db-instance-identifier "$DB" \
      --new-db-instance-identifier "$DB-old" --apply-immediately \
      --query DBInstance.DBInstanceStatus --output text
    until aws rds describe-db-instances --db-instance-identifier "$DB-old" >/dev/null 2>&1; do
      sleep 15
    done
    aws rds wait db-instance-available --db-instance-identifier "$DB-old"
  2. Rename the copy to the original identifier.

    aws rds modify-db-instance --db-instance-identifier "$DB-restore" \
      --new-db-instance-identifier "$DB" --apply-immediately \
      --query DBInstance.DBInstanceStatus --output text
    until [ "$(aws rds describe-db-instances --db-instance-identifier "$DB" \
        --query 'DBInstances[0].DbiResourceId' --output text 2>/dev/null)" = "$NEW" ]; do
      sleep 15
    done
    aws rds wait db-instance-available --db-instance-identifier "$DB"
    aws rds describe-db-instances --db-instance-identifier "$DB" \
      --query 'DBInstances[0].Endpoint.Address' --output text

    It prints the same address as step 2. The database_url secret names that address, so the services find the copy.

If you restore the app database too, repeat steps 2 and 5 with DB=$NAME-appdb and the same T.

6. Swap the file system

Skip this step when you restore the database alone.

Destructive: this moves the repositories the installation uses. The old tree stays on the file system as code-storage-root-before-restore until step 10.

efs_task 'set -eu
cd /mnt/efs
set -- aws-backup-restore_*
[ "$#" -eq 1 ] && [ -d "$1/code-storage-root" ] || { echo "want one restore directory"; exit 1; }
[ ! -e code-storage-root-before-restore ] || { echo "code-storage-root-before-restore exists"; exit 1; }
mv code-storage-root code-storage-root-before-restore
mv "$1/code-storage-root" code-storage-root
rmdir "$1"/aws-backup-lost+found_* "$1" || true
ls -la'

AWS Backup leaves an empty aws-backup-lost+found_<time> directory inside the restore directory. The script removes it and the restore directory. It lists code-storage-root and code-storage-root-before-restore, then the exit code 0. Any other exit code moved nothing; read the message.

7. Bring back deleted objects

The restored database may name bundles and uploads that were deleted after the restore point. Both buckets are versioned, so a deleted object has a delete marker over its last version. Removing the marker brings the object back.

  1. List the markers made after the restore point.

    for purpose in releases uploads; do
      aws s3api list-object-versions --bucket "$NAME-$ACCOUNT-$REGION-$purpose" \
        --output json \
      | jq -r --arg t "$T" '.DeleteMarkers[]?
          | select(.IsLatest and .LastModified[0:19] >= $t[0:19])
          | [.Key, .VersionId] | @tsv' >"undelete-$purpose.tsv"
      wc -l <"undelete-$purpose.tsv"
    done

    It prints how many objects each bucket would get back.

  2. Remove those markers.

    for purpose in releases uploads; do
      while IFS=
    #x27;\t' read -r key version; do aws s3api delete-object --bucket "$NAME-$ACCOUNT-$REGION-$purpose" \ --key "$key" --version-id "$version" >/dev/null done <"undelete-$purpose.tsv" done

    It prints nothing. It deletes only delete markers, never data.

8. Start the services

Run upgrade at the running release. It sets each service's count and waits until every service answers. For a failed upgrade, use the failed release instead.

$VD upgrade "$TAG"
$VD check --wait 15m

You see a check mark on every line. If the standby pool does not refill, replace the runtimes with $VD roll --runtimes --yes.

Write down the time. The downtime is the time since step 4.

9. Verify

  1. Search the logs for data the key ring could not open.

    for svc in api worker; do
      aws logs filter-log-events --log-group-name "/ecs/$NAME/$svc" \
        --start-time $(( ($(date +%s) - 900) * 1000 )) \
        --filter-pattern '"decryption failed"' --query 'length(events)'
    done

    Each prints 0. Any other count means a key is missing; see step 1.

  2. Send an agent one message, and open a site and an app. Each shows its state at the restore point.

  3. Restart each app whose environment changed after the restore point, with valet ps restart --app <app>. A running app keeps the variables it started with until it restarts.

  4. Set again any setting changed after the restore point. $VD settings history <name> lists the changes, but the restored database has only those before the restore point.

10. Remove the old copies

Destructive: after this step you cannot undo the restore. Wait until your people confirm the installation works.

  1. Delete the old database. It keeps a final snapshot, $DB-old-final, and its automated backups.

    aws rds modify-db-instance --db-instance-identifier "$DB-old" \
      --no-deletion-protection --apply-immediately \
      --query DBInstance.DBInstanceStatus --output text
    aws rds delete-db-instance --db-instance-identifier "$DB-old" \
      --final-db-snapshot-identifier "$DB-old-final" --no-delete-automated-backups \
      --query DBInstance.DBInstanceStatus --output text
    aws rds wait db-instance-deleted --db-instance-identifier "$DB-old"
  2. Delete the old repositories and any leftover restore directory.

    efs_task 'rm -rf /mnt/efs/code-storage-root-before-restore /mnt/efs/aws-backup-restore_* && ls -la /mnt/efs'
  3. Remove the helper task and the local files.

    aws ecs deregister-task-definition --task-definition "$(aws ecs describe-task-definition \
      --task-definition "$NAME-efs-restore" --query taskDefinition.taskDefinitionArn --output text)" \
      --query taskDefinition.status --output text
    rm -f efs-restore.json efs-restore-task.json undelete-*.tsv
  4. Run terraform plan. It must show no change to the database or the file system. If it does, stop and ask Valet.

Terraform during a restore

Do not run terraform plan or apply from step 5 until step 10 deletes $DB-old. Terraform tracks the original instance by its resource ID, not its name. While $DB-old exists, an apply would rename it back and fail. Once it is gone, Terraform finds the restored instance by its identifier and adopts it. No import or state change is needed.

Undo a restore

Before step 10, you can put the original data back.

Destructive: the installation is down until the last command.

  1. Stop the services, as in step 4.

  2. Swap the database back.

    aws rds modify-db-instance --db-instance-identifier "$DB" \
      --new-db-instance-identifier "$DB-failed" --apply-immediately \
      --query DBInstance.DBInstanceStatus --output text
    until aws rds describe-db-instances --db-instance-identifier "$DB-failed" >/dev/null 2>&1; do
      sleep 15
    done
    aws rds modify-db-instance --db-instance-identifier "$DB-old" \
      --new-db-instance-identifier "$DB" --apply-immediately \
      --query DBInstance.DBInstanceStatus --output text
    until aws rds describe-db-instances --db-instance-identifier "$DB" >/dev/null 2>&1; do
      sleep 15
    done
    aws rds wait db-instance-available --db-instance-identifier "$DB"
  3. Swap the file system back, if you swapped it.

    efs_task 'set -eu
    cd /mnt/efs
    mv code-storage-root code-storage-root-failed
    mv code-storage-root-before-restore code-storage-root
    ls -la'
  4. Start the services, as in step 8.

The undo does not delete objects that step 7 brought back. They are harmless; nothing names them. Delete $DB-failed and code-storage-root-failed once the installation works, as in step 10.