Operator guide for Valet Dedicated 0.10.1
Observe
This page says what the installation watches, where its logs go, and how long it keeps them. Alarms, access logs, and telemetry export are off until you set their inputs. Log retention is on.
Alarms
Set alerts_topic_arn to an SNS topic in the installation's region. The
module then creates CloudWatch alarms and notifies the topic when each
goes to ALARM and back to OK. Alarm names start with the installation
name. Tune the thresholds with the alarms input; see the
reference.
<name>-alb-target-5xx
- Fires when
- Targets return more than 10 5xx responses in 5 minutes
<name>-alb-elb-5xx
- Fires when
- The load balancer returns more than 10 5xx responses of its own in 5 minutes
<name>-api-unhealthy-hosts, <name>-dashboard-unhealthy-hosts
- Fires when
- A target group has had an unhealthy target for 5 minutes
<name>-<service>-running
- Fires when
- The service reports no running task for 5 minutes. One alarm each for api, worker, inference, dashboard, and build
<name>-db-cpu
- Fires when
- The database's CPU averages above 80% for 15 minutes
<name>-db-storage
- Fires when
- Free storage falls below 10% of the allocated storage
<name>-db-connections
- Fires when
- Connections stay above 80% of 225 for 15 minutes
<name>-appdb-cpu, -appdb-storage, -appdb-connections
- Fires when
- The same three, for the app database. Created only when
app_database.enabledistrue
<name>-efs-burst-credits
- Fires when
- The code-storage file system's burst credit balance falls below 1 TB
The service alarms catch a service with no running task. They cannot
catch a service that runs fewer tasks than desired but more than none.
Container Insights would report that count, and the module turns it off
because it costs about $1.40 a day per cluster. Until the first
valet-dedicated upgrade starts the services, the service alarms are in
ALARM.
The module creates no IAM for the alarms. CloudWatch publishes to the topic with its own service principal. A topic with the default policy accepts it. If you encrypt the topic with a customer managed KMS key, the key policy must let CloudWatch publish:
{
"Sid": "AllowCloudWatchAlarms",
"Effect": "Allow",
"Principal": { "Service": "cloudwatch.amazonaws.com" },
"Action": ["kms:Decrypt", "kms:GenerateDataKey*"],
"Resource": "*"
}Wire the topic to your channel
Create the topic yourself and pass its ARN. Then subscribe a channel to it.
- Email. Subscribe an address to the topic with protocol
emailand confirm the message AWS sends. - PagerDuty. Create an Amazon CloudWatch integration on a PagerDuty
service. Subscribe the integration URL it gives you to the topic with
protocol
https. - Slack. In AWS Chatbot, configure a Slack channel and add the topic to it. Or subscribe a Lambda function that posts to a Slack webhook.
Send metrics and traces to your collector
Set telemetry to export Valet's own metrics and traces over OTLP/HTTP.
The api, worker, inference, dashboard, and build services export. Agent
runtimes do not. Without telemetry, nothing exports.
- Create a Secrets Manager secret in the installation's region. Its plain-text value is the token your collector expects.
- Set the input.
endpointis an https URL with a host and an optional port, and no path. Valet adds/v1/metricsand/v1/traces.
telemetry = {
endpoint = "https://otel.example.com:4318"
token_secret = "arn:aws:secretsmanager:us-east-2:123456789012:secret:otlp-token-AbCdEf"
}The module passes the endpoint to each task as VALET_OTLP_ENDPOINT and
the secret as VALET_OTLP_TOKEN. It never puts the token in plain
environment or state. It grants each task's execution role
secretsmanager:GetSecretValue on that one secret. You can pass the
secret's name instead of its ARN; the module then grants the name with
Secrets Manager's six-character suffix.
Every export carries one header, auth_header, set to auth_prefix
followed by the token. The defaults send Authorization: Bearer <token>.
An empty auth_prefix also sends Bearer , so the header always carries
a prefix.
The tasks read the secret when they start. After you change the secret or
the input, run valet-dedicated upgrade to start new tasks.
The tasks need a route to the endpoint. The workload subnets' NAT gateway provides one to the public internet. For a collector on your own network, make sure the platform security group can reach it on the port you set.
Datadog
Datadog's direct OTLP intake wants the API key bare in the dd-api-key
header. Valet always sends a prefix, so point Valet at a collector that
holds the key. Run the OpenTelemetry Collector (contrib) or the Datadog
Agent with its OTLP receiver on your network. Use the Collector's
bearertokenauth extension on the receiver, and the datadog exporter
with your API key. Then set endpoint to the Collector's https address
and token_secret to the bearer token you gave the extension.
extensions:
bearertokenauth:
token: ${env:VALET_INGEST_TOKEN}
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
auth:
authenticator: bearertokenauth
exporters:
datadog:
api:
key: ${env:DD_API_KEY}
site: datadoghq.com
service:
extensions: [bearertokenauth]
pipelines:
metrics: { receivers: [otlp], exporters: [datadog] }
traces: { receivers: [otlp], exporters: [datadog] }Terminate TLS in front of the Collector, because Valet always exports over https.
A generic OTLP collector
Any collector that accepts OTLP/HTTP over https works. Match its
authentication to Valet's one header. For a bearer token, keep the
defaults. For a collector that reads another header, set auth_header,
such as X-Api-Key, and auth_prefix. Valet adds a service.name
resource attribute to every metric and span, such as valet-api.
Logs
Each service and task family writes to the log group /ecs/<name>/<family>
in CloudWatch. The families are api, worker, inference, dashboard, build,
migrate, runtime, and builder. ECS Exec sessions write to
/ecs/<name>/platform/exec and /ecs/<name>/workloads/exec.
The module keeps each group's events for log_retention_days, 365 by
default. Raise it to keep more, or set 0 to keep them forever. Lowering
it deletes older events when you apply.
The database's PostgreSQL log goes to the group
/aws/rds/instance/<name>-db/postgresql. It records each connection and
disconnection with its role. The module keeps its events for
audit_retention_days. See what the database
records.
Audit events
Valet records each security decision, such as a sign-in or a permission
check, in its database and in a CloudWatch log group named
/valet/<name>/audit. Each event is one JSON line with a version field
and no secret values. The api writes the lines to a stream named
api/<hostname>, one stream per running task, within seconds of the
decision. The audit log page lists every field and how to read
or forward the group.
The group includes allowed and refused operator and superuser writes,
including settings, platform roles, caps, source edits, and runtime
operations. Their permission names start with operator., such as
operator.set_setting and operator.suspend_user. Settings events carry
detail.name and optional detail.reason, never the value. The top-level
reason describes the admission decision: empty on allow,
permission_not_held on denial, or authorization_read_failed on error.
Operator text stays in detail.reason for every outcome.
Other write events carry target identifiers and reviewed non-secret changes:
platform role, org role enforcement, billing exemption, and runtime image.
Each request-derived detail string is capped at 256 bytes on a UTF-8
boundary. The fixed operator-token actor identifies installation-token
calls. GitHub catalog and deployment calls use actor_kind=ci; detail
carries verified repository, workflow, and ref claims, never a token.
These events record admission; an allowed request can still fail in its
handler. Missing or unverified credentials produce no row and increment
audit.events{outcome=unauthenticated_skipped}. Read-only procedures,
including settings and audit-log reads, produce no event. Source clone and
release download URL minting record admission without the URL or token.
Suspension produces one operator.suspend_user admission event with the
target email in detail. The former post-success user.suspend event is
retired. Impersonation remains a session event.
The module keeps the group's events for audit_retention_days, 400 by
default, and no other log group uses that input. Raise it to match your
compliance window, or set 0 to keep them forever. Lowering it deletes
older events when you apply. Only the api task can write to the group. You
can read it, or forward it, with the permissions you already hold on your
account. The database copy is the source of truth; the group is a
best-effort second copy, and a crash can lose the last few seconds.
Load balancer access logs
Set alb_access_logs_bucket to the name of an S3 bucket you own. The load
balancer writes its access logs there under the prefix <name>. The module
does not create the bucket or its policy. Create both first, because AWS
checks the policy when it turns logging on.
The bucket must use SSE-S3 encryption, not a KMS key, and must be in the
installation's region. Its policy must let the Elastic Load Balancing log
delivery service write under the prefix. In regions opened in or after
August 2022, name the service principal. Replace BUCKET, NAME, and
ACCOUNT:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Service": "logdelivery.elasticloadbalancing.amazonaws.com" },
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::BUCKET/NAME/AWSLogs/ACCOUNT/*"
}
]
}In older regions, such as us-east-1, us-east-2, and us-west-2, name the
regional load balancer account as the principal instead:
"AWS": "arn:aws:iam::ELB_ACCOUNT:root". The AWS documentation lists
ELB_ACCOUNT for each region.