The deployment, as code: seven files that stand the stack up from nothing, verify it, and take it down again.
Every script below is the one the deployment actually runs, copied from the repository at the commit named in each heading. The link above each goes to the current file, which may be newer than what is shown here.
They are written to be adapted. Every tunable is an environment variable with a prefixed name and a default — project, region, zone, instance type, image tag, memory — and nothing in them names a specific account, key or address. Secrets are read from files the scripts themselves write, outside any repository, and are never printed. A run against a fresh account creates what is missing and leaves alone what exists.
The decisions behind each script — why the memory limit needs a unit, why an update re-sends every variable, why the Instance sits behind a security group that drops inbound — are in the header comment of each file and, at length, in the deployment guide.
provision.sh — Phase 1 — the scoped project, the database, and an identity that can reach that database and nothing else.deploy_container.sh — Phase 2 — the application as a Serverless Container, from a version tag.verify_deployment.py — Phase 3 — the gate: signs in, checks the image, ingests a document and retrieves it semantically. Exits non-zero on failure.deploy_embeddings.sh — Phase 4 — an Instance serving embeddings over a private network, wired to the container.ollama-cloud-init.yaml — The cloud-init that Phase 4 hands to that Instance. Installs Ollama and deliberately pulls no model.verify_database.py — A standalone check of a managed database: TLS required, pgvector present, a setting that survives a transaction boundary.panic_teardown.sh — The cost kill switch — deletes everything billable in one project, most expensive first. Dry run unless CONFIRM=DESTROY.provision.shPhase 1 — the scoped project, the database, and an identity that can reach that database and nothing else.
#!/usr/bin/env bash
#
# Stands up the LocalChat test stack's Phase 1 on Scaleway: a scoped project, a
# Serverless SQL Database, and an IAM identity that can reach that database and
# nothing else. See docs/DEPLOYMENT_SCALEWAY.md §4 and §10.
#
# Idempotent at every level except one. Re-running finds what exists and leaves
# it alone; only the API key cannot work that way, because Scaleway shows a
# secret key once and never again. That asymmetry is the whole reason this is a
# script rather than a list of commands in a runbook:
#
# everything -> look it up by name, create only if absent
# the API key -> create only when the application has none, and write the
# secret straight to PROVISION_ENV_OUT. If a key already exists its
# secret is unrecoverable, so this says so and refuses
# rather than quietly creating a second one.
#
# Every tunable is PROVISION_-prefixed on purpose: the application's own
# .env defines APP_NAME, and an unprefixed name silently became the name of
# the IAM application it created.
#
# Usage:
# bash scripts/scaleway/provision.sh # create or verify
# PROVISION_ENV_OUT=~/localchat-db.env bash scripts/scaleway/provision.sh
# PROVISION_ROTATE_KEY=1 bash scripts/scaleway/provision.sh # replace the key
#
# The secret is never printed. It goes to PROVISION_ENV_OUT (mode 600), which defaults to
# a path outside this repository so it cannot be committed by accident.
#
# Needs `scw` (authenticated — `scw login`) and Python 3.
# -e matters more here than anywhere: every lookup runs in a command
# substitution, and `die` inside one only exits the subshell. Without -e a
# refusal printed its message and then provisioned anyway.
set -euo pipefail
die() { printf 'error: %s\n' "$*" >&2; exit 1; }
note() { printf ' %s\n' "$*"; }
PROVISION_PROJECT_NAME="${PROVISION_PROJECT_NAME:-localchat-test}"
PROVISION_DB_NAME="${PROVISION_DB_NAME:-localchat}"
PROVISION_APP_NAME="${PROVISION_APP_NAME:-localchat-app}"
PROVISION_POLICY_NAME="${PROVISION_POLICY_NAME:-localchat-db-readwrite}"
PROVISION_CPU_MIN="${PROVISION_CPU_MIN:-0}"
PROVISION_CPU_MAX="${PROVISION_CPU_MAX:-1}"
PROVISION_KEY_EXPIRES="${PROVISION_KEY_EXPIRES:-+365d}"
PROVISION_ENV_OUT="${PROVISION_ENV_OUT:-$HOME/.config/scw/localchat-db.env}"
PROVISION_ROTATE_KEY="${PROVISION_ROTATE_KEY:-}"
# ServerlessSQLDatabaseReadWrite covers data *and* DDL, which the app needs: it
# runs its own Alembic chain and _ensure_extensions_and_tables() on boot.
# FullAccess would also permit deleting the database, which it never does.
PROVISION_PERMISSION_SET="${PROVISION_PERMISSION_SET:-ServerlessSQLDatabaseReadWrite}"
command -v scw >/dev/null 2>&1 || die "scw not found on PATH"
PYTHON=""
for c in python3 python; do
if command -v "$c" >/dev/null 2>&1; then PYTHON="$c"; break; fi
done
[[ -n "$PYTHON" ]] || die "no python3 on PATH"
PROFILE_ARGS=()
if [[ -n "${SCW_PROFILE:-}" ]]; then
PROFILE_ARGS=(--profile "$SCW_PROFILE")
fi
scw_json() {
local raw
if ! raw=$(scw "${PROFILE_ARGS[@]}" "$@" -o json 2>&1); then
die "'scw $*' failed:"$'\n'"$raw"
fi
printf '%s' "$raw"
}
# Prints the id of the row whose `name` matches, or nothing. Exits non-zero on
# an unreadable reply so that "cannot tell" never reads as "does not exist".
read -r -d '' PICK <<'PY' || true
import json, sys
sys.stdout.reconfigure(newline="\n")
want = sys.argv[1]
try:
rows = json.load(sys.stdin)
except Exception as exc:
sys.exit("reply was not JSON: %s" % exc)
if isinstance(rows, dict):
rows = next((v for v in rows.values() if isinstance(v, list)), [])
for row in rows:
if not isinstance(row, dict):
sys.exit("reply contained a non-object entry")
if row.get("name") == want:
print(row.get("id", ""))
break
PY
read -r -d '' FIELD <<'PY' || true
import json, sys
sys.stdout.reconfigure(newline="\n")
try:
obj = json.load(sys.stdin)
except Exception as exc:
sys.exit("reply was not JSON: %s" % exc)
for key in sys.argv[1:]:
value = obj.get(key)
if value in (None, ""):
sys.exit("reply carried no %s" % key)
print(value)
PY
read -r -d '' COUNT_KEYS <<'PY' || true
import json, sys
sys.stdout.reconfigure(newline="\n")
rows = json.load(sys.stdin)
if isinstance(rows, dict):
rows = next((v for v in rows.values() if isinstance(v, list)), [])
print(len(rows))
PY
read -r -d '' LIST_ACCESS_KEYS <<'PY' || true
import json, sys
sys.stdout.reconfigure(newline="\n")
rows = json.load(sys.stdin)
if isinstance(rows, dict):
rows = next((v for v in rows.values() if isinstance(v, list)), [])
for row in rows:
if row.get("access_key"):
print(row["access_key"])
PY
read -r -d '' REFRESH_ENV <<'PY' || true
"""Rewrite the credential file's connection details, keeping the stored secret.
A database can be destroyed and recreated while the API key survives, and then
the key is still valid but the host in the file names a database that no longer
exists. Writing the file only when a key is created — as this script used to —
leaves the summary correct and the file wrong, which is the worst combination:
nothing looks broken until a deployment cannot connect.
"""
import os, pathlib, stat, sys
out = pathlib.Path(sys.argv[1])
host, dbname, user = sys.argv[2], sys.argv[3], sys.argv[4]
if not out.exists():
sys.exit("no credential file to refresh")
existing = out.read_text(encoding="utf-8").splitlines()
def field(name):
for line in existing:
if line.startswith(name + "="):
return line.split("=", 1)[1].strip()
return ""
secret = field("PG_PASSWORD")
if not secret:
sys.exit("the credential file holds no PG_PASSWORD")
previous = field("PG_HOST")
out.write_text(
"\n".join(
[
"# Scaleway Serverless SQL credential for the LocalChat test stack.",
"# Written by scripts/scaleway/provision.sh - do not commit.",
"# Connection details refreshed; the stored secret was kept.",
"PG_HOST=" + host,
"PG_PORT=5432",
"PG_DB=" + dbname,
"PG_USER=" + user,
"PG_PASSWORD=" + secret,
"PG_SSLMODE=require",
"",
]
),
encoding="utf-8",
newline="\n",
)
os.chmod(out, stat.S_IRUSR | stat.S_IWUSR)
print("changed" if previous != host else "unchanged")
PY
read -r -d '' WRITE_ENV <<'PY' || true
import json, os, pathlib, stat, sys
out, host, dbname = pathlib.Path(sys.argv[1]), sys.argv[2], sys.argv[3]
try:
key = json.load(sys.stdin)
except Exception as exc:
sys.exit("api-key create did not return JSON: %s" % exc)
if "secret_key" not in key:
sys.exit("api-key create returned no secret_key")
out.write_text(
"# Scaleway Serverless SQL credential for the LocalChat test stack.\n"
"# Written by scripts/scaleway/provision.sh - do not commit.\n"
"# Created %s\n"
"# EXPIRES %s <- the deployment stops working on this date.\n"
"PG_HOST=%s\n"
"PG_PORT=5432\n"
"PG_DB=%s\n"
"PG_USER=%s\n"
"PG_PASSWORD=%s\n"
"PG_SSLMODE=require\n"
% (
key.get("created_at", "?"),
key.get("expires_at", "never - the organisation allows unbounded keys"),
host,
dbname,
key.get("application_id", ""),
key["secret_key"],
),
encoding="utf-8",
newline="\n",
)
os.chmod(out, stat.S_IRUSR | stat.S_IWUSR)
print(" created %s, secret written to %s (mode 600)" % (key.get("access_key"), out))
PY
find_by_name() { # find_by_name <name> -- <scw list args...>
local want="$1"; shift 2
local raw found
raw=$(scw_json "$@")
if ! found=$(printf '%s' "$raw" | "$PYTHON" -c "$PICK" "$want" 2>&1); then
die "could not read the reply to 'scw $*' ($found)"
fi
printf '%s' "$found"
}
echo "Project '$PROVISION_PROJECT_NAME':"
PROJECT_ID=$(find_by_name "$PROVISION_PROJECT_NAME" -- account project list)
if [[ -n "$PROJECT_ID" ]]; then
note "exists ($PROJECT_ID)"
else
PROJECT_ID=$(scw_json account project create name="$PROVISION_PROJECT_NAME" \
description="LocalChat test stack" | "$PYTHON" -c "$FIELD" id) \
|| die "project creation returned no id"
note "created ($PROJECT_ID)"
fi
echo "Database '$PROVISION_DB_NAME':"
DB_ID=$(find_by_name "$PROVISION_DB_NAME" -- sdb-sql database list project-id="$PROJECT_ID")
if [[ -n "$DB_ID" ]]; then
note "exists ($DB_ID)"
else
# cpu-min/cpu-max are required by the CLI, so the Terraform provider's
# max_cpu = 15 default cannot silently apply on this path (§4).
DB_ID=$(scw_json sdb-sql database create name="$PROVISION_DB_NAME" \
cpu-min="$PROVISION_CPU_MIN" cpu-max="$PROVISION_CPU_MAX" project-id="$PROJECT_ID" \
| "$PYTHON" -c "$FIELD" id) || die "database creation returned no id"
note "created ($DB_ID) with cpu $PROVISION_CPU_MIN-$PROVISION_CPU_MAX"
fi
DB_ENDPOINT=$(scw_json sdb-sql database get database-id="$DB_ID" \
| "$PYTHON" -c "$FIELD" endpoint) || die "could not read the database endpoint"
# postgres://<host>:<port>/<db>?sslmode=require -> host
DB_HOST=$(printf '%s' "$DB_ENDPOINT" | sed -e 's#^postgres://##' -e 's#[:/?].*$##')
[[ -n "$DB_HOST" ]] || die "could not parse a host out of: $DB_ENDPOINT"
note "endpoint $DB_HOST"
echo "IAM application '$PROVISION_APP_NAME':"
APP_ID=$(find_by_name "$PROVISION_APP_NAME" -- iam application list)
if [[ -n "$APP_ID" ]]; then
note "exists ($APP_ID)"
else
APP_ID=$(scw_json iam application create name="$PROVISION_APP_NAME" \
description="LocalChat test stack - owns the database credential" \
| "$PYTHON" -c "$FIELD" id) || die "application creation returned no id"
note "created ($APP_ID)"
fi
echo "Policy '$PROVISION_POLICY_NAME' ($PROVISION_PERMISSION_SET, scoped to the project):"
POLICY_ID=$(find_by_name "$PROVISION_POLICY_NAME" -- iam policy list)
if [[ -n "$POLICY_ID" ]]; then
note "exists ($POLICY_ID)"
else
POLICY_ID=$(scw_json iam policy create name="$PROVISION_POLICY_NAME" \
description="Scoped to the $PROVISION_PROJECT_NAME project only" \
application-id="$APP_ID" \
rules.0.permission-set-names.0="$PROVISION_PERMISSION_SET" \
rules.0.project-ids.0="$PROJECT_ID" \
| "$PYTHON" -c "$FIELD" id) || die "policy creation returned no id"
note "created ($POLICY_ID)"
fi
echo "API key:"
KEY_COUNT=$(scw_json iam api-key list application-id="$APP_ID" \
| "$PYTHON" -c "$COUNT_KEYS")
if [[ "$KEY_COUNT" -gt 0 && -z "$PROVISION_ROTATE_KEY" ]]; then
note "$KEY_COUNT key(s) already exist; a secret is shown once and cannot be read back."
# The key survives a database being destroyed and recreated, so the details
# around the secret must be refreshed even when the secret itself is not.
if ! refreshed=$("$PYTHON" -c "$REFRESH_ENV" "$PROVISION_ENV_OUT" "$DB_HOST" "$PROVISION_DB_NAME" "$APP_ID" 2>&1); then
die "$KEY_COUNT key(s) exist but their secret is not in $PROVISION_ENV_OUT ($refreshed).
A secret cannot be read back, so the only way forward is to replace the key:
PROVISION_ROTATE_KEY=1 bash scripts/scaleway/provision.sh
Rotating invalidates the old credential - redeploy the container afterwards."
fi
if [[ "$refreshed" == "changed" ]]; then
note "reused the stored secret; connection details updated to the current database"
else
note "reused the stored secret; connection details already correct"
fi
else
if [[ "$KEY_COUNT" -gt 0 ]]; then
note "rotating: deleting $KEY_COUNT existing key(s)"
while IFS= read -r ak; do
[[ -n "$ak" ]] || continue
if scw "${PROFILE_ARGS[@]}" iam api-key delete access-key="$ak" >/dev/null 2>&1; then
note " deleted $ak"
else
note " FAILED to delete $ak"
fi
done < <(scw_json iam api-key list application-id="$APP_ID" \
| "$PYTHON" -c "$LIST_ACCESS_KEYS")
fi
umask 077
mkdir -p "$(dirname "$PROVISION_ENV_OUT")" || die "cannot create $(dirname "$PROVISION_ENV_OUT")"
# The secret goes from scw straight into the file. It is never echoed and
# never held in a shell variable.
if ! scw "${PROFILE_ARGS[@]}" iam api-key create application-id="$APP_ID" \
expires-at="$PROVISION_KEY_EXPIRES" \
description="LocalChat test stack database credential" -o json \
| "$PYTHON" -c "$WRITE_ENV" "$PROVISION_ENV_OUT" "$DB_HOST" "$PROVISION_DB_NAME"; then
die "api-key creation failed; the application may now have no usable key.
Re-run with PROVISION_ROTATE_KEY=1 once the cause is fixed."
fi
fi
cat <<SUMMARY
Phase 1 is provisioned.
project $PROVISION_PROJECT_NAME $PROJECT_ID
database $PROVISION_DB_NAME $DB_ID
host $DB_HOST
identity $PROVISION_APP_NAME $APP_ID (policy $POLICY_ID, $PROVISION_PERMISSION_SET)
env file $PROVISION_ENV_OUT
The credential is scoped to this project and this database. It cannot reach the
default project, and it cannot delete the database.
Next: docs/DEPLOYMENT_SCALEWAY.md Phase 2 - the container, from a version tag.
To tear all of it down: docs/COST_KILL_SWITCH.md.
SUMMARY
deploy_container.shPhase 2 — the application as a Serverless Container, from a version tag.
#!/usr/bin/env bash
#
# Phase 2 of docs/DEPLOYMENT_SCALEWAY.md: the Serverless Container namespace and
# the container itself. Idempotent — a second run updates the existing container
# rather than creating a second one.
#
# Everything here was learned by deploying once by hand (D3). Three details cost
# a failed attempt each, and none of them is guessable from the CLI help:
#
# * `memory-limit-bytes` does NOT take bytes. It requires a G/GB unit —
# 3072000000 is refused outright with "size must be defined using the G or
# GB unit". Pass 3GB.
# * An update REPLACES the environment map rather than merging into it, so
# every variable and every secret is re-sent on every run. Sending only the
# one you changed silently drops the rest.
# * Secrets belong in `secret-environment-variables`. Scaleway stores those
# separately and reads them back as argon2 hashes, so `container get` never
# echoes one. Ordinary environment variables are returned in clear.
#
# Usage:
# bash scripts/scaleway/deploy_container.sh
# DEPLOY_IMAGE_TAG=3.0.1 bash scripts/scaleway/deploy_container.sh
#
# Reads the database credential written by provision.sh, and generates the
# application secrets on first run into a second 600-mode file outside the repo.
# ADMIN_PASSWORD in that file is how you sign in to the deployed app.
#
# Needs `scw` (authenticated) and Python 3.
set -euo pipefail
die() { printf 'error: %s\n' "$*" >&2; exit 1; }
note() { printf ' %s\n' "$*"; }
DEPLOY_PROJECT_NAME="${DEPLOY_PROJECT_NAME:-localchat-test}"
DEPLOY_NAMESPACE="${DEPLOY_NAMESPACE:-localchat}"
DEPLOY_CONTAINER="${DEPLOY_CONTAINER:-localchat}"
DEPLOY_IMAGE_TAG="${DEPLOY_IMAGE_TAG:-3.0.0}"
DEPLOY_IMAGE="${DEPLOY_IMAGE:-ghcr.io/jwvanderstam/localchat:$DEPLOY_IMAGE_TAG}"
DEPLOY_REGION="${DEPLOY_REGION:-fr-par}"
DEPLOY_MEMORY="${DEPLOY_MEMORY:-3GB}"
DEPLOY_MVCPU="${DEPLOY_MVCPU:-1000}"
DEPLOY_PORT="${DEPLOY_PORT:-5000}"
DEPLOY_DB_ENV="${DEPLOY_DB_ENV:-$HOME/.config/scw/localchat-db.env}"
DEPLOY_SECRET_ENV="${DEPLOY_SECRET_ENV:-$HOME/.config/scw/localchat-deploy.env}"
# How often the readiness loop polls. Only a test has a reason to lower it.
DEPLOY_POLL_SECONDS="${DEPLOY_POLL_SECONDS:-20}"
# Empty by default and set by deploy_embeddings.sh. Without it the app boots and
# serves everything except chat and document ingest, which both need Ollama.
DEPLOY_OLLAMA_URL="${DEPLOY_OLLAMA_URL:-}"
# Set by deploy_embeddings.sh. Handing the network to this script rather than
# applying it in a second call is what keeps the two from colliding: Scaleway
# refuses an update while the previous one is still applying, with
# "transient state error for resource 'container'".
DEPLOY_PRIVATE_NETWORK_ID="${DEPLOY_PRIVATE_NETWORK_ID:-}"
command -v scw >/dev/null 2>&1 || die "scw not found on PATH"
PYTHON=""
for c in python3 python; do
if command -v "$c" >/dev/null 2>&1; then PYTHON="$c"; break; fi
done
[[ -n "$PYTHON" ]] || die "no python3 on PATH"
PROFILE_ARGS=()
if [[ -n "${SCW_PROFILE:-}" ]]; then
PROFILE_ARGS=(--profile "$SCW_PROFILE")
fi
# Secrets are passed to `scw` as arguments, so a failure that echoes the command
# publishes every one of them — to the terminal, and to whatever collects it.
# That happened once. The arguments are still shown, because a failure with no
# context is unfixable, but every secret value is replaced first.
_redact() {
printf '%s' "$*" | sed -E 's/(secret-environment-variables[.][A-Za-z0-9_]+=)[^ ]*/\1***/g'
}
scw_json() {
local raw
if ! raw=$(scw "${PROFILE_ARGS[@]}" "$@" -o json 2>&1); then
die "'scw $(_redact "$@")' failed:"$'\n'"$raw"
fi
printf '%s' "$raw"
}
read -r -d '' PICK <<'PY' || true
import json, sys
sys.stdout.reconfigure(newline="\n")
want = sys.argv[1]
try:
rows = json.load(sys.stdin)
except Exception as exc:
sys.exit("reply was not JSON: %s" % exc)
if isinstance(rows, dict):
rows = next((v for v in rows.values() if isinstance(v, list)), [])
for row in rows:
if isinstance(row, dict) and row.get("name") == want:
print(row.get("id", ""))
break
PY
read -r -d '' FIELD <<'PY' || true
import json, sys
sys.stdout.reconfigure(newline="\n")
try:
obj = json.load(sys.stdin)
except Exception as exc:
sys.exit("reply was not JSON: %s" % exc)
for key in sys.argv[1:]:
value = obj.get(key)
if value in (None, ""):
sys.exit("reply carried no %s" % key)
print(value)
PY
read -r -d '' MAKE_SECRETS <<'PY' || true
import os, pathlib, secrets, stat, string, sys
from cryptography.fernet import Fernet
out = pathlib.Path(sys.argv[1])
if out.exists():
print("exists")
raise SystemExit(0)
out.parent.mkdir(parents=True, exist_ok=True)
alphabet = string.ascii_letters + string.digits
out.write_text(
"# LocalChat deployment secrets. Mode 600, outside the repo, never commit.\n"
"# ADMIN_PASSWORD is how you sign in to the deployed application.\n"
"SECRET_KEY=%s\n"
"JWT_SECRET_KEY=%s\n"
"ENCRYPTION_KEY=%s\n"
"METRICS_TOKEN=%s\n"
"ADMIN_PASSWORD=%s\n"
% (
secrets.token_hex(32),
secrets.token_hex(32),
Fernet.generate_key().decode(),
secrets.token_hex(16),
"".join(secrets.choice(alphabet) for _ in range(24)),
),
encoding="utf-8",
newline="\n",
)
os.chmod(out, stat.S_IRUSR | stat.S_IWUSR)
print("created")
PY
find_by_name() { # find_by_name <name> -- <scw list args...>
local want="$1"; shift 2
local raw found
raw=$(scw_json "$@")
if ! found=$(printf '%s' "$raw" | "$PYTHON" -c "$PICK" "$want" 2>&1); then
die "could not read the reply to 'scw $*' ($found)"
fi
printf '%s' "$found"
}
[[ -f "$DEPLOY_DB_ENV" ]] || die "no database credential at $DEPLOY_DB_ENV
Run scripts/scaleway/provision.sh first."
echo "Application secrets:"
umask 077
made=$("$PYTHON" -c "$MAKE_SECRETS" "$DEPLOY_SECRET_ENV")
if [[ "$made" == "created" ]]; then
note "generated -> $DEPLOY_SECRET_ENV (mode 600)"
note "ADMIN_PASSWORD in that file is your login. It is not printed here."
else
note "reusing $DEPLOY_SECRET_ENV"
fi
# shellcheck disable=SC1090
set -a; . "$DEPLOY_DB_ENV"; . "$DEPLOY_SECRET_ENV"; set +a
for v in PG_HOST PG_PORT PG_DB PG_USER PG_PASSWORD SECRET_KEY JWT_SECRET_KEY \
ENCRYPTION_KEY ADMIN_PASSWORD METRICS_TOKEN; do
[[ -n "${!v:-}" ]] || die "$v is missing from the credential files"
done
echo "Project '$DEPLOY_PROJECT_NAME':"
PROJECT_ID=$(find_by_name "$DEPLOY_PROJECT_NAME" -- account project list)
[[ -n "$PROJECT_ID" ]] || die "project '$DEPLOY_PROJECT_NAME' does not exist — run provision.sh"
note "$PROJECT_ID"
echo "Namespace '$DEPLOY_NAMESPACE':"
NAMESPACE_ID=$(find_by_name "$DEPLOY_NAMESPACE" -- container namespace list \
project-id="$PROJECT_ID" region="$DEPLOY_REGION")
if [[ -n "$NAMESPACE_ID" ]]; then
note "exists ($NAMESPACE_ID)"
else
NAMESPACE_ID=$(scw_json container namespace create name="$DEPLOY_NAMESPACE" \
project-id="$PROJECT_ID" region="$DEPLOY_REGION" \
description="LocalChat test stack" | "$PYTHON" -c "$FIELD" id)
note "created ($NAMESPACE_ID)"
for _ in $(seq 1 30); do
st=$(scw_json container namespace get namespace-id="$NAMESPACE_ID" \
region="$DEPLOY_REGION" | "$PYTHON" -c "$FIELD" status)
[[ "$st" == "ready" ]] && break
sleep 5
done
note "status $st"
fi
# Every variable is re-sent on every run, because an update replaces the map.
ENV_ARGS=(
environment-variables.APP_ENV=production
environment-variables.SERVER_PORT="$DEPLOY_PORT"
environment-variables.UVICORN_WORKERS=1
environment-variables.REDIS_ENABLED=false
environment-variables.MCP_ENABLED=false
environment-variables.PG_HOST="$PG_HOST"
environment-variables.PG_PORT="$PG_PORT"
environment-variables.PG_DB="$PG_DB"
environment-variables.PG_USER="$PG_USER"
environment-variables.PG_SSLMODE="${PG_SSLMODE:-require}"
)
NETWORK_ARGS=()
if [[ -n "$DEPLOY_PRIVATE_NETWORK_ID" ]]; then
NETWORK_ARGS=(private-network-id="$DEPLOY_PRIVATE_NETWORK_ID")
fi
if [[ -n "$DEPLOY_OLLAMA_URL" ]]; then
ENV_ARGS+=(
environment-variables.OLLAMA_BASE_URL="$DEPLOY_OLLAMA_URL"
environment-variables.OLLAMA_EMBEDDING_MODEL="${DEPLOY_EMBEDDING_MODEL:-nomic-embed-text}"
)
fi
SECRET_ARGS=(
secret-environment-variables.PG_PASSWORD="$PG_PASSWORD"
secret-environment-variables.SECRET_KEY="$SECRET_KEY"
secret-environment-variables.JWT_SECRET_KEY="$JWT_SECRET_KEY"
secret-environment-variables.ENCRYPTION_KEY="$ENCRYPTION_KEY"
secret-environment-variables.ADMIN_PASSWORD="$ADMIN_PASSWORD"
secret-environment-variables.METRICS_TOKEN="$METRICS_TOKEN"
)
echo "Container '$DEPLOY_CONTAINER' ($DEPLOY_IMAGE):"
CONTAINER_ID=$(find_by_name "$DEPLOY_CONTAINER" -- container container list \
namespace-id="$NAMESPACE_ID" region="$DEPLOY_REGION")
if [[ -n "$CONTAINER_ID" ]]; then
note "exists ($CONTAINER_ID) — updating in place"
scw_json container container update container-id="$CONTAINER_ID" \
region="$DEPLOY_REGION" image="$DEPLOY_IMAGE" \
"${NETWORK_ARGS[@]}" "${ENV_ARGS[@]}" "${SECRET_ARGS[@]}" >/dev/null
else
note "absent — creating"
# min/max scale 1 is D1 and D2: no scale-to-zero cold start on a 3 GB image,
# and no way for concurrency to multiply the bill.
CONTAINER_ID=$(scw_json container container create \
namespace-id="$NAMESPACE_ID" name="$DEPLOY_CONTAINER" region="$DEPLOY_REGION" \
image="$DEPLOY_IMAGE" port="$DEPLOY_PORT" protocol=http1 privacy=public \
min-scale=1 max-scale=1 \
memory-limit-bytes="$DEPLOY_MEMORY" mvcpu-limit="$DEPLOY_MVCPU" \
timeout=300s \
"${NETWORK_ARGS[@]}" "${ENV_ARGS[@]}" "${SECRET_ARGS[@]}" | "$PYTHON" -c "$FIELD" id)
note "created ($CONTAINER_ID)"
fi
echo "Waiting for the container (a 3 GB pull takes a few minutes):"
# `error` is not final. Scaleway reports it while a slow first boot is still
# failing its early liveness probes: on 2026-09-14 the container read `error`
# for 15 s and was `ready` 22 s before the app answered /api/health, and this
# loop had given up on the first sighting. Only an error that stays is one, so
# it has to be seen on consecutive polls before the script believes it.
STATUS=""
ERRORS=0
for _ in $(seq 1 60); do
STATUS=$(scw_json container container get container-id="$CONTAINER_ID" \
region="$DEPLOY_REGION" | "$PYTHON" -c "$FIELD" status)
case "$STATUS" in
ready) break ;;
error) ERRORS=$((ERRORS + 1)); if [[ $ERRORS -ge 3 ]]; then break; fi ;;
*) ERRORS=0 ;;
esac
sleep "$DEPLOY_POLL_SECONDS"
done
note "status $STATUS"
[[ "$STATUS" == "ready" ]] || die "container did not become ready (status: $STATUS)"
ENDPOINT=$(scw_json container container get container-id="$CONTAINER_ID" \
region="$DEPLOY_REGION" | "$PYTHON" -c "$FIELD" public_endpoint)
cat <<SUMMARY
Phase 2 is deployed.
namespace $DEPLOY_NAMESPACE $NAMESPACE_ID
container $DEPLOY_CONTAINER $CONTAINER_ID
image $DEPLOY_IMAGE
endpoint $ENDPOINT
Validate it: python scripts/scaleway/verify_deployment.py $ENDPOINT
Add embeddings: bash scripts/scaleway/deploy_embeddings.sh
Tear it down: docs/COST_KILL_SWITCH.md
SUMMARY
verify_deployment.pyPhase 3 — the gate: signs in, checks the image, ingests a document and retrieves it semantically. Exits non-zero on failure.
#!/usr/bin/env python3
"""Phase 3: prove a deployed LocalChat actually works, and gate on the answer.
Every check here was run by hand against the first Scaleway deployment. Three of
them found something the deployment plan had wrong, which is why they are a
script now rather than a checklist someone reads:
* The running version is **not** on ``/api/status`` — that endpoint carries no
version at all. It is on ``/api/settings/stats``, which is admin-only.
* ``/api/health`` reporting a healthy database was, until it was fixed, an echo
of a boot-time flag. This asks for a live answer by exercising a route that
touches the database.
* The document path needs an embedding model. Without one, upload fails cleanly
and nothing is stored — so "retrieval works without Ollama" is false, and this
reports that as a skip rather than a pass.
Usage:
python scripts/scaleway/verify_deployment.py <endpoint>
python scripts/scaleway/verify_deployment.py <endpoint> --pull-model nomic-embed-text
Reads ADMIN_PASSWORD from the deployment secrets file (``DEPLOY_SECRET_ENV``,
default ``~/.config/scw/localchat-deploy.env``). Prints no secret. Exits
non-zero when a check fails, so it can stand in front of a release.
"""
from __future__ import annotations
import argparse
import json
import os
import pathlib
import re
import sys
import urllib.error
import urllib.request
DEFAULT_SECRET_ENV = pathlib.Path.home() / ".config" / "scw" / "localchat-deploy.env"
PROBE_NAME = "deployment-probe.md"
CANARY = "quiet-utility-marmoset-4417"
PROBE_BODY = f"""# LocalChat deployment probe
This document exists to prove that ingest and retrieval work on the deployed stack.
The canary phrase is: {CANARY}.
The Serverless SQL Database routes connections by TLS SNI, which is why an
unencrypted connection cannot name the database it wants.
"""
class Report:
def __init__(self) -> None:
self.failed = 0
self.skipped = 0
def ok(self, label: str, detail: str = "") -> None:
print(f" PASS {label}{f' — {detail}' if detail else ''}")
def bad(self, label: str, detail: str) -> None:
self.failed += 1
print(f" FAIL {label} — {detail}")
def skip(self, label: str, detail: str) -> None:
self.skipped += 1
print(f" SKIP {label} — {detail}")
def note(self, text: str) -> None:
print(f" {text}")
class Client:
"""The smallest HTTP client that keeps a session cookie."""
def __init__(self, base: str) -> None:
self.base = base.rstrip("/")
self.cookie: str | None = None
def request(
self,
method: str,
path: str,
body: bytes | None = None,
content_type: str | None = None,
timeout: int = 120,
) -> tuple[int, bytes]:
req = urllib.request.Request(f"{self.base}{path}", data=body, method=method)
if content_type:
req.add_header("Content-Type", content_type)
if self.cookie:
req.add_header("Cookie", self.cookie)
try:
with urllib.request.urlopen(req, timeout=timeout) as resp: # noqa: S310
set_cookie = resp.headers.get("Set-Cookie")
if set_cookie:
self.cookie = set_cookie.split(";", 1)[0]
return resp.status, resp.read()
except urllib.error.HTTPError as exc:
return exc.code, exc.read()
def json(self, method: str, path: str, payload: dict | None = None, timeout: int = 120):
body = json.dumps(payload).encode() if payload is not None else None
status, raw = self.request(
method, path, body, "application/json" if body else None, timeout
)
try:
return status, json.loads(raw)
except Exception:
return status, raw.decode("utf-8", "replace")
def load_admin_password() -> str:
path = pathlib.Path(os.environ.get("DEPLOY_SECRET_ENV", str(DEFAULT_SECRET_ENV)))
if not path.exists():
sys.exit(f"no secrets file at {path} — run scripts/scaleway/deploy_container.sh")
for line in path.read_text(encoding="utf-8").splitlines():
if line.startswith("ADMIN_PASSWORD="):
return line.split("=", 1)[1].strip()
sys.exit(f"{path} has no ADMIN_PASSWORD")
def multipart(filename: str, content: str) -> tuple[bytes, str]:
boundary = "----localchatprobe"
body = (
f"--{boundary}\r\n"
f'Content-Disposition: form-data; name="files"; filename="{filename}"\r\n'
"Content-Type: text/markdown\r\n\r\n"
f"{content}\r\n"
f"--{boundary}--\r\n"
).encode()
return body, f"multipart/form-data; boundary={boundary}"
def repo_app_version() -> str:
"""APP_VERSION as this checkout declares it in src/config.py.
The default was the literal "3.0.0", and the release that moved the app to 3.1.0 did not
move it — so every deployment from 2026-09-17 failed this check while being correct, and
a check that always fails is one nobody reads.
"""
text = (pathlib.Path(__file__).resolve().parents[2] / "src" / "config.py").read_text(encoding="utf-8")
match = re.search(r"^APP_VERSION:\s*str\s*=\s*os\.environ\.get\(\s*'APP_VERSION'\s*,\s*'([^']+)'", text, re.MULTILINE)
if not match:
sys.exit("src/config.py no longer declares APP_VERSION the way this script reads it")
return match.group(1)
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("endpoint", help="https://…functions.fnc.fr-par.scw.cloud")
parser.add_argument(
"--pull-model",
metavar="NAME",
help="pull this embedding model through the app before testing ingest",
)
parser.add_argument(
"--expect-version", default=repo_app_version(),
help="version the deployed image must report (default: this checkout's APP_VERSION)",
)
args = parser.parse_args()
client = Client(args.endpoint)
report = Report()
print(f"Verifying {client.base}\n")
status, health = client.json("GET", "/api/health", timeout=60)
if status != 200 or not isinstance(health, dict):
report.bad("health answers", f"HTTP {status}")
return 1
checks = health.get("checks", {})
report.ok("health answers", health.get("status", "?"))
db_up = checks.get("database", {}).get("healthy")
(report.ok if db_up else report.bad)("database reachable", checks.get("database", {}).get("status", "?"))
ollama_up = bool(checks.get("ollama", {}).get("healthy"))
(report.ok if ollama_up else report.skip)(
"ollama reachable",
checks.get("ollama", {}).get("status", "?")
+ ("" if ollama_up else " — ingest and chat cannot work without it"),
)
status, _ = client.json(
"POST", "/api/auth/login", {"username": "admin", "password": load_admin_password()}
)
if status != 200:
report.bad("admin can sign in", f"HTTP {status}")
return 1
report.ok("admin can sign in")
status, stats = client.json("GET", "/api/settings/stats")
version = (stats or {}).get("system", {}).get("app_version") if isinstance(stats, dict) else None
if version == args.expect_version:
report.ok("deployed image is the expected one", f"app_version {version}")
else:
report.bad(
"deployed image is the expected one",
f"reports {version!r}, expected {args.expect_version!r}",
)
if args.pull_model and ollama_up:
print(f" .... pulling {args.pull_model} (minutes, streams progress)")
status, _ = client.request(
"POST",
"/api/models/pull",
json.dumps({"model": args.pull_model}).encode(),
"application/json",
timeout=1800,
)
(report.ok if status == 200 else report.bad)(
f"pulled {args.pull_model}", f"HTTP {status}"
)
if not ollama_up:
report.skip("document ingest", "no embedding model reachable")
report.skip("semantic retrieval", "nothing can be ingested")
else:
body, content_type = multipart(PROBE_NAME, PROBE_BODY)
status, raw = client.request(
"POST", "/api/documents/upload", body, content_type, timeout=900
)
text = raw.decode("utf-8", "replace")
if status == 200 and '"success": true' in text:
report.ok("document ingest", PROBE_NAME)
else:
detail = next(
(line for line in text.splitlines() if "message" in line), f"HTTP {status}"
)
report.bad("document ingest", detail[:160])
# Semantic, not lexical: the query shares no word with the canary, so a
# hit can only come from the embedding.
status, found = client.json(
"POST", "/api/documents/test", {"query": "which animal is named in this document?"}
)
chunks = (
(found or {}).get("results", {}).get("semantic_only", {}).get("chunks", [])
if isinstance(found, dict)
else []
)
if chunks and CANARY in json.dumps(chunks):
report.ok("semantic retrieval", f"similarity {chunks[0].get('similarity')}")
report.note("The query shares no words with the canary, so this is the vector arm.")
else:
report.bad("semantic retrieval", "the probe document did not come back")
print()
if report.failed:
print(f"{report.failed} check(s) failed — this deployment is not good.")
return 1
if report.skipped:
print(f"All checks passed, {report.skipped} skipped. Phase 3 is incomplete "
"until an embedding model is reachable (Phase 4).")
return 0
print("All checks passed. Phase 3 is complete.")
return 0
if __name__ == "__main__":
raise SystemExit(main())
deploy_embeddings.shPhase 4 — an Instance serving embeddings over a private network, wired to the container.
#!/usr/bin/env bash
#
# Phase 4 of docs/DEPLOYMENT_SCALEWAY.md: an Ollama Instance that serves
# embeddings, so document ingest and retrieval work. Generation is Phase 5's
# question and belongs on a GPU.
#
# Idempotent: a second run finds the network, the security group and the
# instance, and re-wires the container to whatever private IP it has.
#
# Three decisions that came out of doing this by hand, each non-obvious:
#
# * The security group sets `inbound-default-policy=drop`. Scaleway's default
# is `accept`, which would put Ollama's port 11434 on the public internet.
# The cost is that there is no way in if something goes wrong: the cloud
# image sets no password for the serial console to log in with, and a key
# only reaches the box if it is registered in *this* project (both reasoned,
# neither tried — DEPLOYMENT_SCALEWAY.md §11, 2026-09-14). A box that fails
# is replaced, not read; whether to change that is §12's.
# * The model is pulled through the application's own POST /api/models/pull,
# not from cloud-init. The cloud-init pull failed silently on the first run
# and left a box with zero models and no way in to find out why. Pulling
# through the app is idempotent, streams its progress, and exercises the
# private network that had to be proven anyway.
# * Everything goes in fr-par-2, the one zone offering both the L4 and the
# larger L40S — so Phase 5's GPU lands beside this box rather than in
# another zone with a private network already wired.
#
# Usage:
# bash scripts/scaleway/deploy_embeddings.sh
#
# Needs `scw` (authenticated), Python 3, and a container from deploy_container.sh.
set -euo pipefail
die() { printf 'error: %s\n' "$*" >&2; exit 1; }
note() { printf ' %s\n' "$*"; }
EMB_PROJECT_NAME="${EMB_PROJECT_NAME:-localchat-test}"
EMB_NETWORK_NAME="${EMB_NETWORK_NAME:-localchat-backend}"
EMB_SECGROUP_NAME="${EMB_SECGROUP_NAME:-localchat-ollama}"
EMB_INSTANCE_NAME="${EMB_INSTANCE_NAME:-localchat-embeddings}"
EMB_NAMESPACE="${EMB_NAMESPACE:-localchat}"
EMB_CONTAINER="${EMB_CONTAINER:-localchat}"
EMB_TYPE="${EMB_TYPE:-DEV1-M}"
EMB_IMAGE="${EMB_IMAGE:-ubuntu_jammy}"
EMB_ZONE="${EMB_ZONE:-fr-par-2}"
EMB_REGION="${EMB_REGION:-fr-par}"
EMB_MODEL="${EMB_MODEL:-nomic-embed-text}"
EMB_CLOUD_INIT="${EMB_CLOUD_INIT:-$(dirname "$0")/ollama-cloud-init.yaml}"
command -v scw >/dev/null 2>&1 || die "scw not found on PATH"
PYTHON=""
for c in python3 python; do
if command -v "$c" >/dev/null 2>&1; then PYTHON="$c"; break; fi
done
[[ -n "$PYTHON" ]] || die "no python3 on PATH"
[[ -f "$EMB_CLOUD_INIT" ]] || die "no cloud-init file at $EMB_CLOUD_INIT"
PROFILE_ARGS=()
if [[ -n "${SCW_PROFILE:-}" ]]; then
PROFILE_ARGS=(--profile "$SCW_PROFILE")
fi
scw_json() {
local raw
if ! raw=$(scw "${PROFILE_ARGS[@]}" "$@" -o json 2>&1); then
die "'scw $*' failed:"$'\n'"$raw"
fi
printf '%s' "$raw"
}
read -r -d '' PICK <<'PY' || true
import json, sys
sys.stdout.reconfigure(newline="\n")
want = sys.argv[1]
try:
rows = json.load(sys.stdin)
except Exception as exc:
sys.exit("reply was not JSON: %s" % exc)
if isinstance(rows, dict):
rows = next((v for v in rows.values() if isinstance(v, list)), [])
for row in rows:
if isinstance(row, dict) and row.get("name") == want:
print(row.get("id", ""))
break
PY
read -r -d '' FIELD <<'PY' || true
import json, sys
sys.stdout.reconfigure(newline="\n")
try:
obj = json.load(sys.stdin)
except Exception as exc:
sys.exit("reply was not JSON: %s" % exc)
obj = obj.get(sys.argv[2], obj) if len(sys.argv) > 2 else obj
value = obj.get(sys.argv[1])
if value in (None, ""):
sys.exit("reply carried no %s" % sys.argv[1])
print(value)
PY
# The private IPv4 the container will talk to. IPAM is the only place that knows
# it; the server object reports the NIC but not its address.
read -r -d '' PRIVATE_IP <<'PY' || true
import json, sys
sys.stdout.reconfigure(newline="\n")
want = sys.argv[1]
rows = json.load(sys.stdin)
if isinstance(rows, dict):
rows = next((v for v in rows.values() if isinstance(v, list)), [])
for row in rows:
resource = row.get("resource") or {}
address = row.get("address", "")
if resource.get("name") == want and ":" not in address:
print(address.split("/")[0])
break
PY
find_by_name() { # find_by_name <name> -- <scw list args...>
local want="$1"; shift 2
local raw found
raw=$(scw_json "$@")
if ! found=$(printf '%s' "$raw" | "$PYTHON" -c "$PICK" "$want" 2>&1); then
die "could not read the reply to 'scw $*' ($found)"
fi
printf '%s' "$found"
}
echo "Project '$EMB_PROJECT_NAME':"
PROJECT_ID=$(find_by_name "$EMB_PROJECT_NAME" -- account project list)
[[ -n "$PROJECT_ID" ]] || die "project '$EMB_PROJECT_NAME' does not exist — run provision.sh"
note "$PROJECT_ID"
echo "Private network '$EMB_NETWORK_NAME':"
NETWORK_ID=$(find_by_name "$EMB_NETWORK_NAME" -- vpc private-network list \
project-id="$PROJECT_ID" region="$EMB_REGION")
if [[ -n "$NETWORK_ID" ]]; then
note "exists ($NETWORK_ID)"
else
NETWORK_ID=$(scw_json vpc private-network create name="$EMB_NETWORK_NAME" \
project-id="$PROJECT_ID" region="$EMB_REGION" | "$PYTHON" -c "$FIELD" id)
note "created ($NETWORK_ID)"
fi
echo "Security group '$EMB_SECGROUP_NAME' (inbound drop):"
SECGROUP_ID=$(find_by_name "$EMB_SECGROUP_NAME" -- instance security-group list \
project-id="$PROJECT_ID" zone="$EMB_ZONE")
if [[ -n "$SECGROUP_ID" ]]; then
note "exists ($SECGROUP_ID)"
else
SECGROUP_ID=$(scw_json instance security-group create name="$EMB_SECGROUP_NAME" \
project-id="$PROJECT_ID" zone="$EMB_ZONE" \
inbound-default-policy=drop outbound-default-policy=accept \
description="Ollama embeddings: reachable on the private network, nowhere else" \
| "$PYTHON" -c "$FIELD" id security_group)
note "created ($SECGROUP_ID)"
fi
echo "Instance '$EMB_INSTANCE_NAME' ($EMB_TYPE, $EMB_ZONE):"
SERVER_ID=$(find_by_name "$EMB_INSTANCE_NAME" -- instance server list \
project-id="$PROJECT_ID" zone="$EMB_ZONE")
if [[ -n "$SERVER_ID" ]]; then
note "exists ($SERVER_ID)"
else
SERVER_ID=$(scw_json instance server create name="$EMB_INSTANCE_NAME" \
type="$EMB_TYPE" image="$EMB_IMAGE" zone="$EMB_ZONE" \
project-id="$PROJECT_ID" security-group-id="$SECGROUP_ID" ip=new \
cloud-init=@"$EMB_CLOUD_INIT" | "$PYTHON" -c "$FIELD" id)
note "created ($SERVER_ID)"
scw_json instance private-nic create server-id="$SERVER_ID" \
private-network-id="$NETWORK_ID" zone="$EMB_ZONE" >/dev/null
note "attached to the private network"
scw "${PROFILE_ARGS[@]}" instance server start "$SERVER_ID" zone="$EMB_ZONE" >/dev/null 2>&1 || true
note "powering on"
fi
for _ in $(seq 1 30); do
STATE=$(scw_json instance server get "$SERVER_ID" zone="$EMB_ZONE" \
| "$PYTHON" -c "$FIELD" state)
[[ "$STATE" == "running" ]] && break
sleep 10
done
note "state $STATE"
[[ "$STATE" == "running" ]] || die "instance did not reach 'running' (state: $STATE)"
PRIVATE=$(scw_json ipam ip list private-network-id="$NETWORK_ID" \
| "$PYTHON" -c "$PRIVATE_IP" "$EMB_INSTANCE_NAME")
[[ -n "$PRIVATE" ]] || die "the instance has no private IPv4 yet — re-run in a moment"
note "private IP $PRIVATE"
echo "Wiring the container to http://$PRIVATE:11434:"
NAMESPACE_ID=$(find_by_name "$EMB_NAMESPACE" -- container namespace list \
project-id="$PROJECT_ID" region="$EMB_REGION")
[[ -n "$NAMESPACE_ID" ]] || die "no namespace '$EMB_NAMESPACE' — run deploy_container.sh first"
CONTAINER_ID=$(find_by_name "$EMB_CONTAINER" -- container container list \
namespace-id="$NAMESPACE_ID" region="$EMB_REGION")
[[ -n "$CONTAINER_ID" ]] || die "no container '$EMB_CONTAINER' — run deploy_container.sh first"
# Owning the whole definition cuts both ways: anything not passed to
# deploy_container.sh falls back to *its* defaults, and DEPLOY_IMAGE_TAG defaults
# to a release tag. Rewiring a container must not also roll its image back to
# that default, which is what this did until a stack deployed on an explicit tag
# came back reporting the release one. So the deployed image is read first and
# handed straight back. Any other non-default knob (memory, mvCPU) has the same
# shape; the image is the one that changes which code runs.
CURRENT_IMAGE=$(scw_json container container get container-id="$CONTAINER_ID" \
region="$EMB_REGION" | "$PYTHON" -c "$FIELD" image) \
|| die "could not read the image deployed on container $CONTAINER_ID"
# One update, not two. Attaching the network separately and then re-running
# deploy_container.sh made Scaleway reject the second call outright:
# "transient state error for resource 'container' ... current_state: updating".
# deploy_container.sh owns the whole container definition anyway — an update
# replaces the environment map wholesale — so the network is handed to it.
DEPLOY_OLLAMA_URL="http://$PRIVATE:11434" DEPLOY_PRIVATE_NETWORK_ID="$NETWORK_ID" \
DEPLOY_IMAGE="$CURRENT_IMAGE" \
bash "$(dirname "$0")/deploy_container.sh" >/dev/null
note "container redeployed on the private network, with OLLAMA_BASE_URL set"
note "image kept at $CURRENT_IMAGE"
cat <<SUMMARY
Phase 4 is deployed.
network $EMB_NETWORK_NAME $NETWORK_ID
security $EMB_SECGROUP_NAME $SECGROUP_ID inbound drop
instance $EMB_INSTANCE_NAME $SERVER_ID $PRIVATE
The instance is billing now (DEV1-M is about EUR 0.02/h). Tear it down when you
stop testing — see docs/COST_KILL_SWITCH.md.
Ollama has no model yet. Pull one through the application, which is idempotent
and the only path that does not need SSH:
python scripts/scaleway/verify_deployment.py <endpoint> --pull-model $EMB_MODEL
SUMMARY
ollama-cloud-init.yamlThe cloud-init that Phase 4 hands to that Instance. Installs Ollama and deliberately pulls no model.
#cloud-config
# Phase 4: an Ollama host that serves embeddings for the LocalChat test stack.
#
# It installs Ollama and stops there. It deliberately does NOT pull a model:
# on the first live run the cloud-init pull failed silently and left a box with
# zero models, and with inbound traffic dropped by design there was no way in to
# find out why. Pulling through the application's own POST /api/models/pull is
# idempotent, streams its progress back, needs no SSH, and exercises the private
# network that has to be proven anyway. See deploy_embeddings.sh.
#
# OLLAMA_HOST=0.0.0.0 is required: Ollama binds to 127.0.0.1 by default, which a
# container on the private network cannot reach. The exposure that would create
# is closed at the network edge instead — this instance sits behind a security
# group whose inbound default policy is `drop`, so 11434 answers on the private
# network and nowhere else.
package_update: true
runcmd:
- curl -fsSL https://ollama.com/install.sh | sh
- mkdir -p /etc/systemd/system/ollama.service.d
- |
cat > /etc/systemd/system/ollama.service.d/override.conf <<'EOF'
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
EOF
- systemctl daemon-reload
- systemctl enable --now ollama
- systemctl restart ollama
# A marker readable from the serial console. If this file is absent, Ollama
# never came up and nothing else on the box is worth reading.
- |
for i in $(seq 1 60); do
if curl -sf http://127.0.0.1:11434/api/tags >/dev/null; then
touch /var/log/ollama-ready
break
fi
sleep 2
done
verify_database.pyA standalone check of a managed database: TLS required, pgvector present, a setting that survives a transaction boundary.
#!/usr/bin/env python3
"""Prove the Scaleway Serverless SQL Database can actually host LocalChat.
Four things decide that, and only one of them is obvious:
1. The credential connects at all, and the connection is really encrypted.
2. ``CREATE EXTENSION vector`` succeeds — a managed Postgres may not permit it,
and the whole retrieval layer depends on it.
3. ``SET hnsw.ef_search`` survives into a *later* transaction. ``src/db/connection.py``
sets it once per physical connection in the pool's ``configure`` callback and
assumes it sticks. Behind a transaction-pooling proxy it does not, and nothing
errors — retrieval just quietly gets worse. This is §4's open caveat.
4. The identity is scoped: it can read and write, and it cannot drop the database.
Run it after ``provision.sh``:
python scripts/scaleway/verify_database.py
Reads the credential from ``PROVISION_ENV_OUT`` (default
``~/.config/scw/localchat-db.env``). Prints no secret. Exits non-zero if any
check fails, so it can gate a deployment.
"""
from __future__ import annotations
import os
import pathlib
import sys
try:
import psycopg
except ImportError: # pragma: no cover - the message is the whole value here
sys.exit("psycopg is not installed: pip install 'psycopg[binary]'")
DEFAULT_ENV = pathlib.Path.home() / ".config" / "scw" / "localchat-db.env"
def load_credentials(path: pathlib.Path) -> dict[str, str]:
if not path.exists():
sys.exit(f"no credential file at {path} — run scripts/scaleway/provision.sh first")
values: dict[str, str] = {}
for line in path.read_text(encoding="utf-8").splitlines():
line = line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, value = line.split("=", 1)
values[key.strip()] = value.strip()
missing = {"PG_HOST", "PG_PORT", "PG_DB", "PG_USER", "PG_PASSWORD"} - values.keys()
if missing:
sys.exit(f"{path} is missing {', '.join(sorted(missing))}")
return values
class Report:
def __init__(self) -> None:
self.failed = 0
def ok(self, label: str, detail: str = "") -> None:
print(f" PASS {label}{f' — {detail}' if detail else ''}")
def bad(self, label: str, detail: str) -> None:
self.failed += 1
print(f" FAIL {label} — {detail}")
def note(self, text: str) -> None:
print(f" {text}")
def main() -> int:
env_path = pathlib.Path(os.environ.get("PROVISION_ENV_OUT", str(DEFAULT_ENV)))
creds = load_credentials(env_path)
report = Report()
print(f"Verifying {creds['PG_HOST']}/{creds['PG_DB']} as {creds['PG_USER'][:8]}…\n")
conn_kwargs = {
"host": creds["PG_HOST"],
"port": int(creds["PG_PORT"]),
"dbname": creds["PG_DB"],
"user": creds["PG_USER"],
"password": creds["PG_PASSWORD"],
"sslmode": creds.get("PG_SSLMODE", "require"),
"connect_timeout": 20,
}
try:
conn = psycopg.connect(**conn_kwargs)
except Exception as exc:
report.bad("connect", str(exc).strip().splitlines()[0])
return 1
with conn:
conn.autocommit = True
with conn.cursor() as cur:
cur.execute("SELECT version()")
row = cur.fetchone()
report.ok("connect", (row[0] if row else "?").split(" on ")[0])
# 1. Encryption. `pg_stat_ssl` is the wrong probe here: it reports the
# pooler's own connection to Postgres, which is internal and shows
# no TLS even when the client connection is fully encrypted. The
# honest test is whether the server refuses an unencrypted one.
try:
psycopg.connect(**{**conn_kwargs, "sslmode": "disable"}).close()
report.bad("server requires TLS", "an sslmode=disable connection was accepted")
except Exception as exc:
first = str(exc).strip().splitlines()[0]
report.ok("server requires TLS", "sslmode=disable is refused")
if "ServerNameIndication" in str(exc) or "hostname wasn't sent" in str(exc):
report.note("Routing is by TLS SNI — the database is selected by the")
report.note("server name in the handshake, so TLS is structural here,")
report.note("not merely advisable. PG_SSLMODE=require is mandatory.")
else:
report.note(first[:100])
# 2. pgvector.
try:
cur.execute("CREATE EXTENSION IF NOT EXISTS vector")
cur.execute("SELECT extversion FROM pg_extension WHERE extname = 'vector'")
ver = cur.fetchone()
if ver:
report.ok("pgvector available", f"version {ver[0]}")
else:
report.bad("pgvector available", "CREATE EXTENSION ran but pg_extension has no row")
except Exception as exc:
report.bad("pgvector available", str(exc).strip().splitlines()[0])
# 3. The caveat. Set it, end the transaction, then read it back on
# what the pooler may or may not consider the same session.
try:
cur.execute("SET hnsw.ef_search = 100")
cur.execute("COMMIT")
cur.execute("BEGIN")
cur.execute("SHOW hnsw.ef_search")
seen = cur.fetchone()
observed = seen[0] if seen else "?"
cur.execute("COMMIT")
if str(observed) == "100":
report.ok("hnsw.ef_search persists across transactions", observed)
report.note("The pool's configure callback is enough; §4's caveat does not bite.")
else:
report.bad(
"hnsw.ef_search persists across transactions",
f"read back {observed!r}, expected '100'",
)
report.note("Set it per checkout or per query in src/db/connection.py.")
except Exception as exc:
report.bad("hnsw.ef_search persists across transactions", str(exc).strip().splitlines()[0])
# 4. Scope: writing is allowed, dropping the database is not.
try:
cur.execute("CREATE TABLE IF NOT EXISTS _localchat_probe (id int)")
cur.execute("INSERT INTO _localchat_probe VALUES (1)")
cur.execute("SELECT count(*) FROM _localchat_probe")
cur.execute("DROP TABLE _localchat_probe")
report.ok("DDL and DML permitted", "create/insert/select/drop table")
except Exception as exc:
report.bad("DDL and DML permitted", str(exc).strip().splitlines()[0])
try:
cur.execute(f'DROP DATABASE "{creds["PG_DB"]}"')
report.bad("cannot drop its own database", "the drop succeeded — the identity is too broad")
except Exception as exc:
report.ok("cannot drop its own database", str(exc).strip().splitlines()[0][:70])
print()
if report.failed:
print(f"{report.failed} check(s) failed — do not deploy against this database yet.")
return 1
print("All checks passed. The database is ready for Phase 2.")
print("Note: verify-full needs Scaleway's CA in sslrootcert; require does not.")
return 0
if __name__ == "__main__":
raise SystemExit(main())
panic_teardown.shThe cost kill switch — deletes everything billable in one project, most expensive first. Dry run unless CONFIRM=DESTROY.
#!/usr/bin/env bash
#
# Scaleway cost kill switch: delete everything billable in ONE project.
# See docs/COST_KILL_SWITCH.md.
#
# Scaleway has no spend cap (§8 of DEPLOYMENT_SCALEWAY.md). A budget alert is an
# estimate that lags the invoice and stops nothing. This is what actually stops
# the burn, and it stops it by deleting — not stopping — because a stopped
# Instance keeps billing for the volumes and the reserved IP it leaves behind.
#
# That is only acceptable because the test stack holds nothing worth keeping:
# the image is redeployable and the data is test data. Do not point this at a
# project where that stops being true.
#
# Usage:
# PROJECT_ID=<uuid> bash scripts/scaleway/panic_teardown.sh # dry run
# PROJECT_ID=<uuid> CONFIRM=DESTROY bash scripts/scaleway/panic_teardown.sh
#
# Without CONFIRM it only lists what it would delete, so the panic path is:
# run it, read it, run it again with CONFIRM=DESTROY.
#
# Deliberately no `set -e`: one failed delete must not stop the sweep. Failures
# are collected and the script exits non-zero with them named.
set -uo pipefail
die() { printf 'error: %s\n' "$*" >&2; exit 1; }
: "${PROJECT_ID:?set PROJECT_ID — the project to empty. This never guesses.}"
CONFIRM="${CONFIRM:-}"
command -v scw >/dev/null 2>&1 || die "scw not found on PATH"
PYTHON=""
for c in python3 python; do
if command -v "$c" >/dev/null 2>&1; then PYTHON="$c"; break; fi
done
[[ -n "$PYTHON" ]] || die "no python3 on PATH"
PROFILE_ARGS=()
if [[ -n "${SCW_PROFILE:-}" ]]; then
PROFILE_ARGS=(--profile "$SCW_PROFILE")
fi
# The default project shares its id with the organisation. Emptying that is a
# different and much larger act than emptying a scoped project, so it is refused
# outright rather than confirmed — there is no flag for it here.
ORG_ID=$(scw "${PROFILE_ARGS[@]}" config get default-organization-id 2>/dev/null || true)
if [[ -n "$ORG_ID" && "$PROJECT_ID" == "$ORG_ID" ]]; then
die "PROJECT_ID is the organisation's default project. Refusing.
Create a scoped project and put the stack there instead."
fi
read -r -d '' READ_ROWS <<'PY' || true
import json, sys
sys.stdout.reconfigure(newline="\n")
scope_key = sys.argv[1]
try:
rows = json.load(sys.stdin)
except Exception as exc:
sys.exit("reply was not JSON: %s" % exc)
if isinstance(rows, dict):
# Some list verbs wrap the array; take the first list-valued field.
rows = next((v for v in rows.values() if isinstance(v, list)), [])
for row in rows:
if not isinstance(row, dict) or not row.get("id"):
continue
print("%s\t%s\t%s" % (row["id"], row.get(scope_key, ""), row.get("name", "-")))
PY
FAILED=()
PLANNED=0
# rows <scope-key> -- <scw list args...>
rows() {
local scope_key="$1"; shift 2
local raw
if ! raw=$(scw "${PROFILE_ARGS[@]}" "$@" project-id="$PROJECT_ID" -o json 2>&1); then
printf 'warning: could not list via "%s" — check by hand:\n%s\n' "$*" "$raw" >&2
FAILED+=("list: $*")
return
fi
printf '%s' "$raw" | "$PYTHON" -c "$READ_ROWS" "$scope_key"
}
# Deleting a resource returns before its network attachments are released, so the
# Private Network — which depends on everything above it having let go — can fail
# with "must be empty to be deleted" while being seconds away from deletable.
# Observed on the first live teardown, 2026-09-05: it failed, then succeeded on a
# second run. Nothing is burning while this retries — a Private Network is free.
# Six, not three: three landed on the last attempt on every live run and ran out
# on 2026-09-14, so the budget sat exactly at the observed release time with no
# headroom. The network is the last thing deleted, so waiting here delays nothing.
PN_RETRIES="${PN_RETRIES:-6}"
PN_RETRY_DELAY="${PN_RETRY_DELAY:-5}"
# kill_with_retry <label> <scw delete args...>
kill_with_retry() {
local label="$1"; shift
PLANNED=$((PLANNED + 1))
if [[ "$CONFIRM" != "DESTROY" ]]; then
printf ' would delete %s
' "$label"
return
fi
local attempt out
for attempt in $(seq 1 "$PN_RETRIES"); do
if out=$(scw "${PROFILE_ARGS[@]}" "$@" 2>&1); then
if [[ "$attempt" -gt 1 ]]; then
printf ' deleted %s (attempt %s)
' "$label" "$attempt"
else
printf ' deleted %s
' "$label"
fi
return
fi
if [[ "$attempt" -lt "$PN_RETRIES" ]]; then
sleep "$PN_RETRY_DELAY"
fi
done
printf ' FAILED %s
%s
' "$label" "$out" >&2
FAILED+=("$label")
}
# kill <label> <scw delete args...>
kill_one() {
local label="$1"; shift
PLANNED=$((PLANNED + 1))
if [[ "$CONFIRM" != "DESTROY" ]]; then
printf ' would delete %s\n' "$label"
return
fi
local out
if out=$(scw "${PROFILE_ARGS[@]}" "$@" 2>&1); then
printf ' deleted %s\n' "$label"
else
printf ' FAILED %s\n %s\n' "$label" "$out" >&2
FAILED+=("$label")
fi
}
if [[ "$CONFIRM" == "DESTROY" ]]; then
printf 'DESTROYING everything billable in project %s\n\n' "$PROJECT_ID"
else
printf 'Dry run for project %s — nothing will be deleted.\n' "$PROJECT_ID"
printf 'Re-run with CONFIRM=DESTROY to actually stop the burn.\n\n'
fi
# Ordered by burn rate. Instances first: a GPU Instance is EUR 0.787/h and up,
# which is more than everything below it combined.
echo "Instances (the expensive ones):"
while IFS=$'\t' read -r id zone name; do
[[ -n "$id" ]] || continue
# with-volumes and with-ip are the whole point: a plain delete leaves block
# volumes and a reserved IP behind, and both keep billing silently.
kill_one "instance $name ($id, $zone)" instance server delete \
server-id="$id" zone="$zone" with-volumes=all with-ip=true force-shutdown=true
done < <(rows zone -- instance server list zone=all)
echo "Serverless Containers (namespace delete removes the containers in it):"
while IFS=$'\t' read -r id region name; do
[[ -n "$id" ]] || continue
kill_one "namespace $name ($id, $region)" container namespace delete \
namespace-id="$id" region="$region"
done < <(rows region -- container namespace list region=all)
echo "Serverless SQL databases:"
while IFS=$'\t' read -r id region name; do
[[ -n "$id" ]] || continue
kill_one "database $name ($id, $region)" sdb-sql database delete \
database-id="$id" region="$region"
done < <(rows region -- sdb-sql database list region=all)
# A managed PostgreSQL instance bills per hour whether or not anything connects —
# unlike Serverless SQL, it never scales to zero. Added 2026-10-06, the day a
# deployment needed one (Serverless SQL cannot create the row-level-security role)
# and this sweep reported "nothing billable" with the instance still running.
echo "Managed PostgreSQL instances (bill hourly, never scale to zero):"
while IFS=$'\t' read -r id region name; do
[[ -n "$id" ]] || continue
kill_one "managed database $name ($id, $region)" rdb instance delete \
instance-id="$id" region="$region"
done < <(rows region -- rdb instance list region=all)
echo "Block volumes left behind (detached volumes bill at full price):"
while IFS=$'\t' read -r id zone name; do
[[ -n "$id" ]] || continue
kill_one "volume $name ($id, $zone)" block volume delete volume-id="$id" zone="$zone"
done < <(rows zone -- block volume list zone=all)
echo "Reserved IPs (a flexible IP bills whether or not it is attached):"
while IFS=$'\t' read -r id zone name; do
[[ -n "$id" ]] || continue
kill_one "ip $id ($zone)" instance ip delete ip="$id" zone="$zone"
done < <(rows zone -- instance ip list zone=all)
echo "Private networks (free, but they hold references that block other deletes):"
while IFS=$'\t' read -r id region name; do
[[ -n "$id" ]] || continue
kill_with_retry "private network $name ($id, $region)" vpc private-network delete \
private-network-id="$id" region="$region"
done < <(rows region -- vpc private-network list region=all)
echo
if [[ "$PLANNED" -eq 0 ]]; then
echo "Nothing billable found in this project."
elif [[ "$CONFIRM" != "DESTROY" ]]; then
printf '%d resources would be deleted. Re-run with CONFIRM=DESTROY.\n' "$PLANNED"
fi
if [[ ${#FAILED[@]} -gt 0 ]]; then
printf '\n%d did NOT go away — delete these in the console now:\n' "${#FAILED[@]}" >&2
printf ' - %s\n' "${FAILED[@]}" >&2
exit 1
fi
if [[ "$CONFIRM" == "DESTROY" ]]; then
echo
echo "Verify with: scw billing consumption list -o json"
echo "Consumption lags, so re-check in an hour before believing it is zero."
fi