What each session did, what it found that the plan did not predict, and what it cost.
One entry per session, oldest first, exactly as kept in the repository — with one difference: resource identifiers, private addresses and endpoints are replaced by <id>, <address> and <endpoint>. Everything described here is torn down; the placeholders keep this page about what happened rather than where.
A record of every session that touched the Scaleway deployment: what was done, what it found, and what it cost. DEPLOYMENT_SCALEWAY.md is the plan and the reference; this is the history of running it.
It exists because the stack is ephemeral. Deploy, test, destroy is the pattern, so nothing in the account is evidence of what happened — the resources are gone by the time anyone asks. Without a log, each rebuild starts from the same blank page the last one did.
No secrets, ever
No key, password, token, connection string or secret value appears in this file, and none may be added to it later. Incidents involving secrets are recorded by what happened and what was done about it, never by what the value was. Credentials live only in mode-600 files outside the repository, written by
provision.shanddeploy_container.sh.Resource IDs, hostnames, prices and timings are fine, and are what make an entry useful.
Done. Authenticated the CLI via browser SSO. Rewrote the billing guardrail against the real CLI surface. Created the scoped project, the Serverless SQL Database and a project-scoped IAM identity. Built and tested the cost kill switch. Deployed the container, added a CPU Instance serving embeddings, and proved the document path end to end. Tore it all down again.
Found.
Half the scw billing surface the guardrail assumed |
did not exist — no budget-alert list, and a budget has no name |
scw sdb, and Terraform's max_cpu = 15 default |
wrong command; the CLI requires both CPU bounds, so the trap is Terraform-only |
hnsw.ef_search through the pooler (§4's open caveat) |
survives — the feared defect does not exist |
| TLS to the database | structural, not hardening: routing is by TLS SNI |
| "Retrieval works without Ollama" | false — ingest embeds through Ollama, so nothing is stored at all |
| The version endpoint | not on /api/status; it is admin-only on /api/settings/stats |
memory-limit-bytes |
needs a G/GB unit, not bytes |
| Compressed image size | 2.99 GB over 10 layers, of which one layer is 2.96 GB |
| GPU price | L4-1-24G is €0.787/h ≈ €575/month |
Kill switch, first live run. Five of six resources gone on the first pass; the private network refused because deletes release their network attachments asynchronously. It named the survivor and exited non-zero, which is the design. A retry was added afterwards.
Cost. €0.13 for the whole exercise.
Done. Fixed two generic application defects the deployment exposed. Scripted phases 2 to 4 from what the manual run actually did. Gave the landing host TLS and a security group of its own. Rebuilt the entire stack from nothing using only the scripts, which is what exposed the rest of this entry.
Application defects fixed.
/api/health echoed a boot-time flag and reported the database healthy through a total
outage. It now probes the pool, cached for five seconds.UPLOAD_FOLDER is a staging area, not a
store — a correction to an earlier claim in this project's own documentation that
uploads were lost on restart and needed object storage. They are not, and it does not.Script defects, all found by running from nothing.
scw takes secrets as command arguments, and
the error handler printed the failing command. One transient-state rejection disclosed
every application secret and the database credential. All were rotated immediately.
The error path now redacts them; the underlying exposure through the process table is
documented in §10c and accepted for a single-operator laptop, not for shared CI.deploy_embeddings.sh raced itself, issuing a second container update while the first
was still applying.Landing host. atospoc.solbyco.nl served a placeholder over plain HTTP on Scaleway's
Default security group, which accepts all inbound — port 22 open to the internet. Now:
Let's Encrypt with renewal proven by dry-run, and a security group allowing 80 and 443
publicly with SSH from the operator's address alone. Enforcement on the running instance
was verified by listening on an unlisted port and confirming it unreachable.
The UI was unusable, and the assets were fine. /static/css/style.css returned 200,
text/css, 37012 bytes over HTTPS — while the page asked for it over http. Starlette's
url_for builds an absolute URL from the request, and behind TLS termination that is
http, which browsers block as mixed content. Every stylesheet and every script refused;
the "Loading…" that never resolved was the same cause, since the JavaScript never ran.
Fixed by referencing same-origin assets root-relative, which needs no proxy trust at all.
Kill switch, second live run. Clean, exit 0 — and the private network deleted itself on attempt three, so the retry added after the first run did its job unattended.
Done. Found the 2026-09-06 stack still standing and still billing. Tore it down with
the kill switch, rebuilt the whole thing from the three scripts onto the current main
build, verified it end to end, and ran the rate-limiting check §11 had been carrying since
the plan was written. Then found chat broken on the running stack, traced it to a defect in
the application, wrote it up, and tore the stack down.
Found.
| The previous session's stack | still running, 35 hours after this log recorded a clean teardown. The teardown happened; the rebuild that followed it was never written down |
deploy_embeddings.sh |
silently rolled the container back to the default image tag. Phase 4 rewires the container by re-running Phase 2, passing it two variables — every other knob, DEPLOY_IMAGE_TAG among them, fell back to Phase 2's default of 3.0.0. The inner run is redirected to /dev/null, so nothing said so |
verify_deployment.py's image check |
cannot see that. app_version is 3.0.0 in every build since the tag, so "deployed image is the expected one" passed against the wrong image |
X-Forwarded-For at the edge (§7, §11) |
Everyone shares one bucket. Measured, not inferred — see below |
| The kill switch, third live run | clean, exit 0. The Private Network again needed the retry, landing on attempt 3; the instance delete took its volume and its IP with it, so the later passes found nothing to do |
| The stale-credential fix | works. The database was recreated with a new host while its API key survived, and provision.sh reused the stored secret and rewrote the connection details — the exact combination that broke on 2026-09-06 |
| Chat, on the verified stack | broken, and nothing said so. An embedding model had been made the active chat model — see below |
| The mixed-content fix (#364) | holds in the real environment: /login on the deployed image references /static/css/style.css root-relative, where the 3.0.0 image emitted an absolute http:// URL |
The image revert, and why nothing caught it. Phase 2 was deployed on sha-359069a,
Phase 4 reported success, and the container was afterwards running 3.0.0 — a release tag
four fixes behind main. Two independent guards should have caught it and neither could:
the deploy script prints nothing about the inner run, and the verifier's version check reads
app_version, which is a constant in config.py rather than anything derived from the
build. deploy_embeddings.sh now reads the deployed image and hands it back, prints what it
kept, and tests/unit/test_deploy_scripts.py asserts the update carries it. The verifier is
deliberately left as it is: over HTTP there is nothing to compare against, since the
application does not know which image it came from. Treat its version row as "an app
answered", not "the right build is deployed".
The rate-limiting check (§11, §7). 14 login attempts with one fixed
X-Forwarded-For gave 9 x 401 then 429 — the 10/min limit. Eight more with a rotating
header, and three with no header at all, stayed 429 throughout. A forged header therefore
buys nothing: TRUSTED_PROXY_IPS is empty on this deployment, so no ProxyHeadersMiddleware
is mounted and the limiter keys on Scaleway's ingress address for every caller. D8's
shared-bucket failure mode is the live one, which is the outcome §7 recommends accepting.
What the check does not settle is §7's other half — whether the edge would pass a forged
header through if the app were told to trust it. It cannot: with no proxy trusted, "the edge
stripped it" and "the app ignored it" look identical from outside. Settling that means
setting TRUSTED_PROXY_IPS deliberately and repeating the probe, which is worth doing only
if anyone proposes to fix the shared bucket that way.
The billing guardrail warns at €20, not €40 — and until today it warned nobody at all.
Two things about budget-alert that this session established:
threshold is a percentage. Nothing renders a unit — the API returns a bare
"threshold": 40, and consumption_limit comes back with an empty currency_code — so
looking at it in the console or the API settles nothing. The constraint does:
threshold=101 and threshold=150 are both refused with "must be lower than or equal
to 100". The guardrail created on 2026-09-05 therefore fires at 40% of €50 = €20,
and this project's documentation called it €40 in five places for three days. Earlier and
more conservative than intended, which is exactly why nobody noticed.notifications: []), so firing it would have
produced nothing anywhere. budget-alert-notification create accepts
email-addresses and sms-phone-numbers alongside webhook-urls; an email notification
is now attached.The webhook is dropped, and the payload check with it. The shape only matters to code
that parses it, and that code should not exist: a consumer that tears a stack down on an
unauthenticated POST is a liability, and a budget alert lags consumption by hours, so it can
never be the brake — panic_teardown.sh with a human in front of it is. Email needs no
endpoint, no public surface and no third party.
Whether the alert actually delivers is still unproven. A 1% alert (€0.50) against ~€5 of consumption, with email attached, produced no mail in 2 h 33 min. Two candidates, neither established: the evaluation runs on a slow cadence, or an alert fires on a crossing and one created above the line never fires at all. If it is the second, the guardrail is a tripwire that must be armed before the spend — and this test was the wrong shape rather than the notification being broken.
Chat was broken, and the cause is in the application, not the deployment. Every chat
request came back {"error": "GenerationError", "message": "Failed to generate response"}
while upload and retrieval worked and /api/status reported ready: true. The reason
appeared in one place only, the log:
ERROR src.ollama_client | Ollama API error 400: {"error":"\"nomic-embed-text:latest\" does not support chat"}
The active chat model was the embedding model. _init_ollama_service() picks an active
model at startup when none is set; get_first_available_model() filters embedding families
out and then — this is the defect — falls back to the unfiltered list when the filter leaves
nothing. Phase 4 pulls nomic-embed-text and deliberately nothing else, so the fallback is
guaranteed to fire on exactly the stack this project builds. It prefers a wrong answer to no
answer, and the wrong answer reaches the user as five opaque words.
Pulling llama3.2:1b afterwards did not fix it: the active model is chosen only when unset,
so it stayed on the embedding model. POST /api/models/active did fix it — the same
question then streamed tokens and cited the right chunks in 32 s.
Written up in TROUBLESHOOTING.md and beside Phase 4 in the plan. Not fixed in code — the fix is a two-line change (drop the fallback, or refuse to make an embedding model active) plus a decision about what the app should do when it has no model it can chat with, and that decision is worth making deliberately rather than at the end of a deployment session.
Fixed 2026-09-09, after this session. The decision turned out to be already made and already in the code:
api_chatreturns a 400NoModelConfigurednaming the remedy when no model is active, and the fallback was the only thing routing past it. Dropping it was enough./api/statuswas corrected in the same change:readynow includes having a model that can chat, so the stack no longer reportsready: truefor a chat it cannot do./api/healthdeliberately still tracks only the backing services — a container must not be restarted for having no chat model.tests/unit/test_active_model_is_chat_capable.pyholds both. The operator note above still stands — a model pulled into a running instance is not selected automatically.
A second thing the chat showed, and it is not a defect. With llama3.2:1b active, the
model answered that the canary phrase "is not mentioned in the provided document" — while
the chunk containing it was the second source it cited. Retrieval did its job; a 1B model on
three vCPUs did not. That is Phase 5's question, and this is not the instrument for it.
What was built, and then destroyed.
project localchat-test <id>
database localchat <id> ready, scaled to zero
namespace localchat <id>
container localchat <id> sha-359069a
endpoint <endpoint>
network localchat-backend <id>
security localchat-ollama <id> inbound drop
instance localchat-embeddings <id> DEV1-M, fr-par-2, <address>
Every Phase 3 check passed on this stack, ingest and semantic retrieval included, and chat worked once the active model was corrected by hand.
Torn down at the end of the session, and it took two runs. The instance, the namespace
and the database went on the first pass; the Private Network did not, and this time the
three bounded retries were not enough — the script named the survivor and exited 1, which
is the design. A second run deleted it and exited 0. Every resource type in the project then
listed zero. So three attempts at five seconds is the usual case, not a guarantee:
PN_RETRIES and PN_RETRY_DELAY exist for that, and re-running remains the first thing to
try. Nothing was billing while it retried — a Private Network is free.
Cost. EUR 1.48 in the project this billing period, covering the stack found running and
this DEV1-M 0.83, database 0.31, IPv4 0.21, block storage 0.06, containers 0.07. The
container line is small only because the Serverless Containers free tier absorbed 2.25 of
2.30 — and memory has now passed 400,000 GB-s, so the tier is spent for this month and the
next day of min_scale=1 bills in full.
Done. Merged the three PRs left open by the previous session and fixed the two application defects it had found and deferred. No stack was built. Nothing in Scaleway was created, changed or destroyed, and nothing billed.
This is the first entry for a session that touched the deployment only through the repository. It is here because the 2026-09-08 entry recorded a defect as open, and the record of a finding is incomplete until the record of its fix sits beside it.
| PR | What landed |
|---|---|
| #368 | Phase 4 no longer rolls the container back to the default image; the 2026-09-08 findings and plan corrections |
| #367 | alembic 1.19.1 -> 1.19.2, pypdf 6.16.2 -> 6.17.0, grouped into one PR as intended |
| #369 | An embedding model can no longer become the active chat model, and /api/status stops claiming a readiness it cannot deliver |
Found.
| The decision the last entry deferred | did not exist. It recorded the chat fix as "a two-line change plus a decision about what the app should do when it has no model it can chat with". The decision was already made and already in the code: api_chat returns a 400 NoModelConfigured naming the remedy. The fallback was the only thing routing past it. It was the two-line change, and no more |
/api/status's ready |
the same lie one layer up. Computed as Ollama up AND database up — both true on a box holding only an embedding model — so a Phase 4 stack reported ready: true for a chat whose every request is a 400. It now also requires an active model |
/api/health |
never read that flag, which is the only reason correcting it is safe. compute_health_status derives from the database and Ollama directly. Had it read ready, a Phase 4 stack would have gone unhealthy and the container would have been restart-looped for the crime of having no chat model — turning an honest status field into an outage |
verify_deployment.py |
reads /api/health and /api/settings/stats, never /api/status. The Phase 3 gate is untouched by the change |
startup_status["ready"] |
write-only. Initialised in app_fastapi.py and written in app_bootstrap.py, monitoring.py and services/chat.py; read by nothing in src/. Only two tests assert on it. Left alone deliberately — it is pre-existing, and this was not its cleanup |
| Why the fallback survived a green suite | no test had ever installed only embedding models. The two that existed covered "a chat model is present" and "the list is empty", so a filter with a fallback that defeats it looked covered. That is the MISSING-NEGATIVE-SPACE shape .claude/rules/testing.md names |
None of this has run on a Scaleway stack, and that is the point of saying so. The fixes are proven by nine unit tests, each confirmed red with its fix reverted, and by CI. That is evidence about the code; it is not evidence about the deployment. §10b's rule is explicit — whose behaviour is this, and did I watch it? — and the honest answers here are "the application's" and "no". §11 carries the row; the next stack settles it.
The specific thing still unwatched: that a Phase 4 stack now boots with no active model rather than a wrong one, and that ingest and retrieval are unaffected by that. Every part of it follows from code that is tested, and the whole of it has never been observed together on a real container against a real Ollama box.
Cost. EUR 0.00. Nothing was running.
Done. Merged the three PRs left open (#370, #371, #372 — the log entry, a pip group
carrying onnxruntime, and the codeql-action set) and deleted twelve merged branches that
had outlived their PRs. Rebuilt the stack from the scripts onto sha-af6012b, which is that
main. Settled all three of §11's "needs the next live stack" rows — the ones #369 created
on 2026-09-09 — by watching them. The stack was left standing for the operator to look
at, then torn down at the end of the session on their word.
project localchat-test <id>
database localchat <id>
namespace localchat <id>
container localchat <id> sha-af6012b
endpoint <endpoint>
network localchat-backend <id>
instance localchat-embeddings <id> DEV1-M <address>
models nomic-embed-text, llama3.2:1b (active)
The three rows, watched.
| §11 row | Expected | Observed |
|---|---|---|
| A Phase 4 stack boots with no active model | ready:false, active_model:null, ollama:true, database:true; /api/health healthy |
Exactly that. /api/models listed one model, nomic-embed-text, and nothing had made it active |
| Ingest and retrieval are unaffected | The Phase 3 gate passes before any chat model exists | Passed: pull, ingest, semantic retrieval at similarity 0.4851 — with active_model still null |
| Chat works once a model is pulled and made active | Answer cites the ingested document | llama3.2:1b pulled in 13 s; POST /api/models/active flipped ready to true; the answer quoted the canary and cited deployment-probe.md. TTFT 33.9 s on the CPU box |
Found.
deploy_container.sh |
declared the container dead while it was still starting. Scaleway reported error ("unable to start OR not listening on port 5000") at 12:11:21; the script broke its wait loop on that first sighting and exited 1. The container was ready at 12:11:36 and /api/health answered at 12:11:43. error is transient while a slow first boot fails its early liveness probes. Fixed in this PR: three consecutive polls now, with tests |
| The first Ollama box | never came up. 26 minutes after power-on, /api/tags was refused — not dropped — on both the private and (via a temporary rule) the public address, so the SYN reached the kernel and nothing was listening. A reboot changed nothing, so the unit was never installed: cloud-init failed at or before install.sh. The user-data was on the server and intact. Deleted and rebuilt; the second box answered in 2 min 30 s. The cause was not read, and the next row is why |
"The serial console is the way in" (§5, deploy_embeddings.sh) |
Almost certainly not true, and never tried. The image is Ubuntu Jammy cloud, which sets no password, and a serial console needs one. The only SSH key on the account (Scaleway-poc) is registered in the default project, and Scaleway injects keys per project, so the box in localchat-test had no authorised key either. Both ways in were closed by construction. A working way in is: a key registered in localchat-test, plus a temporary security-group rule on 22 from one IP — the same shape as the diagnostic used above. Whether to register one is in §12 now |
| The cloud-init the box receives | carries \r. core.autocrlf=true on this checkout makes every file under scripts/scaleway/ CRLF, and deploy_embeddings.sh uploads the yaml verbatim, so the server held #cloud-config\r\n. Probably harmless — cloud-init matches the header with startswith, YAML normalises line breaks, and this same file built the 2026-09-08 stack — so it is not established as the cause above. Fixed regardless with a .gitattributes forcing LF on the directory: a Linux-bound file should not depend on the checkout it left |
budget_mb: 0 on every model |
the fit check measures the wrong machine. /api/models computes the budget with detect() inside the container, whose CPU backend reports nothing to spend. On a topology where Ollama is elsewhere — which is every Scaleway topology — every model not already loaded shows fits:false with a reason naming the container's memory. Cosmetic: load_model_guard is called by nothing in src/, so no activation is refused by it. Left alone; recorded |
| The 1% budget alert from 2026-09-08 | never delivered, six days on, with the consumption standing above its line the whole time. So an alert fires on a crossing, not on a state: the guardrail is a tripwire that has to be armed before the spend. The €50/40 % alert was created on 2026-09-05 with nothing spent, so it is armed correctly; the 2026-09-08 test design was what was wrong |
| The two temporary rules | were both created — the first create-rule call also succeeded; its JSON is nested under rule, which the parser missed. Both deleted; the group is back to inbound-drop only, verified |
Torn down at the end of the session. The kill switch, fifth live run: clean, exit 0, six resources, the Private Network again on attempt 3 — that retry has been needed on every run since it was added. Every resource type in the project then listed zero.
Cost. €1.89 in the project for the billing period, against the €1.48 the 2026-09-08 entry recorded for the same period — about €0.41 for this session, before the lag catches up. The stack ran for roughly three hours, most of it standing for a look rather than being tested.
Done. Rebuilt the stack onto sha-37c7bab — main with #373, #374 and #375 merged an
hour earlier — ran the Phase 3 gate, the status and chat checks from the afternoon entry,
confirmed the two Docs-viewer fixes are what the image serves, and tore it down. The point
was to run the afternoon's fixes through the whole chain unattended, and it did.
database <id> namespace <id> container <id> sha-37c7bab
instance <id> DEV1-M <address> network <id>
| Afternoon | Evening | |
|---|---|---|
| Phase 2 wait loop | quit on a transient error; container fine 15 s later |
waited it out — ready after 4.5 min, no hand on it |
| Ollama box | the first never started; rebuilt | up on the first poll, about 6 min after creation, LF cloud-init |
| Phase 3 gate | pass, similarity 0.4851 | pass, similarity 0.4851 |
/api/status before / after activation |
ready:false, null → true, llama3.2:1b |
same |
| Chat | 36 s, canary quoted, cited | 24 s, canary quoted, cited |
| Docs viewer | fixes not in the image | docs.js carries the link rewrite and style.css the sticky rule — served, not just merged |
Found.
| The kill switch's retry budget | ran out. PN_RETRIES defaulted to 3 with 5 s between, 10 s in all; every live run since the retry was added had landed on attempt 3, and this one needed more. The script named the survivor and exited 1, as designed; a second run deleted it. A default that equals the observed need has no headroom — raised to 6, with a test that fails at 3 |
| One clean box start | is not evidence about the LF change. The afternoon's failure was never read, so nothing links it to \r. What two runs in one day do establish is that the chain runs from nothing without intervention |
Torn down. Two runs of the kill switch: the first left the free Private Network, the second removed it. Every resource type in the project then listed zero.
Cost. €2.01 in the project for the period, against €1.89 read after the afternoon teardown — about €0.12 for this run, before the lag.
Afterwards. The §12 decision the afternoon left open was taken: the operator's public
key is registered in localchat-test as localchat-operator, so the next box that fails
can be read instead of replaced. Nothing was deployed to try it; §11's row stays open.
Done. Deployed main for a hands-on test: Phase 1 from provision.sh, Phase 2 from
deploy_container.sh onto sha-19db1c6, Phase 4 from deploy_embeddings.sh. The database
came up down. After two fixes the stack ran on sha-921c1e4 against a managed
PostgreSQL instead, passed the Phase 3 gate (ingest, semantic retrieval at 0.4851) and
answered a RAG question citing deployment-probe.md — 32 s to first token on
llama3.2:1b on the DEV1-M. The maintainer judged that too slow to test anything
properly; the next test is on a GPU. Torn down the same day.
Found.
| Serverless SQL cannot run P2-1b | Row-level security (#400–#407, after the last deployment on 2026-09-14) switches every scoped transaction into localchat_scoped. Serverless SQL's IAM identity may not create roles at all (permission denied to create role), so the role can never exist and the app reports the database down. No Serverless SQL deployment can run current main. |
| D4's premise is gone | D4 chose Serverless SQL because Managed PostgreSQL lacked pgvector. A db-dev-s PostgreSQL 16 instance created today installed pgvector 0.8.2, and its user has CREATEROLE. That is D4's own reversal condition |
| The scoped-role grant only ever worked for a superuser | On the managed database, GRANT localchat_scoped TO CURRENT_USER is refused (cannot use special role specifier), and a plain grant of the membership the creator already holds leaves it without SET, so SET LOCAL ROLE was denied. Compose and CI connect as postgres, which needs no membership. Fixed in #430: GRANT … WITH SET TRUE to the user by name, at boot and in 0018 |
| A CVE published mid-session blocked every PR | pip-audit failed on CVE-2026-104874 in multidict 6.7.1; #431 moved it to 6.9.1 before #430 could merge |
| A push event never fired | The #428 merge produced no Tests and no Publish run, so main had no image; merging #429 republished. The same dropped-event shape as #426's PR the day before |
| The verify gate always failed the version check | Its default was a literal 3.0.0; the app has been 3.1.0 since 2026-09-17. It now reads APP_VERSION from the checkout |
| The kill switch missed the managed database | It sweeps Serverless SQL but not rdb instances, and would have reported "nothing billable" with one running. Deleted by hand today; the script now covers them |
An incident. Inspecting the CLI profile with scw config info printed the account's API
secret key into the working session's transcript. Nothing was published, but by §10c's rule
a secret that reached a transcript has leaked: the key is to be rotated.
Not done. provision.sh still provisions Serverless SQL. The managed database was
created by hand (db-dev-s, PG 16, backups off, password generated into a 600-mode file),
and moving the script to it is the first step of the next deployment.
Torn down. The managed database by hand, then the kill switch: instance, namespace, Serverless SQL database, private network. A dry run afterwards reported nothing billable.
Cost. Not yet read — consumption lags by about an hour.