Automatic upstream updates for Botensten Mail and iCharles Newsletters

Scope: marclou/mailcheap main -> botensten.com/mail and icharles.com/newsletter-app.
The owner authorized ongoing updates and the second installation on October 17, 2026.
The iCharles Canvas launcher is https://icharles.com/apps/newsletters.
Both services share the reviewed source, with independent databases, cookie names,
signing secrets, sender configuration, webhook endpoints, backups and rollback state. The existing Resend provider, subpath changes,
R2 gateway, allowlist, credentials, subscriber data and unrelated sites remain.

The Mac's launchd checks hourly and once when the job loads. No Codex session or
model is used. The Mac and its logged-in service account must stay available.
No-change runs are quiet. status.json contains the latest outcome; events.jsonl
records meaningful state changes only. No email or external notification is sent.

The shared runner lives in mailcheap-botensten-local/deploy. Its private instances.json
lists both deployment folders. Each instance.json selects its repository, environment,
host, basePath, appPort, gatewayPort, publicPort, process names, archive name and mongod.
The iCharles instance additionally names peerManifest for the reviewed Botensten source.
Missing instances.json retains the original single-service behavior. Never publish real
instance configuration or credentials in a public source archive.

Example instance.json (replace paths and ports for your own installation):
{"repository":"/srv/mailcheap","environment":"/srv/private/icharles.env","appName":"mailcheap-icharles","gatewayName":"icharles-newsletters-gateway","host":"icharles.com","basePath":"/newsletter-app","appPort":3032,"gatewayPort":4192,"publicPort":4319,"sourceArchive":"mailcheap-icharles-source.tar.gz","retireCommunity":false,"mongod":"/srv/mongodb/bin/mongod"}
Set MAILCHEAP_DEPLOY_DIR on each gateway process to its own deployment folder.
The optional Canvas/auth configuration is described in docs/canvas-integration.md.

Files relative to each deployment folder:
  update-mailcheap.py       orchestrator, activation and rollback
  update-candidate.mjs      isolated build and HTTP/asset verification
  update-backup.mjs         consistent MongoDB snapshot before activation
  updater-config.json      private reviewedUpstream boundary (not a credential)
  ../updater/status.json    current status, previous/live revision, log location
  ../updater/runs/          per-run tests, build, checks and recovery receipts
  ../updater/events.jsonl   changes, completions and blockers

Pipeline:
1. Lock against overlapping runs, recover an interrupted activation, and require
   a clean local main. Fetch only the original upstream main. Reject rewritten
   history. Stop if the candidate is identical to the deployed source. Unpublished
   local commits require an exact reviewedSourceCommit in the private config;
   merely committing local work does not authorize this job to publish it.
2. Keep the local customization commits, merging upstream into an isolated main
   checkout with its own frozen npm ci install. Never reset/force the local main.
3. Run the complete original tests and lint, including Resend/subpath regressions.
   Build with production public URLs but a disposable MongoDB and disabled external
   sending/storage keys. Probe actual sign-in, docs, auth discovery, asset URLs,
   protected MCP and rejection of unsigned jobs. No real email is sent.
4. Keep all required source/license/customization files. Scan the public source
   archive for configured credentials. Preserve the complete corresponding source.
5. Make a consistent read-only snapshot of the app database, including index
   metadata, plus a separate private backup of signing secrets. EJSON is round-trip
   validated. This is a pre-update recovery copy, not an offsite recovery service.
6. Preserve the old manifest, source archive and process configuration. Activate
   only MailCheap and its gateway, verify origin routing, authentication rejection,
   public endpoints and exact public source archive. Automatically restore the
   previous app release on a failed activation. Never roll a database back.
7. Fast-forward the clean main checkout after successful live verification. Keep
   the previous release and backups. Remove only the new run's disposable cache.

Automatic review holds:
Merge conflicts, rewritten history, dirty main, insufficient disk space, failed
checks, changed schemas/migrations/authentication/provider contracts, new environment
requirements, package/runtime contracts or deployment configuration stop deployment.
These boundaries require an explicit technical review, not blind conflict resolution.
After review, commit any compatibility adjustment on local main, set reviewedUpstream
to the exact reviewed upstream hash and reviewedSourceCommit to that reviewed local
commit, and run update-mailcheap.py --retry. Never move
that boundary simply to suppress a failure. The unchanged failed revision is not
rebuilt every hour; a new revision/configuration or --retry triggers another attempt.

A failure in one installation is recorded independently; the other still receives
its scheduled check. Changes to the Life OS Canvas host require its separate verified
release procedure and are not overwritten by upstream MailCheap updates.

Current provider limitation: upstream's new quota panel supports SES/PostShiba only.
Resend continues sending through its existing adapter and retains the panel's original
unavailable state, without querying another provider or fabricating quota data.

On the deployment Mac:
  python3 update-mailcheap.py                  check now
  python3 update-mailcheap.py --retry          retry both after addressing a failure
  python3 update-mailcheap.py --deployment /absolute/instance/deploy --retry
                                             retry only the named installation
  python3 test-updater.py                     test safety/rollback boundaries
  cat ../updater/status.json                  inspect status
  touch ../updater/paused                     pause future updates
  rm ../updater/paused                        resume on the next scheduled check

The LaunchAgent is com.botensten.mailcheap-updater. launchctl print gui/$(id -u)/
com.botensten.mailcheap-updater shows the registered job (write it on one line).
Existing prepare-release.py/activate-release.py are historical manual helpers;
use this updater for releases because it has isolated dependencies and rollback.

Production credentials are copied unchanged only after the candidate passes.
Never publish .env files, database backups, state/run logs or private updater-config.
Application rollback restores binaries/configuration, not subscriber data. A database
restore is a separate reviewed operation. Release/backup pruning is not automatic;
the free-disk guard stops safely before storage is exhausted. An update may briefly
restart the application; signed queue retries remain handled by the original app.
