Cron jobs

The IBA API ships a set of scheduled background jobs: currency-expiry alerts, payment reminders, and a data-integrity safety net. Each job is a standalone Node script under api/src/crons/, registered as a row in the op_cron_configs table. This page covers what runs, why, and how it’s triggered.

This is the conceptual reference. The auto-generated endpoint schema for the manual trigger lives at API → IBA API → Crons; don’t hand-edit that page — it’s regenerated from the Bruno collection.

How crons run

Scheduling and execution are two decoupled halves:

1. Scheduler — a Cloudflare Worker. workers/cron-scheduler makes one authenticated GET to the API’s trigger endpoint for the due job. Schedules live in wrangler.toml; CRONS_API_KEY is a Worker secret delivered by an Infisical Secret Sync scoped to that one key.

   crons.yml is manual-dispatch only — the break-glass trigger for any job when the Worker is unavailable. Do not re-add a schedule: block to it: a job scheduled in both places fires twice, which for payment_reminder means duplicate member emails.

2. Executor — the API. GET /crons/run/:cron_name looks the job up in op_cron_configs, runs node <cron_path> as a child process via exec() inside the API App Platform container, and writes a row to op_cron_logs when it finishes.

A third mechanism exists but is dormant: the in-process node-cron scheduler (api/src/crons/index.js) that cronInit() would start is commented out in api/index.js.

schedule_pattern in op_cron_configs used to be documentation only. It is now load-bearing: the cronFreshness health check measures each job’s last successful run against it, so a pattern that disagrees with the scheduler produces a false “late” or “missing” alert. All three — wrangler.toml or crons.yml, SCHEDULE_TO_CRON in the Worker, and schedule_pattern — have to agree.

Rollout status (2026-09-23). Complete. All eight scheduled jobs, health included, fire from the Cloudflare Worker. Both crons.yml and health-monitor.yml are manual-dispatch only.

The test_cron canary ran 64 of 64 slots over 15.75 hours, mean dispatch lag 7.7s, max 12s, zero missed. It has been retired; its op_cron_configs row stays active as a fixture for manual triggering.

History. The scheduler used to be a hand-maintained crontab on the Dev-DB-Redis DigitalOcean droplet that curled the same endpoint, with the token in cleartext on the box. The GitHub Actions workflow replaces it: version-controlled, secret-managed, no SSH.

The last job to leave was the hourly health monitor, which ran as a standalone app under that crontab until 6 Aug 2026. A monitor must not share fate with what it monitors — on the droplet, a droplet failure took the alerting down with it, silently. See Health monitor below.

Why the second move. GitHub’s scheduler is not late occasionally; it is late constantly. Measured over eight days in September 2026, every daily job ran 4-6 hours behind — payment_reminder went out at 10:40-11:34 UTC against an 06:00 schedule, and db-backup at ~07:13 against 02:00. The hourly health workflow managed 4-7 runs a day against 24.

One cause, two symptoms: a constant ~5h queue delay still lands a daily job before its next window, so it runs, merely late. On an hourly schedule those same hours swallow five windows, and GitHub drops rather than stacks a delayed run. Which is why the workflow best placed to notice this is the one the fault silences.

Cloudflare, for comparison, has fired the canary within 7 seconds of the scheduled minute on every run.

Either way: a schedule does not imply execution. Check op_cron_logs, or let cronFreshness do it for you.

Because each job is launched as node <script>, every handler bootstraps its own environment (pulling secrets from Infisical when not in production), guards on IS_PRODUCTION === 'Yes' before it mutates anything, and exits 0 on success / 1 on failure.

At a glance

Cadences are UTC. The name is the value passed to /crons/run/<name>; the currency names carry a _currency suffix. Fired by is mid-migration — see Rollout status.

reconcile_parent_skills, health and chatbot_ingest are registered via committed migrations (0003, 0007, 0016). The currency and payment rows pre-date the migration system and live directly in the production op_cron_configs table.

health has its own workflow rather than a schedule entry in crons.yml. It runs 24 times a day against the others’ daily-or-rarer cadence, so sharing a workflow would bury them in its run history, and a monitor’s failures want their own notification stream. It also runs at :07 rather than the top of the hour — GitHub documents scheduled runs as delayed under load, and the start of every hour is the worst window for it.

reconcile_parent_skills was previously dormant. Under the old droplet crontab it had no trigger line (and cronInit() is off), so until the GitHub Actions workflow only the one-shot back-fill in migration 0001 had ever run. The workflow now schedules it daily at 04:00 UTC (0 4 * * *), closing that gap.

Currency-expiry alerts

The four currency jobs (trainer, coach, instructor, military) share one engine — CurrencyCronClass in api/src/crons/currency/index.js — and differ only in the role they target. A fifth currency job, flyer (regular flying members), is wired separately to flyer.service.js under the newer api/src/features/shared/currency/ implementation rather than CurrencyCronClass. For members whose currency is expiring or has expired, the job sends localised notifications (Mandrill email + app push) to a fan-out of recipients:

• the member themselves, in their own language;
• their manager(s), grouped by tunnel (military skips this — it has a different approval structure);
• the regional ops team (USA / EU / AU split);
• admin (info@tunnelflight.com).

It runs in two modes driven by how far out the expiry is: a renewal reminder ahead of the deadline, and an expired notice on/after it. When configured to, it also disables the lapsed account. The instructor variant adds a lastSafetyPeriodActive() check so a still-valid safety period doesn’t get revoked by accident.

Payment reminders

payment_reminder queries members with upcoming payment deadlines (joining fees where status is succeeded against fees_mapping where the mapping is Active), computes days remaining, and sends one of three tiered notices: 31 days (last month), 7 days (last week), and overdue. A single run handles all three tiers — the tier is chosen per member from the days-remaining figure, not from separate schedules.

reconcile_parent_skills

The daily safety net behind the parent-skill auto-approval work (TNBUGS-1728). It delegates to ReconciliationService.run(), which heals any member-upgrade rows left inconsistent, and logs the count it fixed. Like the others, it only mutates data in production. This is the one cron registered through a database migration, so it travels with the code.

Health monitor

health is the odd one out: it doesn’t act on member data, it watches the platform. Fourteen checks run per pass — API / WWW / Admin endpoints, MySQL, Redis, the S3 backup, Stripe webhooks and payment validation, Mandrill, Twilio, the DigitalOcean droplet and App Platform APIs, DataDog ingestion, and dailyCron and cronFreshness. Each writes a row to op_health_checks tagged with a shared group_id, so one pass reads as a unit.

One failing check never stops the rest — a broken Stripe webhook must not hide a broken database — and the script exits 1 if anything failed, which is what op_cron_logs records.

cronFreshness is the one that watches the schedulers. It reads every active op_cron_configs row and reports both missing (no run within the expected interval) and late (the last run started well after its slot), measured against schedule_pattern.

It counts runs at any status, not just successes. “Did the scheduler fire it” and “did it work” are separate questions, and the second is already alerted per-run by postCronAlert. Measuring on success conflated them and closed a loop: cronFreshness failing the health run meant health recorded no success, so the next run reported health as never having run, with the age climbing indefinitely. health is now excluded outright — a check cannot meaningfully assess the run it is part of. Lateness needs its own verdict because the older dailyCron — two hardcoded job ids, asking only “did it run yesterday” — passes cleanly while a job runs five hours late every day, which is how the GitHub drift went unnoticed for two months. Jobs on a monthly or minutely pattern are skipped rather than guessed at.

Nothing watches the watcher

health moved to the Worker on 2026-09-23 along with everything else. GitHub was delivering 4-7 of 24 hourly runs, which is poor monitoring of S3 backups, Mandrill, Twilio and the databases — worse than the risk the split avoided.

The consequence is explicit: the monitor now shares fate with its scheduler. If the Worker stops, the health checks stop with it, and the failure mode is silence — the same shape as the droplet arrangement health was moved off in August, with a different provider.

Nothing inside this system can detect its own absence. cronFreshness reports a job that did not run only if something runs it, and it excludes health for exactly that reason — a check cannot assess the run it is part of. Closing this needs an external dead-man’s switch: the Worker pings a third-party endpoint each run, and that service alerts when the pings stop. Tracked as a follow-up.

Until then the practical signal is the daily digest. If it stops arriving, something upstream of every job has failed.

Alerting

Everything goes to the crons Slack channel. There is no email — the Mandrill summary was retired in TUN-845 because it sent to TO_EMAIL, which was unset in production, so it fell back to a hardcoded personal address that nobody on the team was reading.

• On failure, a summary naming every failing check and its reason.

• Once every 24 hours, a digest of every scheduled job — run count, last status, duration, and what it did:

  :clipboard: Daily cron digest — 24h to 22:10 UTC
  :white_check_mark: Health checks 15/15 passing
  
  :white_check_mark: `payment_reminder` — 1 run, last 06:04, 12s — {"payments":38}
  :white_check_mark: `reconcile_parent_skills` — 1 run, last 04:00, 42s — {"reconciled":0}
  :white_check_mark: `flyer_currency` — 1 run, last 08:31, 1m10s — _no summary_
  :zzz: no runs: `trainer_currency`

  It replaced a health-only all-clear, under which seven of the nine jobs were invisible unless they broke and “no news” could not be told from “nothing ran”. It fires regardless of the health verdict — a day with a failing check still wants one — and its cadence is tracked by a slackCronDigest row, because the script is a fresh process each run with nowhere else to remember.

A failing run also produces the generic Cron failed: health alert described in Run outcomes & failure alerts. That overlap is deliberate — it’s the backstop for a crash that happens before the summary can be posted.

Skips expire

Some checks don’t run every pass. A skip is legitimate while the check is still attempting on its own cadence, but a check that reports skipped forever is indistinguishable from a healthy one, because only failures alert. Any check whose last real verdict is more than 48 hours old is therefore promoted to a failure.

This exists because of a real two-week outage. twilioSms gated itself on getHours() === 0. The droplet’s 0 * * * * crontab hit that window every night; GitHub’s scheduler delays runs 25–58 minutes and drops the 00:07 slot outright, so across 200 consecutive runs not one landed in hour 00 UTC. The check reported skipped every time, and its underlying failure went unreported until someone noticed the alerts had stopped.

The rules now live as pure, unit-tested functions in crons/health/domain/staleness.js. Periodic checks are paced off their last real attempt, not the wall clock — and off the last attempt rather than the last success, so a failing check alerts once per interval instead of once an hour.

Don’t gate a check on wall-clock time. Anything of the form “only run during hour N” assumes a punctual scheduler, and this one isn’t. Pace off the last recorded attempt instead.

op_health_checks retention

The table is append-only and nothing in api/, admin/ or www/ reads it, so it grew past 124,000 rows unchecked. Each run now prunes rows older than 90 days (HEALTH_CHECK_RETENTION_DAYS) in 5,000-row batches — batched rather than one large DELETE so a backlog drains without holding locks. That holds the table near ~34k rows.

Notes are kept small on purpose. The DigitalOcean checks used to store every app’s id, URLs, timestamps and a live memory/CPU sample — about 1.5KB a row, and over half of everything in the table — for data nothing reads and that DO’s own monitoring page already holds. They now record names and status only, which also saves 16 DigitalOcean API calls per run.

Configuration & logging

• op_cron_configs — one row per job: name (unique), cron_path (relative to api/src/crons/), schedule_pattern, is_active.
• op_cron_logs — one row per run: cron_config_id, start_at, end_at, duration_seconds, status, summary. Written by the HTTP trigger after the child process exits; status (success / failure / unknown) is derived from the child’s exit code (see below). A run interrupted by a signal — a deploy recycling the container mid-run, or output exceeding the exec buffer — is recorded as unknown (not failure), so an interruption isn’t alerted as a real failure. summary records what the run did — see below.
• Admin view — run history (including Status) is surfaced in the admin app’s logs section, backed by GET /admin/logs/cron-logs (filterable by cron_name).
• Secrets (both in Infisical):
  • CRONS_API_KEY — trigger auth. Three places, all the same value: /github-ci (GitHub sends it), /api (the API validates it), and /cloudflare (synced into the Worker). The Cloudflare copy arrives via an Infisical Secret Sync scoped to that one key — a sync pushes everything at its source path, and pointing one at /github-ci would put the INFISICAL_* bootstrap credentials and a Workers-edit token inside a Worker that reads exactly one secret.
  • SLACK_CRONS_WEBHOOK_URL — the incoming webhook for failure alerts (#cron-alerts), in /api. Optional — alerts are best-effort and no-op if unset. Distinct from the DigitalOcean-deploys SLACK_WEBHOOK_URL. It is also the channel the health monitor reports to; if it goes missing, the only trace is a summary_post_failed log line, so treat it as required in practice.
  • HEALTH_CHECK_RETENTION_DAYS — optional, in /api. How long op_health_checks rows are kept. Defaults to 90.

To activate or pause a job without a deploy, flip is_active (and ensure the matching schedule is or isn’t enabled in whichever scheduler owns it).

What a run did — summary

status says whether a job ran; it cannot say whether it achieved anything. A reconciliation that heals nothing and one that is quietly broken both log success in a second.

Scripts print one line — CRON_SUMMARY {"reconciled":4} — and the trigger scrapes it into op_cron_logs.summary. stdout is the channel because the scripts are separate processes with no database handle.

That output used to be discarded. exec() buffers the child’s stdout rather than inheriting it, and the callback read the buffer only on failure — so every count a job computed on a successful run was captured into a variable and dropped, never reaching the container’s stdout or DataDog. The trigger now re-emits it through the logger as well as storing it.

NULL means the job is not instrumented, which is deliberately different from one reporting zero. The currency jobs are in that state; their services return no count.

Run outcomes & failure alerts

The trigger is fire-and-forget, so a green scheduler run only means “the trigger was accepted” — not that the job succeeded. The real outcome is determined in the API’s exec callback from the child process exit code:

• Cron scripts MUST exit(0) on success — including a non-prod skip, or when there is simply no work to do (an empty result set is a quiet run, not a failure) — and exit non-zero only on genuine failure. A script that exits non-zero on success (as several historically did) makes its outcome unreadable — use test-cron / reconcile-parent-skills as the template.
• On a failure the API posts a Slack alert (#cron-alerts) with the cron name, duration, and a scrubbed output excerpt — failures only, no success spam. The excerpt is scrubbed of secrets (URL-embedded credentials, key=value secrets, and long token-like strings) and truncated before it is sent. Alerting is best-effort: it needs SLACK_CRONS_WEBHOOK_URL in the API env and never breaks the run if the webhook is missing or unreachable.

Triggering a cron manually

curl --request GET \
  --url 'https://api.tunnelflight.com/api/crons/run/test_cron' \
  --header 'token: <CRONS_API_KEY>'

The endpoint authenticates on a token header matched against the CRONS_API_KEY environment variable. It returns immediately (success: true) after spawning the child process — it does not wait for the job to finish, so check op_cron_logs (or the admin view) for the outcome. Full request/response schema: API → IBA API → Crons.

You can also trigger a job from the crons workflow  with Run workflow (workflow_dispatch) and pick the cron name — handy for testing without waiting for the schedule.

Gotchas

• The DB schedule isn’t the trigger, but it is no longer inert. While cronInit() is commented out, what actually fires a job is wrangler.toml or crons.yml (see How crons run). schedule_pattern is what cronFreshness measures lateness against, so a disagreement now produces a false alert rather than being harmless. They did disagree, undetected, for two months: op_cron_configs said reconcile_parent_skills was hourly while the workflow fired it daily. Keep all three in step.
• Most currency jobs run only twice a year. Per the workflow schedule, the trainer/coach/instructor/military currency jobs fire on the 16th of a few specific months (e.g. Jan & Dec for trainers), not continuously. Only flyer_currency and payment_reminder run daily. If you expected rolling daily currency checks, that is not what’s scheduled.
• The trigger token is a secret. The workflow passes CRONS_API_KEY (a repo secret) in the token: header. Rotate it in Infisical and update the GitHub Actions secret together. (The legacy droplet crontab held this token in cleartext — rotate it if that box is still reachable.)
• Nothing runs outside production. Every handler exits early unless IS_PRODUCTION === 'Yes', so triggering one locally or in a preview is a safe no-op (handy for testing the harness with test_cron).
• Fire-and-forget. The HTTP trigger returns before the job completes. A success: true response means “started”, not “succeeded”.
• Most configs aren’t in the repo. Only reconcile_parent_skills has a migration. The currency and payment rows exist only in the production DB, so they won’t appear in a fresh local database.

Assistant crons (separate)

The Assistant app has its own, unrelated cron infrastructure (a NodeCronScheduler, a JobRegistry, and a token-guarded POST /api/cron/run/[name] endpoint that logs to its own MySQL table). It ships with no jobs registered yet — the scheduling machinery is in place for future phases. It does not share op_cron_configs or the IBA API harness described above.

See also

• API → IBA API → Crons — generated endpoint schema for the manual trigger.
• Database migrations — how reconcile_parent_skills is registered.
• GitHub Actions workflows — the other scheduled/automated automation in the platform.