Cron jobs

The IBA API ships a set of scheduled background jobs: currency-expiry alerts, payment reminders, and a data-integrity safety net. Each job is a standalone Node script under api/src/crons/, registered as a row in the op_cron_configs table. This page covers what runs, why, and how it’s triggered.

This is the conceptual reference. The auto-generated endpoint schema for the manual trigger lives at API → IBA API → Crons; don’t hand-edit that page — it’s regenerated from the Bruno collection.

How crons run

Scheduling and execution are two decoupled halves:

1. Scheduler — GitHub Actions. .github/workflows/crons.yml runs on a schedule: (cron) and curls the API’s trigger endpoint for the due job, authenticating with CRONS_API_KEY (fetched at runtime from Infisical /github-ci via OIDC — not a stored GitHub secret). The schedule is in version control — there’s no scheduler box to SSH into. It can also be fired on demand from the Actions tab (Run workflow → pick a job).
2. Executor — the API. GET /crons/run/:cron_name looks the job up in op_cron_configs, runs node <cron_path> as a child process via exec() inside the API App Platform container, and writes a row to op_cron_logs when it finishes.

A third mechanism exists but is dormant: the in-process node-cron scheduler (api/src/crons/index.js) that cronInit() would start is commented out in api/index.js. So schedule_pattern in op_cron_configs is documentation only — the workflow’s cron: lines are the real cadence.

Rollout status (2026-08-06). Complete. Every currency and payment line on the Dev-DB-Redis crontab is commented out — verified directly on the box — so all scheduled jobs now fire from GitHub Actions. test_cron remains manual-dispatch only.

History. The scheduler used to be a hand-maintained crontab on the Dev-DB-Redis DigitalOcean droplet that curled the same endpoint, with the token in cleartext on the box. The GitHub Actions workflow replaces it: version-controlled, secret-managed, no SSH.

The last job to leave was the hourly health monitor, which ran as a standalone app under that crontab until 6 Aug 2026. A monitor must not share fate with what it monitors — on the droplet, a droplet failure took the alerting down with it, silently. See Health monitor below.

Caveat: GitHub’s scheduler is not punctual. Scheduled runs are delayed under load and dropped entirely when runners are scarce. During the Actions outage on 6 Aug the hourly health monitor produced 7 runs in 18 hours, one of which failed with “job was not acquired by Runner”. The droplet crontab was reliable to the minute; this is the cost of the independence gained. For jobs that must not be missed, check op_cron_logs rather than assuming a schedule implies execution.

Because each job is launched as node <script>, every handler bootstraps its own environment (pulling secrets from Infisical when not in production), guards on IS_PRODUCTION === 'Yes' before it mutates anything, and exits 0 on success / 1 on failure.

At a glance

Cadences below are the workflow’s cron: schedules (UTC). The name is the value passed to /crons/run/<name>; the currency names carry a _currency suffix.

reconcile_parent_skills and health are registered via committed migrations (0003, 0007). The currency and payment rows pre-date the migration system and live directly in the production op_cron_configs table.

health has its own workflow rather than a schedule entry in crons.yml. It runs 24 times a day against the others’ daily-or-rarer cadence, so sharing a workflow would bury them in its run history, and a monitor’s failures want their own notification stream. It also runs at :07 rather than the top of the hour — GitHub documents scheduled runs as delayed under load, and the start of every hour is the worst window for it.

reconcile_parent_skills was previously dormant. Under the old droplet crontab it had no trigger line (and cronInit() is off), so until the GitHub Actions workflow only the one-shot back-fill in migration 0001 had ever run. The workflow now schedules it daily at 04:00 UTC (0 4 * * *), closing that gap.

Currency-expiry alerts

The four currency jobs (trainer, coach, instructor, military) share one engine — CurrencyCronClass in api/src/crons/currency/index.js — and differ only in the role they target. A fifth currency job, flyer (regular flying members), is wired separately to flyer.service.js under the newer api/src/features/shared/currency/ implementation rather than CurrencyCronClass. For members whose currency is expiring or has expired, the job sends localised notifications (Mandrill email + app push) to a fan-out of recipients:

• the member themselves, in their own language;
• their manager(s), grouped by tunnel (military skips this — it has a different approval structure);
• the regional ops team (USA / EU / AU split);
• admin (info@tunnelflight.com).

It runs in two modes driven by how far out the expiry is: a renewal reminder ahead of the deadline, and an expired notice on/after it. When configured to, it also disables the lapsed account. The instructor variant adds a lastSafetyPeriodActive() check so a still-valid safety period doesn’t get revoked by accident.

Payment reminders

payment_reminder queries members with upcoming payment deadlines (joining fees where status is succeeded against fees_mapping where the mapping is Active), computes days remaining, and sends one of three tiered notices: 31 days (last month), 7 days (last week), and overdue. A single run handles all three tiers — the tier is chosen per member from the days-remaining figure, not from separate schedules.

reconcile_parent_skills

The daily safety net behind the parent-skill auto-approval work (TNBUGS-1728). It delegates to ReconciliationService.run(), which heals any member-upgrade rows left inconsistent, and logs the count it fixed. Like the others, it only mutates data in production. This is the one cron registered through a database migration, so it travels with the code.

Health monitor

health is the odd one out: it doesn’t act on member data, it watches the platform. Fourteen checks run per pass — API / WWW / Admin endpoints, MySQL, Redis, the S3 backup, Stripe webhooks and payment validation, Mandrill, Twilio, the DigitalOcean droplet and App Platform APIs, DataDog ingestion, and dailyCron (which asserts that yesterday’s flyer_currency and payment_reminder actually logged a run). Each writes a row to op_health_checks tagged with a shared group_id, so one pass reads as a unit.

One failing check never stops the rest — a broken Stripe webhook must not hide a broken database — and the script exits 1 if anything failed, which is what op_cron_logs records.

Alerting

Everything goes to the crons Slack channel. There is no email — the Mandrill summary was retired in TUN-845 because it sent to TO_EMAIL, which was unset in production, so it fell back to a hardcoded personal address that nobody on the team was reading.

• On failure, a summary naming every failing check and its reason.
• Once every 24 hours when everything passes, an all-clear. Failure-only alerting cannot distinguish a healthy platform from a dead monitor; the heartbeat is what makes the silence between them meaningful. Its cadence is tracked by a slackHeartbeat row, because the script is a fresh process each run with nowhere else to remember.

A failing run also produces the generic Cron failed: health alert described in Run outcomes & failure alerts. That overlap is deliberate — it’s the backstop for a crash that happens before the summary can be posted.

Skips expire

Some checks don’t run every pass. A skip is legitimate while the check is still attempting on its own cadence, but a check that reports skipped forever is indistinguishable from a healthy one, because only failures alert. Any check whose last real verdict is more than 48 hours old is therefore promoted to a failure.

This exists because of a real two-week outage. twilioSms gated itself on getHours() === 0. The droplet’s 0 * * * * crontab hit that window every night; GitHub’s scheduler delays runs 25–58 minutes and drops the 00:07 slot outright, so across 200 consecutive runs not one landed in hour 00 UTC. The check reported skipped every time, and its underlying failure went unreported until someone noticed the alerts had stopped.

The rules now live as pure, unit-tested functions in crons/health/domain/staleness.js. Periodic checks are paced off their last real attempt, not the wall clock — and off the last attempt rather than the last success, so a failing check alerts once per interval instead of once an hour.

Don’t gate a check on wall-clock time. Anything of the form “only run during hour N” assumes a punctual scheduler, and this one isn’t. Pace off the last recorded attempt instead.

op_health_checks retention

The table is append-only and nothing in api/, admin/ or www/ reads it, so it grew past 124,000 rows unchecked. Each run now prunes rows older than 90 days (HEALTH_CHECK_RETENTION_DAYS) in 5,000-row batches — batched rather than one large DELETE so a backlog drains without holding locks. That holds the table near ~34k rows.

Notes are kept small on purpose. The DigitalOcean checks used to store every app’s id, URLs, timestamps and a live memory/CPU sample — about 1.5KB a row, and over half of everything in the table — for data nothing reads and that DO’s own monitoring page already holds. They now record names and status only, which also saves 16 DigitalOcean API calls per run.

Configuration & logging

• op_cron_configs — one row per job: name (unique), cron_path (relative to api/src/crons/), schedule_pattern, is_active.
• op_cron_logs — one row per run: cron_config_id, start_at, end_at, duration_seconds, status. Written by the HTTP trigger after the child process exits; status (success / failure / unknown) is derived from the child’s exit code (see below). A run interrupted by a signal — a deploy recycling the container mid-run, or output exceeding the exec buffer — is recorded as unknown (not failure), so an interruption isn’t alerted as a real failure.
• Admin view — run history (including Status) is surfaced in the admin app’s logs section, backed by GET /admin/logs/cron-logs (filterable by cron_name).
• Secrets (both in Infisical):
  • CRONS_API_KEY — trigger auth; in /github-ci (the workflow sends it) and /api (the API validates it).
  • SLACK_CRONS_WEBHOOK_URL — the incoming webhook for failure alerts (#cron-alerts), in /api. Optional — alerts are best-effort and no-op if unset. Distinct from the DigitalOcean-deploys SLACK_WEBHOOK_URL. It is also the channel the health monitor reports to; if it goes missing, the only trace is a summary_post_failed log line, so treat it as required in practice.
  • HEALTH_CHECK_RETENTION_DAYS — optional, in /api. How long op_health_checks rows are kept. Defaults to 90.

To activate or pause a job without a deploy, flip is_active (and ensure the workflow’s cron: schedule is or isn’t enabled for it).

Run outcomes & failure alerts

The trigger is fire-and-forget, so a green scheduler run only means “the trigger was accepted” — not that the job succeeded. The real outcome is determined in the API’s exec callback from the child process exit code:

• Cron scripts MUST exit(0) on success — including a non-prod skip, or when there is simply no work to do (an empty result set is a quiet run, not a failure) — and exit non-zero only on genuine failure. A script that exits non-zero on success (as several historically did) makes its outcome unreadable — use test-cron / reconcile-parent-skills as the template.
• On a failure the API posts a Slack alert (#cron-alerts) with the cron name, duration, and a scrubbed output excerpt — failures only, no success spam. The excerpt is scrubbed of secrets (URL-embedded credentials, key=value secrets, and long token-like strings) and truncated before it is sent. Alerting is best-effort: it needs SLACK_CRONS_WEBHOOK_URL in the API env and never breaks the run if the webhook is missing or unreachable.

Triggering a cron manually

curl --request GET \
  --url 'https://api.tunnelflight.com/api/crons/run/test_cron' \
  --header 'token: <CRONS_API_KEY>'

The endpoint authenticates on a token header matched against the CRONS_API_KEY environment variable. It returns immediately (success: true) after spawning the child process — it does not wait for the job to finish, so check op_cron_logs (or the admin view) for the outcome. Full request/response schema: API → IBA API → Crons.

You can also trigger a job from the crons workflow  with Run workflow (workflow_dispatch) and pick the cron name — handy for testing without waiting for the schedule.

Gotchas

• The DB schedule isn’t the trigger. While cronInit() is commented out, schedule_pattern is documentation only. The cadence that actually fires a job lives in the workflow’s cron: lines (see How crons run), not in op_cron_configs. The two can and do disagree.
• Most currency jobs run only twice a year. Per the workflow schedule, the trainer/coach/instructor/military currency jobs fire on the 16th of a few specific months (e.g. Jan & Dec for trainers), not continuously. Only flyer_currency and payment_reminder run daily. If you expected rolling daily currency checks, that is not what’s scheduled.
• The trigger token is a secret. The workflow passes CRONS_API_KEY (a repo secret) in the token: header. Rotate it in Infisical and update the GitHub Actions secret together. (The legacy droplet crontab held this token in cleartext — rotate it if that box is still reachable.)
• Nothing runs outside production. Every handler exits early unless IS_PRODUCTION === 'Yes', so triggering one locally or in a preview is a safe no-op (handy for testing the harness with test_cron).
• Fire-and-forget. The HTTP trigger returns before the job completes. A success: true response means “started”, not “succeeded”.
• Most configs aren’t in the repo. Only reconcile_parent_skills has a migration. The currency and payment rows exist only in the production DB, so they won’t appear in a fresh local database.

Assistant crons (separate)

The Assistant app has its own, unrelated cron infrastructure (a NodeCronScheduler, a JobRegistry, and a token-guarded POST /api/cron/run/[name] endpoint that logs to its own MySQL table). It ships with no jobs registered yet — the scheduling machinery is in place for future phases. It does not share op_cron_configs or the IBA API harness described above.

See also

• API → IBA API → Crons — generated endpoint schema for the manual trigger.
• Database migrations — how reconcile_parent_skills is registered.
• GitHub Actions workflows — the other scheduled/automated automation in the platform.