Secrets and personal data are kept out of logs, and failures are logged in full
Shipped 2026-10-22
Production request logs, error reports and the connector audit table were keeping secrets in plain text, and the API logger was throwing away most error detail.
Secrets
- Only password fields were removed from the request body logged on every production request. Datadog received login and verification passcodes, two-factor and backup codes, security-question answers, WebAuthn credentials and Convergence member PINs. The same gap applied to stored-procedure arguments and SQL parameters in that log, and to the body sent to Jira, Sentry and the error email on a 5xx.
connector_historystored each vendor request body as sent, includingmember_pinand the vendorsecret.- A failed Convergence PIN check printed the submitted and stored PIN.
- Push notifications logged the member’s device token.
All of these now go through one redact(). A value whose key is a secret
word (pin, passcode, answer, token, secret, code, …) is replaced,
at any depth, by a description of its shape: [redacted: 4-char string] or
[redacted: number]. An empty value is left as it is. That is enough to tell a
missing PIN or one that lost its leading zero from a wrong PIN, without storing
the PIN.
Personal data
The same logs also carried member contact and profile details. Anything leaving our systems (Datadog, Jira, Sentry and the error email) now has:
- emails and phone numbers masked, e.g.
ja******@g****.com - names, usernames, date of birth, address, IP-address arguments and social IDs replaced by their shape
- free text members type (messages, comments, enquiries, reasons, Mandrill events) replaced by its shape
member_id and other IDs, tunnels, skills, dates, amounts and country stay
visible, so a member can still be looked up in admin. connector_history is
our own table and keeps the personal details a vendor sent; only secrets are
removed there. The “email sent” info line now logs the masked recipient with
member_id instead of the full address.
Values already in Datadog age out with retention; history rows written before this keep their PINs.
Logger
- Every log call is now one JSON line. Info and warn lines with extra fields were two lines, which Datadog counted as two events.
- Errors are logged with their message and stack, as Datadog’s standard
error.kind,error.messageanderror.stack. Before,{ error: e }logged{}. Axios errors keep the method, URL and status only, never the request headers.error.kindis the error’s class, so Stripe errors group asStripeInvalidRequestErrorand the like rather than all asError. - Every line has a searchable
message, taken from the text, the error, or theeventname. - Unhandled promise rejections are logged as
process.unhandled_rejection, and failures in the after-response work ashttp.finish_failed.
Failures that were silent
Each of these now logs, with an event to search or alert on:
| Event | Level | When |
|---|---|---|
sms.send_failed | warn for a bad member number, error otherwise | Twilio refused an SMS. The member still sees “invalid phone”; the number is masked, including inside Twilio’s own message |
security_answers.rate_limit_unavailable | error | Redis is down, so the security-answer attempt limit is off. Requests are still allowed |
password_reset.token_read_failed, password_reset.token_store_failed | error | The password-reset verification token couldn’t be read or saved in Redis |
redis.connect_retry | warn | A Redis connection attempt failed and will be retried (was logged twice, as error) |
redis.unavailable | error | Redis gave up after its retries |
cron.currency.send_failed | error | A currency cron run had at least one failed send |
The currency crons (instructor, trainer, coach, military, flyer) now print a
CRON_SUMMARY with members, deactivated, emails_sent and
emails_failed, which is stored in op_cron_logs. Members still lose
currency whether or not their email or push notification sends.