Skip to Content

Secrets and personal data are kept out of logs, and failures are logged in full

Shipped 2026-10-22

Production request logs, error reports and the connector audit table were keeping secrets in plain text, and the API logger was throwing away most error detail.

Secrets

  • Only password fields were removed from the request body logged on every production request. Datadog received login and verification passcodes, two-factor and backup codes, security-question answers, WebAuthn credentials and Convergence member PINs. The same gap applied to stored-procedure arguments and SQL parameters in that log, and to the body sent to Jira, Sentry and the error email on a 5xx.
  • connector_history stored each vendor request body as sent, including member_pin and the vendor secret.
  • A failed Convergence PIN check printed the submitted and stored PIN.
  • Push notifications logged the member’s device token.

All of these now go through one redact(). A value whose key is a secret word (pin, passcode, answer, token, secret, code, …) is replaced, at any depth, by a description of its shape: [redacted: 4-char string] or [redacted: number]. An empty value is left as it is. That is enough to tell a missing PIN or one that lost its leading zero from a wrong PIN, without storing the PIN.

Personal data

The same logs also carried member contact and profile details. Anything leaving our systems (Datadog, Jira, Sentry and the error email) now has:

  • emails and phone numbers masked, e.g. ja******@g****.com
  • names, usernames, date of birth, address, IP-address arguments and social IDs replaced by their shape
  • free text members type (messages, comments, enquiries, reasons, Mandrill events) replaced by its shape

member_id and other IDs, tunnels, skills, dates, amounts and country stay visible, so a member can still be looked up in admin. connector_history is our own table and keeps the personal details a vendor sent; only secrets are removed there. The “email sent” info line now logs the masked recipient with member_id instead of the full address.

Values already in Datadog age out with retention; history rows written before this keep their PINs.

Logger

  • Every log call is now one JSON line. Info and warn lines with extra fields were two lines, which Datadog counted as two events.
  • Errors are logged with their message and stack, as Datadog’s standard error.kind, error.message and error.stack. Before, { error: e } logged {}. Axios errors keep the method, URL and status only, never the request headers. error.kind is the error’s class, so Stripe errors group as StripeInvalidRequestError and the like rather than all as Error.
  • Every line has a searchable message, taken from the text, the error, or the event name.
  • Unhandled promise rejections are logged as process.unhandled_rejection, and failures in the after-response work as http.finish_failed.

Failures that were silent

Each of these now logs, with an event to search or alert on:

EventLevelWhen
sms.send_failedwarn for a bad member number, error otherwiseTwilio refused an SMS. The member still sees “invalid phone”; the number is masked, including inside Twilio’s own message
security_answers.rate_limit_unavailableerrorRedis is down, so the security-answer attempt limit is off. Requests are still allowed
password_reset.token_read_failed, password_reset.token_store_failederrorThe password-reset verification token couldn’t be read or saved in Redis
redis.connect_retrywarnA Redis connection attempt failed and will be retried (was logged twice, as error)
redis.unavailableerrorRedis gave up after its retries
cron.currency.send_failederrorA currency cron run had at least one failed send

The currency crons (instructor, trainer, coach, military, flyer) now print a CRON_SUMMARY with members, deactivated, emails_sent and emails_failed, which is stored in op_cron_logs. Members still lose currency whether or not their email or push notification sends.

Last updated on