This morning, a third-party package repository used by our virus scanning infrastructure experienced an outage following an attack on its systems. Because our virus scanning instances rely on this repository at startup to pull the latest files and configurations, affected instances were unable to start. This reduced the capacity of our virus-scanning cluster below demand.
When a virus scan cannot complete, our system quarantines the email as a precaution. As a result, emails processed during this window were quarantined rather than delivered to inboxes. No emails were lost. Affected emails can be released from quarantine by users or admins.
Resolution
The virus scanning cluster has been reconfigured to pull packages directly from the source repository, eliminating the dependency on the third-party intermediary. Monitoring and alerting thresholds have also been updated to surface this class of issue more quickly in the future.