One of the inbound transport servers was automatically rebooted for maintenance and could not handle the imposed load as it scaled back up. A server reboot is typically infrequent and mundane, but this particular transport server did not respond as typically expected. While the other servers could handle their current load, it was decided to resize in order to handle unexpected periods of higher demand in the future.
All inbound transport servers have been resized to prevent similar delays, and more monitoring/alerting metrics have been added.
Posted Jan 29, 2024 - 18:57 EST
Monitoring
Maintenance was performed on the impacted instance. It has been added back into production and we are no longer seeing delivery delays. We will continue to monitor performance.
Posted Jan 29, 2024 - 16:56 EST
Investigating
The team is currently investigating inbound mail delays. Delayed mail is primarily associated with a singular instance, and the instance has been temporarily removed from production. This should improve delivery times while the team continues to track down the root of the issue.
Posted Jan 29, 2024 - 15:32 EST
This incident affected: CloudFilter and SafeSend Email Security.