Mail flow slowed during peak hours
Incident Report for Mailprotector
Postmortem

On May 30, 2023, mail flow monitoring systems recorded pressure during peak loads at the top of each hour. The volume of messages was higher than usual, not unexpectedly, returning from a holiday weekend. However, the mail processing speed was not performing as expected.

Monitoring metrics indicated messages took over a minute to a few minutes to pass through the gateway and transports. The team increased resources and identified the bottleneck at the logging layer. Additional metrics have been added to analyze performance. However, some metrics caused more load on the system and slowed mail processing.

The logging changes were adjusted and improved mail processing performance quickly. The mail processing slowdown was not anticipated but has allowed the collection of valuable information to continue improving the performance and infrastructure of mail processing in the future.

Reports from partners indicate that CloudMail experienced the most effect from the performance degradation. However, a few reports have correlated to delayed emails sent and received that were likely affected by this issue. Monitoring during peak hours today, May 31, 2023, has confirmed yesterday's changes resolved the issue.

Posted Jun 01, 2023 - 13:27 EDT

Resolved
Mail delays of several minutes were observed during peak hours yesterday, May 30, 2023. The slowdown in processing messages was determined and resolved by the close of business. Changes to the logging system required adjustment, and the resolution has been confirmed as of the early afternoon on May 31, 2023.
Posted May 30, 2023 - 17:30 EDT