On May 30, 2023, mail flow monitoring systems recorded pressure during peak loads at the top of each hour. The volume of messages was higher than usual, not unexpectedly, returning from a holiday weekend. However, the mail processing speed was not performing as expected.
Monitoring metrics indicated messages took over a minute to a few minutes to pass through the gateway and transports. The team increased resources and identified the bottleneck at the logging layer. Additional metrics have been added to analyze performance. However, some metrics caused more load on the system and slowed mail processing.
The logging changes were adjusted and improved mail processing performance quickly. The mail processing slowdown was not anticipated but has allowed the collection of valuable information to continue improving the performance and infrastructure of mail processing in the future.
Reports from partners indicate that CloudMail experienced the most effect from the performance degradation. However, a few reports have correlated to delayed emails sent and received that were likely affected by this issue. Monitoring during peak hours today, May 31, 2023, has confirmed yesterday's changes resolved the issue.