Inbound email delay
Incident Report for Mailprotector
Postmortem

The inbound gateway servers experienced degraded performance on the morning of Friday, March 31, 2023. The volume of emails was not unusually high, and further investigation into the performance degradation revealed a poorly constructed custom rule. The associated message rule was removed, and additional system resources were allocated to expedite email processing.

For almost three years, Mailprotector has allowed partners to add regular expressions to Message Rules. Since introducing the feature, poorly written regular expressions have caused several performance issues. In the interest of consistent and predictable email delivery performance, Mailprotector has removed the option to add regular expressions to message body searches.

In addition to the Message Rules modification, the team is working on enhancing the infrastructure to eliminate disruptions or delays in mail flow during peak hours. These enhancements include improving monitoring and scaling capabilities to the infrastructure, allowing for increased elasticity to adapt to the volatile and unpredictable nature of SMTP traffic.

Mailprotector understands the critical nature of healthy email flow to businesses. The difficult decision to reduce a partner feature and continued infrastructure optimization are small pieces of the commitment to predictable email security performance. If you have questions about the Message Rule change, please get in touch with the Partner Success team.

Thank you.

Posted Apr 04, 2023 - 14:53 EDT

Resolved
This incident has been resolved.
Posted Mar 31, 2023 - 15:08 EDT
Monitoring
Surge queues have improved, and we will continue monitoring the traffic.
Posted Mar 31, 2023 - 12:52 EDT
Identified
Surge queues are high again, and the team is working on managing the mail flow.
Posted Mar 31, 2023 - 12:24 EDT
Monitoring
The surge queue has been recovered, and inbound email delivery is operating at expected levels. We will continue closely monitoring the traffic for a couple of more hours.
Posted Mar 31, 2023 - 12:11 EDT
Investigating
We are investigating inbound email delays associated with a high volume of traffic.
Posted Mar 31, 2023 - 11:41 EDT
This incident affected: CloudFilter and SafeSend Email Security.