Mail Flow Problem
Incident Report for Mailprotector
Postmortem

A code change occurred during normal operations that introduced a bug affecting the perimeter servers' routing tables, causing addresses for active users to be removed. The issue was identified by our internal monitoring systems immediately, and our operations team was notified and rolled back the change. A repair procedure was then run on the routing tables. Upon completion, all services returned to normal operation, and the incident was resolved.

All inbound and outbound emails through the perimeter were rejected during the initial part of the incident. No emails were queued; therefore, senders will need to resend any emails that were rejected. The incident lasted 27 minutes from origin to complete resolution, but not all users were affected the entire time.

Email flow is critical to most businesses, and we understand the immediate impact that can have on you and your clients. This incident was caused by human error, and the team is reviewing process to prevent a similar occurrence. We appreciate the quick feedback from all partners that contacted us. Thank you.

Posted 9 months ago. Aug 22, 2018 - 12:24 EDT

Resolved
Mail flow continues to operate as expected. Thank you.
Posted 9 months ago. Aug 21, 2018 - 17:54 EDT
Monitoring
The routing table problem has been fixed. Email is flowing through the perimeter as expected.

We will continue to monitor the situation.
Posted 9 months ago. Aug 21, 2018 - 17:04 EDT
Identified
A routing table problem has caused email flow to stop at the perimeter. The repair is underway and we expect mail flow to be back shortly.
Posted 9 months ago. Aug 21, 2018 - 16:58 EDT
Investigating
We are investigating a mail flow problem. An update on the situation will be posted as soon as possible.
Posted 9 months ago. Aug 21, 2018 - 16:52 EDT
This incident affected: CloudFilter Email Security and SafeSend Outbound.