Inbound email delivery delays
Incident Report for Mailprotector
Postmortem

Inbound email delivery has been stable since Monday afternoon, August 8.

Last week's delivery delay appeared to be a disk constraint. The team scaled the system and moved processes to faster disk resources. Unfortunately, increasing the disk performance moved the resource constraint to another layer of the system, and the problem was repeated on Monday morning.

Mailprotector understands the importance of consistent email flow, and delays can create more business disruption than many other technical issues.

The team has over-provisioned the systems to ensure the delivery delays do not repeat. At the same time, new monitoring and warning alerts are being developed to scale resources better ahead of noticeable performance changes. Historically, the transport systems have run smoothly since they do nothing more than complete the delivery of a message to a destination email server. The monitors and alerting added to the transport servers will provide the team more visibility to react in a timelier manner and prevent delays as we continue to grow with our partners.

The inbound email delays are resolved, and we do not anticipate further performance problems.

Posted Aug 11, 2022 - 11:45 EDT

Resolved
Inbound email delivery is stable and performing as expected. The team is still working on further investigation. A post-mortem will be posted as soon as possible.
Posted Aug 08, 2022 - 17:30 EDT
Monitoring
All transport queues have caught up, and new emails are being delivered in a few seconds. The team will continue to monitor the transport system.
Posted Aug 08, 2022 - 13:24 EDT
Update
We are continuing to investigate the issue and work to reduce delays as quickly as possible.
Posted Aug 08, 2022 - 12:50 EDT
Investigating
We are investigating inbound email delivery delays. The issue is intermittent. Emails are being delivered within 30 seconds upwards to 90 minutes. The average delay is 10 to 20 minutes.

Last week's system upgrades were successful, but the issue is still under investigation.
Posted Aug 08, 2022 - 11:46 EDT
This incident affected: CloudFilter and SafeSend Email Security.