Inconsistent inbound logs through CloudFilter

Incident Report for Mailprotector

Resolved

An issue that started at 12:21 PM ET resulted in lost logs from 12:20 PM to 2 PM and likely duplicated some logs in the early morning.

AWS performed a maintenance task that restarted a transport server in an ungraceful manner. This caused the logging service to lose its place in the log that it was duplicating. The logging service started at the beginning of the log file and writing logs. The problem was discovered, and it forced a rotation of the mail log (to start over fresh) and a re-addition of the affected server to the load balancer.

The event may have duplicated some log messages, dropped others, and created odd-looking data in the mail flow logs in the Mailprotector Console during the event. However, mail flow was not affected by this event.

Posted Oct 17, 2024 - 12:30 EDT