Problems with mail delivery
Incident Report for SuperOffice
Postmortem

Description 

On Friday 1 October, Super Office Online experienced an incident that caused incoming mail to our service solution to arrive without any content in the content section of the email. 

Email messages are received by our mail service provider as a MIME message and transmitted using the SMTP protocol.  The message is then transmitted to our ticket service solution using an HTTP POST, where the body of the HTTP message contains the original MIME message. 

Root cause 

On October 1st at 00:00 UTC +2, our mail service provider rolled out a modification in the transfer encoding of emails that changed the coding. 
The HTTP message transmitted to our ticket service solution had a HTTP Header value indicating that there were no HTTP message body present.  This resulted in our ticket service solution accepting the HTTP message, but not processing the message body containing the original e-mail message in MIME format. The effect of this in our ticket service solution was that an empty message was received. 

Limitation and solution 

The solution to the problem was twofold.  Firstly, our ticket service solution was modified to accept misleading HTTP Header values indicating that no HTTP message bodies were present.  This resulted in successfully receiving e-mail messages from our mail service provider.  Secondly, e-mail messages that was not correctly transmitted by our mail service provider to our ticket service solution were retransmitted.  This resulted in the messages that was previously received as empty messages were received as e-mail messages.  

Our fault tolerance has been enlarged and work in a way to be more robust and prevent similar cases happening again.

Posted Oct 08, 2021 - 16:15 CEST

Resolved
This event is now resolved
All email is imported into the solution. A very small number of emails received between 01:00 and 01:28 CET 1st of October may have come in as duplicates.
Posted Oct 02, 2021 - 17:08 CEST
Update
The import of all email sent on the 1st of October from CET 01:30 to CET 10:00 has started for all customer databases, and is expected to be finished this afternoon.
Posted Oct 02, 2021 - 13:24 CEST
Monitoring
A fix has been implemented and we are monitoring.

Email sent to Service mailboxes from CET 10:00 to now are being imported. Emails sent between CET 01:25 and CET 10:00 will be imported during the weekend
Posted Oct 01, 2021 - 15:41 CEST
Identified
The issues has been identified and our engineers is working on a fix.
Posted Oct 01, 2021 - 13:43 CEST
Update
We are continuing to investigate this issue.
Posted Oct 01, 2021 - 13:16 CEST
Update
We are continuing to investigate.
The admin client for service is now open again, but we blocking incoming emails to the service solution so they are queued up until we have a solution.
Posted Oct 01, 2021 - 11:13 CEST
Update
We have temporarily stopped the admin functionality in the service client during the incident.
Posted Oct 01, 2021 - 10:09 CEST
Investigating
Some of our customers are reporting that incoming mail in Service are not delivered correctly. We are investigating the issue
Posted Oct 01, 2021 - 08:48 CEST
This incident affected: SuperOffice CRM Cloud Mail Service (mailgun Inbound email processing).