Description and consequence of the incident

SuperOffice Online was hit by two different incidents on Wednesday February 7th 2024.  There was one isolated incident with the SuperId login service (id.superoffice.com), and another incident related capacity of outbound network traffic in the Visma datacenter.  The SuperId incident resulted in users struggling to log in.  This incident was identified 08:44 and lasted until about 12:00.

Indications of the incident related to capacity of outbound network traffic in the datacenter was first observed 05:50.  The bottleneck for the of the outbound network traffic was identified at 18:15.  A preliminary fix resolving the bottleneck was in place 18:45.

The Root cause

The root cause of the incident related to the Superid login service was a network degradation incident in Azure west Europe.  This aligns with Microsoft Incident Tracking ID: 6V3Z-VZZ

The root cause of the incident related to capacity of outbound network traffic was caused by an increased amount of open outbound network connections to queue services in Azure.  The increase of the amount of connections were greater than anticipated.  All outbound network traffic in SuperOffice Online pass through a proxy service, as part of the security measures.  This proxy service was not capable of handling this unexpected increase of outbound network connections.

Mitigation

The incident related to SuperId was resolved by scaling out and restarting the service in Azure 9:10. 

The incident related to capacity of outbound network traffic was resolved using several action: 1) The services were reconfigured to handle a greater number of connections.  2) The service was scaled out to handle additional load. 3) The feature in version 10.3.1 causing an unexpected number of outgoing connections where rolled back in a HotFix of the product.