An analysis of the application logs and system behavior indicates that the incident was caused by an OutOfMemoryException encountered by the ASP.Net Core application. This exception occurred due to a memory leak, where the application failed to release memory and clean up resources properly after encountering the exception.

Key details of the issue:

1. The ASP.Net Core application running under IIS did not fully clean up after itself following the OutOfMemoryException.
2. The internal port allocated by the ASP.Net Core runtime remained bound to the previous process after the crash, even though the process terminated.
3. This lingering port prevented new ASP.Net Core processes from starting, which not only caused the primary application to become unresponsive but also caused dependent web tools that relied on its services to fail.

Side Effect:

The web tools associated with the ASP.Net Core application were dependent on its internal APIs and services to function. As a result, when the primary application failed to restart, the web tools also became non-functional, causing additional disruptions:

Mitigation and Resolution:

Despite various restart attempts, the application continued to fail to start due to the port being bound by the previous instance, which caused both the application and the associated web tools to remain down.

After a period of time, the issue resolved itself without any manual intervention. Upon further investigation, it was found that no specific action or system change directly correlated with the application recovery. Explanations include:

• System-level memory management cleaning up orphaned resources after a timeout.
• Azure platform services reclaiming the bound port.
• IIS or Azure-managed auto-restart mechanisms eventually freeing up the port and allowing the application and web tools to come back online.

Conclusion:

The incident was caused by an OutOfMemoryException that resulted in a memory leak and improper resource cleanup, causing the ASP.Net Core runtime to leave the internal port bound, which prevented new processes from starting. This not only affected the main application but also rendered associated web tools non-functional.

The team will continue to monitor the situation closely and implement the recommended improvements to prevent future occurrences of this nature.