Lessons Learned From the CrowdStrike Outage

It’s been a little over a week since the CrowdStrike outage hit 8.5 million Windows devices across the globe. While the initial disruption has passed, businesses are still grappling with the far-reaching impacts. As they recover, many are rethinking their cloud-based cybersecurity strategies and update processes.

Today, we’re discussing the causes of the outage and its ongoing impact. We’ll also share some general advice for protecting your company from cloud outages. Keep reading if your organization is moving any of their systems or services to the cloud.  

Causes of the CrowdStrike Outage​

The root cause of the outage was identified as a software update error made by CrowdStrike, an endpoint security vendor. Windows systems were the only systems affected because the CrowdStrike platform only integrates with Windows environments.  

The Immediate Impact​

Affected organizations were temporarily unable to receive real-time alerts and threat intelligence. This increased their vulnerability to cyber-attacks during the downtime.

This downtime felt like an eternity for many organizations. According to TechTarget, IT staff had to not only respond to the crisis, but they had to manually reboot all affected systems to delete the problematic file and restore normal operations.

The Ongoing Impact

Businesses with complex IT infrastructure may be working for months to recover all affected systems.

For example, many airlines are still reconfiguring their flight scheduling systems, while many healthcare organizations are still restoring access to critical patient records​.

Protecting Your Business from Cloud Outages

While the CrowdStrike situation was more of an update error than a true cloud outage, it still serves as a reminder of the importance of a robust cloud strategy. To mitigate the risks associated with cloud outages, businesses should consider the following best practices:

Rigorous Testing and Validation​

Before deploying any software updates, thorough testing and validation are crucial. We recommend testing updates in isolated environments and using automated testing tools to simulate various scenarios.

When working on implementation projects, our ERP software consultants like to involve end-users in testing to ensure any new software or updates meet all functional requirements.

Considering Hybrid and Multi-Cloud Strategies​

A multi-cloud strategy involves using multiple cloud service providers to host different parts of your IT infrastructure.

This approach can enhance redundancy and reduce the risk of a single point of failure. If one provider experiences an outage, the other can take over, ensuring continuity of services.

For many small to mid-sized businesses, the complexity and cost of managing multiple cloud environments can outweigh the benefits. However, if you do decide to adopt this approach, here are some best practices to follow:

• Data Synchronization: Ensure real-time data synchronization across different cloud environments.
• Unified Management Tools: Leverage tools for centralized management of your entire cloud environment.
• Automated Monitoring Solutions: Deploy automated monitoring for real-time performance and health tracking.

Selecting Reliable Cloud Service Providers (CSPs)​

You should evaluate CSPs based on their uptime guarantees and the clarity of their Service Level Agreements (SLAs). We recommend asking providers for examples of acceptable uptime percentages and response times.

If these seem reasonable, you can move on to assessing the CSP’s infrastructure and redundancy measures as well as their data center locations, backup power supplies, and disaster recovery plans.

Even if you do have a reliable provider, it’s still important to test all updates before deployment, as mentioned earlier. This means minimizing your use of automated updates, especially for mission-critical systems.

Monitoring and Alerting Systems

Advanced monitoring and alerting systems can provide early warnings of potential issues, allowing you to address them before they escalate into full-blown outages. These systems should be capable of real-time monitoring and integrated with automated response protocols.

Regular Audits and Risk Assessments

Conducting regular audits and risk assessments of your IT infrastructure can help identify vulnerabilities and areas for improvement. Third-party experts can provide an objective view and highlight risks that internal teams might overlook.

One of our software expert witness cases involved a government entity that caused its own demise (at least in terms of payroll). While they were implementing a new payroll system, one of their biggest mistakes was failing to take action in response to recommendations from a third-party independent verification & validation (IV&V) team.

This, in addition to other missteps, resulted in the issuance of incorrect paycheck amounts and inaccurate calculation of vacation pay to a large percentage of the government entity’s employees.  

Enhancing Data Backup and Recovery Strategies

With a backup plan and recovery strategy in place, you can more quickly bounce back from an outage when it comes your way.

Backup systems make it easier to continue operations after a system failure, while a recovery plan with clear roles and responsibilities can make it easier to get back to business as usual.

Mitigate the Risks of Cloud Software and Services​

With the right cloud strategy, organizations can safeguard themselves against the adverse impacts of cloud service disruptions. For business leaders, these measures are not just about preventing outages but about maintaining trust and ensuring continuous protection of their digital assets.

Whether you’re evaluating cloud ERP vendors or cloud service providers, our ERP consultants can help you make the right decision. Contact us below for a free consultation.