In 2020, the SolarWinds attack shook the foundations of corporate IT and cybersecurity. What made it so devastating wasn’t just that a major software vendor had been breached; it was who they were and what their product touched.

SolarWinds’ flagship software was designed to monitor the health of entire corporate networks. It tracked whether servers were up or down, measured performance, and triggered alerts when something looked off. To perform that function, SolarWinds was granted some of the deepest levels of access imaginable inside corporate and government systems. When that system was compromised by a nation-state actor, it meant that tens of thousands of networks, both private and public, were suddenly vulnerable.

Control SolarWinds, control the network. That nightmare became a reality.

Fast Forward to 2025: The F5 Big IP Breach

Now, five years later, another storm is brewing, and it feels eerily familiar.

Cybersecurity firm F5 has disclosed that its flagship Big IP product was compromised by a sophisticated nation-state threat actor. While F5 says its operations remain unaffected, the implications reach far beyond its corporate walls.

Big IP is not just another network device. It is one of the foundational technologies behind how modern networks are protected and managed. F5 pioneered the concept of virtualized firewall appliances, and its products are deeply entrenched across industries, including government agencies and Fortune 500 companies.

Much like SolarWinds, Big IP sits at the heart of digital infrastructure: decrypting traffic, managing load balancing, and deciding what gets in or out of a network. In many ways, it is a digital gatekeeper, and now that gatekeeper has been breached.

The Alarming Details

According to F5’s disclosure, portions of their source code were accessed and exfiltrated by the attackers. That detail cannot be understated. Source code is the blueprint of any software system. With it, an adversary can study the internal workings of a product, replicate its behavior, or identify vulnerabilities hidden deep within. Once a threat actor holds that level of insight, the playing field changes completely.

F5’s guidance so far has been straightforward: update immediately to the latest version of Big IP software. That directive alone signals how serious the situation is. When a company urgently tells customers to patch, it typically means that viable attack paths are known internally, even if not publicly disclosed.

What We Don’t Know Yet

The uncertainty is what makes this breach especially concerning. We do not yet know how this source code theft will be weaponized, or whether it already has been. Will attackers use the code to craft targeted exploits? Could they engineer counterfeit versions of F5 software? The answers remain unknown, but history has shown that such breaches often echo for years.

F5’s prompt patching effort is a good sign, but it is also a reminder of how fragile digital trust has become. When the very systems designed to protect networks are themselves compromised, the ripple effects are immense and long-lasting.

Hoping for Containment

For now, the best defense is vigilance: apply patches, monitor logs, and assume that attackers are already studying what they have obtained.

If we are lucky, this will end as a contained event, a cautionary tale rather than the next SolarWinds-scale catastrophe. But as we have learned before, the real test comes months later, when the first signs of exploitation begin to surface.

Let’s hope this time, the patch really is the end of the story.

Somewhere along the line in the past few years, I got my real estate license. My brief exposure to the industry has shown me that pound for pound, no sector may be more exposed to cybersecurity threats than real estate.

At my brokerage, an incident occurred where a threat actor obtained information off Zillow about an unsold land listing. They reached out to our brokerage requesting to relist with us. They had done their research. Using the same public records available to us, they identified the legitimate owner of the land. Then, they impersonated them, creating a Gmail address that closely resembled the owner’s name, and initiated contact.

They used the same social engineering techniques common in corporate attacks. They created a sense of urgency, insisting the property be listed immediately. Not wanting to lose what seemed like a legitimate client, our broker obliged and listed the property. A few days later, the actual owner of the land contacted us, and the listing was quickly removed without financial loss. Still, the experience gave our brokerage pause and raised serious questions about how to handle similar situations in the future.

For me, it was a moment of reflection, connecting my experience in corporate cybersecurity with the vulnerabilities in real estate.

Why Real Estate Is a Prime Target for Cyber Threats

1. Real Estate Agents Handle Extremely Sensitive Data
Agents collect names, phone numbers, addresses, pre-approval letters, and even copies of escrow checks containing account details. For a threat actor, a real estate transaction is a goldmine of personal and financial information, all bundled together.

2. Agents Operate as Independent Contractors
Unlike corporate employees, most agents use their own laptops, personal email accounts, and whatever level of security knowledge they happen to have. Many do not use enterprise-grade tools, and in some cases, they may not even have up-to-date antivirus software. This means that when you email a scanned escrow check, there is no guarantee it is being stored or transmitted securely.

3. Rogue Attacks and Long-Tail Consequences
Imagine becoming a victim of identity theft months or even years after closing a home. A new credit card appears on your credit report, or a fraudulent loan surfaces. Would you ever trace it back to a real estate agent’s compromised inbox? Probably not. This lack of visibility makes real estate an especially appealing vector for cybercriminals.

4. Lack of Cybersecurity Training
In the corporate world, we sit through monotonous but necessary cybersecurity training sessions. Real estate agents, on the other hand, are required to complete Continuing Legal Education (CLE) to maintain their license. Yet cybersecurity awareness is not part of the curriculum. This gap leaves agents unprepared to identify phishing attempts, social engineering tactics, or malware threats.

What Can Be Done?

We could shoot for the stars and require brokerages, many of which operate on the independent contractor model, to begin treating agents like full-time employees. This would include providing secure devices, standardized email platforms, and mandatory training. But that is unlikely to happen overnight.

A more practical step would be to integrate cybersecurity training into the real estate licensing process and CLE requirements. Even basic awareness could help prevent many of the attacks currently slipping through unnoticed.

In cybersecurity, you eventually learn to stop blaming end users. Incidents are inevitable. What matters is how organizations prepare for and respond to them. Real estate has not reached that stage of maturity yet. The industry still treats cyber incidents as anomalies instead of inevitabilities. Until that mindset shifts, it will remain one of the most exposed sectors.

John Connor also carried the lifelong burden, never escaping it in the sequels, of being the only one who knew how to fight the machines as humanity’s savior. For the record, I hope no one expects me to fill that role when the machines turn on us, because I’m out. Still, with the rapid advancement of AI and humanoid robots, that day feels closer than ever. It may not look exactly like Terminator, but there will come a moment when we think: “Wow… we really should have seen that coming.”

How Could This Actually Happen?

We’ve already seen incidents reported out of China involving humanoid robots behaving aggressively toward humans.

From the first time I saw progress in AI, I thought: “Oh boy, they’re going to put an AI brain into a humanoid robot sooner or later.” And here we are.

In a lab, a glitch like that might just be chalked up as a code malfunction. Send it back to the developers, patch it, and move on. Do I expect a simple malfunction to launch Skynet? Not exactly. But human error, negligence, and shortsightedness certainly play a role in my thinking.

The Path to the Living Room

When humanoid robots first enter everyday life, they will be an exclusive luxury. The wealthy will use them as housekeepers, cooks, or even babysitters. But over time, as competition increases and production scales, prices will fall. Eventually, humanoid robots will become common household appliances.

Here’s the catch: in order to make them affordable, production costs must come down. And if there is one area companies love to cut corners, it is cybersecurity.

Imagine the “Amazon Special” version of a humanoid robot walking into all households. How do you think updates and patches will be delivered? My suggestion would be to require in-store updates, similar to how you take your car to the dealer for certain proprietary fixes.

But realistically, that will not happen. Instead, these robots will almost certainly connect to Wi-Fi for updates and maintenance. Once that happens, they are subject to the same vulnerabilities as your phone or work laptop. If the security is not airtight, a breach could hand control of your robot to a malicious actor.

And from there? I will let you imagine your own nightmare scenario.

Opening a ticket to get help from IT can feel like the most annoying part of your day. You just want someone to pick up the phone, fix the problem, and let you get back to work. Instead, you are forced to fill out forms, describe the issue in detail, and wait for a reply. It can feel slow, bureaucratic, and completely unnecessary, until you understand the real reason behind it. More on that later.

The Incident

A recent disclosure by Google revealed that attackers successfully infiltrated a Google Salesforce environment, one user at a time.

Credit where it’s due: Google’s disclosure was clear, detailed, and left little unanswered. The methods and motives of the attackers were well documented. This is how you disclose a breach in a thorough, airtight way, allowing your organization to move on.

The attack vector was not the most efficient, but it was effective. The threat actors used social engineering vishing calls, impersonating tech support. Once on the line, they convinced victims to launch a Salesforce data extractor, which enabled the download of a large portion of the environment, as outlined in the report.

Leaking Salesforce data is not inherently disruptive to business operations, and scraping it one victim at a time is tedious. So why bother?

Extortion. Salesforce data can include client lists, pipeline details, financial information, and more. The attackers were likely betting that some companies would pay a ransom to prevent public release or to have the stolen data deleted. While the ransom success rate may be low, the payouts from a single lucrative target can be significant if the stolen information is sensitive enough.

Google also noted that exfiltration attempts continued for months after the initial incidents. This suggests the attackers collect data from many organizations during a campaign, then sift through it later for the most valuable details.

How can you prevent this kind of breach?

Every breach is preventable with the right awareness and processes. Training users to recognize and resist fake tech support calls is critical. This is the reason IT insists on strict ticketing procedures. “We will not call you unless you open a ticket” is not just us being annoying; it is a safeguard for your company data. It is the last line of defense against an increasingly sophisticated AI powered vishing campaigns.

One of the phrases I hate reading in breach disclosures is “at this time.” It’s a placeholder that sounds cautious but signals uncertainty. In the latest update from Tea, it shows up four times.

As I wrote last week, the Tea dating app breach appears to have started with a legacy system, left online for compliance. They seemed confident that the data exposure was limited to that outdated system, not affecting newer users. That distinction mattered. It helped them avoid notifying their growing user base and kept the scope of the incident tightly controlled.

But yesterday, private message archives from a much newer database were leaked.

These messages aren’t just metadata or email addresses. They contain deeply personal, private conversations. Some people are calling this a second breach. That might be true, but based on the timing and nature of the data, it’s more likely part of the same compromise.

Still, the company repeats: “At this time, we have found no evidence of access to other parts of our environment.”

This is the nightmare scenario in cybersecurity. You think you’ve contained the breach, and then something surfaces that tells you the hole is deeper than expected. The messages leaked yesterday will likely result in more disclosures and require additional explanation from the company. For this reason Tea is likely hoping this is the last time they need to update the disclosure.

Oh boy. I wish I didn’t have to write about this one, but news is news.

For those who don’t know, there’s a social media app out there (female only) that lets users upload names and photos of men from dating apps or social media. That info gets cross-referenced with criminal records, marital status, and crowd-sourced stories from other women who have dated them.

Here’s an ad pulled from their website:

I’m not going to get into the problems with online dating or the ethics of what this app is doing. This is a cybersecurity blog. But the reality is, the service Tea offers is controversial. Most apps are focused on defending against threat actors who want to steal data for profit. Apps like this also have to defend against people who attack for fun, to make a point, or just because they can.

And someone just did exactly that.

The Breach

Tea was recently breached. Selfie photos and government IDs of the women using the app were leaked online. The very people the app was built to protect.

This wins the award of being the first disclosure I’ve seen coming from a TikTok post. Not a news outlet, not a press release. You can only find the website link to the disclosure from their TikTok post.

What Went Wrong?

The root cause is one I’ve seen too many times: an old archive system. It’s no longer tied to revenue, so no one prioritizes securing it. But they keep it online anyway, for the sake of compliance…but is that really necessary?

Just because you’re required to store data for legal reasons doesn’t mean it has to be online and accessible at all hours of the day for the rest of time. It doesn’t. Archive it properly. Store it offline. Bring it back only when a formal legal request comes in. What you shouldn’t do is leave it sitting on a live server forever just so you can run a report when needed.

My guess is that’s what happened here.

And now, all the personal info meant to be protected is sitting in one central place. That central place got breached. Now that data is out there, and it’s easy to use for the wrong reasons.

Will Tea Recover?

Probably.

We still have a long way to go when it comes to cybersecurity. But not if consumers don’t care.

The top comment on that TikTok expresses my concerns.

Want to read more about cybersecurity and data breaches?

Subscribe now

If you work in corporate America, chances are mergers and acquisitions (M&A) affect your work whether you want them to or not. At some point, a company becomes so successful that it saturates its growth opportunities in its core product. When that happens, the easiest way to add new revenue streams is often by acquiring or merging with another company rather than building something internally.

Every merger or acquisition includes a due diligence process to verify that the companies involved are what they claim to be. Among the many items reviewed is the cybersecurity posture of the target company. In an acquisition, this is typically a one-sided review. However, cybersecurity is rarely prioritized highly enough to stop a deal. It is unlikely that an M&A transaction will fail solely because of cybersecurity risks.

That means if you are responsible for cybersecurity during the M&A process, you should be prepared to make it work, one way or another.

After the deal closes, there is usually an “earn-out” period during which the acquired or merged company continues to operate as usual. During this time, security leaders from both organizations must work together to create a plan to merge the two networks and security operations into a single environment.

This is when gaps appear, and they can be costly if overlooked.

Knowledge gaps

One of the hard realities of M&A is redundancy. When two companies combine, fewer staff are needed. Morale is often low during the transition, and employees who know they may be leaving are not always motivated to help. Knowledge transfers frequently leave holes, and it is not uncommon to discover a server or device months later that no one understands because the person who knew about it was let go. You cannot secure what you do not even know exists.

Data gaps

Even in well-run IT environments, documentation is rarely perfect. Do you have a full inventory of internal and external IP addresses? Are you confident that your vulnerability scans cover all of them? Can you produce an accurate list of servers and their patch statuses? How certain are you that the list is complete? Or will you stumble upon virtual machines that were forgotten? These are common issues that must be addressed during the integration.

Technology gaps

One company uses Gmail, while the other uses Microsoft 365. One side has Palo Alto firewalls, and the other uses Cisco. Eventually one approach has to prevail, but sometimes companies try to maintain both at least for a while. When technology stacks do not align, users and administrators both face a steep learning curve with unfamiliar tools and settings, which increases risk.

Configuration standard gaps

Even when two companies use the same technology, the way they configure it can be very different. Over time, each IT team develops its own practices, and administrators are often very opinionated about the “right way” to do things. To maintain strong security, consistent configuration standards need to be established and enforced. This can be challenging but is critical.

When you look at many of the high-profile breaches in recent years, it is hard not to wonder how many were caused by weaknesses introduced during the M&A process. Large organizations often have dozens of past acquisitions layered into their operations, each with its own IT department and its own bad habits. Without a disciplined approach to cybersecurity in M&A, security hygiene can quickly erode.

Fresh off being featured in Beyond The Firewall for the Medicare breach disclosure, I went looking for a new breach to analyze. I can’t say I’m surprised, but once again we find ourselves in the healthcare industry. And once again it is a subsidiary of UnitedHealth Group (UHG) that has fallen victim to a cybersecurity incident that exposed the personally identifiable information (PII) of millions of customers.

In 2024, another UHG subsidiary was breached in a much more high-profile incident that received far more media attention. More on that later.

The Disclosure: Timing and Gaps

The disclosure itself offers little to analyze beyond the timeline and the notification process.

“On February 6, 2025, we found unusual activity in our computer systems. We quickly took steps to stop the activity. We began investigating right away and hired a special team to help us. We also called law enforcement. We turned off our computer systems to help protect our customers and their patients and members. We learned that a criminal was able to see and take copies of some data in our computer systems. This happened between January 27, 2025 and February 6, 2025. To date, we are not aware of any misuse of the data.

What Information Was Involved: On April 23, 2025, we began informing customers about what specific data may have been involved. The data that may have been seen and taken was not the same for everyone and includes contact information (such as name, address, phone number, and email), plus one or more of the following:

• Health insurance data (such as health plans or policies, insurance companies, member or group ID numbers, and Medicaid, Medicare, or government payor ID numbers)
• Health data (such as medical record numbers, doctors, diagnoses, medicines, test results, images, care, and treatment)
• Other personal data such as date of birth”

On February 6, UHG claims it identified and contained the breach. Over two months later, in April, it began informing affected customers. Yet it was not until this past week, nearly six months later, that the company made the breach public by filing disclosures with the states of Vermont and California. Further research suggests that other states may have received notifications as well.

Why No SEC Filing This Time?

During the 2024 breach, UHG filed an 8-K with the SEC. This time, however, they only notified individual states. Why did they consider the 2024 breach significant enough to report to the SEC but not this 2025 incident, which involved data loss affecting 5.4 million patients?

The only plausible answer is that they did not believe it was “material.”

The SEC requires an 8-K filing for material cybersecurity events. To be fair to UHG, the rules around cybersecurity disclosures on Form 8-K are still new and vague. Some have even argued that these requirements should be eliminated or culled. Notably, UHG even tried to walk back the materiality of the 2024 breach, despite the fact that it disrupted insurance payments and operations for weeks.

What Does “Material” Really Mean?

At its core, materiality means that investors need to know about an event because it impacts the company’s bottom line.

In the 2024 incident, UHG experienced widespread disruption. Systems were down and insurance payments were not being processed. That clearly affected operations and investor confidence.

But the 2025 incident? They “only” lost the personal information of 5.4 million patients.

No big deal, right?

We Still Have a Long Way to Go. How Do We Fix Disclosures?

Cybersecurity disclosures remain inadequate, not just in healthcare but across all industries. Accountability and transparency are still sorely lacking.

Only requiring disclosures when an incident threatens the stock price is not the right approach. For starters, only publicly traded companies fall under these disclosure requirements. Private companies are only required to notify states in the way UHG did for this breach. Don’t even get me started on small business and independent contractors. Many of whom possess very sensitive data. I’m looking at you attorneys and real estate agents.

I am not calling for heavy-handed regulation, and I respect the privacy of businesses. Disclosing every single incident across the board is neither reasonable nor feasible.

The Question Is: Do You Really Care?

Looking back, the uproar from the 2024 UHG incident was not because personal information was lost. It was because insurance payments were not being made. Yet we know that data for more than 100 million users was stolen.

If it had only been data for 100 million users, would it even have made the news outside of cybersecurity circles? Probably not.

That tells me that as a whole we have become numb to our personal data being stolen.

It does not have to be this way. But it will continue if we do not demand more accountability when handing over our data.

Not all days in cybersecurity are exciting. This is especially true in the case of leadership positions. Gone are the days of fixing vulnerabilities, responding to SOC alerts, otherwise generally being responsible for pressing the buttons to make the environment more secure. In leadership, these tasks are replaced by reviewing reports, assessing trends, and working with counsels to ensure we can agree to the security requirements of a new client contract. These can be tedious days. This all changes the moment you are breached.

You were hopeful this day would never come but now that it has you instantly become the center of attention for the entire company. No time to stress, this is what they paid you for. You call your family and tell them you’ll be working late knowing that late probably means through the night. Until the situation is contained and systems are back online, it’s a difficult situation to excuse yourself from.

When will we be back online?
How did it happen?
Do you know what they took?

These are all questions you will be asked multiple times and you will likely not have answers to all of them. All you can do is work with the information you have at the moment and follow your incident response plan. You communicate clearly, even if the answer is “we don’t know yet,” and you keep everyone focused on the process: contain, eradicate, recover, and learn. Staying calm under pressure and projecting confidence, even in uncertainty, is your most critical contribution. Every decision is scrutinized, and how you handle these moments will define how your leadership is remembered.

During the height of a breach, the CISO becomes the anchor in a storm of chaos. You are not just managing a technical incident…you’re managing people, expectations, and fear. Your role shifts into that of a crisis manager: coordinating teams, briefing executives, keeping stakeholders informed without creating panic. You have to translate technical realities into business impacts in real time, often while making tough calls with incomplete data. The hours blur together, but you maintain discipline, ensuring logs are collected, decisions are documented, and the narrative stays under control. You are the one who ensures the company has a fighting chance to emerge intact.

The end of the breach allows the company to move on from the issue but your job may continue for several more weeks if not months. Does someone need to be held accountable? I often find breaches are a result of oversight or failure to enforce policy. What information was lost and what is our responsibility to our clients for notification? You will work through all the intense details of the breach with your legal counsel to ensure that your message is accurate while limiting exposure. And of course, you take the necessary steps to ensure this does not happen again.

Cybersecurity leadership is hard. And breach is a humbling experience that will teach you this quickly.

Now that I have addressed the Medicare disclosure, let’s return to the Inside the Breach series. Today, we examine the Chief Legal Officer’s (CLO) role during a cybersecurity breach.

Outside of the CISO, no executive may have a more immediate and impactful role during a breach than the Chief Legal Officer.

On the Clock from the Start

If a cybersecurity breach is disruptive enough, the outside world will notice quickly. From the moment the news breaks, the CLO must work with PR to craft a statement that acknowledges the disruption, updates stakeholders on the progress of restoring services, and reassures investors.

In the early hours or days of the breach, details are often unclear, and rumors of worst-case scenarios may already be spreading. Even so, a statement still needs to be written based on the facts you have at hand.

Navigating Legal Obligations

As more details come in, the CLO moves quickly into action. Reviewing client contracts to determine breach notification requirements, meeting with company leadership to assess the materiality of the breach, and, if the company is publicly traded, preparing to file an 8-K disclosure with the SEC all become priorities.

These steps require both speed and precision because mistakes at this stage can erode trust and increase legal risk.

Closing the Loop

Once the investigation is complete and all facts are known, it is time to draft the final notifications. The 8-K filing becomes part of the public record, explaining the breach and its impact. In addition, every affected client must be formally notified, often within strict regulatory deadlines.

Why the CLO Matters

The CLO plays a critical role throughout the breach. Writing accurate statements that stand up to both legal and technical scrutiny is a challenge that cannot be underestimated.

This is why I offer a consulting service specifically designed for general counsels and law firms, helping them respond to breaches with confidence and clarity.

While the HR department may not be the first phone call during a cybersecurity breach, it undoubtedly has a vital role to play.

Operational Disruption

During a breach, critical systems may be taken offline. Employees may find themselves unable to perform their duties or meet performance objectives. As the lead HR representative, you may need to collaborate with management to revise performance metrics, ensuring fairness in light of the disruption.

Employee Support and Well-Being

On the technology side, your IT or Security team may be working long hours, possibly for several days straight, trying to contain the incident and restore systems. If team members have worked 48 hours without rest, how should that be reflected in time-off policies? HR must consider how to recognize their efforts, support recovery, and ensure compliance with labor guidelines and well-being standards.

Policy Violations and Accountability

If the breach resulted from a policy or procedural violation, HR may need to be involved in the investigation and any disciplinary action. Did the responsible employee complete the required security awareness training? Was the breach caused by an honest mistake, or was company policy intentionally circumvented? HR must help answer these questions with objectivity and consistency.

When IT Is the Source

What if the breach originated within the IT department? If an employee with elevated access violated policy, HR must manage the delicate balance of supporting the investigation while continuing to work with the department to restore systems. This requires coordination with legal, compliance, and leadership teams.

HR's Role in Crisis Management

These are just a few of the complex challenges HR may face during a cybersecurity breach. Such events often create unprecedented and uncomfortable situations, requiring HR to act with flexibility, sound judgment, and a clear understanding of both employee and organizational needs.

Many organizations have their top IT position reporting to the CFO. If that’s you, as the CFO, you now carry the significant burden of leading during a cybersecurity breach.

Already deeply involved in risk management, the CFO must continuously assess threats based on technical feedback from the IT team. Questions arise rapidly:

• Can we recover?

• Should we consider paying the ransom?

• How much will downtime affect revenue?

These are just a few of the high-stakes decisions you must confront. As the person closest to the CEO with direct insight into the IT team’s latest updates, you play a critical role in real-time decision-making. Meanwhile, calls with legal counsel start piling up as you assess potential fines and SEC reporting obligations.

As you begin to emerge from the crisis, you recognize a fiduciary duty to shareholders: preventing this from ever happening again. This typically leads to unplanned spending to patch the vulnerabilities that allowed the breach. Cybersecurity consultants may seize the moment, proposing solutions that significantly increase your technology spend.

You trust your IT team, but the technical details behind the breach are complex. You are not convinced the root cause was the absence of certain tools. Your instincts to protect the organization’s budget are valid. In many cases, breaches stem from failures in policy or procedure rather than missing technology. Every improvement in tooling helps, but you will remain vulnerable if you do not first address weaknesses in governance.

That is where I come in at MoveOn. I help identify the policy and procedural gaps that led to breaches, ensuring the fundamentals are solid before you commit to expensive new tools.

This is a series on what it looks like behind the scenes for corporate roles during a cybersecurity breach.

Maybe you’ve seen news articles written about cybersecurity breaches and the outages and data loss that come along with them. But did you ever wonder what goes on behind closed doors at the executive level of companies? Each company should have an incident response plan that defines what happens. But those plans don’t prepare you for the stressful moments you will encounter.

CEO – The CEO is pissed, first and foremost. They have a business to run, and getting the computers back online is not something that was on the agenda today. At the beginning of the breach, the CEO always has one question: How long will it take to fix? At this point, it doesn’t matter what data was lost if the company can’t make any money.

As the breach continues and the CEO meets with the other executives, they start to realize that this incident is now something that will have to be dealt with head-on. If the impact is customer-facing, the CEO may be getting calls and requests for private updates on the situation from clients.

The longer this continues, the harder the CEO’s job becomes. They know the company is on the clock in more ways than one, and not having answers for clients is not an option. The CEO will be requesting regular updates from technology leadership and will take a keen interest in their plan to bring systems back online, often with plenty of advice.

As hard as the CEO’s job is during the breach, it becomes even harder afterward. Now they have to win back the trust of all their customers. There may be media appearances and interviews, which means constantly working with the PR and legal teams. They also have to tend to the IT department and make sure the necessary changes to prevent this from happening again are actually implemented. That’s bandwidth they would rather use elsewhere, but putting this nightmare behind them becomes the top priority.

Next Article: The CFO

Backups are a bit of a lost art. I could write an entirely separate article on why that is, but that’s for another article. Let’s revisit the GitHub outage of 2017, read between the lines, and see what lessons we’ve learned from the chaos and the importance of backups and testing.

To Be Fair To Github

This incident took place in 2017, a very different time for data security and compliance. The level of transparency GitHub demonstrated during the outage was commendable, though it likely wouldn’t be legally advisable by today’s standards. Having been in similar situations myself, I have a lot of empathy for teams who sacrifice everything to restore systems. It’s both poor form and bad karma to be overly critical of our peers during moments like these. You’ve been there too. So, credit where it’s due: well done to the GitHub team for getting things back online, and apologies for featuring you in one of my articles.

What Happened?

On January 31, 2017, GitLab.com went offline for nearly 18 hours. An engineer trying to fix a database issue accidentally deleted data from the primary database server, thinking it was the backup server. As a result, a large amount of user data was permanently lost. This included around 5,000 projects, 5,000 comments, and 700 user accounts.

Why didn’t backups save them?

GitLab had multiple backup systems in place, but none of them worked when needed:

• Their main backup process, using a tool called pg_dump, had been failing silently for weeks due to a version mismatch. No one knew because the alert emails were being rejected by the mail server.

• Live database replication had already failed before the incident.

• Snapshot backups were either too old, too slow to use quickly, or not enabled at all for the most critical systems.

In the end, the only usable backup was a manual snapshot made six hours earlier. Everything created after that point was permanently lost.

What went wrong with recovery?

Restoring the system took so long because the GitLab team had to copy hundreds of gigabytes of data across very slow cloud storage. That alone took 18 hours. To make matters worse, the recovery tools and instructions were incomplete or unclear, which caused further delays and uncertainty during a high-pressure situation.

The Incident was not the issue

The reason I like using this incident as an example is because the level of detail shared during the outage paints a clear picture of what really happens behind the scenes during a technology crisis. Whatever your disaster recovery plan was, if you even had one, you might find yourself tossing it out the window once you realize the procedure you wrote down doesn’t actually work in practice. That’s exactly what happened here, for a variety of reasons.

Put simply, the GitHub team had never tested a full recovery of their database environment before they were forced to. The incident was the result of both technical issues and human error. But if you work in technology, you have to expect that kind of thing. People make mistakes. Software has bugs. %#!@ happens. That’s why we have contingency plans.

Here are two questions every executive should be asking their technology teams:

1. If we lost everything, what does recovery look like?

2. When was the last time we tested that?

I can almost guarantee the first answer will be “restore from backup” and the second will be “never.”

Testing full recoveries is not so simple

There are a number of problems with most backup architectures out there.

Infrastructure is expensive – Whether it’s cloud or on-prem, I know more than one CFO who nearly had a heart attack at the cost of fully upgrading our core infrastructure, where backups must connect to. Premium storage capable of handling a live workload is a huge cost driver.

Backup software can be even more expensive – I remember receiving a true-up bill from my backup software salesman after an infrastructure upgrade and thinking, that’s more expensive than the hardware! Now I had to go to an already-angry CFO and show him a new backup bill.

Backup storage is slow – Our production storage is costing us a fortune, and our software costs are just as high. We have to start saving money somewhere, right? Buying the cheapest disk possible for backups is usually a no-brainer. Sure, it may take 3 to 5 days to perform your first full backup, but after that, deduplication takes over and it gets much faster. The problem is that you can afford to be patient with backups, but not with restores.

You can’t test without being disruptive – Remember that infrastructure upgrade that made your CFO angry? Well, it’s sized appropriately and for growth. It is not sized to have a full copy of your backup restored alongside it for testing. In order to test without being disruptive, you need another copy of your infrastructure, completely separate.

Don’t most companies require testing for compliance?

Yes, testing is required, but the definition of a "test" is often determined by the organization itself. As technology engineers, we make do with what we have. A common way to test is through a partial recovery. This is typically done by restoring a much smaller portion of your environment, maybe a handful of servers or even a single database. As long as you're able to do that, you can meet the compliance requirements for most environments.

What can be learned?

How many organizations are in the position of never having tested a full recovery? If I had to guess, just looking at the Fortune 500, I would estimate 70 to 80 percent. But the real question is: how many executives in those organizations actually realize this?

If I take my technology cap off and put on my business hat, I really don’t want to deal with technology. I just want it to work and be secure. My interest in the details of our disaster recovery testing is low, and I assume my tech team has it handled. My interest in hearing about additional spend on technology is even lower.

Now, if I put my tech hat back on, I know that proposing a comprehensive disaster recovery testing plan is going to be a hard conversation. We will either need to increase spending to expand the backup and testing environment, or we will have to take production offline to make room for testing. That test could take days, because we bought the cheapest storage possible for our backups.

Neither side wants to have this conversation. And that is how we end up where 70 to 80 percent of companies are: with a partial disaster recovery test that does not resemble what recovery from a full-blown disaster would actually look like.

So we wing it when it happens, just like GitHub did.

What can be done?

Let’s assume we can’t fix the disconnect between the business side of the house and the tech side through better communication. We are tech people, after all, and not all of us are people persons.

Backups often fall outside the purview of cybersecurity and into IT operations. As a result, when it comes to compliance checks and audits, they are frequently overlooked. We tend to be far more concerned about preventing breaches than about recovering from them. This is a mistake. We need to assume a breach will eventually happen, and we must treat recovery as equally important as prevention.

Backups deserve more attention in both the NIST and ISO standards. A full recovery test requirement should be included in the controls. The scope of disaster recovery testing should not be left entirely to individual organizations to define.

On the legal side of the house, we should push for full recovery testing to be included in contract language.

One of the most important parts of securing your environment is putting into writing what secure actually means. Security is not just about prevention. It also means being able to recover from the worst-case scenario, and that capability must be documented and tested.

One of my talents is being able to see how technology evolves from its infancy to broader adoption. But as I heard Garth Brooks once say, “For every blessing, there is a curse.” Having this talent while working in cybersecurity means I often find myself worrying about how to defend against attacks before the rest of the world even sees them coming.

Having seen how effective phishing can be using nothing more than text, one of my biggest concerns now is AI impersonation during live video calls. I found one major example, where a finance worker in Hong Kong transferred 25 million dollars during a video call with what he thought were co-workers and executives.

The most chilling part of the report:

The elaborate scam saw the worker duped into attending a video call with what he thought were several other members of staff, but all of whom were in fact deepfake recreations, Hong Kong police said at a briefing on Friday.

Imagine getting a meeting invite from a spoofed email address that appears to come from your boss. You join the call and see your manager and several colleagues, all speaking normally. What would you do?

I would like to believe I could spot subtle cues. Maybe the lack of micro-expressions or off cadence in speech. But the truth is, this technology is getting dangerously good.

This Is Not Difficult to Do

So how easy is it to pull this off? Let me walk you through a simple demo I created using ChatGPT and HeyGen.

In this case, I asked ChatGPT:

"What is the single most important thing organizations can do to secure themselves in cybersecurity?"

ChatGPT responded with a pretty good answer.

I then used HeyGen to clone my voice and mimic my behavior based on a video I recorded months ago. I pasted the AI-generated script into the HeyGen tool and produced a polished, believable video without speaking a single word into a microphone.

Now imagine this process happening in real time.

Real Time AI Integration into Video Calls is the risk

While my example was built manually, we are quickly moving toward real-time integration. Tools like ChatGPT are already being embedded into voice assistants and live chat platforms. Soon, we will see AI agents join video meetings, responding and adapting on the fly.

In the Hong Kong attack, the threat actor built an elaborate multi-person deepfake simulation. But they did not have to. A simpler approach would be to use an AI agent to join the meeting silently, just to listen in and collect intelligence.

There are already many legitimate services that do this for note-taking. I use one. But it is not a stretch to imagine those same tools being used by attackers.

How Companies Can Protect Themselves

Here are six practical strategies organizations should adopt right now:

1. Implement Multi-Channel Verification for Sensitive Actions

If a request comes through video, confirm it via another medium. Require a phone call or secure messaging confirmation for tasks involving financial transfers or sensitive information.

2. Lock Down Calendar Access and Meeting Links

Ensure calendars are not publicly indexed or visible. Disable "join before host" and always verify who is joining.

3. Train Teams on Deepfake Awareness

Just like phishing simulations, run video-based social engineering drills. Teach employees how to question even familiar faces when high-stakes decisions are involved.

4. Use Code Words or Security Phrases

For executive-level communication, establish internal passphrases or behavioral cues that AI would struggle to replicate. I’m sure many of my former colleagues and I would have some fun with this one!

This is far from a complete list but it’s a start. We’re entering a brave new world with AI being part of lives and our jobs. It’s best to prepare rather than react!