COMMENTARY, ANALYSIS, AND UPDATES FROM OUR EXPERTS
Alexa Wehsener and Philip Reiner
Technology for Global Security (T4GS) is passionate about providing opportunities through which the public and private sectors can interact on critical issues, ideally fostering relationships and developing a bridge between communities in the process. As we recently noted, hosting cybersecurity tabletop exercises (CTTXs) is one way we accomplish this bridge-building.
Over the past year, T4GS has hosted CTTXs with a genesis around the threats posed by distributed-denial of service (DDoS) attacks. Last summer in Washington D.C., we hosted a group of experts representing the tech industry, federal government, and cybersecurity policy communities. These experts participated in a scenario in which a simulated cyberattack occurred on U.S. and allied critical infrastructure. The simulation was designed to explore the inevitable vulnerabilities, biases, and communication challenges that have yet to be adequately addressed in the current cyber crisis response protocol. This exercise was conducted under the Chatham House Rule, so participants will only be identified by their assigned roles, not by name or institutional affiliation.
T4GS staff, including senior former national security officials and elite cyber technical operators, constructed the simulation with the aim of emphasizing two themes central to understanding current cybersecurity challenges:
In the exercises these themes interact in novel ways, demonstrating how public and private institutions can collaborate, or not, during a real-world cyber crisis.
It became evident during the simulation, and even clearer in the after-action analysis, that narrow thinking steered the decisions of many actors, driven by incentives to protect their organizations and limit liability. This substantially impeded cooperation and produced flawed decision-making at key points, which resulted in a high-cost outcome in the simulation.
The exercise was governed by a control team made up of industry experts deeply embedded in the trust group communities responsible for identifying and mitigating these types of attacks on a day-to-day basis. Participants were divided into five teams representing the U.S. Government (USG), content providers and distribution, internet service providers (ISP), domain name system providers (DNS), and DDoS mitigation firms.
These teams were presented a situation at the Saudi Aramco facilities in the Persian Gulf which had caused major physical damage, to include the loss of life. Shortly after the scenario began, it was determined that the events at the facilities were the result of a plausible cyberattack. As they analyzed the digital forensic evidence surrounding the attack, the teams were confronted with an escalating DDoS attack targeting Saudi emergency response capabilities, which dramatically increased the scope of the potential threat. As teams worked through attribution, mitigation, and remediation efforts over the course of the next couple of days, DDoS attacks began also targeting major North American retailers during the Black Friday/Cyber Monday shopping weekend. The threat had apparently come home to the United States--and the teams needed to figure out the relationships while limiting the scope of the damage.
In response, participants worked to investigate what happened in Saudi Arabia and stop the DDoS targeting U.S. companies. While the teams were challenged to identify and eliminate the malicious activity that caused the various attacks, a command-and-control (C2) server was identified by several teams, and ultimately hijacked by the DDoS mitigation firms team in an effort to prevent further damage elsewhere against similar industrial control systems around the world. In doing so, however, this attempt to eliminate the threat accidentally triggered explosions at American oil refineries that maintained older versions of the same industrial control systems (ICSs).
Order was ultimately restored, but not without loss of life and billions of dollars in economic damage. When the dust had settled, signals and human intelligence strongly suggested that Hizbullah had orchestrated the sophisticated, multi-stage attack to punish the United States and Saudi Arabia for cooperating with Israel to contain Iran.
Players and the control team concurred on the relative implausibility of Hizbullah conducting attacks at this level of sophistication, but also concurred that such capabilities may well be handed off by state actors to their proxies increasingly over the coming years for a variety of purposes - the most critical of which is their ability to accomplish much of the same aims while benefiting from enhanced confusion, deflection, and deniability.
The (Majority of) Customers Are Always Right
A recurrent structural theme that emerged quickly in the course of the simulation was that for the owner-operators of infrastructure, even a crisis with enormous geopolitical implications is likely to directly involve only one or two of a firm’s hundreds, thousands, or tens of thousands of paid clients. In preparing for attacks on their networks, firms have, broadly, two options:
Because there is a highly competitive market for cyber-talent, most firms, especially smaller- and medium-sized firms, have opted for the latter strategy. As a result, firms may have to choose between continuing to provide stable services and support to their customer base as a whole, as opposed to responding to a dire situation that may only be affecting one or two clients during a crisis.
In the T4GS simulation, firms prioritized the needs of their customer base as a whole, even at the risk of prolonging or worsening a serious international cyberattack aimed at causing kinetic harm to critical infrastructure. In particular, content providers within the game feared that pulling resources away from their systems over a “Cyber Monday” weekend deeply important to their clients’ bottom lines could do permanent damage to their reputations and businesses. This moral hazard stems in part from a poor public understanding of cyber infrastructure: content providers were unlikely to bear significant blame for the catastrophe regardless of their conduct, so they saw little reason to stick out their necks. Rather, content providers generally preferred to keep their heads down and their customers happy until the storm had blown over. Significantly, one of the few actions that the content provider team took quickly was to demonetize questionable content potentially associated with the attack, so as to avoid the perception that they had profited from cyberterrorism.
A second dynamic that emerged quickly in the simulation was insufficient communication between teams. This breakdown was particularly problematic across sectors: while communication between firms responsible for different portions of the internet ecosystem ranged from strong to medium, communication between the U.S. government and owner-operators of critical infrastructure was almost universally poor. Realistic simulation of communication is a challenge in tabletop exercises, but participants in this exercise found that the communication challenges imposed by the rules of the game appropriately stood-in for the communication challenges they have faced in the real world, in both government and industry.
Two specific challenges were particularly apparent: communication authentication and liability concerns. First, requests for information sent from the U.S. government to tech firms involved in the crisis went unheeded on numerous occasions, on the basis that they came through unexpected channels and could not be authenticated in a timely fashion. In the event of a national-level cyber-driven crisis, tech companies should expect inauthentic communiques purporting to be from governments. These might come from unscrupulous independent journalists, from hostile groups or governments, or from other interested parties. In some cases, protocols for official USG outreach do not exist; in others, they are simply not well-known or followed. In the cases in which government outreach did occur through pre-established channels, classification challenges, and the general risk-aversion that surrounds them, prevented government expertise and context from percolating down to infrastructure operators.
Second, information-sharing was also impinged by long-standing concerns about liability issues such as user privacy and personally identifiable information. Because the primary targets of the attack were energy company hardware, rather than, say, healthcare infrastructure or corporate databases, the industry-side information liability concerns were somewhat muted. Still, with classification issues preventing the flow of information in one direction, firms saw little reason to risk legal jeopardy by sharing data with the government, knowing that their in-house counsel, their boards, and their customers all might punish them for it. The exception to this trend was the DNS team, who, because of the public nature of their data, were more comfortable sharing data with both government and other industry teams than the rest.
Location, Location, Location
Jurisdiction and territoriality are perennial challenges in fighting all manner of malign global networks, both on and offline. The Pentagon’s new “Defend Forward” doctrine aims to reduce the ability of bad actors to evade countermeasures by simply basing themselves, or even just their servers, outside of the United States in countries with weak institutional relationships with U.S. law enforcement and intelligence communities. Despite this new approach, territoriality still presents enormous challenges.
During the game, the location of the initial kinetic manifestations of the attack (eastern Saudi Arabia) limited the options of the U.S. government when it came to employing legal powers based on “exigent circumstances.” Though the situation was dire and fatal casualties had already occurred, the USG had little standing to demand user info from, for instance, a social network, because the attack had occurred outside the United States and no U.S. citizens or residents were known to be involved, either as victims or perpetrators. While “Defend Forward” provides U.S. Cyber Command with new options with respect to monitoring and disabling foreign networks, the posture does not change the underlying legal regimes that make it difficult for the USG to subpoena or otherwise demand access to other copies of servers or data absent clear evidence of a specific crime in or against the United States. In some cases, absent a subpoena, U.S. law not only permits firms to withhold customer data, but actually demands that they do.
Law of the Instrument
The likely responses in any crisis depend on the tools at hand. While a Defend Forward policy offers potentially more options for dealing with a foreign cyberthreat, it also presents major risks, including biases to action that could be exploited by a sophisticated foreign actor. In the course of this exercise, the USG team faced strong pressure from U.S. industry to take decisive action, as a sophisticated DDoS attack was costing U.S. companies billions in lost sales over a major shopping weekend. DDoS mitigation firms were under the same pressure from their clients. This pressure, which could have been engineered by a clever adversary, led the DDoS firms to disable a C2 server based overseas, despite having a poor understanding of its precise role in the attack
This C2 setup turned out to be a trap laid by the attackers. The server had been supporting a deadman switch, and disabling it activated malware payloads already implanted in ICSs based in Texas and Louisiana and caused further destruction. In this scenario, the DDoS team moved quickly, walking into the trap before the USG could, but it is easy to imagine an assertive government team, committed to Defend Forward, blundering into it as well. Such a gambit is not necessarily beyond the capability of sophisticated sub-state actors, and demonstrates the ways in which a greater number of tools at the disposal of both government and industry can also mean a greater threat surface, and more severe consequences for missteps.
Alexa Wehsener and Philip Reiner
At Technology for Global Security (T4GS), we do not accept the argument that there is a divide between technology companies and Washington D.C. Quite simply, there are differing cultures across all domains — we embrace that fact and thrive on it. More importantly, we accept that technology is moving faster than policy can keep up — which requires earnest, trustworthy venues for the honest exchange of ideas and tools.
As part of our suite of solutions to these challenges, T4GS conducts public-private cybersecurity tabletop exercises (CTTXs) to examine current trends and potential future crises. These games force players to actively imagine what is possible, question their assumptions, and collaborate with others outside their comfort zones. We aren’t the only ones who do this — but our approach is a reinvigorated approach to solving 21st century challenges through a time-tested and proven method.
These simulations require participants to respond to complex national-level contingencies driven by cyber attacks, but also to consider their actions and solutions in light of much broader consequences that demand public-private collaboration. This means vertical integration between network operators implementing ground-level technical solutions and government officials responsible for higher-level public protection, geopolitical responses, and policy decisions. At the same time, players must reach out horizontally across sectors to coordinate their response, share information, and utilize all available tools — even devise new ones. Perhaps most importantly, the exercises build the trusted personal relationships and deepened understanding across sectors that are necessary to be prepared at all levels to face the inevitable crises of the future.
The exercises are teaching us valuable lessons. We have routinely found that the introduction of a geo-strategic lens is at odds with typical industry technical and business approaches. In large part, players from industry excel at what they do best: protecting their customers and dealing with technical problems. Stepping out from this tactical and technical level of thinking to broader, more strategic considerations is important and often challenging. Vice-versa, those attending from national security policy circles tend never to have seen how security companies actually manage a crisis — to include seeing just how rapidly and effectively massive online threats are addressed and ameliorated by networks of trusted actors across industries — usually before many national security professionals are aware of the problem. Our CTTXs tie together the strategic and tactical levels in dynamic ways to encourage participants to actively practice and develop new relationships, ascertain and learn about new tools, and otherwise break the mold of existing assumptions about threats and solution sets.
In 2019, T4GS hosted several different Distributed Denial of Service (DDoS) -centric CTTXs. Participants often push back against a scenario when they feel that certain elements are so unrealistic that they cannot place themselves within the scope of the game. By combining open-source research with information from subject matter experts within our network, we produced a scenario that participants found not only plausible, but highly likely to actually occur in the ‘real-world’. Our game designs incorporate flexibility, allowing for small scale to massive engagements, and are built and run by teams that are composed of highly-regarded subject matter experts who make these games as real as imaginable. This experience and expertise enables technical deep dives and realistic injects throughout our games — keeping us technically honest while challenging the assumptions and standard relationships that experts tend to rely on in real-life crisis scenarios.
Part of T4GS’s mission is to build ever-stronger relationships between the public and private sector. Threats are too broad and increasingly at such a velocity we cannot think otherwise. As a non-profit operating at the nexus of technology and global security, T4GS engages leading experts through various means — including these unique exercises. The real value of these exercises rests in building greater levels of trust. Tabletop exercises force participants to actually do something challenging, together, and to think critically in unorthodox ways. While these exercises are run to address real-world issues, they are done so within the confines of a ‘game’ where mistakes do not result in large-scale consequences. As the threat environment continues to intensify, it is best we build this trust, and these relationships, before we are confronted with these crises in the ‘real world’.