In the past year, cyber adversaries have started to exploit artificial intelligence (AI) technology to execute a wide range of attacks. AI is helping them accelerate, optimize, and scale up their techniques. Agentic AI lowers the work factor for savvy adversaries, and it lowers the barrier to entry for new entrants. There is already an increase in the extent, tempo, and sophistication of attacks, and they are targeting diverse sectors. There are the well-known Salt Typhoon attacks on multiple prominent telecommunications providers and, more recently, attacks on water systems attributed to Iran.

AI and other new technologies are also creating opportunities for system developers and operational defenders. Large language models (LLMs) are now highly capable at rapidly finding vulnerabilities and, additionally, developing candidate repairs. In operational settings, AI can support more effective and rapid triage in security operations centers.

This combination of challenge and opportunity is creating an undeniable urgency in both executing and adapting the national cybersecurity mission. Leaders in cyber research, operations, and missions for national security must focus their efforts where they can rapidly and effectively bolster our country’s cyber safety.

In response to these disruptive changes, the SEI has recently released a report, Advancing Cybersecurity: A Framework of Cyber Research Priorities, that identifies seven areas of cybersecurity focus and elaborates on potential directions for research. In developing the framework, we led a team of more than 25 technical experts who identified the 37 highest priority opportunities within those seven areas that hold the greatest potential cyber benefit for national security:

• Analytic and Operational Tradecraft for Cybersecurity
• Securing AI-Based Systems and Workflows
• Cyber-Physical System Security
• Cybersecurity for Complex Integrated Systems
• Insider Threat and Human–Systems Interaction
• Secure Engineering and Mission Confidence
• Modeling and Simulation in Support of Security

Our team selected the particular topics on the basis of an explicit set of criteria. We aimed not to be fully comprehensive, but rather to identify topics that have the greatest potential impact on cyber operations and envisioned cyber capabilities in the Department of War (DoW), the intelligence community, the Department of Homeland Security, and other federal agencies. The set of topics considered in this framework focuses specifically on technical challenges that have the following characteristics:

1. Significant for national security and civil infrastructure.
2. Progress is likely to make the greatest difference.
3. Emphasizes both cybersecurity and also resilient response of complex systems under attack.
4. The talents, resources, and leadership of CERT and partners can be applied to best advantage.
5. Areas where the CERT Division and the SEI can build strong collaborative relationships.
6. Areas where both immediate and longer-term benefits can be realized.

The sections below take a closer look at the seven framing areas. The full report is available online and includes numerous references to the technical literature.

1. Analytic and Operational Tradecraft for Cybersecurity

Tradecraft encompasses operationally focused topics, combining computational and human elements to support secure operation of systems in an environment of active threats. We distinguish operational tradecraft from related topics associated with the design, engineering, evaluation, and sustainment of systems.

The tactics, techniques, and procedures of cybersecurity tradecraft to protect data and systems must constantly evolve to keep up with adversarial tactics and with the rapid advance in computational capabilities, most recently AI-based system elements, highly distributed system designs, and modern heterogeneous hardware designs.

The scope of the tradecraft challenge is extensive. In developing recommendations for research areas, the following issues were identified in our analysis process as particularly pressing and relevant:

• Event Analysis for Networked and Distributed Systems. Event analysis for networked and distributed systems includes practices and concepts for monitoring, detecting, analyzing, and diagnosing cybersecurity-relevant events in the operation of a system.
  
• Data Retention and Analysis. Event analysis is an issue at nearly all levels of scale, ranging from individual applications and systems to infrastructures and nation-state capabilities. In nearly all cases, data retention poses challenges because instrumentation associated with system and network processes usually generates a large amount of highly granular data, and organizations often can’t retain all of it.
  
• Analysis of Cyber Adversary Tactics, Techniques, and Procedures (TTPs). A full-scope analysis of cyber-adversary TTPs builds on data from threat intelligence, system operations, and software understanding (such as reverse engineering). Such analysis is critical to provide a better reckoning of adversarial capabilities, strengths, and weaknesses; to detect cyber campaigns; and to identify potential future targets before adversarial strikes.
  
• Reverse Engineering and Software Understanding. Reverse engineering is a traditional focus in defensive cyber operations. Goals in reverse engineering support response, recovery, and prevention through a (sometimes painstaking) analytic process that reveals the function and design of malware.
  
• Secure System Operations. Very large-scale systems can operate on premises, in a cloud, and in distributed and heterogeneous configurations. Secure operations for very large systems must be continually responsive to the rapidly evolving cyber environment, including both emerging threats and newly disruptive technologies.

Vision and Research Opportunities for Analytic and Operational Tradecraft for Cybersecurity:

Advancement of tradecraft must occur on many fronts:

• Efficient and Confident Analysis and Response
• TTPs for Efficient and Capable Cyber Red-Team Operations
• Capacity to Find and Diagnose Weaknesses and Vulnerabilities
• Rapid Creation and Validation of Software
• Cyber Hygiene for AI-Based Systems

2. Securing AI-Based Systems and Workflows

Modern AI computations are based on neural networks whose implementations range in scale from embedded sensor devices to giant data centers, with billions, or even trillions, of parameters. Attempts to support assurance-related judgments through analysis of these parameters, while showing partial success at smaller scale, have not been sufficient to support a principled understanding of outcomes at LLM scale.

In other words, it is generally not possible to make confident judgments that very large models behave in particular ways and, consequently, that incorrect and dangerous results can be fully avoided at the model level. These points illustrate the potential high complexity of engineering trade spaces for AI-based systems and the present infeasibility of establishing a universal cybersecurity rubric for such systems. This issue suggests that cybersecurity goals should be focused on intended system use cases, and that this understanding should guide system designers in making choices regarding both when and how to make use of the power of modern AI, and, additionally, how to appropriately evaluate whether the AI-based system is fit for use.

It is also essential, with agentic AI, to focus cybersecurity scrutiny on the harnesses and plug-in systems that connect the LLM with external tools and networks. These expose new attack surfaces, and they require principled engineering for security and, importantly, to support effective instrumentation with immutable logging.

We identify the following principal elements of scope in our analysis for cybersecurity issues associated with AI:

• Weaknesses and Vulnerabilities for Modern ML Models. Modern machine learning and LLMs make use of a range of statistical techniques to find patterns that are encoded during the training process into models that can support classification, detection, generation, and prediction.
  
• Cybersecurity for AI-Based Systems and Workflows. AI-based systems can have rich supply-chain structures, including multiple models and diverse sources of data for training and testing. Such structures create an extraordinary richness and diversity of attack surfaces. Importantly, the richness of attack surfaces motivates a three-level approach to AI engineering and AI-focused security analyses. These levels are as follows: (1) the individual AI models used within a system; (2) the design of the overall system that incorporates those models, including consideration of how that design can affect the exposure of potential attack surfaces; and (3) the role of the system within an operational workflow in a mission environment where there may be human operators and users.
  
• Techniques to Protect Against Attacks. Many of the weaknesses and vulnerabilities intrinsic to modern machine learning models do not have perfect remedies. In the case of data poisoning, for example, development of potential techniques for “unlearning” has not been fully successful, and sometimes the only mitigation (that doesn’t overly impair model behavior) is full retraining. Similar challenges exist in the prevention of patch attacks, where training techniques can be used to improve robustness in the presence of known categories of patches. These examples illustrate how operational suitability can be difficult to achieve for AI-based systems and workflow designs. There is general agreement in the AI community that close attention to the risk characteristics of specific use cases should guide decisions about using AI. When a use case has potential high consequence and potential threat, development teams must make careful choices about system design to limit the consequences of weaknesses and exploitable vulnerabilities in any incorporated AI capabilities.
  
• Red Teaming for AI. AI red-team practices must continue to evolve as new weaknesses, vulnerabilities, and associated assays and mitigations are developed. Because of this pace of progress, there is no static playbook of tactics, techniques, and procedures for AI red teams. AI red teams should engage throughout the lifecycle of conceptualizing, developing, and testing AI-based systems, including the earliest stages when training data corpora are identified, alignment issues are addressed, and model architectures are designed. Early engagement is also important at the systems level, where designs, for example, must be evaluated to assure insulation of internal attack surfaces from adversaries and to limit the effects of potential erroneous outputs from incorporated AI models.
  
• Special Considerations for AI Models Trained with Mission Data. There are applications where the mission data used to train AI models is highly sensitive. In general, models derived from sensitive data must operate at the same level of sensitivity that is applied to the data itself. Meeting operational requirements for models to operate at lower levels of sensitivity can present a significant technical challenge—indeed, at this stage in vulnerability research, researchers have little confidence that models can fully protect sensitive training data.
  
• Incident Response for AI-Based Systems and Workflows. Incident response for AI-based systems and workflows presents special challenges. These challenges include the usual elements of incident response—preparation and prevention, detection, analysis and diagnosis, containment, eradication and repair, recovery, and post-incident analysis. But there are additional difficulties in log reviews and regression analysis because outputs of machine learning models are the results of statistical model building rather than exact encoding of rules and patterns. New tools and TTPs must be created to offset this challenge.
  
• Mission Engineering for AI-Based Systems and Workflows. Mission engineering presents a challenge for AI-based systems and associated operational workflows since it must be informed by deep understanding of the special characteristics of not just AI models but also the design of AI-based systems (based on AI engineering principles). Mission engineering includes making decisions about how those systems are employed in operational workflows, possibly with human operators and users. (In advancing AI cybersecurity, the CERT Division has the advantage of close partnerships in academia, government, military, and industry as well as collaborations with the SEI AI Division and Software Solutions Division.)

Vision and Research Opportunities: AI-Based Systems and Workflows

The specific topics we identify to advance these goals and address these challenges include the following:

• Mission-Acceptable AI-Based Systems
• Security-Focused Test and Evaluation, Including Red-Team Operations
• Direct Analytics for Small-Scale ML Models
• Resilient and Secure System Architecture Design Patterns
• Relevant Best Practices and Standards

3. Cyber-Physical System Security

Use cases for cyber physical systems (CPSs ) include individual devices such as wearables, car key fobs, pacemakers, and prosthetic devices as well as networked CPSs for industrial control systems (ICSs) such as those for chemical plants, refineries, warehouse operations, and scientific laboratory instrumentation.

CPSs are pervasive in many sectors, and they are also pivotal elements in many military mission platforms and the multiple onboard capabilities that reside on those platforms as payloads. The span of scale for CPSs is broad, ranging from individual devices (e.g., cameras, health wearables, pacemakers, and vehicles) to societal-scale networked CPSs (e.g., electric grids, air traffic control, and weather sensor networks). Networked CPSs can be distributed and dynamic, ranging from sensor nets to dynamic swarms of autonomous elements. In many system contexts, CPS elements are used to manifest security functionalities, such as monitoring, logging, and identity and authorization.

This section outlines the following elements of scope to frame the most pressing issues related to cybersecurity for CPSs:

• CPS Design, Security, and Assurance. CPS designs are often driven by considerations of size, weight, power, and cost (SWaP-C). SWaP-C considerations of networked CPSs often encompass very large numbers of system elements in highly distributed settings. CPSs have additional technical challenges due to timing requirements and their often central role in control-systems engineering. These issues have fundamental impacts on choices regarding system design and development as well as on how testing and evaluation is undertaken. An example of a beneficial design pattern for CPS elements is rate monotonic analysis (RMA). It is important to note that adherence to this pattern—and realization of the benefits—can be nontrivial to assure.
  
• Legacy CPS Challenges. CPSs also have legacy challenges. Many early CPS-based systems were developed under the assumption that they were not on a public network, even when they were connected to a local network, because they were not connected to the open internet. Therefore, many of these early systems lack even basic cyber protections. SWaP-C considerations, noted above, could also impede security enhancements. Even when a networked CPS is not intended to be connected to the open internet, it is important to adopt resilient architectural designs that avoid propagating and cascading failures and compromises. Resiliency becomes more important with increases in scale and interconnection since a greater attack surface—with more expansive potential consequence—may be exposed. Such designs become particularly important when there’s a possibility that network bridges will be introduced to the system at some later point.
  
• CPS and OT Resilience and Autonomy. Resilience is essential for CPSs and OT. Highly distributed systems for national security missions can operate in challenging domains—land, surface, underwater, air, and space. Individual elements within these systems can be subject to failures, attacks, and compromise. This means that system-level design goals must be defined with relation to, for example, resilience and accommodation of attrition of system elements. These may depend on issues such as the kinds of communication modalities available among the system elements. Many CPS-based system designs involve semi-autonomy, with human operators in the loop or in supervisory roles on the loop. This means that designs must accommodate both the challenges of assuring autonomy and the realities of human weaknesses and adverse behaviors, whether those are the result of negligence or malfeasance.

Vision and Research Opportunities: Cyber-Physical System Security

The specific topics we identify to advance these goals and address these challenges include the following:

• Securable CPS Elements
• Networked CPSs and Communications
• CPSs and Autonomy
• Infrastructures
• CPSs and Edge AI

4. Cybersecurity for Complex Integrated Systems

Many of the most consequential national security, infrastructural, and enterprise systems are integrated from diversely sourced elements. Sources for systems elements can include custom developers, commercial vendors, open source projects, and cloud and software-as-a-service (SaaS) services provisioned through application programming interfaces. These supply chains pose special challenges such as unpredictable update cycles for independently developed system elements, complexity of interdependencies, and limited visibility into system elements, which may often remain opaque or only partly transparent even to system owners and cybersecurity evaluators. Diverse sourcing can mean not only rich supply chains but also high levels of complexity, distributed implementations, and long lifetimes of development, operation, and evolution.

The technical and business characteristics of complex integrated systems complicate the process of reaching confident judgments regarding cybersecurity characteristics that are essential for operational deployment.

This topic area addresses complex integrated systems of all kinds and the cybersecurity challenges associated with supply chains and integration. Certain systems have additional special characteristics that merit focused attention, however. These include, for example, AI-based and cyber-physical systems. The scope of this topic area includes the following:

• Complexity, Abstractions, and Vulnerabilities. Systems developers have, since the earliest days of software development, struggled with management of complexity. One of the principal ways to gain control of complexity at gross scale is through architectural design. Careful architectural design can minimize coupling among system elements and facilitate reuse of libraries, enabling a building-block approach. Fine-grained abstraction mechanisms in programming languages also play essential roles for gaining control of complexity. These mechanisms include typing mechanisms in programming languages, API designs, rules in protocols, and metadata in data stores. Abstraction mechanisms enable implementors to elevate their design thinking to concepts more aligned, for example, with a mission domain. This is accomplished in coding by separating decisions about how the abstractions are used from decisions about how they are represented at lower levels in an implementation, even down to the level of individual bits. This separation enables clarity of focus in ongoing engineering activity, even as overall system complexity grows.
  
• Operational Scoping, and Functional and Quality Requirements. Choices regarding operational scoping and functional and quality requirements strongly influence cybersecurity risks, particularly threat and consequence. This means that cybersecurity stakeholders should be engaged from the earliest stages of mission conceptualization, architecting, requirements development, and the workflow and user experience (UX) design process. For example, small adjustments to the UX design can help improve resistance to social engineering attacks on operators and users.
  
• Integration with Complex Supply Chains. Larger mission systems are typically integrated from components and services drawn from diverse sources. The software bill of materials (SBOM) was conceptualized to provide partial transparency into sourcing within the supply chain for complex integrated systems. A complementary approach, Supply-chain Levels for Software Artifacts (SLSA), is intended to prevent tampering within a partially opaque supply chain.
  
• Interconnected Systems and Data Exchange. Integrated systems increasingly feature distributed implementations with complex communications architectures, such as those for networked CPSs with dynamic configurations of interacting small platforms. These architectural choices can enhance security and resilience but also create challenges in T&E due to the attack surface that communication links and data-exchange formats and protocols expose. When integrated systems interconnect with other systems to form systems of systems, resiliency becomes paramount. Cascading failures among the interconnected systems can become increasingly consequential with scale. Interconnection apertures generally comprise multiple levels of abstraction, and each level can have different kinds of vulnerabilities. The lowest level involves the physics of low-level communication links. Just above this are protocols for the control and data aspects of those links. At higher levels, structured data formats can be used to represent imagery, signals, data tables, and other abstractions—all of which may require complex parsers that introduce new attack modalities.
  
• Long Lifespans and Information Loss. Integrated systems can have long lifespans and, over time, engineering information—models, design rationale, and the like— can be lost, withheld, or become inconsistent with evolving as-built system baselines. This means that sustainment teams for many mission systems must engage in significant reverse engineering efforts analogous to the kind of work that cyber analysts perform to understand the function and structure of incoming malware. It can take a high level of effort to recover, represent, and record information about complex integrated systems that’s been lost. When this effort is not undertaken, the risks and costs to repair and enhance existing systems can increase, which impairs systems evolution. The common elements of these “software understanding” challenges are detailed in a short report recently issued in a joint venture by CISA, DARPA, OUSW(R&E), and NSA called “Closing the Software Understanding Gap.”

Vision and Research Opportunities: Cybersecurity for Complex Integrated Systems

We identified the following elements as the primary areas of focus to address the issues we introduced:

• Systems Integration Practices
• Models for Integrated Systems and their Operating Environments
• Systems Engineered for Cybersecurity
• Sourcing Practices that Create Incentives for Security
• Capacity to Monitor Configuration Integrity
• Security and Integrity of Open Source System Elements
• Security for Cloud-Based Services

5. Insider Threat and Human-Systems Interaction

Human operators and users form part of the attack surface for most systems. A good example of exploitation of human weaknesses is the well-known phishing campaign called Operation Aurora, which targeted large firms. Social engineering attacks are often reported as the most frequently exploited pathway for adversaries to gain access to critical systems.

A key challenge in systems and workflow design is to reduce the opportunity for attackers to gain access to systems and data by exploiting human weaknesses and vulnerabilities. To address this challenge, we consider the following elements of scope:

• Cybersecurity Applied to Human–System Interaction (HSI). There are multiple kinds of human behaviors that can lead to adverse consequences, both intended and unintended. These behaviors can include negligence by human operators and users, which can be triggered by a motivated attacker taking methodical and informed steps. System developers and designers benefit from understanding that human weaknesses and cognitive biases are key considerations in the design for operator and user interactions of all systems, including AI systems. There are inventories of the various cognitive biases that can affect how people might behave as system operators and users, such as the one compiled by Wikipedia in its article Cognitive Bias Codex. An example of a cognitive bias is automation bias, which can cause human operators to uncritically defer to advisory inputs from an AI-based system even when operators are fully accountable for outcomes.
  
  It is well documented (see here and here) that the “human attack surface” is the dominant access pathway for cyber attacks. This means that HSI must be a consideration not just in operations but also in design and engineering. Design choices include determining how a system is integrated into operational workflows that include human operators and users. Design choices must also be informed through consideration of the full range of cognitive biases. Many scammers, for example, work to create a sense of urgency in their victims because people may bypass safeguards in a rush to comply. Urgency drives reactive behaviors, which can create a narrowing of attention known as cognitive tunneling.
  
  AI can facilitate attacks that capitalize on human vulnerabilities. Studies have shown, for example, that individually targeted and personalized phishing attacks are more effective.
  
• Anthropomorphization. With the emergence of LLMs and other AI systems that interact using natural language, it becomes important to understand anthropomorphization, which is the attribution of human characteristics to non-human entities. The effect of anthropomorphization is that people may impute meanings, emotions, and phenomenology to natural language texts generated by automated tools such as LLMs—even though the generated text derives from textual patterns in the training corpus rather than from underlying meanings. Human users may find that anthropomorphization is difficult to overcome even when they understand the mechanism by which the texts are generated.
  
• Engineering Designs that Accommodate HSI Realities. HSI-related design choices need to include consideration of the various kinds of human weaknesses, cognitive biases, and potential reactive behaviors relevant to the operational context of the planned system. This applies to both human operators in the loop, who are integral elements of system operations, and human operators on the loop, where supervisory attention and occasional intervention from a human are elements of an otherwise autonomous workflow.
  
  There are often tradeoffs in the respective roles of humans and systems in operational workflows with regard to security and usability. Designs should support usability but also minimize the opportunity for humans to bypass or evade security controls due to in-the-moment expediency, usability difficulties, or intentional malfeasance. Importantly, there are cases where advances in research can enable an escape from a tradeoff previously perceived as fundamental. An example is password practices, where new empirical research results caused the National Institute of Standards and Technology (NIST) to revise guidance to support passphrases, enabling higher levels of both usability and security.

Vision and Research Opportunities: Insider Threat and Human-Systems Interaction

The principal vision for HSI in cybersecurity is to develop designs for systems and workflows that reduce opportunities for human actions to create significant consequences—whether accidentally or through intentional malfeasance—in the operational context of the system. HSI poses special challenges because of the interplay of technical systems engineering, the modeling of intended and unintended human behaviors, and UX design. Elements to advance this vision for HSI include the following:

• HSI Design Practices for Systems and Their Workflows
• Models to Inform HSI Design Practices
• Data from Operational Monitoring of Critical Systems
• AI Models in Advisory Roles
• Red-Team Practices that Facilitate Risk Assessment

6. Securing Engineering and Mission Confidence

Weaknesses and vulnerabilities in systems are, for the most part, consequences of unintentional flaws and expedient decisions in the process of engineering those systems and their elements. Secure engineering practices are intended to reduce the extent of these weaknesses and vulnerabilities, ideally with minimum, or ideally, negative net cost. Secure practices range from architecture and coding to modeling and analysis.

The emergence of AI is significant both as an element of modern systems and as an enhancement to secure engineering practice. Secure engineering practices can reduce the extent of weaknesses and vulnerabilities in systems and, perhaps even more importantly, they can increase the work factor—the challenge and cost to attackers—to access and exploit any vulnerabilities that remain. The scope of this topic area includes the following elements:

• Trustworthy Critical Systems. There is a broad and growing diversity of applications that demand high levels of trustworthiness, ranging from faster-than-thought autonomy to security-critical system elements such as the secure enclaves used in edge deployments and commodity clouds. Secure engineering practices should not only enhance security but also support the affordable development, rapid assurance, and ongoing evolution of critical systems that must be trustworthy. An important additional challenge is how to enhance existing legacy systems so they can be secure in the face of emerging threats.
  
  Achieving trustworthiness includes establishing security-enhancing design principles for system elements, tools, and practices in a way that ensures that those design principles are respected as system developers integrate elements into the system. As noted earlier, these architectural choices affect not only security but also resilience, fault tolerance, and AI robustness.
  
• Engineering practices for critical systems. There are diverse practices for the design, development, and evaluation of critical systems. At code level, practices can include use of programming languages with security-enhancing features such as type safety, memory safety, and data-race prevention. Languages such as Rust and TypeScript provide security-relevant benefits. These languages have become widely adopted, and both are on many top-10 lists of popular programming languages. (See here and here.) This popularity is driven by a coincident property, which is productivity enhancement for software developers. The typing and race-prevention features enable them to find defects both more reliably and much earlier in the development process when those defects are easier to understand and repair.
  
• Formal Methods. Organizations are increasingly adopting formal methods and sound software analysis practices to overcome barriers related to scale, composition, usability, cost, and ease of integration into workflows. Formal methods can allow software developers to exclude certain categories of vulnerabilities, enabling human evaluators to focus their attention on other areas of concern. In fact, formal methods are already pervasively adopted, but mostly in the “invisible” form of programming language typing, memory safety, ownership (in Rust), and other attributes that are built into modern programming languages, their compilers, intermediate languages, and runtime systems. These properties are expressed succinctly in the language syntax, are supported by sound analyses, and are composable across multiple compilation units.
  
• Security-Focused Architectural Designs. Architectural choices can have enormous impact on security and resilience outcomes. Poor architecture choices can effectively preclude the possibility of resilient response and greatly hinder confident security evaluations. There are a number of technical architecture and design principles that can enhance security and resilience. For example, good architectural planning helps localize and reduce the extent of system elements that require the closest scrutiny in cybersecurity evaluations—the trusted computing base (TCB). Architectural choices are the principal means to minimize coupling and privilege sharing among all system elements, such as through Zero Trust. Importantly, architectural choices are the principal means to create resilience, enabling systems to “operate through” attacks on individual system elements through survivable and adaptive designs.
  
• Roles for AI. It is well understood that modern AI is creating enormous opportunities to enhance engineering and assurance practices. Consequently, roles for AI in the engineering of trustworthy and adaptable systems are rapidly broadening and diversifying, but there are also significant challenges.
  
  Roles for AI in secure engineering can range from coding and test development to inspection and advisory support for threat modeling and hazard analysis. In assessing prospective roles for AI, it is important to understand the specific capabilities and limitations of current AI-based tooling. Modern AI-supported software engineering can create order-of-magnitude improvements in the velocity of code generation and in vulnerability finding, as has become widely understood in recent months. Unfortunately, these speed-ups are not matched with corresponding speed ups in repairing defects and in developing evidence in support of assurance judgments. This AI Pacing Gap is a particular challenge for critical systems where positive evidence must be presented to support granting authority to operate (ATO). The harmonization of AI development tooling with formal methods and other techniques is a promising direction.

Vision and Research Opportunities: Cybersecurity for Complex Integrated Systems

Secure engineering practices, and particularly those involving formal methods and AI, are undergoing rapid evolution and advancement. Directed research is important to hasten this progress and enable its effective application to mission systems. Areas for research that we identified include the following:

• Memory-Safe Programming Languages and Beyond
• Secure Coding and Application Programming Interface (API) Design
• Architecture Design and Modeling
• Integration of Secure Engineering, AI, and Formal Methods
• Supply-Chain Practices
• Means to Organize and Analyze Bodies of Evidence

7. Modeling and Simulation in Support of Security

Modeling and simulation (M&S) and the related concept of digital twins have wide-ranging roles in advancing cybersecurity and resilient operations. These techniques support activities ranging from training and mission rehearsals to the development of mission architectures and operational concepts, and they provide rapid and affordable feedback throughout a system’s lifecycle. M&S enables proactive security assessments, accelerating innovation and enhancing preparedness.

Many kinds of models are significant throughout the lifecycle of a system. We identified the following specific elements of scope that relate to M&S in support of cybersecurity:

• M&S of Cybersecurity Systems and Operations. Cyber M&S can assist in both system development and mission operations. Cyber defense teams can be confronted by numerous threat vectors, ranging from high-frequency probes to more advanced and sophisticated attacks, including AI-created attacks. Realistic mission rehearsals enabled by M&S can build operator experience across a wide range of cyber scenarios. High-quality simulations can improve readiness by supporting the full spectrum of training exercises from basic training to complex, full-scale mission rehearsals. Here are some examples of M&S applications for cybersecurity:
  
  • Network attack simulations—Modeling of large-scale networks, such as those featuring Border Gateway Protocol (BGP) behavior and global latencies. These simulations can help defenders test resilience against denial of service attacks, routing manipulation, and other network-based threats. Tools like NS-3 and Mininet are commonly used for these simulations.
  • Insider threat modeling—Simulating actions of malicious or compromised insiders can reveal vulnerabilities in access-control mechanisms and data-protection policies. This application could involve agent-based modeling to represent employee behavior and data flow.
  • Vulnerability assessment—Using M&S to replicate attack vectors against a system or network can identify weaknesses before they are exploited. This assessment includes using M&S for multiple concurrent attacks to enable evaluation of security and resiliency.
  • Incident response mission rehearsals—M&S platforms can provide a safe and realistic environment for incident response teams to practice detection, containment, and recovery procedures.
    
• M&S Integration and Interoperation. Models are selective by nature. They represent certain features of the underlying phenomenon within the modeling formalism while excluding or abstracting others. This structure enables a particular model to focus on aspects of a phenomenon or system that are salient to the needs of the model builders, such as a particular set of analyses, simulations, or visualizations. Simulations relevant to cybersecurity may focus on aspects of concern to developers, operators, mission owners, and other stakeholders. Simulations could focus on the interactions of human operators, users, or adversaries within a system, and they could do so even before the system is built as a way to help conceptualize its user experience and operational designs.
  
• Model Realism and Validity. The validity of models, which is the extent to which the model reflects truths about both the system and its operating environment, is a key driver for acceptance of the results of simulation exercises. Digital twins in systems engineering and sustainment, for example, must accurately capture the salient aspects of the actual system and its operations in a mission environment. This fidelity to the system heightens the importance of managing the consistency of engineering data underlying a model with the underlying reality of the as-built system and its operating environment. Importantly, mission architectures, mission threads, and similar constructs are themselves models, and their consistency must be continually managed and assessed, especially when simulations are based on these other models. This continual management includes three facets: (1) verification, which is the assurance that modeling is consistent with the system and its specification; (2) calibration, which pertains to the adjustment of model parameters to align with observed worldly data; and (3) validation, which is the alignment of a model to mission operations and real-world phenomena.
  
• Human Elements. Developers of simulations that involve human operators and users often assume that exact fidelity of the visual and interactive experience is an essential requirement for a useful simulation. However, correspondence of “real-life” experience with the simulation experience may not require fully faithful renderings. For example, gaps in visual exactness may be acceptable because the simulation may compensate for those limitations in other ways, such as through close alignment on the tactical aspects of simulated action, which can lend validity to a necessarily abstracted model.
  
  There are multiple aspects of human behavior that bear on M&S success. Cognitive modeling involves the development of computational models to represent human cognitive processes such as decision making and maintenance of situational awareness.

Vision and Research Opportunities: Modeling and Simulation

Modeling and simulation models designed for cybersecurity simulation can support purposes ranging from requirements engineering to operator training and mission rehearsal. The following research areas support the advancement of this vision:

• M&S as a Key Linking Element
• Experience-Based Capability Development
• M&S Standards and Requirements

Integrating CERT, AI, and Software Solutions for Advanced Cybersecurity Defense

Although the work described in this report focuses on the CERT Division and its mission partners, there are significant areas of overlap with the work of the two other SEI divisions: the Software Solutions Division and the AI Division. The three SEI divisions have complementary strengths that can be integrated to advantage in advancing the key cybersecurity topic areas. Examples include the development of defenses against adversarial use of AI, the defense of AI-specific vulnerabilities, use of AI in cyber defense and offense, secure software development including for systems with AI capabilities, reverse engineering, resilient architecture designs, and more.