Open Problems in Frontier AI Risk Management

A Test1 task in setting the scope is specifying the intended application of an AI system. What are the functional domains, test2policy sectors, intended operating conditions, and boundaries of the system? Why is the system being developed; what problems does it solve? What aspects of internal context (e.g., AI models, organizational features, data) and external context (downstream uses, socio-economic or political factors) are relevant to consider? Below, we review some of the open problems in setting the scope for frontier AI.

Intended (and unintended) use of the system. NIST’s “MAP” function asks organizations to specify why the system is being developed, the system-specific features and requirements, who the relevant stakeholders are, and what context-specific settings will matter (NIST, 2023). Considerations of intended use also depend on system affordances and limitations, which is why NIST’s Generative AI Profile encourages capability- and modality-specific considerations (NIST, 2024). They also provided prior guidance to create sector-specific (e.g., finance, healthcare) profiles where usage and context vary (cite). This challenge is exacerbated when attempting to scope intended uses of frontier AI systems, which may be further fine-tuned, given additional affordances e.g. tools and integrated into workflows across innumerable use cases (citations). Additionally, for risk management activities to be sufficiently responsive to possible frontier AI risks, decision-makers must look beyond intended uses (Boine & Rolkin, 2023; Gailmard et al., 2025; Galinkin, 2022; Mylius, 2025). Scope consideration thus also requires determination of unintended uses as well as “foreseeable misuses” of AI systems, which could lead to intended or unintended harms. Frameworks for anticipating misuse remain immature, particularly for general-purpose systems whose emergent capabilities depend on context, prompting, tool integrations, and downstream system interactions (Campos et al., 2025; Raman et al., 2025; Anderljung et al., 2023; Bengio et al., 2024). A growing number of systematic taxonomies of risks and harms is available, (Shelby et al., 2023; Slattery et al., 2024; Bagehorn et al., 2025) with increasing attention to societal, user-centered, and technically-oriented risks, as well as taxonomies focused on individual contexts, like chatbots, mental health, or deepfakes (Coeckelbergh, 2025; Bird et al., 2023; Zhang et al., 2025). Additional methods exist to support consideration of adversarial uses (Nganyewou Tidjon & Khomh, 2022; Kumar et al., 2019;  Perez et al., 2022). While documentation practices such as Model Cards (Mitchell et al., 2019) and emerging System Cards (OpenAI, 2024) attempt to codify intended uses and limitations, adoption is uneven (De Laat, 2021; Bommasani et al., 2024), and most organizations do not systematically enumerate misuse scenarios.

Boundary determination. Beyond use, another key aspect is to determine the boundaries of the system. While standards like ISO/IEC 42001 instruct organizations to define the scope of their AI management systems, this offers little guidance on drawing system boundaries for composite or vendor-integrated systems. Despite calls for attention to supply chain dynamics (Balayn et al., 2025; Widder & Nafus, 2023) or proposals like an AI bills of materials (e.g., BSI & ACN, 2025), there is no standardized way for organizations to enumerate dependencies while determining the scope. This leads to additional blind spots at integration interfaces and weakens downstream accountability. Notwithstanding local or sectoral recommendations (e.g., UK Local Government Association, 2025), this also creates informational asymmetries during procurement for officers involved in purchasing and contracting-out AI-based technologies (Studman & Marchiori, 2024). Overall then, scope determination is complex, covering broader social and institutional ecosystems, potentially multiple deployment contexts, and intended and unintended impacts from human or system interaction. 

Classification regimes as heuristics. A prominent institutional strategy for facilitating more efficient scoping is classification or risk tiering. For instance, the European Union’s AI Act introduces a tiered approach, differentiating prohibited, high-risk, limited-risk, and minimal-risk systems (European Union, 2024). The United Kingdom previously emphasized context-specific regulation (UK Government, 2024), and China relies on application-specific laws (e.g., for recommender systems or deepfakes) (Roberts et al., 2023). These kinds of classification efforts – whether focused on parsing AI tools by sectoral context or by overall riskiness – can streamline placement of an AI system into a relevant bucket, fast-tracking scoping decisions about relevant uses, plausible unintended uses, common stakeholders, boundaries, relevant objectives and risk mitigation strategies, and so on. While such an approach can simplify scoping (and help analysts determine what can be safely ignored), the utility of these heuristics is contingent on the quality of the classification schema, and potentially vulnerable to actors that might attempt to game the schema (Mokander et al., 2023;  Tlaie, 2024; Veale & Borgesius, 2022). For instance, if a certain system is only associated with one industry’s regulatory regime or inappropriately classified as low risk, determinations of context and riskiness may be similarly inappropriate. Further, given that different classifications and requirements are promoted by different jurisdictions, and because relevant frameworks or regulatory regimes may emerge from external (“non-AI”, sectoral) contexts, organizational scoping decisions are further complicated. This makes establishing a single risk management system not only difficult but potentially also highly contentious – something that organisations which aim to help companies navigate AI compliance, such as the AI Verify Foundation, Saidot and Credo, as well as various “cross walk” exercises for standards and regulations across jurisdictions are attempting to help organisations navigate (cite). In the absence of an international standard for frontier AI and in the midst of regulatory uncertainty, there remains significant uncertainty about how to define or constrain scope.

Stakeholders and affected parties. Beyond defining the intended purpose and deployment context, scope-setting involves identifying relevant stakeholders (Deshpande & Sharp, 2022). This exercise is similarly complex. Among the list of stakeholders the organisation should care about, risk management standards on AI like ISO/IEC 23984 mention customers, partners and third parties, suppliers, end users; regulators, civil organizations; individuals, affected communities, and societies (ISO/IEC 23894, 6.3.3). Standards like IEEE 7010-2020 require identification of directly impacted as well as indirectly impacted stakeholders  (IEEE 7010-2020). In the case of autonomous vehicles, for instance, relevant stakeholders might include drivers, pedestrians, ridesharing companies, taxicab unions, urban planners, parking enforcers, safety inspectors, transportation regulators, and disability advocates, each with potentially unique impacts to consider (IEEE 7010-2020). Along these lines, the OECD’s framework for the classification of AI systems (OECD, 2022) encourages not just looking at an AI model or associated tasks and outputs, but also understanding underlying data and input, economic context, and even considerations of “people and planet.” ISO/IEC 23894 also mentions to take into account whether AI systems can harm human beings, deny essential services or infringe human rights, and to consider external and internal expectations for the organization’s social and environmental responsibility (ISO/IEC 23894, 6.3.3). Impacts can thus pertain not only to individuals, but also communities, social groups, municipalities, political systems, and ecosystems. Stakeholder identification is thus closely tied with understanding AI systems’ penetration within and beyond their immediate context. It can be unclear what the downstream implications of an AI system are, or what impacts will be borne by users or other stakeholders, making scope setting uncertain and dynamic. 

Some prominent frameworks and tools encourage such scoping while also providing structured strategies to interrogate scope. GDPR mandates Data Protection Impact Assessments for high-risk data processing (European Parliament, 2016), Canada requires an Algorithmic Impact Assessment for federal automated decision-making (Treasury Board of Canada, 2020), and the Netherlands developed the Fundamental Rights and Algorithms Impact Assessment (FRAIA) (Dutch Government, 2021). IEEE 7010 encourages internal and external consideration of possible impacts and stakeholder engagement prior to setting objectives (IEEE 7010-2020). Yet, these assessments (under a variety of names) focus on a wide variety of topics: human rights, well-being, fairness, privacy, safety, transparency, hallucinations, or toxicity, with a variable focus on different stages of the AI lifecycle and technical versus sociotechnical dimensions, creating confusion around which one to apply and their implementation in practice (Deshpande & Sharp, 2022). Additionally, as we will see in Section X, research shows that in practice, most assessment tools adopted for frontier AI, such as safety evaluations and red teaming efforts, focus on risks closely related to model outputs (Feffer et al., 2024; Rauh et al., 2024) rather than user interaction harms or broader societal harms. Finally, while many frameworks urge participatory scoping, methods for involving affected stakeholders systematically remain limited or underutilized (Young et al., 2024), despite evidence that stakeholder engagement shapes fairer and more context-sensitive problem formulations (Passi & Barocas, 2019, Hanwen Deng et al., 2025). Overall, while impact assessments can be useful tools in contextualizing model uses, misuses, stakeholders, and boundaries, the diversity of tools hardly removes open-endedness of choices involved in scoping.

Closely connected to scope setting is the determination of clear and actionable objectives, needed to underpin choices throughout AI risk management, both at the level of the organisation and of the AI system. Below, we review some of the open problems in setting objectives at the level of the organisation and of the frontier AI system.

Organisational objectives. Organisational objectives are multiple, are up to the organisation to be determined and may differ in tenor: some organizations may wish to emphasize regulatory compliance while others may have prosocial or economic goals. Relevant organisational objectives specified in ISO/IEC 42001 vary from accountability to AI expertise (ISO/IEC 42001, Annex C). Depending on the breadth and depth of articulated objectives, organizations face many decisions, as each objective carries potentially different implications for risk management strategy (Schiff, 2025; Alvarez, 2025). Some objectives may be legally binding, and others aspirational, some carried out through programmatic risk management activities and others through high-level organizational decisions (Manning, 2017).Given the general-purpose nature of frontier AI, objectives can be too numerous, poorly defined, and hard to select between thereby undermining risk management activities, muddying their purpose and making it harder to evaluate success. The question of timing is also a confounding factor (Logan et al., 2021). Given the adaptable and updatable nature of frontier AI, when should objectives be set and how can organizations be confident they have enough information? While many frameworks treat scope identification and objective setting as early-stage activities (e.g., ISO/IEC 42001), many also recommend continuous and iterative development, which is easier said than done. For instance, the NIST RMF’s Govern and Measure functions articulate a feedback loop (NIST RMF), while IEEE 7010 encourages multiple internal feedback loops (IEEE 7010-2020). As we’ve seen from our discussion, setting objectives as well as scope setting are also entangled with risk identification and risk assessment activities, conceptually and procedurally, meaning it is not a given that scope setting can be simply performed a priori.

AI System’s objectives. Beyond purely organisational objectives, ISO/IEC 42001 also covers objectives related to AI systems per se, such as robustness, fairness, transparency and explainability (ISO/IEC 42001, Annex C). NIST RMF MAP too describes that system requirements (e.g., privacy) ought to be elicited from and understood by relevant AI actors, and that design decisions be made that take into account socio-technical implications in addressing AI risks (NIST RMF). At the same time, companies developing frontier AI systems formulate specifications for how their models should behave (e.g., OpenAI, 2025). Technical teams and data labellers adhere to this specification when designing and providing feedback and instructions to the system, in order to ensure that a system is aligned with specified objectives and values. This has raised questions over what set of human values AI systems should be aligned with (Korinek and Balwit, 2022; Lazar, 2023) especially when value alignment may be skewed by either the often limited diversity of companies’ teams relative to the world at large or, corporate profit motives (e.g., Abdulla and Chahal, 2023; Singh et al., 2024; Maslej et al., 2025). To mitigate these kinds of problems, some researchers have proposed paradigms for designing AI systems that balance competing views (Kirk et al., 2023; Sorensen et al., 2023; Sorensen et al., 2024; Caputo, 2024; Ali et al., 2025). Inspiration can also be taken from how other institutions develop normatively-accepted ways to choose what principles to align with (e.g., democracy) (Gabriel, 2020). However, other researchers have expressed concerns about “pluralism washing,” arguing that technical approaches to pluralistic alignment fail to address deeper problems related to systematic biases and social power dynamics in the AI ecosystem (Kalluri, 2020; Dobbe et al., 2021; Birhane et al., 2022; Sloane et al., 2022). Overall, the problem of pluralism poses challenges at multiple levels, requiring research on refining pluralistic alignment processes, the implementation of pluralistic alignment approaches, and institutional work to have meaningful representation from different stakeholders.

An important step before proceeding with the process of risk assessment is to set criteria for decisions which will be taken later on in the process (IEC 31010:2019, 6.1.6). This includes criteria for deciding whether risk is acceptable, criteria for measuring risk, and criteria for deciding between options (e.g., deployment decisions). Below, we review open problems for these steps. 

Setting risk acceptance criteria. Risk acceptance criteria are the terms of reference against which the level of risk is evaluated (ISO 31073, 3.3.6). Risk management standards listing specific methods for setting them (31010) propose techniques such as Risk Bearing Capacity (RBC) for risks related to the organisation – how much risk the organisation can take without without jeopardizing its overall stability, performance, or long-term goals -, and ALARP/ALARA for ensuring that safety-related risks are “As Low As Reasonably Practicable” or “As Low As Reasonably Achievable” (HSE). Risk acceptance criteria in the safety-critical industry evaluate the extent of harm, expressed as “severity x likelihood”, from potential accidents or losses on people, society, infrastructure, etc., within a specific intended context. For instance, as expressed by (Koessler et al., 2024), in the U.S. aviation industry the probability of “failure conditions which would prevent continued safe flight and landing” should not exceed 1 x 10−9 (one in a billion) per flight-hour (FAA, 1988). In the UK nuclear industry, the risk of death is “unacceptable” above 1 x 10−4 per plant-year and “broadly acceptable” if below 1 x 10-6 per plant-year (ONR, 2020). 

In frontier AI, the role of these criteria is often played by capability thresholds, which are “levels of an AI system’s performance or capabilities that, when reached, require implementation of specific mitigation measures to prevent risk exceeding established risk tolerance” (Campos et. al 2025, p.15). In order to set criteria for risk acceptance, these thresholds use capabilities as a proxy for risk. While they are not expressed in quantitative terms of likelihood and severity of harm, several tend to be linked to generalised harm scenarios (e.g., Meta (2025) links capabilities to high-level threat scenarios, and OpenAI (2025) links its capability thresholds to associated risk of severe harm). Given the nascent stage of AI risk assessments, capabilities are currently considered better proxies for risk than compute power and are currently perceived as easier to implement than risk acceptance criteria as described above (Koessler et al. 2024). Notwithstanding their increasing link with harm scenarios, capability thresholds remain limited as risk criteria insofar as they address the possibility rather than likelihood of harmful outcomes, and/or neglect other external factors (e.g., the threat landscape) that can influence both the risk scenario leading to the harm as well as risk levels (Koessler et. al. 2024, Caputo et al. 2025). As a consequence, also mitigation measures become harder to justify and calibrate because they are no longer directly tied to concrete accident or harm scenarios. To address these limitations, recent work offers guidance on the way capability thresholds can be operationalised more robustly, by using quantitative risk tiers and scenario-based risk modeling (Caputo et al. 2025; Koessler et. al. 2024; FMF, 2025; Wisakanto et al. 2025). One suggested approach involves introducing key risk indicators and key control indicators as measurable signals that can be traced to keep risks within an acceptable range (Campos et al., 2025). Additionally, a capability-centered criterion is not well-suited for some types of risks which are not made apparent through model capabilities, such as risks to fundamental rights. This could be facilitated though harm modeling and associated thresholds for intolerable risks including toxicity, deception, discrimination and socioeconomic harms (Raman et al., 2025). At the same time, some researchers point at the theoretical, conceptual, interdisciplinary and practical challenges involved in tackling fundamental rights through risk management concepts (Yeung, 2025). Other propose that these should be informed by input from the public (OECD, 2025) e.g. via multistakeholder panels (Schuett et al. 2025). 

Measuring/defining risk. Beyond setting criteria for risk acceptance, it is also important to define how one will measure risk. Traditionally, safety-relevant risk management measures risk in terms of severity and likelihood of harm (cite). In organisational risk management, risk is typically expressed as the effect of uncertainty on objectives (e.g., ISO 31000). While the connotation of risk in safety tends to be inherently negative, in organisational risk management it can be both positive and negative, and can create threats as well as opportunities (ISO 31000). Both types of risk management plan their processes to measure inherent risk, which refers to the foundational level of risk before mitigations (cite), and residual risk, which refers to the level of risk after mitigations (cite). Few industries use the concept of marginal risk, which refers to a difference in risk relative to a baseline (Williams et al., 2025). European Transport regulation, for example, uses GAMAB (“Globalement Au Moins Aussi Bon” in French), which translates to “globally at least as good” (Tchiehe & Gauthier, 2017). This specifies that a newly introduced transport system should not introduce more risk than the state-of-the-art, which is their baseline. The concept of marginal risk is used by multiple frontier AI companies, which may use different baselines (e.g., human (Wei et al., 2025), the state of the world at a previous point in time (Alaga & Chen, 2025; FMF, 2025), the companies’ own previous model (AISI, 2024), or a competitors’ model (Williams et al., 2025)). The widespread use of marginal risk in frontier AI presents a set of open challenges. First, the reliance on the concept itself, especially on baselines post-general-purpose-AI (Alaga & Chen, 2025), might lead to a boiling frog scenario, where overall risk is increased across the AI ecosystem without us noticing. Additionally, the lack of a shared baseline used across frontier AI companies (FMF, 2025), and especially the increasing tendency to adopt competitors’ models (Williams et al., 2025) might exacerbate this while promoting a false sense of safety. There also remains the question of the marginal risk posed by open foundation models compared to closed models and non-AI sources for which Kapoor et al. (2024) present an initial risk assessment framework. 

[add similar figure from Guide 51 showing how a risk is made up of severity and likelihood components]

Criteria to decide between multiple options. Another set of important criteria, as mentioned above, are those that determine how to decide between multiple options (IEC 31010:2019, 6.1.6.4). An organization will be faced with many decisions where often competing objectives are potentially affected (e.g. see Deployment decision, Section X), and there are both potential adverse outcomes and potential benefits to consider.  For such decisions, several criteria might need to be met and trade-offs between competing objectives might be required (IEC 31010:2019, 6.1.6.4). One prominent technique in risk management  is cost-benefit analysis (CBA) (cite), which bases decisions on expected financial or utility loss or gain. Other criteria cited in standards are decision-tree analysis (Kirkwood, 2002), which helps organizations express the utility of their decisions through a tree-like structure and multi-criteria analysis (Velasquez & Hester, 2013), thereby allowing different criteria to be weighted against each other. Criteria such as Robust Decision Making (RDM), used in Climate Impact and Disaster risk assessment, is suitable for systems involving significant uncertainty. It entails using simulation models to stress test one or a few selected options (e.g., policies or investments) across a large set of plausible futures and according to a list of multiple success metrics (cite) . While these techniques might prove relevant, their specific applicability or suitability for taking decisions around frontier AI remains untested to our knowledge, and will largely depend on application contexts. Some companies’ frontier AI frameworks make some far reference to CBA (e.g., Meta (2025) includes a Benefits Assessment alongside its Risk Assessment, Anthropic (2025) considers risks and benefits when developing their ASL standards) Still, CBA does not suit situations of high-uncertainty and struggles to express costs and benefits for the public good which are not expressed in terms of utility (cite), aspects which are particularly important in frontier AI.

A key aspect in the process of risk identification is to identify risk sources related to the development and use of frontier AI (cite). This should be informed by and in line with the defined scope (Section 1.1) and objectives (Section 1.2) presented above. Below, we review open problems related to the steps involved.

Considering AI risk sources. In risk management, a risk source is defined as an “element which alone or in combination has the potential to give rise to risk” (ISO 31073:2022, 3.3.10). A hazard, defined as “source of potential harm” (ISO 31073:2022. 3.3.12), is a particular class of risk source characterized by the presence of an inherent property, condition, or state that can cause harm if activated or realized. Hazards are often in the form of energy, agents, or conditions, e.g., high voltage, pathogenic agents, and loose cables on the floor. By contrast, risk sources that are not hazards do not necessarily possess inherent harmful properties; instead they give rise to risk through structural, behavioural, informational, or organizational characteristics, such as ambiguous decision authority, poor data quality, mis-specified system objectives, or inadequate change-control processes (cite). International standards on AI systems include as risk sources aspects like the complexity of the environment in which they operate, system lifecycle and hardware issues, lack of transparency and explainability and the readiness of the technology given the application context (see ISO/IEC 23894:2023, Annex B), to cite a few. Regarding frontier AI, the EU’s General-Purpose AI Code of Practice (European Commission, 2025), includes as risk sources model capabilities (e.g., offensive cyber capabilities), model propensities (e.g., tendency to hallucinate), model affordances and so-called “other systemic risk sources” (e.g., model configurations, model properties and context). NIST RMF Generative AI Profile also considers as sources aspects like human behaviour (e.g., abuse, misuse and unsafe repurposing) and interaction between humans and an AI system (NIST, 2024). There are also repositories of AI-related threats that can be used to consider risk sources. For example, the ATLAS Matrix catalogues tactics used in attacks of AI systems across various phases of its lifecycle (MITRE, n.d.). By considering these capabilities alongside the system’s characteristics and exposures, risk sources such as actors, assets, or conditions that give rise to harm can be identified. While many current AI companies’ safety frameworks (Anthropic, 2025; OpenAI, 2025; Google DeepMind, 2025) list model capabilities and propensities, there is often a lack of emphasis on specific model affordances (e.g., configurations) and context (e.g. way in which is made available on the market) as risk sources. It is key that the latter aspects are considered to be able to form a full picture of how these elements might give rise, or also increase, possible risks.

Selecting techniques for identifying AI risk sources. Many classical risk management techniques include identification of risk sources. For example, Hazard Identification (HAZID) is a structured brainstorming session typically held in the early stages of a project that aims to identify all relevant hazards and hazardous scenarios. Although the name emphasizes hazards, HAZID sessions commonly also elicit potential initiating events, high-level consequences, and existing or proposed controls to provide contextual information for subsequent risk assessment (Golwalkar & Kumar, 2022). A similar technique is Hazard and Operability study (HAZOP), which is a systematic examination of a process or a system that identifies potential deviations from the design intent, as well as their possible causes and consequences. HAZOP typically uses guidewords (e.g. “no”, “more”, “less”) applied to various parameters related to the system to surface potential deviations for analysis (IEC 30101). For example, a loss of cooling water (“no” cooling water) could lead to an increase in vessel pressure (deviation) resulting in a release of chemicals (Mocellin et al., 2022). Structured What If Technique (SWIFT), similar to HAZOP, is a structured brainstorming session that uses a predetermined set of guidewords (e.g. timing, amount, etc.), but also combined with prompts from participants in the format of “what if?” and “how could?”. Discussions in a SWIFT workshop include known risks, risk sources and drivers, previous successes and incidents, as well as known controls (IEC 30101). For example, in a clinical setting, a prompt can be in the form of “what if the patient list was not obtained?”, to which a risk of “failure or delay in referral” may be identified (Potts et al., 2014). While HAZID, HAZOP, and SWIFT are primarily forward-chaining techniques where the events and consequences are identified from the sources, there are other techniques in risk identification that are primarily backward-chaining where the sources are identified through the events or the consequences. For example, Ishikawa analysis (also known as the fishbone method) is used to identify possible causes of an event, by depicting the causes as the ‘bones’ of a ‘fish’ with the ‘head’ as the event (IEC 30101). Additionally, there is also failure mode and effects analysis (FMEA), where the system or process is subdivided into elements, and for each element its failure modes, its causes, and its effects, and the existing controls are considered. 

For frontier AI risk management, Wisakanto et al. (2025) proposes aspect-oriented hazard analysis, which provides systematic hazard coverage guided by a first-principles taxonomy of AI system aspects (e.g. capabilities, domain knowledge, affordances) by identifying critical risks through the system’s characteristics and context. Koessler and Schuett (2023) also provided examples on how frontier AI companies can apply techniques such as the fishbone method or risk taxonomies to frontier AI. They cite the fishbone method as useful to identify frontier AI risk sources for highly uncertain risks in the abstract, and in a systematic way working backwards from their possible consequences. However, they advise against using it for risk where sources may have a non-linear relationship, as the method is not able to handle risks that involve complex interactions between events, such as in competitive dynamics (Koessler & Schuett, 2023).  They also recommend risk taxonomies to avoid blind spots in the process of identification and to foster a common understanding of the risk landscape across multiple stakeholders. Yet, taxonomies are time-consuming and lack of empirical data might convey a sense of comprehensiveness while providing an incomplete picture. Notwithstanding this important work, the risks sources identified for most AI companies, often expressed as capabilities, tend to fall under several consensus risk domains (Frontier Model Forum, 2024, Jones et al., 2024), and it is not explicit how these risk domains were determined (Ziosi et al., 2025). An open challenge is therefore to ensure that frontier companies’ practices take stock of existing work, and that these techniques are properly applied to arrive at the relevant risk sources for the system at hand, rather than these being set beforehand.

Beyond – and sometimes alongside – identifying risk sources, it is key that one also identifies relevant elements like events, controls and consequences (cite). We review open problems for each in turn below.

Identifying potential events. An event is defined as occurrence or change of a particular set of circumstances (ISO 31073:2022. 3.3.11) and it can have several causes and several consequences. Classical risk management techniques such as HAZID, HAZOP, and SWIFT are also used to identify events that result from the hazards identified. Once hazards are established, these methods naturally take the next step of outlining the credible events these hazards can lead to, where specific initiating conditions are mapped to system deviations, failures, or losses (cite). In other words, these processes do not stop at naming hazards but follow through to the concrete ways risks might manifest. Among the relevant methods and sources to identify events, risk management standards for AI systems cite elements like market data or incident reports on similar systems, usability studies, and interviews and reports from internal and external experts (ISO/IEC 23894:2023). There are a set of AI incident databases which can be useful in this respect. These include the AI Incident Database (AIID, n.d.) and the AI Incidents and Hazards Monitor (OECD, n.d.). These historical incidents serve as a reference for potential events should the identified risks not be appropriately managed. Nevertheless, as many of these databases are built from voluntary incident reports, they may not be representative of the full range of potential events, as the events reported may be those that receive the most public attention instead of having the highest probability or severity. Furthermore, just as past performance in financial markets may not be indicative of future performance (cite), historical AI incidents will necessarily not be reflective of future incidents that have yet to occur. Thus, on top of the problem of ensuring past recorded incidents are representative of the actual distribution, another open problem is with identification of potential events that do not have precedence. 

Identifying controls. Controls are defined as measures that maintain and/or modify risk (ISO 31073:2022. 3.3.33). This is not to be confused with risk treatment (Section X), which is a process and not a measure. Risk treatment may include measures that implement, modify, or remove controls, which will be further discussed in Section 5. Risk Treatment. Additionally, while controls may be identified during risk identification, particularly in classical safety engineering methods (cite), controls can also be systematically selected, analyzed, and evaluated in subsequent stages of the risk management process (cite). Classical risk management techniques such as HAZID, HAZOP, SWIFT, and FMEA are used to identify controls as part of its process. As failure modes and causal pathways are identified, these methods also outline planned controls that are part of the system’s design to mitigate these pathways to harm (cite). In the context of frontier AI risk management, there are also existing repositories of AI-related controls such as the MIT Risk Mitigation Taxonomy (MIT, n.d.) the Secure AI Framework (Google, n.d.), and other academic sources (Gipiškis et al., 2024). While existing risk management methods and control repositories support the identification of candidate controls, the boundary between controls and hazards is often fuzzy in frontier AI. In contrast to traditional safety-relevant contexts – where hazards and controls are typically distinct (e.g., the fuel is the hazard, the fire extinguisher is a control) – many AI system potential controls, such as model capabilities or design choices, may also represent hazards. For example, code generation or reasoning capabilities may enable harmful misuse in some settings (cite) while serving as safeguards in others (cite), such as vulnerability detection or safety monitoring. Moreover, interventions introduced as controls, including fine-tuning for safety or human-in-the-loop oversight, can themselves introduce new risks by creating additional attack surfaces, reinforcing problematic incentives, or increasing reliance on fallible human judgment (cite). This ambiguity blurs the boundary between hazards and controls and complicates the application of classical risk management approaches to frontier AI systems.

Identifying consequences. Consequences are outcomes of an event affecting objectives (ISO 31073:2022. 3.3.18), where the objectives would be set in the previous phase as discussed in Section 1.2 Setting Objectives. Classical risk management techniques involving consequences identification are largely similar to those for identification of hazards, since identification of hazards naturally yields their associated events and consequences. Additionally, scenario analysis, which covers a range of techniques that involve developing models of how the future might turn out, can also be used as a forward-chaining technique to identify consequences (IEC 31010). This can include extrapolating past trends for the short-term and building imaginary but credible scenarios for the long-term (IEC 31010). This technique is also useful to inform risk analysis techniques later on, such as risk modelling (Section 3.3 Understanding/Assessing consequences and likelihood). In frontier AI, efforts have been made to explore various AI-related scenarios, such as those created by DSIT (2024). These scenarios picture various positive and negative consequences in the social, economic, technological, and environmental domains, ranging from limited AI adoption in the business community to catastrophic incidents on a global scale. Given the rapid pace of advancement and a limited understanding of its real-world effects, scenario analyses like these carry significant uncertainty (cite?). This is further amplified by model development itself sitting at the beginning of the lifecycle of general-purpose models, but many harms only emerging further downstream when models are integrated into AI applications (cite). Consequences may, therefore, be difficult to foresee at the model development phase (Touzet et al., 2025) and methods should be developed to bridge this gap.

In order to further comprehend the nature of risk, we here review approaches that contribute to gathering relevant information about the identified risk sources or hazards, events and consequences as  available to model providers through internal sources and methods.

Assessing an AI system’s properties/Internal Assessments. Internally, one can gather relevant information about the nature of risk and its relevant identified elements by further analysing risk sources related to the organisation and/or the product, possible events and potential impacts. In risk management, this can be done by gathering data on the type or the significance of identified risk sources, or analysing potential consequences through impact assessments, such as the intended impact of the development or use of the AI on relevant individuals or society (ISO/IEC 23894). When it comes to frontier AI, there is a strong focus on analysing the type and significance of risk sources or hazards posed by an AI system. As mentioned in frontier AI frameworks such as NIST Generative AI Profile (2024) and the EU GPAI Code of Practice (2025), the main approach is through conducting capability assessments or evaluations as a first step in analysing an AI system’s internal properties. Capability assessments are currently used to determine whether capability thresholds have been crossed and safeguards should be implemented (OpenAI 2025, Anthropic 2025, DeepMind 2023); whether the system possesses vulnerabilities that could enable harmful misuse (Carlini et. al 2024, Anil et al. 2024, Sharma et al. 2025); or whether the underlying model demonstrates undesirable propensities, values, bias or discriminatory tendencies (Greenblatt et al. 2024, Solaiman et al. 2024, Meinke et al., 2025; Huang et al 2025). This evidence can be used to inform a wide variety of decisions related to governing the risk posed by the system, such as whether to adopt certain technical mitigations, restrict access internally, delay or refrain from deployment, or pursue further investigation (cite). It remains an open question the extent to which evaluations can be relied on to inform these multiple, crucial risk-relevant decisions. Beyond concerns about their quality and robustness, as we will see below, it is still unclear how evaluations results should be both informed by and integrated within a broader risk model to reliably inform such decisions. The role of capabilities as potential risk sources or hazards, especially in their relation to risk, needs to be further defined in order to be able to rely on them for risk relevant decisions (e.g., for deployment or risk treatment decisions). As some authors remind us, in fact, evaluations assess capabilities, which are not directly translatable into risks (cite, e.g., Koessler). 

Ensuring quality, coverage, and robustness. While such assessments have improved in quantity and quality over the last few years, there remains a significant shortage of widely adopted and sufficiently high quality ones, and best practices are still evolving (Apollo 2024). Many existing assessments vary substantially in their methodological rigor, scope, and reproducibility (cite). Moreover, assessments might not elicit or represent a system’s full capabilities. For instance, evaluations might assess models with less access to inference time compute, fewer attempts, or less effective tools & scaffolding than future deployments of the system might realistically have access to (Barnett and Thiergart 2024, Volk et al. 2025, Götting et al., 2025).  It is also challenging to construct capability assessments that are robust for systems with increasing capabilities (Summerfield et al 2025, McKee-Reid et al. 2024, METR 2025) in ways that might undermine assessments’ accuracy (Greenblatt et al. 2024), particularly in more agentic scenarios (Anthropic 2025). Assessments’ results may also be highly sensitive to small differences in prompting and implementation (Schaeffer et al. 2023; Biderman et al., 2024; Burden, 2024; Robinson and Burden 2025, Sun et al. 2025), thereby hampering their reliability and reproducibility. Taken together, these issues complicate comparisons across models and over time, limit confidence in reported results, and weaken the role of evaluations as stable inputs into downstream risk analysis. This is made even more difficult by a lack of detail and comprehensiveness in how results of safety evaluations are reported publicly (McCaslin et al 2025).

Linking results to real-world behavior and harms. Building robust predictive models of AI system capabilities (Zhou et al. 2025, Hofstätter et al., 2025), and how these might map to real-world scenarios and harms (Barnett and Thiergart 2024, Mukobi 2024) remains difficult. First, evaluations should aim to approximate and thereby better measure performance on fuzzier real-world tasks with hard-to-measure outcomes (Phan et al. 2025). While some assessments are created after thorough risk modelling, many are not designed in a way that allows researchers to tie capability or propensity measurements to concrete risks and real-world harms (Righetti 2024, Williams et al 2025). As a result, the external validity of many evaluations – e.g., their ability to support inference about real-world impacts – remains limited. A further challenge is that most current evaluations focus on isolated individual models or systems. Multi-agent environments are still nascent (Hammond et al 2024, Agent Village), and assessments including representative numbers of human participants are generally rare beyond basic red-teaming exercises (cite). Frontier AI company assessments also often underinvestigate the effects of directly uplifting human capabilities, including in crucial, high-risk domains such as cyber offense (Righetti 2024). Finally, more realistic evaluations are frequently slow and costly – especially when they require large numbers of qualified participants or substantial compute – and, particularly in national-security-relevant domains, may pose security risks.

Beyond gathering information internally, it is also important that one engages with external actors or implements mechanisms to gather external information to inform a better understanding of the risks (cite). This is not only important to gather more information, but also to indirectly validate or stress-test the information gathered so far (e.g., capability assessment results), a key step in risk management (ISO 31010:2019). This step can also help identify new risks, trigger another round of risk assessments or revise the plan for risk management. 

External assessments. Risk management standards refer to engagement with external actors at a high-level, e.g., through communication with or participation of external stakeholders (e.g., 31000, ISO/IEC 23895). ISO/IEC 31010 specifically refers to “independent review processes” to verify and validate risk analysis (cite, 6.4.1). Regarding frontier AI, both NIST AI RMF Generative AI Profile (2024) and the EU GPAI Code of Practice (2025) prescribe that independent external assessors are involved in regular assessments, under specific conditions. These external technical evaluations are referred to by some as ‘external model or system evals,’ (see Xia et al., 2024; Sharkey et al., 2023), ‘external access’ (Bucknall & Trager, add ore) or technical audits (see Kluge, 2023). Such assessments focus on technical functionality, performance metrics or test for potential risks such ranging from bias to CBRN risks (citation to add, Mokander? Raji?). While they are a widespread practice in frontier AI, there is no consensus at any level of external assessment on the protocols these investigations should follow, their scope, or how results should be disclosed and implemented (see Cattell et al. 2025, Lam et al., 2024). Another problem concerns the depth of access (Kembery et al. 2024, Homewood et al. 2025). Research papers have proposed ‘structured access controls’ in which the depth of access is adjusted to the risk and enforced through technical controls – from API sampling to secure enclaves that enable parameter-level verification (Shevlane 2022, Bucknall and Trager 2023). However, it remains unclear how to strike a balance between meaningful access and preventing misuse, theft, or disclosure of sensitive research materials. Casper et al. (2024) even argue that black-box/output-only audits are insufficient for rigorous safety assessment, while outside-the-box access (to training and deployment information) can allow to scrutinise the development process and design more targeted assessments, and white-box access (to the system’s inner workings) can allow to interpret models more thoroughly, conduct more robust attacks as well as fine-tuning. Technical studies further show that some tampering attacks or safeguard-bypass behaviours only surface under weight- or activation-level interventions, reinforcing that output-only testing under-detects risk (Che et al., 2025). The resulting problem of ‘mutual privacy’ – protecting both proprietary designs and the results of independent testers – remains broadly unexplored despite proposals for secure technical environments, encrypted test setups, and contractual safeguards (Bucknall et al., 2025). Even as some governments propose strictly controlled access for professional evaluation agencies, developers remain cautious given the risks related to intellectual property, security, and misuse, leaving the central design problem of enabling sufficiently thorough testing without undermining safety unresolved (see Anderljung et al., 2023).

Ensuring structural independence. Another challenge is ensuring that evaluators are structurally independent in order to ensure credible results. Studies of early AI audit practices show that many contracts described as ‘external’ were in fact hybrid forms, in which the developer selects and finances the assessor, defines the scope, and may be able to veto the publication of negative results (Raji et al., 2022). Sandoval and Jing (2025) point out that these conditions also skew the choice of methodology in favour of fast, scalable tools over slower socio-technical investigations, as speed/scaling incentives favour automation and quantitative techniques. This mirrors the conflicts of interest in US financial auditing where auditing firms were financially dependent on the clients they were supposed to be auditing; a problem partially mitigated but not eliminated by later reforms (Moore et al., 2006). External assessments may face the question of how to prevent similar capture dynamics. In AI auditing, some actors are experimenting with mitigations such as independent audit committees or public declarations of conflicts of interest (Lam et al, 2024). Some proposed measures go even further: public or pooled funding of audits, lists maintained by regulatory authorities with rotating assignments to reduce familiarity bias, and a legal ‘right to publish’ to protect critical findings (Raji et al., 2022). For AI evaluations more specifically, some propose legal and technical safe harbour agreements that aim to address the documented deterrent effects on independent testing and disclosures (Longpre et al., 2024). Under these agreements large developers would commit not to take retaliatory action (e.g., account suspension) or assert claims based on terms of service against good-faith evaluators. Whether such protections can be operationalised for the external assessment of AI – and who will enforce them – remains unclear. Furthermore, the lack of specialised expertise, computational infrastructure, and reproducible methods means that the ability to conduct rigorous external evaluations remains concentrated in a handful of well-resourced organisations, raising further concerns about the independence and diversity of oversight (Costanza-Chock et al. 2022; Busuioc 2022; Anderljung et al. 2023).

Post-deployment assessments. Risk management standards (ISO 31000, ISO.IEC 23894) generally refer to the importance of “ongoing monitoring” and “periodic review” of the risk management process and its output as a transversal approach. Standards prescribing specific risk assessment techniques, such as IEC 31010, refer to the importance of monitoring and reviewing with regards to the process of analysis, especially by comparing actual outcomes with predicted results (2019, 6.4.3). Regarding frontier AI, both NIST AI RMF Generative AI Profile (2024) and the EU GPAI Code of Practice (2025) refer to “post-deployment AI system monitoring” and “post-market monitoring” respectively, as a specific step in the process of risk analysis. This is another way to gather external information which, depending on how models are deployed and data is shared, can be carried out by providers and third parties. Post-deployment monitoring mainly encompasses three categories of information; (1) model integration and usage data, (2) application usage data, and (3) impact and incident data (Stein et al., Pratt and Tanjaya). Model integration and usage data describe where and how models are deployed by developers or integrators, disaggregated by sector, geography, and application type. In practice, this information is limited and largely reconstructed from voluntary self-reports, surveys, national accounting frameworks, and public registries (Tamkin et al., Lopez et al., Highfill et al., Municipality of Amsterdam, Government of UK). Some model developers, such as Anthropic and OpenAI, have released high-level anonymized usage statistics, (Anthropic, Tamkin et al., OpenAI), but disclosure is overall limited (Bommasani et al). Application-usage data encompasses information about how users interact with applications deploying models in the real world, and could include AI application and AI agent activity logs or indices (MIT), user feedback channels, and coordinated flaw reporting by application developers (Catell et al., NIST, Chan et al.). These mechanisms are inconsistently applied and rarely standardized. For instance, some developers use OpenTelemetry instrumentation to track model calls or agent decision steps, but these practices vary widely and raise unresolved privacy concerns (Stein and Dunlop, Pratt and Tanjaya). Coordinated mechanisms for reporting application flaws to centralized bodies could help regulators and civil-society organizations identify patterns of failure and target interventions (Longpre et al., Gailmard et al., Richards and Zilka). 

Impact and incident data capture real-world harms and broader social or economic effects of AI systems, including reports of adverse incidents and sociotechnical field evaluations (Agarwal et al.). Current practices rely mainly on voluntary reporting to incident repositories such as the AI Incident Database and OECD AI Incidents Monitor (AIID, OECD, AIAAIC, AVID), and sociotechnical field evaluations. These independent repositories mirror precedents in other safety-critical sectors such as aviation and cybersecurity and demonstrate proof of concept for shared reporting, but remain voluntary and lack interoperability (Turri and Dzombak, Stein et al., McGregor, Agarwal and Nene). Sociotechnical field evaluations have generated meaningful insights into foundation model impacts on users (Zhao et al., Vaccaro et al., Tahaei et al) and subsequently informed model providers safety protocols (Hendrix). More robust analysis of model impacts and incidents will require sustained funding and interdisciplinary collaboration (Bengio et al.). In addition to reporting, post-deployment monitoring also encompasses model forensics: traceability techniques for analyzing model usage and outputs to identify misuse or harmful behavior. Current model forensics methods include embedding identifiable patterns into model outputs to make models uniquely traceable (Yu et al., Boenisch, Fernandez et al., Christ et al.,  Xu et al.), and watermarking methods which help to either uniquely identify certain models or verify that content is generated from a specific model (Li et al.,, Block et al., Gloaguen et al., Wan et al., Liu et al., Yang et al.) Metadata standards can also record contextual traces such as time, device, and location (Khan et al.). Further research is needed to evaluate how to standardize and integrate traceability tools into post-deployment monitoring practices, balance traceability with privacy considerations, and how regulators and auditors might use model forensics to attribute responsibility for reported harms (Cao, Goel et al,. Klasen et al, Hilgert et al.). 

In some situations, it is useful to provide a measure of risk as some combination of the magnitude/severity of potential consequences and the likelihood of those consequences (ISO 31010:2019, 6.3.7.1). As mentioned above, how “binding” this step is varies from organisational (ISO 31000, ISO/IEC 23894) to safety risk management (Guide 51), as well as on the types of risks at hand. All the information collected internally and externally can be used, as appropriate, to arrive at an understanding of the relationships within and between different risks as well as the severity of their consequences and likelihood. Below, we review open problems related to modelling risks and their interdependencies, as well as choosing the appropriate measures for risk.

Modelling risks and interdependencies. Risk modelling is an important part of the risk management practices of several high risk industries. Nuclear risk modeling for example combines Fault Tree Analysis (top-down deductive analysis tracing undesired events to root causes) with Event Tree Analysis (bottom-up mapping of potential outcomes following initiating events) (NRC, 2016). In cybersecurity, threat modeling scopes the scenario space by adopting an attacker’s perspective to identify assets, trust boundaries, and plausible attack vectors (OWASP, n.d.). In frontier AI, risk modelling features in frontier AI frameworks such as the EU’s Code of Practice on GPAI (2025) and it is used to analyse “how risks could materialize into concrete harms”  (Campos et al., 2025). This step, aided by techniques such as scenario analysis, connects technical system assessment with broader societal risk assessment by focusing on chains of causal steps linking model properties, user behaviour, and downstream impacts (Wisakanto et al., 2025)*. However, AI risk modeling remains nascent compared to established practices in other high-risk industries, though several approaches are emerging. 

Rodriguez et al. (2025) analyze how AI lowers attacker costs by identifying representative attack scenarios, locating bottlenecks in attack chains across the threat landscape, and quantifying how AI assistance reduces the costs of these bottlenecks. Wisakanto et al. (2025) integrate and adapt aerospace and nuclear safety techniques through a set of complementary methods including; aspect-oriented hazard analysis, identifying hazards by examining AI system “aspects” (such as capabilities, domain-knowledge and affordances), risk pathway modelling, tracing causal sequences from AI aspects to real-world harms, bidirectional analysis, using both forward chaining (similar to event-tree analysis) starting from capabilities and backward chaining (similar to fault-tree analysis) starting from potential harms; and propagation operators, accounting for how accumulation, correlation, adversarial exploitation, and sociotechnical diffusion may amplify and transform risks. Building on these ideas, Murray et al. (2025) propose a six-step risk modelling and quantification framework that selects representative scenarios, decomposes them into event sequences with quantifiable parameters, establishes baseline risk levels without AI involvement, identifies benchmarks and indicators as proxies to estimate scenario parameters, and aggregates individual parameter estimates using statistical tools into high-level risk estimates. This methodology has been applied to create nine quantified risk models in the cybersecurity domain (Barrett et al., 2025). 

Notwithstanding emerging work on the field, open problems remain. One open challenge is to ensure that risk models are sufficiently comprehensive (Shostack, 2014), such as taking into account all the possible interdependencies between the multiple possible components of risks and between different risk pathways. Risk models should draw on all available data sources, including API logs and usage data (a valuable resource for AI developers), incident reports (as mandated by frameworks such as the EU AI Act Code of Practice), AI capability evaluations, and uplift studies measuring how AI systems enhance human ability to cause harm. Accordingly, as mentioned above, further research is needed on methodologies for systematically combining multiple evidence streams into unified risk models. Frameworks that map evaluation results to specific parameters within risk models (building on emerging work such as Murray et al., 2025) offer a promising direction. Additionally, it is challenging to develop risk models in the absence of historical data on risks which have not yet realised, and on risks with low probability yet high-severity (cite). Where empirical data is unavailable, techniques such as expert elicitation – as we will see below – can help capture judgements from domain specialists (ISO 31010:2019, B.1). Conservative assumptions can also be used to simplify the more uncertain components of a model while erring on the side of caution (cite). However, substantial further research is needed in this area, particularly through collaboration with domain experts who understand the specific threat landscapes and can inform realistic risk pathways. 

Determining measures of risk. While risk modelling determines how different risk components interact, the choice of metric determines how those components are represented and quantified. This can entail quantitative, semi-quantitative or qualitative measures (IEC 31010:2019). Techniques for combining qualitative values of consequence and likelihood, for example, include index methods (ISO 31010:2019, B.8.6) and consequence/likelihood matrices (ISO 31010:2019, B.10.3). A single quantitative measure of risk can be produced from a probability distribution of consequences (e.g., VaR, CVar and S-curves). Depending on the situation and context, additionally, risk does not need to be expressed through one single measure or value (31010, and cite). Regardless of which specific approach or technique is chosen, it is important that different values are presented on the same scale or made comparable (cite). Additionally, it is key that an appropriate format is chosen depending on the risk at hand (IEC 31010:2019). While quantitative scores may be easier to handle as they allow for easier comparisons and aggregation, they can also be misleading if used inappropriately (ISO 31000). Extensive research shows that, for example, risk matrices and coarse ordinal scores are prone to mis-ranking risks, compressing uncertainty, and creating a false sense of precision, particularly when used for cross-domain comparisons or resource allocation (Cox, 2008; Aven & Reniers, 2013). 

There is only limited research explicitly referring to risk measures or estimates in frontier AI (Murray et al. 2025, Wisakanto et al. 2025), with some work only mentioning measures for evaluation scores with little or no methodological explanation on how measures are derived (DSIT, 2024, Weidinger et al. 2023, Solaiman et al. 2023). Specific examples of metrics include elicitation probabilities and capability scores (see e.g., Ho et al., 2025) or the recently proposed 50%-task-completion-time-horizon metric for long tasks (Kwa et al., 2025). This metric choice is also important once probabilistic structures are applied because the resulting estimates are only meaningful if the metrics capture a relevant aspect of risk. Additionally, different frontier AI risks present distinct measurement challenges. Some risks, such as discrimination or human-rights harms, are difficult to capture through clear estimates (Yeung, 2025), even when using semi-quantitative or qualitative approaches. Others, such as loss of control, are hindered by the absence of relevant historical data (Chin, 2025). While documents like the EU GPAI Code of Practice (cite) provide high-level examples of frontier AI risk estimates, specific guidance is lacking on how to present these in ways that remain interpretable and decision-useful while faithfully representing relevant aspects such as deep uncertainty, and the plurality of possible harm pathways, rather than encouraging premature closure around simplified risk scores. As frontier AI risks may not admit a single dominant “harmful scenario,” for one risk but rather a family of interacting scenarios (Barrett et al., 2025), it is unclear how these should be expressed in a useful and clear way through a single or multiple risk measures.