How DARPA Can Anticipate Downside Risk

Introduction

Mitigating adversarial technological tail risks by creating technological breakthroughs is core to DARPA’s mission. But the cost of technological development is decreasing for state and non-state actors alike, making effective safeguards around the misuse of DARPA-developed technologies essential. Recent advances in the study of probabilistic forecasting provide DARPA with a powerful tool it can incorporate into its risk assessments. And adjusting the guidance framework for program managers can help them better anticipate DARPA’s role in the technology landscape. The below recommendations will help ensure DARPA is not unintentionally increasing the likelihood of the very tail risks it seeks to mitigate. 

Invest in ELSI forecasting capability

While technological trends can be notoriously difficult to predict beforehand, forecasting is not impossible. Even in a rapidly changing technological landscape, alert actors can spot certain trends and patterns. By identifying and understanding these early signals, DARPA can better anticipate future developments.

In particular, it would be valuable for DARPA to build dedicated internal capacity to anticipate the likely ethical, legal, and societal implications of technologies in the years before they become commercialized. Probabilistic forecasting, a skill that can be cultivated through training,¹ can be leveraged² to assess the risks of various outcomes. This need not involve the use of complex forecasting models to be accurate and useful. Even one hour of training has been demonstrated by the Good Judgment Project to improve forecasting accuracy by about 14% over the course of a year.³ By accurately assigning probabilities to different identified risks and their potential impacts, DARPA can weigh relative risks to decide whether to proceed with a program and strategically influence the chronological ordering of these technologies’ development.

Better forecasting should allow DARPA to anticipate the balance of offense-dominant versus defensive-dominant technologies or sub-techniques of a technology. The precise ordering of when we get these technologies can matter tremendously for their actual impact on society, and DARPA has the potential to deliberately shape this ordering through its technological investments. This strategic approach, sometimes referred to as “differential technology development” or “terrain strategy,” can increase the likelihood that defensive capabilities do not lag behind offensive ones, and that ethically dubious technical approaches are superseded by ethically sound approaches, thereby reducing the potential for misuse or catastrophic accidents. But making these investments in a proactive fashion will likely require in-house technical forecasting expertise. 

To develop technology within ethical and legal standards, we propose a two-step methodology to evaluate ELSI risks during the proposal phase. The first step is qualitative, comprehensive consultation with a diverse array of stakeholders to identify potential risks. The next step, the focus of this comment, involves the quantitative evaluation of these identified ELSI risks. 

DARPA could consider several approaches when integrating forecasting into its existing risk assessment processes:

1. Internal Forecasting Team: A dedicated in-house team could focus on forecasting the ELSI implications of emerging technologies, and more broadly consider trends in emerging technologies and DARPA’s capacity to shape those trends. Such a team would provide consistent, readily accessible forecasting capabilities within DARPA — ensuring a constant stream of informed insight.
2. Tech Council-Led Forecasting: As an existing body with extensive expertise, DARPA’s Tech Council is uniquely positioned to incorporate forecasting into its oversight functions. This approach could provide an extra layer of analysis to the potential impacts of emerging technologies.
3. Program Manager-Led Forecasting: DARPA could leverage the intimate knowledge program managers have on their projects to generate forecasts. By encouraging managers to produce forecasts for their specific projects, we can harness their direct insight into the development process. However, this approach may be difficult to implement without adverse incentives, as managers may face pressure to downplay the risks of the projects they champion.

Any forecasting function should focus primarily on smaller, shorter timeline forecasts that resolve within the tenure of the forecasters. Repeated measurability of performance will allow DARPA to iteratively improve forecasting ability. The accuracy of these shorter-term forecasts should be used to improve DARPA’s institutional forecasting and inform DARPA’s assessment of probabilities assigned to longer-term higher-consequence risks.

Regardless of where forecasting is conducted within DARPA, forecasters should consider a range of potential risks, including cyber concerns, dual-use concerns, and the unintended consequences of the technology itself. They should include probability estimates for:

1. The likelihood that the technology could be misused, stolen, or replicated by an adversary within a given time frame. This probability should consider factors like the technology’s complexity, the availability of skilled personnel, the order in which features of the technology may be developed, and resources required for adversaries to exploit the technology.
2. The potential impact of misuse, if it did occur. This would depend on the destructive potential of the technology, and the damage it could cause if misapplied.

Finally, quantitative forecasts and the rationale behind them should undergo thorough review by red teams and security experts. In areas with sparse data, assessments can rely on expert judgment, simulations, or analogies to historical technologies, such as the management of nuclear technology misuse, or the recent handling of gene editing in the Safe Genes program.⁴

Using probabilistic forecasting to assess the risks of misuse aligns with DARPA’s use of Heilmeier Question #5 to evaluate a program’s overall risks. Moreover, it helps build an internal framework for iteratively improving DARPA’s forecasting ability. By integrating probabilistic forecasting into its risk assessment framework, DARPA can make more informed decisions and better prioritize programs — selecting those that offer the greatest benefits and pose minimal or acceptable risks.

Create an expanded guidance framework

DARPA should create an expanded guidance framework for program managers to better anticipate DARPA’s role in the technology landscape, encompassing proposal development and technology transfer considerations.

Given the emerging technology landscape, DARPA should integrate other frameworks to complement the Heilmeier questions. One approach is creating expanded Heilmeier questions, such as the ARPA-H Heilmeier questions guide, to provide program managers with a comprehensive framework to more accurately anticipate DARPA’s role in the technology landscape. Another similar approach is a set of internal questions a former IARPA director⁶ created to aid in developing new proposals. 

To better mitigate downstream technological risks, DARPA should work with external stakeholders to develop a similar set of questions, such as:

1. How quickly and accurately could an adversary develop your technology? 
2. What is the likelihood of your technology proliferating and what are the consequences if it does?
3. What types of risks are allowable for this project?

This expanded guidance will be instrumental to integrating probabilistic forecasting and quantitative thinking about ELSI risks. This approach can help anticipate potential ELSI issues and provide proactive solutions — thus strengthening the overall development and transition process of emerging technologies. DARPA should apply this expanded framework during the technology transfer process to minimize risks associated with technological proliferation. 

DARPA has previously displayed foresight to minimize risks; for example, placing speed and altitude controls on Global Positioning Systems (GPS) when they became commercially available. As dual-use technology becomes increasingly accessible for potential adversaries and the general public, developing and bolstering this foresight is essential. DARPA is uniquely positioned to identify and mitigate unintentional technological risks, as an organization mandated to prevent technological surprises by creating such surprises itself.