Public Safety

While users can also circumvent safeguards in closed AI models, such as by consulting online information about how to ‘jailbreak’ a model to generate unintended answers (i.e., creative prompt engineering) or, for more technical actors, fine-tuning AI models via APIs,³⁹ methods to mitigate these circumventions for an API-access system, such as moderating data sent to a model and incorporating safety-promoting data during fine-tuning, exist. These same mitigation strategies do not reliably work on AI models with widely available model weights.⁴⁰ Experimentation with available model weights, while often helpful for research to employ defenses against previously unknown attacks, can also illuminate new channels for malicious actors to exploit proprietary models because open models are easier to manipulate and can share properties with closed models.⁴¹

One mitigation that may work is using techniques including tuning a model on distinct objective functions and weakening its ability to produce dangerous information, prior to its weights being made widely available. However, we currently have limited technical understanding of the relative efficacy of different safeguards, and protections available to closed models might end up providing significant additional protection.⁴²

This Report considers two discrete public safety risks discussed in relation to dual-use foundation models with widely available model weights:

1. lowering the barrier of entry for non-experts to leverage AI models to design and access information about chemical, biological, radiological, or nuclear (CBRN) weapons, as well as potentially synthesize, produce, acquire, or use them; and
2. enabling offensive cyber operations through automated vulnerability discovery and exploitation for a wide range of potential targets.

Chemical, Biological, Radiological, or Nuclear Threats to Public Safety

Widely available model weights could potentially exacerbate the risk that non-experts use dual-use foundation models to design, synthesize, produce, acquire, or use, chemical, biological, radiological, or nuclear (CBRN) weapons.

Open model weights could possibly increase this risk because they are:

1. more accessible to a wider range of actors, including actors who otherwise could not develop advanced AI models or use them in this way (either be cause closed models lack these capabilities, or they cannot “jailbreak” them to generate the desired information); and
2. easy to distribute, which means that the original model and augmented, offshoot models, as well as instructions for how to exploit them, can be proliferated and used for harm without developer knowledge.

ACCESSIBILITY

This ease of access may enable various forms of CBRN risk. For instance, large language models (LLMs) can generate existing, dual-use information (defined as information that could support creation of a weapon but is not sensitive) or act as chemistry subject matter experts and lab assistants, and LLMs with open model weights specifically can be fine-tuned on domain-specific datasets, potentially exacerbating this risk.⁴³ However, the CBRN-related information open models can generate compared to what users can find from closed models and other easily accessible sources of information (e.g., search engines), as well as the ease of implementing mitigation measures for these respective threats, remains unclear.⁴⁴

Open dual-use foundation models also potentially increase the level of access to biological design tools (BDT). BDTs can be defined “as the tools and methods that en able the design and understanding of biological processes (e.g., DNA sequences/synthesis or the design of novel organisms).”⁴⁵ Intentional or unintentional misuse of BDTs introduces the risk that they can create new information, as opposed to large language models’ dissemination of information that is widely available.⁴⁶ While BDTs exceeding the 10B parameter threshold are just now beginning to appear, sufficiently capable BDTs of any scale should be discussed alongside dual-use foundation models because of their potential risk for biological and chemical weapon creation.⁴⁷

EASE OF DISTRIBUTION

Some experts have argued that the indiscriminate and untraceable distribution unique to open model weights creates the potential for enabling CBRN activity amongst bad actors, especially as foundation models increase their multi-modal capabilities and become better lab assistants.⁴⁸ No current models, proprietary or widely available, offer uplift on these tasks relative to open source information resources on the Internet.⁴⁹ But future models, especially those trained on confidential, proprietary, or heavily curated datasets relevant to CBRN, or those that significantly improve in multi-step reasoning, may pose risks of information synthesis and disclosure.⁵⁰

FURTHER RESEARCH

Further research is needed to properly address the marginal risk added by the accessibility and ease of distribution of open foundation models. For instance, the risk delta between jailbreaking future closed models for CBRN content and augmenting open models, as well as how the size of the model, type of system, and technical expertise of the actor, may change these calculations remains un clear. Previous evaluations on CBRN risk may not cover all available open models or closed models whose weights could be made widely available.⁵¹ Future analysis should distinguish between and treat separately each aspect of chemical, biological, radiological, or nuclear risks associated with open model weights.

Experts must also assess the amount open models in crease this risk in the context of the entire design and development process of CBRN material. Information about how to design CBRN weapons may not be the highest barrier for developing them. Beyond computational de sign, pathogens, toxins, and chemical agents need to be physically generated, which requires expertise and lab equipment to create in the real world.⁵² Other factors, such as the ease of attaining CBRN material, the incentives for engagement in these activities, and other mitigation measures—i.e., current legal prohibitions on nuclear, biological, and chemical weapons—also determine the extent to which open models introduce a substantive CBRN threat.

Offensive Cyber Operations Risks to Public Safety

Modifying an advanced dual-use foundation model with widely available model weights requires significantly fewer resources than training a new model and may be more plausible than circumventing safeguards on closed models. It is possible that fine tuning existing models on tasks relevant to cyber operations could further aid in con ducting cyberattacks—especially for actors that conduct operations regularly enough to have rich training data and experimentation environments.⁵³

FORMS OF ATTACKS

Cyber attacks that rely on dual-use foundation models with widely available model weights could take various forms, such as social engineering and spear-phishing, mal ware attack generation, and exploitation of other models’ vulnerabilities.

First, open foundation models could enable social engineering (including through voice cloning and the auto mated generation of phishing emails).⁵⁴ Attacks could also take the form of automated cybersecurity vulnerability detection and exploitation.⁵⁵ The marginal cybersecurity risk posed by the wide distribution of dual-use foundation models may increase the scale of malicious action, over whelming the capacity of law enforcement to effectively respond.

Cyber-attackers could also potentially leverage open models to automatically generate malware attacks and develop more sophisticated malware, such as viruses,⁵⁶ ransomware,⁵⁷ and Trojans.⁵⁸ For instance, one of Meta’s open foundation models, Llama 2, may have helped cy ber-attackers illicitly download other individuals’ employee login credentials.⁵⁹

Finally, actors can leverage open models to exploit vulnerabilities in other AI models, through data poisoning, prompt injections, and data extractions.⁶⁰

FURTHER RESEARCH

The marginal cybersecurity risk posed by dual-use foundation models with widely available model weights remains unclear, and likely varies by attack vector and the preexisting capabilities of the cyber attackers in question.⁶¹ For years, tools and exploits have become more readily accessible to lower-resourced adversaries, suggesting that foundation models may not drastically change the state of cybersecurity, but rather represent a continuation of existing trends. In the near term, the marginal uplift in capabilities for cyber attackers that widely available weights introduce to social engineering and phishing uses of foundation models may be the most significant of possible risks.⁶² Closed foundation models and other machine learning models that can detect software vulnerabilities, alongside other cyber-attack tools, such as Metasploit, can also be found online for free, and play a critical role in adversary emulation.⁶³ Further, while open models could provide new instruments for performing offensive attacks, hackers may not want to invest time, energy, and resources into leveraging these models to update their existing techniques and tools.⁶⁴ The extent to which a particular dual-use foundation model with widely available model weights would meaningfully increase marginal risk is therefore uncertain.

When an AI system or tool is built using a foundation model with widely available model weights, the inclusion of the model could introduce unintentional cybersecurity vulnerabilities into the application as well.⁶⁵ Promisingly, it is more readily possible to prevent these types of harms –where the deployer of the model does not desire for the vulnerability to be present – than harms intentionally leveraged by the deployer of the model.⁶⁶ In line with the Cybersecurity and Infrastructure Security Agency’s Secure by Design guidance, developers of AI models – whether open or closed source – can take steps to build in security from the start.⁶⁷

Benefits of Widely Available Model Weights for Public Safety

The open release of foundation model weights also introduces benefits. Specifically, widely available model weights could:

1. bolster cyber deterrence and defense mechanisms;
2. propel safety research and help identify safety and security vulnerabilities on future and existing models; and
3. facilitate transparency and accountability through third-party auditing mechanisms.

Cyber Deterrence & Defense 

Open foundation models can further cyber defense initiatives. For example, various cyber defense models, such as Security-BERT,⁶⁸ a privacy-preserving cyber-threat detection model, are fine-tuned versions of open foundation models.⁶⁹ These models and other systems built on open models provide security benefits by allowing firms, re searchers, and users to use potentially sensitive data with out sending this data to a third-party proprietary model for processing. Models with widely available weights also have more flexibility to be narrowly optimized for a particular deployment context, including through quantization, allowing opportunities for cost savings. Thus, several intrinsic technical benefits of openness allow a wider range of users to benefit from the value foundation models introduce to securing computing systems.⁷⁰ For instance, entities have published cyber-defense toolkits and create open-source channels for collaboration on cyber de fense.⁷¹

Furthermore, any advances in dual-use foundation models’ offensive cyber-attack capabilities may also strength defensive cybersecurity capabilities. If dual-use foundation models develop advanced offensive capabilities, those same capabilities can be used in securing systems and defending against cyberattacks. By detecting and ad dressing otherwise-undetected cybersecurity vulnerabilities, dual-use foundation models with widely available model weights could facilitate stronger cyber-defensive measures at scale.⁷² Parity of licit access to models that have offensive cyber capabilities is also important for ac curate adversary emulation, as advanced international cyber actors may incorporate such models into their own tradecraft. However, these benefits must be contextualized within the larger cyber defense landscape, as many developers perform their most effective cyber defense re search internally.

Safety Research & Identification of Safety and Security Vulnerabilities

Widely available model weights can propel AI safety re search. Open foundation models allow researchers with out in-house proprietary AI models, such as academic institutions, non-profits, and individuals, to participate in AI safety research. A broad range of actors can experiment with open foundation model weights to advance research on many topics, such as vulnerability detection and mitigation, watermarking failures, and interpretability.⁷³

Actors can also tailor safeguards to specific use-cases, thus improving downstream models. Creating external guardrails for dual-use foundation models can pose an abstract, under-specified task; actors that use open models for specific purposes can narrow and concretize this task and add on more targeted and effective safety training, testing, and guardrails. For instance, an actor that fine-tunes a foundation model to create an online therapy chatbot can add specific content filters for harmful mental health content, whereas a general-purpose developer may not consider all the possible ways an LLM could produce negative mental health information.

Open foundation models allow a broader range of actors to examine and scrutinize models to identify potential vulnerabilities and implement safety measures and patches, permitting more detailed interrogation and testing of foundation models across a range of conditions and variables.⁷⁴ This scrutiny from more individuals allows developers to understand models’ limitations and ensure models’ reliability and accuracy in scientific applications. An open model ecosystem also increases the availability of tools, such as open-source audit tooling projects, available for regulators to monitor and evaluate AI systems.⁷⁵

Experimentation on model weights for research may also help propel alignment techniques. Llama 2, for example, has enabled research on reinforcement learning from human feedback (RLHF), though the underlying RLHF mechanism was first introduced by OpenAI, a closed model-weight company.⁷⁶ Open models will likely help the AI community grapple with future alignment issues. However, as models develop, this research benefit should be weighed against the possibility that open model weights could enable some developers to develop, use, or finetune systems without regard for safety best practices or regulations, resulting in a race to the bottom with negative impacts on public safety or national security. Malicious actors could weaponize or misuse models, increasing challenges to effective human control over highly capable AI systems.⁷⁷

Auditing and Accountability

Weights, along with data and source code, are a critical piece of any accountability regime. Widely available model weights more readily allow neutral third-party entities t o assess systems, perform audits, and validate internal developer safety checks. While access to model weights alone is insufficient for conducting more exhaustive testing, it is necessary for most useful testing of foundation models.⁷⁸

Expanding the realm of auditors and allowing for external oversight regarding developers’ internal safety checks increases accountability and transparency throughout the AI lifecycle, as well as public preparedness for harms. This is for three reasons.

First, the developer may be able to use information from external auditors about its model’s robustness to improve the model’s next iteration, and other AI developers may be able to benefit from this information to identify potential vulnerability points to avoid in future models.

Second, third-party evaluations can hold developers ac countable for their internal safety and security checks, as well as downstream deployers responsible for which models they choose to use and how, which could improve accountability throughout the AI lifecycle. Of note, it may be difficult to implement such a third-party evaluation system due to differences in evaluations, the lack of ability to articulate how models can fail, and the scale of potential risks. Accessible model weights, alongside data and source code, facilitate oversight by regulatory bodies and independent researchers, allowing for more effective monitoring of AI technologies.⁷⁹

Finally, a robust accountability environment may increase public trust and awareness of model capabilities, which could help society prepare for potential risks introduced by AI. The public can then respond to and develop resiliency measures to potential harms that have been demonstrated empirically. A foundation model ecosystem in which many models have widely available weights also further promotes transparency and visibility within the field, making it easier for the broader community to understand how models are developed and function.⁸⁰

These community-led AI safety approaches could result in safer models, increased accountability, and improved public trust in AI and preparedness for potential risks. This transparency is vital for fostering trust between AI developers and the public and encourages accountability, as work is subject to scrutiny by the global community.⁸¹

Next: Geopolitical Considerations