Policy Approaches
Restrict the Availability of Model Weights For Dual-Use Foundation Models
The U.S. government could seek to restrict the wide availability of model weights for specific classes of dual-use foundation models through existing authorities or by working to establish new authorities.
Restrictions could take a variety of forms, including prohibitions on the wide distribution of model weights, controls on the exports of widely available model weights, licensing requirements for firms granted access to weights, or the limiting of access to APIs or web interfaces. A structured access regime would determine who can perform specific tasks, such as inference, fine-tuning, and use in third-party applications.227
Another approach could involve mandating a staged release, where progressively wider access is granted over time to certain individuals or the public as the developer evaluates post-deployment risks and down stream effects.228
Additionally, a government agency could require review and approval of model licenses prior to the release of model weights or at other stages in a structured access or staged release regime.
Pros: Proponents of restricting model weights argue that such measures are essential for limiting nefarious actors’ ability to augment foundation models for harmful purposes. For instance, restrictions could reduce the accessibility of specific models trained on biological data, possibly creating a higher barrier to entry for the design, synthesis, acquisition, and use of biological weapons.229 Additionally, limiting the availability of specific advanced open-weight models could potentially limit the ability of countries of concern to build on these models and gain strategic AI re search advantages.230 Restricting the wide availability of model weights could potentially limit the capabilities of countries of concern, as well as non-state actors, from developing and deploying sophisticated AI systems in ways that threaten national security and public safety.
Cons: Restrictions on the open publication of model weights would impede transparency into advanced AI models.231 The degree of this effect, and other negative effects in this section, depend on the types and magnitude of restrictions. Model weight restrictions could hinder collaborative efforts to understand and improve AI systems and slow progress in critical areas of research, including AI safety, security, and trustworthiness, such as bias mitigation and interpretability.232 Restrictions might also hamper research into foundation models, and stifle the growth of the field.233 This could force investment and talent to relocate to more permissive jurisdictions, enhance adversary and competitor capabilities, and limit U.S. and allied autonomy to control the distribution of specific model weights. Targeted restrictions on certain classes of models may impose less of these costs than broader restrictions. Restrictions that use specific benchmarks or are not carefully scoped may not address some key risks and concerns.
For instance, AI-generated CSAM and NCII are created using models with widely available model weights that are well below the 10 billion parameter threshold of a dual-use foundation model.234 Further, if other countries with the current or future capacity to develop dual-use foundation models do not similarly restrict the wide availability of model weights, the risks will persist regardless of U.S. policy. Some commenters have argued that the sharing or open dissemination of model weights would be protected under the First Amendment, similar to protections that have been recognized by some courts for open-source software.235
Continuously Evaluate the Dual-Use Foundation Model Ecosystem and Build & Maintain the Capacity To Effectively Respond
A second approach would require the U.S. government to build the capacity to continuously evaluate dual-use foundation models for evidence of unacceptable risk, and to bolster its capacity to respond to models that present such risk. The U.S. government can leverage the information and research that an open environment fosters to engage in ongoing monitoring of potential risks of dual-use foundation models. By staying up-to-date on model advancements, the U.S. government can respond to current and future risks in an agile and effective manner.
Effective risk monitoring would require access to information on both open and proprietary foundation models, including dual-use foundation models and other advanced AI models, systems, and agents.236 Useful risk evaluation information could include data from foundation model developers, AI platforms, independent auditors, and other actors in the foundation model marketplace,237 238 model evaluations and red-teaming results,239 240 and standardized testing, evaluations, and risk benchmarks.241, 242 It could also include keeping track of key indicators in the economic and social systems that impact and interact with foundation models.
Best practices around evaluation and transparency will change over time, as will society’s perceptions of the most pressing risks, so the U.S. government would need flexibility in future adaptations of evaluation standards and transparency requirements. Monitoring of specific risks, such as CBRN or cybersecurity risks, may require liaising between agencies with specific subject matter expertise.243 In addition, monitoring requires secure storage of the research, including for external research and internal research with proprietary data.244 The risks that arise from open and closed foundation models involve not just the technology itself, but how those models interact with social, legal, and economic systems post-deployment.245 246 Consequently, effective monitoring and responsiveness would require combined technical, social, legal, and economic expertise.
Research and evaluation methods would need to be developed, including benchmarking, evaluation of capabilities, risks, limitations, and mitigations, red-teaming standards, and methods for monitoring and responding when appropriate to the more social, long-term, and emergent risks. International cooperation would also be needed.247 As other nations develop their governance frameworks for foundation models, the U.S. could work to collaborate on interoperable standards and guidelines with like-minded partners.
Pros: A monitoring approach gives time for the U.S. government to develop the staffing, knowledge, and infrastructure to respond to AI’s rapid developments.248 Monitoring allows for a more targeted approach to risk mitigation. If done well, it allows the United States to continue to benefit from the wide availability of model weights, such as through innovation and research, while protecting against both near- and long-term risks. The uses of AI will likely continue to change, as will the technology itself, and the marketplace of model developers, distribution platforms, companies using fine-tuned models, and end users.249 A monitoring approach would give time for the U.S. government to develop the staffing, knowledge, and infrastructure to respond appropriately.250 In addition, the increased AI capabilities that could come from this approach could support continued U.S. leadership on the international AI front.
Cons: Besides the potential risks of not restricting open model weights mentioned above, such as enabling innovation in countries of concern, one major drawback is the cost to the U.S. government. AI will impact many corners of government, so cross-sector monitoring capacity will likely require significant investment. Monitoring imposes obligations on companies, which could be costly, especially for smaller companies in the AI value chain, and burden the U.S. innovation ecosystem. Compelled disclosures to the government and public could also be intrusive and would need to be carefully considered to avoid exposure of proprietary information. If this approach is not done well, it could be a drain on government expenditures with out substantially mitigating risks. For example, as innovation leads to new uses, more unexpected harms will likely arise that require a government response. The U.S. government may also incur extra financial mitigation costs in areas such as cybersecurity defense.
Accept or Promote Openness
The U.S. government has tended toward a laissez-faire approach to many new technologies in order to promote innovation and permit market forces to shape the development of technology.251 On the one hand, a hands-off approach to the wide availability of dual-use foundation model weights can enable different competitive approaches to the development of foundation models252 but would rely on industry and the research community to develop methods for detecting and mitigating risks. Several foundation model developers have already articulated risk detection and mitigation frameworks that could serve as the focus for broader norm development across the industry.253 On the other hand, the U.S. government could further affirmatively promote the wide availability of model weights for dual-use foundation models. Further active steps could be taken, for example government policy could be used to support open foundation models through subsidies, procurement rules, or regulatory sup port for open models.
Pros: An approach involving minimal government action would pose the least risk of regulatory burden on developers of dual use foundation models. It is likely that the main benefits of openness would arise from innovation and research.254 Openness may provide more access for small businesses to access foundation model resources.255 Open resources are the norm among academic researchers, who draw on previous work to build a collective, public body of knowledge.256 In recent years, private companies have overtaken academics in AI research.257 258 Encouraging openness could potentially reverse that trend. In addition, incentives for openness could support greater access for researchers to examine models for safety, security, and trustworthiness, including bias and interpretability.259
Cons: There are several significant drawbacks to a hands off or affirmative promotion approach. There has already been significant involvement by both the U.S. and other allied governments in obtaining industry commitments and developing standards for AI risk management. Also, as discussed, there are significant security, societal, and strategic risks that may yet materialize from dual-use foundation models. This option would constrain the ability of the U.S. government to understand the developing risk landscape or to develop mitigation measures. Incentivizing openness may well exacerbate many of the risks from dual-use foundation models with widely available model weights that have been outlined in this Report.260 For example, without restrictions on sharing model weights, dual-use foundation models that create novel biorisk or cybersecurity threats could be used by a wide range of actors, from foreign nations to amateur technologists. As innovation leads to new uses, new and unexpected harms will likely arise. Besides the negative societal effects that these risks could create, the U.S. government may also in cur extra financial mitigation costs in areas such as cyber security defense.
227 Rand Comment at 4 (“The most common approach to structured access is to create flexible application programming interfaces (APIs) that allow researchers, small businesses, or the public to access the model.”).
228 Anthony Barrett Comment at 3 (“Foundation model developers that plan to provide downloadable, fully open, or open source access to their models should first use a staged-release approach (e.g., not releasing parameter weights until after an initial secured or structured access release where no substantial risks or harms have emerged over a sufficient time period), and should not proceed to a final step of releasing model parameter weights until a sufficient level of confidence in risk management has been established, including for safety risks and risks of misuse and abuse.”) (internal citation omitted).
229 Johns Hopkins Center for Health Security at 6 (“…none of the small studies in the field so far have evaluated how much dual-use foundation models purposefully trained on relevant data (eg, virology literature) will marginally improve bioweapons development or assessed the interaction between LLMs and BDTs. 24 Nor, to our knowledge, have there been any published evaluations of the marginal benefit BDTs like Evo or RFdiffusion could play in bioweapons design.”).
230 Mozur, P. et al. (2024, February 21). China’s Rush to Dominate A.I. Comes With a Twist: It Depends on U.S. Technology. NYTimes.
231 National Association of Manufacturers Comment at 2 (“The availability of model weights allows independent examination of a model to ensure it is fit for purpose and to identify and mitigate its vulnerabilities.”). At the same time, the benefit of transparency may be relative to the availability of other components and resources. See, e.g., CSET Comment at 16 (“Models with publicly available weights fit along a spectrum of openness, and where they fit depends on the accessibility of their components [. . .] More research is needed to gauge how different degrees of access and transparency can impact the ability to scrutinize or evaluate open models. For example, many open model come with documentation and model cards, but the level of detail in these documents can vary dramatically, and they can enable (or not enable) different degrees of evaluation.”); Databricks Comment at 10 (“Making the model code widely available in addition to the model weights provides the benefits of incremental transparency in evaluation the model[. . .].”).
232 See, e.g., EleutherAI Institute Comment at 24 (“Open-weights models allow more researchers than just the small number of at industry labs to investigate how to improve model safety, improving the breadth and depth of methods that can be explored, and also allows for a wider demographic of researchers or auditors of safety.”); Databricks Comment at 5 (“The biggest risks Databricks sees are the risks that would be created by prohibiting the wide availability of model weights: i.e., the risks to economic productivity benefitting a larger swath of society, innovation, science, competition, and AI transparency if Open DUFMs were not widely available.”).
233 See, e.g., EleutherAI Institute Comment at 24 (listing examples of research facilitated by “open-weights foundation models.”); Rishi Bommasani et al. Comment at 3 (“Model weights are essential for several forms of scientific research across AI interpretability, security, and safety) ; CDT Comment at 8 (“Researchers used the model weights of Mistral 7B [. . .] to decrease the computational power required for fine-tuning the model for downstream tasks by a factor of ten.”).
234 For example, Stable Diffusion 3. (2023, February 22). “suite of models currently ranges from 800M to 8B parameters.”
235 See, e.g., CDT Comment at 33-40; The Abundance Institute Comment at 7 (“Like object code, model weights communicate information to a computer – in this case, a computer running an inference engine. [. . .] People and organizations who wish to publish such model weights have a protected speech interest in doing so.”); Mozilla Comment at 10 n.2 (“Further, as U.S. courts have held multiple times, computer source code must be viewed as expressive for First Amendment purposes. [. . .] A similar argument could be made about the importance of protecting the sharing of information about model weights and other AI components.”). Cf. G.S. Hans Comment at 2 (“Regulated AI companies may rely upon the reasoning of [Bernstein v. U.S.] to argue that export restrictions on model weights violate the First Amendment.”). But see Rozenshtein, A. (April 4, 2024). There Is No General First Amendment Right to Distribute Machine-Learning Model Weights. Lawfare. There may also be other constitutional challenges regarding openness in AI models beyond restriction of model weights. See generally G.S. Hans Comment (outlining a range of potential First Amendment challenges related to government requirements on transparency, content moderation, and other topics.).
236 Engler, A. (Jan. 22, 2024). The case for AI transparency requirements.
237 Greene, T., & et al. (2022). Barriers to academic data science research in the new realm of algorithmic behaviour modification by digital platforms. Nature Machine Intelligence, 4, 323–330.
238 Gorwa, R., & Veale, M. (2023, November 21). Moderating Model Marketplaces: Platform Governance Puzzles for AI Intermediaries. ArXiv.org.
239 National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). (2023).
240 Longpre, S., & et al. (2024). A Safe Harbor for AI Evaluation and Red Teaming. ArXiv.
241 Google Comment at 3.
242 National Telecommunications and Information Administration (2024). Artificial Intelligence Accountability Policy Report.
243 IBM Comment at 4.
244 AI Policy and Governance Working Group Comment at 3.
245 Balwit, A., & Korinek, A. (2022, May 10). Aligned with whom? Direct and social goals for AI systems.
246 Sartori, L., & Theodorou, A. (2022). A sociotechnical perspective for the future of AI: narratives, inequalities, and human control. Ethics and Information Technology, 24(4).
247 IBM Comment at 8.
248 AI Accountability Policy Report, National Telecommunications and Information Administration. (2024, March).
249 Gorwa and Michael Veale, ‘Moderating Model Marketplaces: Platform Governance Puzzles for AI Intermediaries’ (2024) 16(2) Law Innovation and Technology.
250 National Telecommunications and Information Administration (2024). Artificial Intelligence Accountability Policy Report.
251 See, for example, previous writings from the Clinton administration about the Internet, which noted that “governments should encourage industry self-regulation wherever appropriate and support the efforts of private sector organizations to develop mechanisms to facilitate the successful operation of the Internet.” 1997 Global Electronic Commerce Framework. Clintonwhitehouse4.Archives.gov.
252 Rodriguez, S. and Schechner, S. Facebook Parent’s Plan to Win AI Race: Give Its Tech Away Free. WSJ. May 19, 2024.
253 See, e.g., Preparedness. (n.d.). OpenAI; Anthropic’s Responsible Scaling Policy. (2023, September 19). Anthropic.
254 See, e.g., AI Policy and Governance Working Group Comment at 3-5 ; Public Knowledge Comment at 11-12 ; Hugging Face Comment at 8-9.
255 See, e.g., Stability AI Comment at 4 (“By reducing these costs, open models help to ensure the economic benefits of AI accrue to a broad community of developers and small businesses, not just Big Tech firms with deep pockets.).
256 See, e.g., AI Policy and Governance Working Group at 1 (“Openly available data, code, and infrastructure have been critical to the advancement of science, technological innovation, economic growth, and democratic governance. These open resources have been built and shared in the context of commitments to open science, to expanding industry and markets, and to the principle that some technologies should be widely available for maximum public benefit, while allowing for control of access to data, code, and infrastructure as necessary for safety and security purposes.”).
257 Nix, N., Zakrzewski, C., De Vynck, G., (2024, March 10) Silicon Valley is pricing academics out of AI research. Washington Post.
258 Affiliation of research teams building notable AI systems, by year of publication. (n.d.). Our World in Data.
259 Rand Comment at 3 (“Publishing foundation model weights can aid in AI safety research.”).
260 Google Comment at 2 (“While the benefits of open AI models are profound, there is also a risk that their use accelerates harms, like deepfake imagery, disinformation, and malicious services.”).