Artificial intelligence (AI) has been a hot topic in state legislatures across the country over the past year. As is common with technology-related issues, California has taken the lead in regulating the AI space. The California legislature recently passed AI bills AB 2013 and SB 1047, on August 26, 2024, and August 28, 2024, respectively. SB 1047 requires developers of the largest AI models to take significant safety measures to mitigate risks to safety that future development of their systems may pose to the public. AB 2013, meanwhile, mandates transparency in the development of generative artificial intelligence (AI) systems. Both bills are now pending Governor Gavin Newsom’s final approval.

California has been at the forefront of regulating industries and technologies, and that appears to be no different with AI. However, it remains unclear whether either bill will be signed into law. The fate of SB 1047 is especially uncertain due to opposition from industry and even California Democrats. However, even if these bills do not pass, companies thinking about developing AI models or fine-tuning them should be aware of their provisions, as there is clearly a trend in AI proposals toward imposing substantive requirements—like safety protocols, testing procedures and third-party audits—for model developers in an attempt to avoid harm as well as improve transparency around the development of these models, including their data sources.

California hasn’t been the only state active on AI issues. A number of states have proposed laws in 2024 aiming to regulate AI. It is possible that the passage of these laws in California will further spark action in other states (as we have seen with other technology-related issues). More movement on AI issues at the state level may also push Congress to seriously consider federal legislation, especially as other jurisdictions (such as the European Union) continue to regulate this space.

Below, we have provided our key takeaways on these bills, as well as an overall summary of these two new potential California AI laws. We will continue to keep you updated on notable developments on this topic through the WilmerHale Privacy and Cybersecurity Blog.

Key Takeaways:

• Reasonable Care Standard: Given the concerns about the potential safety risks of large AI models, there continues to be a focus on imposing a duty of reasonable care on developers of these models to avoid major harm. This is similar to the approach taken in Colorado, where the AI law requires developers to use reasonable care to protect consumers from any known or reasonably foreseeable risks of algorithmic discrimination arising from the intended and contracted uses of high-risk AI systems.
• Policies and Procedures: SB 1047 in particular creates specific documentation obligations for AI developers, including the requirement to develop a separate safety and security protocol. The focus on AI developers documenting their practices through appropriate policies and procedures has been an area of emphasis in other AI proposals as well.
• Employee Protections: SB 1047 is notable because it provides whistleblower protections for employees, which is a somewhat unique provision compared to other AI proposals.
• Training Data: AB 2013 specifically requires transparency about the data that generative AI developers use to train their models. This has been an area of focus for other regulators and legislatures, as well as an area that plaintiffs’ lawyers have scrutinized.

SB 1047: The Safe and Secure Innovation for Frontier Artificial Intelligence Models Act

SB 1047, also known as the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, introduced by state Senator Scott Wiener, would impose various safety restrictions and requirements on advanced AI models, with the stated goal of allowing California to recognize the benefits of the technology while avoiding the most severe risks.

Applicability

The bill applies to AI developers who offer their services in California. A “developer” is defined as “a person that performs the initial training of a covered model either by training a model using a sufficient quantity of computing power and cost, or by fine-tuning an existing covered model or covered model derivative using a quantity of computing power and cost greater than the amount specified in the bill’s definition of a ‘covered model.’”

A covered model is also a defined term, which is intended to apply to large, power models, and it will change over time based on computing power thresholds and costs of the AI system. Prior to January 1, 2027, covered models are defined as AI models that are either trained (1) using computing power “greater than 10^26 integer or floating-point operations” (FLOP) that cost more than $100 million to develop or (2) using fine-tuning with computing power of three times 10^25 integer or FLOP costing more than $10 million. After January 1, 2027, the cost threshold will remain the same, although adjusted for inflation, but the computing power threshold will annually be set by the Government Operations Agency, which the bill also creates.

Key Requirements

The bill requires developers to implement certain safety measures before and after training covered models, and it imposes a duty of reasonable care on developers to prevent covered models and covered model derivatives from posing unreasonable risks of causing or materially enabling critical harm. The bill deems several considerations to be relevant in determining whether a developer exercised reasonable care, including the quality of a developer’s safety and security protocol, the extent to which the developer implemented and followed its safety and security protocol, and the quality and implementation of the developer’s safety and security protocol as compared to those of peers.

Developers are not permitted to use a covered model or its derivative for any purpose other than training, evaluation, or legal compliance or to make it available for commercial or public use if there is an unreasonable risk of causing or enabling critical harm. Before training a covered model, developers are required to implement several safety measures, including reasonable cybersecurity protections to prevent unauthorized access to, misuse of, or unsafe post-training modifications of the covered model and all covered model derivatives controlled by the developer. The bill also requires the implementation of a capability to promptly enact a full shutdown of the model.

In addition, the bill requires developers to implement a written and separate safety and security protocol that complies with the developer’s duty to take reasonable care to avoid producing a model or covered model derivative that poses an unreasonable risk of causing or materially enabling a critical harm, such as the creation or use of a chemical, biological, radiological or nuclear weapon in a manner that results in mass casualties, or other grave harm to public safety and security that causes mass casualties or significant monetary damage. The protocol must identify a testing procedure to evaluate whether the model or its derivatives pose an unreasonable risk of causing or materially enabling a critical harm, and it must describe in detail how the developers will meet their obligations as set forth in the bill. Developers are required to take specified steps to ensure the protocol is implemented as written, including designating senior personnel to oversee implementation and retaining an unredacted copy of the safety and security protocol for as long as the covered model is made available for commercial, public or foreseeably public use plus five years, including records and dates of any updates or revisions. They are also to conduct an annual review of the protocol and conspicuously publish a copy of the safety and security protocol (with redactions allowed for certain types of information) and transmit a copy to the Attorney General upon request, though the bill would make the protocol exempt from the California Public Records Act.

The bill also imposes ongoing requirements on developers and requires third-party audits. Specifically, developers of covered models would be required to annually reevaluate their implemented procedures, policies, protections, capabilities and safeguards adopted pursuant to the bill. Starting January 1, 2026, they are also to annually hire a third-party auditor to independently verify compliance with the bill’s requirements, following auditing best practices. The auditor is to prepare an audit report, including a detailed assessment of the steps the developer has taken to comply with the bill, instances of noncompliance with the bill’s requirements and recommendations for improvement, and a detailed assessment of the developer’s internal controls. Developers are to conspicuously publish a redacted copy of the auditor’s report and provide a copy to the California Attorney General. Unredacted copies of the report are also to be made available to the Attorney General upon request.

Finally, developers of covered models are to annually submit a statement of compliance with the bill to the Attorney General. They are also to report to the Attorney General within 72 hours each “artificial intelligence safety incident” (i.e., an incident that demonstrably increases the risk of a critical harm by several means outlined in the bill) affecting the covered model or any covered model derivatives controlled by the developer. Additionally, they must notify the Attorney General within 30 days if they use the model or its derivative for purposes beyond training, evaluation or legal compliance or if they make it available for commercial or public use for the first time.

Employee Protections

The bill provides whistleblower protections for employees of developers or their subcontractors. Developers are not to prevent their employees from or retaliate against them for disclosing information to the authorities about violations of SB 1047. Developers are also to provide an internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that developer is in violation of SB 1047.

Enforcement and Oversight

The California Attorney General has exclusive enforcement authority and may bring civil actions for specified violations of the bill—for example, violations that cause death or bodily harm to another human, harm to property, or theft or misappropriation of property or that constitute an imminent risk or threat to public safety. In bringing civil actions, the California Attorney General may seek to recover civil penalties, monetary or punitive damages, injunctive relief, attorney’s fees and costs, or any other relief the court deems appropriate.

The bill also creates a consortium called the Government Operations Agency, which will operate independently from the Department of Technology. The agency will be required to annually issue regulations to update the definition of covered models, with its first issue due by January 1, 2027. By January 1, 2026, the agency is also to develop a framework for the creation of a public cloud computing cluster to be known as CalCompute that advances the development and deployment of AI that is safe, ethical, equitable and sustainable, among other things; foster research and innovation that benefits the public; and submit a report on this framework to the legislature.

AB 2013: Artificial Intelligence Training Data Transparency

AB 2013, titled Artificial Intelligence Training Data Transparency, will require developers to publicly release specified documentation regarding the data used to train their generative artificial intelligence systems or services.

Applicability

The bill applies to “Generative AI” “developers,” and both of these terms have specific definitions in the bill. Generative AI is defined as “artificial intelligence that can generate derived synthetic content, such as text, images, video, and audio, that emulates the structure and characteristics of the artificial intelligence’s training data.” The bill further defines “artificial intelligence” as “an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.”

The bill defines a developer as a “person, partnership, state or local government agency, or corporation that designs, codes, produces, or substantially modifies an artificial intelligence system or service for use by members of the public, excluding certain persons like hospital medical staff.”

Key Requirements

Starting January 1, 2026, for any new or significantly updated generative AI systems released after January 1, 2022, developers are required to publish documentation on their public websites about the data used to train these systems. The documentation is to include information on, for example:

• the sources or owners of the training data sets;
• a description of how the data sets further the intended purpose of the artificial intelligence system or service;
• the number and description of data points in the data sets;
• whether the data sets include any data protected by copyright, trademark or patent or whether the data sets are entirely in the public domain; and
• whether the generative artificial intelligence system or service used or continuously uses synthetic data generation in its development, etc.

Enforcement and Oversight

The bill will be embedded in the California Consumer Privacy Act, if passed, and the California Privacy Protection Agency and the California Attorney General will have enforcement authority.