(4) “Department” means the Department of Technology.
(4)“High-risk automated decision system” or “high-risk ADS” means an automated decision system that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for,
free speech, housing or accommodations, education, employment, credit, health care, child welfare, immigration, and criminal justice.
(5) “Substantial factor” means an element of a decisionmaking process that is capable of altering the outcome of the process.
(b) The Following the adoption of regulations by the California Privacy Protection Agency pursuant to paragraph (16) of subdivision (a) of Section 1798.185 of the Civil Code, or following the enactment of similarly comprehensive statewide legislation that establishes a regulatory framework governing the
development and deployment of ADTs, whichever occurs sooner, the department shall develop and adopt regulations to create an AI risk management ADT procurement standard.
(1) To develop regulations related to the AI risk management ADT procurement standard, the department may apply shall consider principles and industry standards
addressed in relevant publications, including, but not limited to, any all of the following:
(A) The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People, published by the White House Office of Science and Technology Policy in October 2022.
(B) The Artificial Intelligence Risk Management Framework (AI RMF 1.0), released by the National Institute of Standards and Technology (NIST) in January 2023.
(C) The Risk Management Framework for the Procurement of Artificial Intelligence (RMF PAIS 1.0), authored by the AI Procurement
Lab and the Center for Inclusive Change in 2024.
(D) The Advancing Governance, Innovation, and Risk Management for Agency Use of Artificial Intelligence Memorandum, published by the Executive Office of the President, Office of Management and Budget, dated March 28, 2024.
(2) The AI risk management ADT procurement standard shall include all of the following:
(A) A detailed risk assessment procedure for procuring ADS that analyzes all of the following:
(i) Organizational and supply chain governance associated with the ADS. ADT.
(ii) The purpose and use of the ADS. ADT.
(iii) Any known potential misuses or abuses of the ADS. ADT.
(iv) An assessment of the legality, traceability, and provenance of the data the ADS ADT uses and the legality of the output of the ADS. ADT.
(v) The robustness, accuracy, and reliability of the ADS. ADT.
(vi) The interpretability and explainability of the ADS.
ADT.
(B) Methods for appropriate risk controls between the state agency and ADS ADT vendor, including, but not limited to, reducing the risk through various mitigation strategies, eliminating the risk, or sharing the risk.
(C) Adverse incident monitoring procedures.
(D) Identification and classification of prohibited use cases and applications of ADS ADT
that the state shall not procure.
(E) A detailed equity assessment that analyzes, at a minimum, all of the following:
(i) The individuals and communities that will interact with the high-risk ADS.
ADT.
(ii) How the information or decisions generated by the ADS ADT will impact an individual’s rights, freedoms, economic status, health, health care, or well-being.
(iii) Any issues that may arise if the ADS ADT is inaccurate.
(iv) How users of
with diverse abilities will interact with the user interface of the ADS ADT and whether the ADS ADT integrates and interacts with commonly used assistive technologies.
(F) An assessment that analyzes the level of human oversight associated with the use of ADS. ADT.
(G) Adherence to data minimization standards, including that an AI or ADS ADT vendor shall only use information provided by or obtained from an agency to provide the specific service authorized by the agency. Further, the data collected may not be used for training of proprietary vendor or third-party systems.
(3) To develop the AI
risk management In developing the ADT procurement standard, the department shall comply with do all of the following:
(A) Collaborate with organizations that represent state and local government employees and industry experts, including, but not limited to, public trust and safety experts, community-based organizations, civil society groups, academic researchers, and research institutions focused on responsible AI ADT
procurement, design, and deployment.
(B) Consult with the California Privacy Protection Agency.
(C) Solicit public comment on the risk management ADT procurement standard.
(4)The department
(4) (A) Subject to subparagraph (B), the department shall adopt regulations pursuant to this subdivision in accordance with the provisions of Chapter 3.5 (commencing with Section 11340) of Part 1 of Division 3 of Title 2 of the Government Code.
(B) Regulations adopted by the department pursuant to subparagraph (A) shall be consistent with both of the following:
(i) Regulations adopted by the California Privacy Protection Agency pursuant to paragraph (16) of subdivision (a) of Section 1798.185 of the Civil Code.
(ii) Statewide legislation that establishes a regulatory framework governing the development and deployment of ADTs.
(5) Starting January 1, 2026, and annually thereafter, the department shall review and update both of the following:
(A) The ADT procurement standard.
(B) Regulations adopted pursuant to this subdivision.
(c) A state agency shall not procure an ADT, enter into a contract for an ADT, or enter into a contract for any service that utilizes an ADT, prior to the adoption of regulations by the department pursuant to subdivision (b).
(c)Commencing six months after the date on which the regulations described in subdivision (b) are approved and final, a
(d) A state agency shall not may enter into a contract for an automated decision system, or any ADT, or a service that utilizes an automated decision system, unless ADT only after the department has adopted regulations pursuant to subdivision (b) and only if
the contract includes a clause that does all of the following:
(1) Provides a completed risk assessment of the relevant ADS ADT that analyzes the items included in subparagraph (A) of paragraph (2) of subdivision (b).
(2) Requires the state agency or the ADS ADT vendor, or both, to adhere to appropriate risk controls.
procurement standards.
(3) Provides procedures for adverse incident monitoring.
(4) Requires authorization from the state agency before deployment of ADS ADT upgrades and enhancements.
(5) Requires the state agency or the ADS ADT
vendor, or both, to provide notice to individuals that would likely be affected by the decisions or outcomes of the ADS, ADT, and information about how to appeal or opt-out of ADS ADT decisions or outcomes.
(6) Provides a termination right in the event of a significant breach of responsibility or violation by the vendor.