The Trust Constraint on Personalized AI: How Transparency and Adaptive Governance Can Unlock AI Productivity
Introduction
Personalization has driven much of the digital economy’s productivity gains over the past two decades. By tailoring services to individual users—ranging from targeted advertising[1] and content recommendations[2] to adaptive education tools[3] and precision health diagnostics[4]—firms have been able to reduce search costs and cognitive burdens while increasing relevance and efficiency across sectors.[5]
Generative artificial intelligence fundamentally alters this model. Unlike these largely reactive examples of personalization, generative AI enables interactive, context-dependent tailoring, such as collaborative document creation. This shift raises the stakes for personalization, while reshaping users’ expectations. In many high-value or high-risk contexts, users will tend to experience interactions with generative AI as private and confidential, even absent any formal grant of privilege. [6] While some uses—such as search substitution or shared enterprise workflows—clearly entail assumed disclosures, users broadly maintain an expectation of contextual privacy in their interactions with AI systems.
These expectations complicate familiar privacy tradeoffs. While users have generally accepted limited data collection to support services like targeted advertising, the use of personal information in ways that exceed these prevailing baseline expectations will continue to provoke consumer backlash and regulatory scrutiny.
This issue brief argues that personalization is not optional for generative AI: It is a structural requirement for the technology’s usefulness and safety. Realizing AI’s full productive potential, however, depends on verifiable trust, grounded in transparency and enforceable guardrails governing data use. Policy debates must therefore move beyond categorical judgments for or against personalization to instead focus on fostering governance frameworks that are capable of sustaining innovation while honoring users’ implicit privacy expectations.
I. Personalization as Capital: How User Context Makes AI Productive
The economic history of the digital age can be understood as a sustained effort to reduce search costs. Early e-commerce lowered the cost of physical search, while search engines reduced the cost of information retrieval. The systematic use of historical user data, contextual signals, and users’ demonstrated preferences in order to tailor delivery of digital services—commonly described as “personalization”—represents the latest stage of this evolution. By anticipating users’ needs and filtering for relevance, personalization increases engagement, improves profitability, and enhances productivity.
For example, Google’s search engine vastly expanded access to information not only by indexing the internet and making it searchable, but by increasingly tailoring results to individual users.[7] As a result, users may receive different answers to the same query based on their prior search history.[8] As Google put it:
For example, if you search for chocolate cake, and then search again for “how to make,” Google might be more likely to predict that you’re searching for “how to make chocolate frosting.” These predictions are based on your past searches to give you better results and help you pick up where you left off.[9]
Similarly, the value of product platforms like Amazon,[10] content providers like Netflix[11] and YouTube,[12] and social-media networks like Facebook,[13] Instagram,[14] TikTok,[15] and X[16] are maximized by personalized recommendations based on user engagement, history, and location.
Generative AI represents a further development in this trajectory. A model is not fully productive unless it can incorporate a user’s history, stylistic preferences, and tacit workflow conventions. A generic foundation model trained solely on public data remains a blank slate with limited utility; it cannot reproduce the formatting norms of a law firm, the architectural practices of a software team, or the stylistic preferences of a designer without awareness of those contexts. In this setting, personalization extends well beyond recommendation. We are moving toward a model in which a user’s AI functions as a powerful cognitive extension of that user him or herself.
One of the most significant gains from this shift is increased productivity. Empirical studies of generative AI document substantial, measurable productivity improvements, particularly in high-wage, information-intensive occupations such as programming and professional services. These gains arise from automating repetitive tasks, synthesizing large volumes of information into user-specific formats, and providing tailored, real-time assistance. A study from the Federal Reserve Bank of St. Louis found “workers are 33% more productive in each hour they use generative AI.”[17] Early evidence from real-world deployments further suggests that AI adoption can expand employment opportunities by lowering operational complexity and enabling participation by workers who might otherwise be excluded.[18]
Generative AI’s usefulness derives in large part from its capacity for interactive learning, which allows it to adapt to user-specific conventions, such as the formatting practices of a particular law firm or the coding style of an individual programmer. This capability depends on access to user-specific contexts, which can enter the system through multiple channels. Some contextual information is incorporated through fine-tuning or other post-training adaptation, while other context is supplied at inference time via system prompts, retrieved documents, or accumulated conversation history. Conversational interfaces further provide a continuous mechanism for user feedback, enabling iterative refinement of personalized outputs. Across these mechanisms, effective operation requires the system to draw on the user’s workflow, language preferences, prior inputs, and domain-specific knowledge.
Observational reports suggest that generative models perform substantially better when aligned with actual user workflows than when deployed in a generic, publicly trained state. Studies of enterprise deployments consistently find that models fine-tuned on internal documents, prior outputs, and domain-specific conventions produce work product that is more accurate, relevant, and stylistically consistent. In law firm settings, for example, fine-tuning on past briefs, clause libraries, and citation norms enables models to draft documents that conform to the firm’s preferred structure and voice.[19] In software-development environments, exposure to prior codebases and architectural patterns allows models to generate code that is not only syntactically correct but also consistent with local style, tooling, and design choices. These gains are not incidental; they result from the incorporation of tacit knowledge embedded in user history that is absent from public training data.[20] Continuous access to such contextual information is what transforms a general-purpose model into a genuinely productive collaborator.
Personalization also has important, but often overlooked, implications for user safety. Public debate has tended to focus on AI models’ contextual-understanding failures, including excessive refusals, inappropriate compliance, sycophantic behavior, or emotionally over-attentive responses.[21] Large language models (LLMs) are predictive systems whose performance depends both on their training data and on the context available at the point of use. Many of the alleged safety failures cited in public discussions of LLMs can be traced to insufficient contextual information for accurately interpreting user intent. When context is lacking, systems are more likely to misclassify benign requests as harmful or to overlook genuinely risky prompts.
Greater personalization enables AI systems to more accurately distinguish benign, professional, or exploratory prompts from genuinely harmful ones, thereby reducing both unwarranted refusals and inappropriate permissions. By incorporating user-specific context, interaction history, and inferred intent, personalized systems can more precisely calibrate tone, boundaries, and safeguards, particularly in high-stakes or sensitive domains. In this sense, personalization serves as a mechanism for reducing ambiguity and improving classification accuracy—an enduring challenge in AI safety.
The central insight is straightforward but consequential: the transformative productivity gains promised by generative AI cannot be fully realized without the ongoing, context-specific use of user data.
II. The Trust Constraint on Personalized AI
Public debate surrounding generative AI often emphasizes speculative informational harms, such as filter bubbles or ideological bias.[22] While these concerns are not trivial, they are in many cases subject to self-correction via competition, user feedback, and accountability mechanisms embedded in public institutions and developer commitments. [23] Professional and other high-agency users rapidly identify irrelevant or distorted outputs, and firms face strong competitive incentives to improve accuracy and utility. In this sense, familiar “marketplace of ideas” dynamics tend to constrain these risks.[24] As a result, such concerns do not pose a fundamental threat to personalization as a productivity-enhancing technology.
By contrast, the more serious risks associated with personalized generative AI are structural in nature. They stem from deep information asymmetries regarding how user data are retained, segmented, repurposed, or disclosed. The most consequential harms include accidental or malicious exposure of highly sensitive prompts; nonconsensual or opaque reuse of user data for model training; and legally compelled disclosure of conversational logs to courts or regulators in the absence of clear protective standards.
These risks are largely opaque to users and difficult to manage individually. When harms occur, recourse mechanisms are often slow, uncertain, or unavailable. Addressing these systemic trust failures—rather than speculative informational harms—therefore represents the central governance challenge to ensure that personalization remains both safe and economically viable.
Against this backdrop, a core trust dilemma emerges: Users experience interactions with generative AI as private and context-bound, yet effective personalization requires systems to ingest and rely on data that feel intimate. As discussed below, this tension can be mitigated through well-calibrated institutional arrangements that operate in tandem with developers’ efforts to provide transparency and user control.
The salience of this concern is illustrated by ongoing debates over LLM developers’ copyright and data-collection practices.[25] Users reasonably fear that highly sensitive prompts—such as unfiltered personal reflections, complex legal analyses, medical information, or confidential business plans—could be disclosed, subpoenaed, or repurposed for legal or regulatory proceedings. Most users would find such outcomes unacceptable, reflecting the widespread expectation that these interactions are private, even in the absence of formal legal privilege.
This reaction reflects two simultaneous realities that define the core policy challenge. First, users expect privacy. The intimate and often high-stakes character of generative AI interactions fosters a strong—if frequently implicit—expectation of confidentiality, with users treating the system as a private interlocutor. Second, users recognize the necessity of data. They understand, at least implicitly, that high-quality, tailored outputs require the system to draw on user-specific context, interaction history, and current inputs, rather than operating as a blank slate.
At the same time, users often operate under miscalibrated assumptions about how generative AI systems handle their information. Empirical research indicates that users adopt intuitive “mental models” of chatbots, the most prevalent of which resembles an “agent model”—the belief that the system functions as an autonomous interlocutor whose exchanges remain private, akin to interactions with a doctor, lawyer, or trusted colleague.[26] In practice, many users therefore behave as if generative AI systems are subject to confidentiality norms, despite the absence of any such contractual, professional, or technical guarantees.[27] This perception is reinforced by the anthropomorphic design of conversational interfaces, which can lead even privacy-literate or technically sophisticated users to disclose highly sensitive information. Because these interactions feel intimate, users infer norms of discretion that the system itself does not—and often cannot—legally guarantee.[28]
This dynamic reflects a broader paradox well documented in the privacy literature. Users consistently report that conversations with generative AI feel more sensitive than email or social-media interactions, yet they nonetheless disclose medical information, financial concerns, and internal business matters in real time.[29] The issue is not user naiveté, but reliance on contextual cues that imply confidentiality in the absence of clear, verifiable information about downstream data practices. Users tend to assume strict purpose limitation and minimal propagation of their prompts, not because they oppose personalization, but because they expect the norms governing private professional exchanges to apply.[30]
More broadly, this conflict reflects a classic problem of information asymmetry, which can give rise to adverse selection. [31] The canonical illustration is the market for used cars. [32] Sellers know the true quality of a vehicle, while buyers observe only a probability that the car is high quality or a “lemon.” Faced with this uncertainty, buyers rationally discount the price they are willing to pay. Sellers of higher-quality vehicles may then exit the market because the discounted price fails to reflect their cars’ true value, leaving a disproportionate share of low-quality vehicles. In the extreme, the market can unravel entirely, with only lemons remaining.
Markets have developed partial solutions to this problem. In used-car markets, vehicle-history reports, such as those provided by Carfax,[33] and independent pre-purchase inspections[34] can help to reduce information asymmetries and support trade by allowing buyers to better assess quality.
A similar asymmetry characterizes personalized generative AI. Model developers possess proprietary knowledge about what data are collected, how long they are retained, whether they are repurposed for model training, what information may be subject to legal disclosure, and what security safeguards are in place. Users, by contrast, lack practical means to verify these claims.[35] This imbalance creates conditions conducive to adverse selection, as providers may face incentives to deploy user data in ways that maximize long-term model improvement or commercial advantage, while exposing users to heightened privacy and disclosure risks.
As in the market for lemons, these dynamics can undermine participation. If users increasingly fear exposure or misuse of sensitive information, they may rationally avoid using generative AI in contexts that require interactive learning and deep personalization, the very features that make the technology most valuable. Over time, this can degrade both trust and functionality.
The problem is particularly acute in regulated, high-value domains where information asymmetries are especially costly. For example, users of AI-enabled advisory tools may be unable to assess whether a system is acting in their interests or optimizing for engagement and data extraction, creating a textbook adverse-selection environment in which opaque or high-risk systems are most likely to proliferate.[36] Similarly, in insurance markets, underwriters may struggle to distinguish between firms with robust AI governance and those deploying unsafe or opaque systems, leading to adverse selection and moral hazard that complicate or preclude effective pricing of AI-related risks.[37]
Interactions with generative AI are therefore better analogized to relationships between patients and physicians or clients and lawyers than to the consumption of ordinary software products. This analogy is likely to become even more salient as AI systems grow increasingly agentic.[38] Absent credible commitments to confidentiality, users will rationally withhold information necessary for accurate diagnosis, effective treatment, or sound representation.
The implication is straightforward: personalization—the primary driver of generative AI’s utility—is sustainable only if providers can make and maintain credible commitments to safeguard user trust. Without such assurances, users are likely to adopt privacy-protective behaviors, including self-censorship, reduced data sharing, or outright abandonment of the technology, thereby constraining its productivity-enhancing potential. In the extreme, substantial social and economic value that generative AI could otherwise generate will fail to materialize.
III. Governing by Outcomes, Not Architecture
Designing effective guardrails for personalized generative AI requires grounding governance in how these systems actually create value. Personalized systems raise distinct privacy concerns because they depend on sustained access to user-specific context. User preferences vary across settings; in some circumstances, individuals may wish to limit or suspend personalization, while in others they may actively seek deeper contextual integration. Effective guardrails should therefore prioritize clear disclosure, meaningful user choice, and credible commitments regarding data use, rather than prescribing specific technical designs or attempting to comprehensively regulate inputs ex ante.
Given the complexity and heterogeneity of data flows in personalized systems, governance approaches are more likely to succeed when they focus on evaluating system outputs and observable harms, while preserving flexibility in how providers achieve those outcomes. Such flexibility allows developers to balance functionality, risk, and user preferences across diverse contexts without undermining the core value of personalization.
To bridge the trust gap created by information asymmetry, generative AI systems—particularly those deployed in consumer-facing contexts—require structural commitments to transparency that enable users to understand how their information is used, coupled with credible assurances against undesirable secondary uses. Existing consumer-protection regimes, including unfair and deceptive acts and practices (UDAP) statutes, already prohibit misrepresentation of data practices and the failure to honor public commitments. The marginal challenge posed by personalized AI is therefore not a lack of legal tools, but the increased difficulty users face in observing, interpreting, and verifying compliance in systems that encourage unusually sensitive disclosures while obscuring downstream data flows.
Effective transparency in personalized AI must focus on mitigating concrete risks through clear, contextual communication about data use. Users must be able to trust that information shared during interactions will not be repurposed in ways inconsistent with their reasonable expectations or stated preferences. Accordingly, transparency mechanisms should operate in context and be tied to practices that meaningfully enhance user understanding and risk assessment, rather than relying on prescriptive disclosure requirements that impose procedural burdens without delivering practical insight.
Transparency also plays a critical role in competition. In enterprise and professional markets, purchasers are likely to evaluate AI systems based not only on technical performance, but also on the clarity and credibility of providers’ commitments regarding data use, retention, and governance. Although it remains uncertain whether transparency will emerge as a dominant source of competitive advantage across all AI markets, it is plausible that greater transparency and user control will be associated with higher rates of adoption and more sustained use.[39] At a minimum, transparency appears to be a necessary condition for credible commitments to safeguard user trust. Absent meaningful visibility into data practices, assurances regarding privacy, purpose limitation, or non-retraining are unlikely to carry weight with sophisticated users.
Reducing the underlying information asymmetry therefore requires effective disclosure. In this context, disclosures should be clear, conspicuous, and intelligible to ordinary users.[40] Relevant information would include what data are collected (such as prompts, outputs, or conversation history), where those data are stored, and for how long (for example, session-based versus persistent account retention). Users should also be informed whether their data are used for model improvement or training, targeted advertising, or other third-party purposes.
Where such disclosures are provided, users may reasonably wish to limit or opt out of particular data uses. Accordingly, systems should offer clear and accessible mechanisms that allow users to align data practices with their preferences, including choices about whether information is retained or deleted after a session.
While disclosure is the primary mechanism for addressing information asymmetries, experience in adjacent data-governance contexts demonstrates that disclosure is effective only when it is both contemporaneous and credible. Market forces—such as reputation, contractual arrangements, and competition—often sustain these commitments, but a minimal legal backstop remains essential. Such a backstop helps to ensure that firms do not retroactively revise terms of service or expand data-use permissions after information has been collected.
Enforcement actions under consumer-protection law, including those brought by the Federal Trade Commission (FTC) in response to retroactive data-use changes, illustrate this role. In practice, background consumer-protection regimes function less as engines of affirmative regulation than as guarantors of promises, reinforcing the expectation that firms must clearly communicate their data practices and honor those commitments once made.[41] Beyond these baseline constraints, market structure itself—through reputational incentives, enterprise contracting, and user preference for trustworthy systems—continues to discipline data-handling behavior.
By contrast, top-down regulatory approaches that seek to prescribe system architecture, documentation requirements, or ongoing monitoring obligations for broadly defined categories of “AI systems” face inherent limitations. Regulators typically lack the information needed to specify ex ante how rapidly evolving technologies should operate. The predictable result is overbreadth and rigidity: compliance regimes that impose high fixed costs, chill experimentation, and deter incremental innovation valued by users. These dynamics are already evident in the extensive obligations imposed on general-purpose AI providers under the EU AI Act, which risk generating burdens disproportionate to any marginal gains in user understanding or trust.[42]
The core elements of a voluntary, outcome-oriented governance approach are reflected in the National Telecommunications and Information Administration’s (NTIA) recent policy on dual-use open models.[43] That framework implicitly recognizes that the relevant tradeoffs among functionality, risk, and innovation differ substantially between the foundation-model layer and the deployed-system layer, where outputs are generated and users interact with AI. Rather than prescribing model architecture, training techniques, or business models, NTIA emphasizes marginal-risk assessment, transparency, and evidence-based monitoring for foundation models whose weights are publicly released.[44] Accountability under this approach is achieved through standardized disclosures, risk–benefit documentation, and independent evaluation, rather than rigid ex ante technical mandates. This allows firms to determine how best to meet trust and safety objectives while preserving flexibility and innovation.
By contrast, more prescriptive regimes—such as the EU AI Act’s requirements that general-purpose AI developers publish public training-data summaries and extensive system documentation—risk imposing substantial compliance costs with limited corresponding gains in user understanding. Such obligations may also chill incremental innovation.[45] A more adaptive approach relies instead on context-appropriate disclosures, supplemented where necessary by independent audits or voluntary certification mechanisms that function as market signals of reliability without imposing one-size-fits-all regulatory requirements.[46] Properly structured, these guardrails can enhance trust while preserving the dynamism valued by both users and developers.
A further advantage of this approach is its scalability across the wide range of generative-AI applications. Indeed, “AI” is not a single technology, but a heterogeneous set of tools deployed in diverse contexts. Attempting to regulate it through a unified, systemwide framework would amount to governing nearly all software-mediated activity—an undertaking that is neither epistemically feasible nor administratively practical. Context-sensitive, purpose-limited disclosure avoids this pitfall by aligning governance with the specific risks of particular user interactions, rather than with abstract system classifications.
The guiding principle, therefore, should be purpose limitation; user data should be used only for the purposes for which it is shared. While users may accept data use as necessary for personalization and improved performance, they may reasonably object to its use for unrelated or unexpected purposes.
Context is central to shaping user expectations about data protection. Sensitive health information shared with an AI system designed to assist in medical diagnosis warrants substantially stronger safeguards than information provided to a general-purpose chatbot for routine consumer searches.[47] Similarly, professionals using generative AI to draft legal memoranda, briefs, or corporate strategy documents are likely to hold heightened expectations of privacy and security.[48] Certain contexts also carry explicit legal requirements. For example, the collection of data from children under the age of 13 triggers federal obligations for verifiable parental consent.[49]
To make privacy commitments credible in these varied settings, self-attestation alone is unlikely to be sufficient. Although markets discipline firms through contract, reputation, and competition, significant information asymmetries surrounding model training, data retention, and data segmentation prevent users from independently verifying compliance. One mechanism to address this gap would be a well-designed certification regime that operates as a safe-harbor disclosure framework. Under such a system, firms that adopt recognized transparency practices and submit to periodic independent audits would receive a credible designation signaling compliance.
Crucially, this approach would not prescribe system architecture or internal processes. Instead, it would verify outcomes: whether providers enforce data segmentation, honor non-retraining commitments in sensitive contexts, implement purpose limitation, and maintain clear accountability for data stewardship. By providing users with a reliable means to distinguish trustworthy providers from mere self-promoters, certification can reduce information asymmetry without constraining innovation or imposing disproportionate burdens on smaller entrants.
This model also preserves the bottom-up character of AI governance. Firms would retain flexibility in how they meet disclosed commitments, while certification bodies would function more like standards-setting organizations than regulatory authorities. Background legal frameworks—such as contract law, tort law, and baseline consumer-protection regimes—serve as guarantors of promises, ensuring accountability for misrepresentation without requiring continuous administrative oversight. In this way, a certification-based ecosystem can address the trust challenges of personalized generative AI while avoiding the rigidity, compliance costs, and innovation-suppressing effects associated with top-down regulatory regimes.
Beyond comprehensive self-certification regimes, experience with sector-specific approaches to data protection offers practical guidance for implementing disclosure and opt-out mechanisms. Historical regulatory and market-based efforts in domains such as finance, health data, and online tracking illustrate how transparency and user choice can be operationalized in ways that are both workable and enforceable.
In the financial sector, for example, the Gramm–Leach–Bliley Act (GLBA) requires financial institutions to disclose their information-sharing practices and provide consumers with the right to opt out of certain disclosures.[50] GLBA limits when firms “significantly engaged” in financial activities may share consumers’ nonpublic personal information with nonaffiliated third parties[51] and mandates “clear and conspicuous” notice of privacy practices and opt-out rights.[52] A comparable framework for generative AI could require disclosure of how user data are used, coupled with meaningful opportunities to opt out of specified secondary uses.
A parallel model exists for sensitive health information under the Health Insurance Portability and Accountability Act (HIPAA) and its Privacy Rule.[53] HIPAA imposes strict purpose limitations on the use and disclosure of protected health information,[54] backed by substantial civil and criminal penalties for knowing violations.[55] Similar purpose-limitation principles could be applied to sensitive health, legal, or otherwise confidential information shared with generative AI systems, supported by clear accountability mechanisms for misuse or breach.
Market-based approaches to data governance also provide instructive lessons. Browser-based privacy signals were developed to allow users to communicate preferences regarding online tracking. Early efforts, such as Do Not Track,[56] failed to gain traction in part because they lacked legal enforceability.[57] By contrast, the Global Privacy Control browser extension[58] has become effective because it is recognized under the California Consumer Privacy Act as a valid mechanism for exercising statutory opt-out rights.[59] The broader lesson is that voluntary tools—such as certification programs, standardized signals, or independent audits—can meaningfully reduce information asymmetry when reinforced by a modest regulatory backstop that ensures compliance and accountability.
The interactive nature of generative AI systems creates new opportunities to implement transparency and guardrails that were largely infeasible under earlier software paradigms. Unlike static services, conversational systems can deliver disclosures, explanations, and feedback mechanisms in context, enabling users to query data practices, contest outputs, or signal preferences during use rather than relying solely on ex ante or ex post policy documents. These interaction-driven feedback loops can accelerate the identification of system failures and reduce the time between harm, detection, and remediation.
At the same time, such mechanisms do not resolve the underlying information asymmetry between developers and users. While interactive transparency can improve comprehension and responsiveness, it cannot replace independent verification or credible external commitments. Instead, it should be understood as a complement to broader governance frameworks, lowering the cost of feedback and increasing the visibility of trust-related failures without substituting for enforceable assurances.
IV. Unlocking Personalization Without Breaking Trust
Public-policy approaches to personalization in generative AI should avoid both uncritical permissiveness and rigid prohibition. Instead, it should adopt an adaptive, trust-enabling governance posture. Three principles should guide this approach.
First, personalization should be presumptively permitted as a core service feature. Policymakers should recognize that personalization is a structural prerequisite for generative AI to deliver its anticipated productivity and safety benefits. Governance frameworks should therefore facilitate, rather than impede, its development and deployment.
Second, transparency and guardrails are essential to assure users that their data will not be used in ways that exceed reasonable expectations. Regulation should focus on addressing information asymmetry through standardized disclosures and credible enforcement of disclosed commitments, while accounting for the context-dependent nature of privacy risks and user expectations.
Third, governance must be adaptive. Given the rapid evolution of AI technologies, regulatory approaches should emphasize outcome-based objectives—such as prohibitions on unauthorized retraining or secondary data use—rather than prescriptive technical requirements, such as mandates on model architecture. Policymakers should consider voluntary certification and auditing mechanisms as flexible tools for accountability. Governance should further proceed from the premise that privacy expectations vary by context and that protecting those expectations is central to sustaining user trust in generative AI systems.
To put these principles into practice, policymakers should consider the following measures:
- Standardized Disclosure Language: Encourage the use of standardized, layered disclosures that allow users to understand key data-use commitments and default settings, particularly whether user interactions are used for model retraining. Disclosures should emphasize observable practices and user-relevant outcomes, rather than require the disclosure of proprietary technical details or security-sensitive information that could compromise trade secrets or system integrity.[60]
- Sectoral Privacy Standards: Develop targeted, high-water-mark privacy standards for generative AI deployed in regulated or high-risk sectors (e.g., law, finance, health care), where the potential harm from disclosure is greatest. Policymakers should avoid overly generalized frameworks that impose uniform privacy requirements across heterogeneous AI use cases.
- Voluntary Certification and Commitment Verification: Establish a federally recognized safe-harbor framework supported by voluntary, independent verification of data-use commitments. Rather than mandate intrusive audits of model internals or performance, certification should focus on whether firms adhere to their stated practices regarding data segmentation, purpose limitation, retention, and non-retraining in designated contexts. By limiting verification to organizational controls and observable practices, this approach protects trade secrets, reduces security and confidentiality risks, and avoids premature or overly burdensome testing regimes. Properly designed, certification can function as a market signal of trustworthiness, offering regulatory predictability for participating firms while preserving flexibility and innovation.
Personalization is a central driver of productivity and utility in generative AI, converting general-purpose models into effective, user-specific tools. In practice, personalization is not merely passive. Many systems evolve through ongoing interaction, user feedback, and iterative refinement, giving users meaningful influence over how outputs are shaped over time. Transparency and user control over how these signals are incorporated can therefore strengthen trust—not only by constraining data use, but by enabling users to understand and participate in the personalization process itself. At present, however, the promise of personalized AI is constrained by a significant trust deficit rooted in information asymmetries between developers and users concerning how highly sensitive data might be handled.
The core challenge facing policymakers is to find a framework that balances the data use required for effective personalization with the implicit privacy expectations users bring to interactions with increasingly sophisticated AI systems. By implementing robust guardrails, ensuring meaningful transparency, and supporting credible governance mechanisms, policymakers can enable the economic and social benefits of personalized generative AI while preserving the trust necessary for its long-term sustainability.
[1] See, e.g., Nidhi Arora et al., The Value of Getting Personalization Right—Or Wrong—Is Multiplying, McKinsey & Co. (Nov. 12, 2021), https://www.mckinsey.com/capabilities/growth-marketing-and-sales/our-insights/the-value-of-getting-personalization-right-or-wrong-is-multiplying (noting 71% of consumers expect AI personalization, 76% get frustrated when they don’t find it, and companies that excel at personalization generate 40% more revenue than average players).
[2] See, e.g., Content Recommendation Engine, Optimizely (last accessed Nov. 12, 2025), https://www.optimizely.com/optimization-glossary/content-recommendation-engine.
[3] See, e.g., Gavoy Small, The Top 12 Adaptive Learning Platforms for 2025, SC Training (Jan. 23, 2025), https://training.safetyculture.com/blog/adaptive-learning-platforms.
[4] See, e.g., Genomics and Your Health, Ctrs. Dis. Ctrl. & Prev’n. (Nov. 13, 2024), https://www.cdc.gov/genomics-and-health/precision-health-treat/index.html (“Precision medicine, also called personalized medicine, helps your healthcare provider find your unique disease risks and treatments that will work best for you. Precision health is broader—it includes precision medicine but also includes approaches that occur outside the setting of a healthcare provider’s office or hospital, such as disease prevention and health promotion activities.”).
[5] See, e.g., Robert Donnelly, Ayush Kanodia, & Ilya Morozov, Welfare Effects of Personalized Rankings, Mark. Sci. (2023), at 1-22, available at https://robdonnelly.me/files/personalized_rankings.pdf.
[6] For purposes of this discussion, “AI system” encompasses both conversational chat-based models and more fully agentic systems capable of acting across tasks, tools, or environments with varying degrees of autonomy. While these systems raise distinct technical and governance considerations, both are commonly experienced by users as interlocutors rather than passive software tools, and both can trigger similar expectations of confidentiality, depending on context.
[7] How Search Works with Your Activity, Google (last accessed Nov. 12, 2025), https://support.google.com/websearch/answer/10909618 (“When you search on Google, your past searches and other info are sometimes incorporated to help us give you a more useful experience.”).
[8] See Why Your Google Search Results Might Differ from Other People, Google (last accessed Nov. 12, 2025), https://support.google.com/websearch/answer/12412910 (“You may get the same or similar results to someone else who searches on Google Search. But sometimes, Google may give you different results based on things like time, context, or personalized results.”).
[9] Google, supra note 7.
[10] See Amazon Personalize, Amazon (last accessed Nov. 12, 2025), https://aws.amazon.com/personalize.
[11] See Recommendations, Netflix Rsch. (last accessed Nov. 12, 2025), https://research.netflix.com/research-area/recommendations.
[12] See Recommendations on YouTube, YouTube (last accessed Nov. 12, 2025), https://www.youtube.com/howyoutubeworks/recommendations.
[13] See How Feed Works, Facebook (last accessed Nov. 12, 2025), https://www.facebook.com/help/1155510281178725/?helpref=hc_fnav.
[14] See What We Mean When We Talk We Talk About Personalization, Meta Priv. Ctr. (last accessed Nov. 12, 2025), https://privacycenter.instagram.com/dialog/what-we-mean-when-we-talk-about-personalization.
[15] See Making Your Feed for You, TikTok Safety Ctr. (last updated Jul. 24, 2025), https://www.tiktok.com/safety/en/making-your-feed-for-you.
[16] See Personalization and Data, X (last accessed Nov. 12, 2025), https://x.com/settings/account/personalization.
[17] Alexander Birk, Adam Blandin, & David Deming, The Impact of Generative AI on Work Productivity, Fed. Res. Bank of St. Louis (Feb. 27, 2025), https://www.stlouisfed.org/on-the-economy/2025/feb/impact-generative-ai-work-productivity.
[18] Gustavo de Souza, Artificial Intelligence in the Office and the Factory: Evidence from Administrative Software Registry Data, (Fed. Res. Bank of Chi. Working Paper No. 2025-11), https://www.chicagofed.org/publications/working-papers/2025/2025-11. Notably, while AI adoption of AI displaces some routine administrative roles—particularly middle-income office positions—the evidence to date suggests these effects are offset by increased demand in production settings, where AI-enabled tools make machines easier to use and expand opportunities for younger and less-skilled workers. The aggregate impact appears to be a reallocation rather than reduction of labor, with overall employment rising as productivity improves.
[19] See Aditya Krishna Sonthy, Lessons Learned: Fine-Tuning a Generative AI Model for Internal Knowledge Management — Pitfalls and Successes, 7 J. Comp. Sci. & Tech. Stud. 5, 46 (2025), https://www.researchgate.net/publication/392221309_Lessons_Learned_Fine-Tuning_a_Generative_AI_Model_for_Internal_Knowledge_Management_-_Pitfalls_and_Successes.
[20] Erik Brynjolfsson, Danielle Li, & Lindsey R. Raymond, Generative AI at Work (NBER Working Paper No. 31161, 2023), https://doi.org/10.3386/w31161.
[21] See, e.g., Mike Caulfield, AI Is Not Your Friend, The Atlantic (May 9, 2025), https://www.theatlantic.com/technology/archive/2025/05/sycophantic-ai/682743.
[22] See Tomo Lazovich, Filter Bubbles and Affective Polarization in User-Personalized Large Language Model Outputs, arXiv (Oct. 31, 2023), available at https://arxiv.org/pdf/2311.14677; Exec. Order No. 14319, Preventing Woke AI in the Federal Government (“Americans will require reliable outputs from AI, but when ideological biases or social agendas are built into AI models, they can distort the quality and accuracy of the output.”); Bobby Allyn, Google Races to Find a Solution After AI Generator Gemini Misses the Mark, Morning Ed. (Mar. 18, 2024), https://www.npr.org/2024/03/18/1239107313/google-races-to-find-a-solution-after-ai-generator-gemini-misses-the-mark.
[23] One might ask whether nonprofessional or lower-agency users—particularly in social or entertainment contexts—are more vulnerable to harms such as filter bubbles, bias reinforcement, or manipulation. It is plausible that children and others who are less aware of how these systems operate could absorb information uncritically. But even if that is a potential vulnerability, market accountability would still be driven primarily by high-agency users, including professionals, enterprises, and expert communities, whose tolerance for error, misuse, or reputational risk is low. This dynamic is familiar from privacy law: While there are relatively few “privacy fundamentalists,” they have historically exerted outsized influence both on firms’ privacy practices and on regulatory outcomes. See, e.g., Alec Stapp, Against Privacy Fundamentalism in the United States, Niskanen Ctr. (Nov. 19, 2018), https://www.niskanencenter.org/against-privacy-fundamentalism-in-the-united-states (quoting studies that find only “25 to 35 percent of Americans are privacy fundamentalists” unwilling to “trade their privacy for economic benefits”). These users exert disproportionate pressure on developers to improve reliability, contextual understanding, and safeguards, and the resulting design and governance improvements tend to benefit all users. The same dynamic is likely to shape privacy and personalization governance in AI systems.
[24] See, e.g., Ben Sperry, The Marketplace of Ideas: Government Failure Is Worse Than Market Failure When It Comes to Social-Media Misinformation, Truth on the Mkt. (Sep. 22, 2023), https://truthonthemarket.com/2023/09/22/the-marketplace-of-ideas-government-failure-is-worse-than-market-failure-when-it-comes-to-social-media-misinformation.
[25] For example, the recent New York Times v. OpenAI litigation included a “preservation order requiring OpenAI to retain all ChatGPT conversation logs – affecting over 400 million users worldwide…” See Jeffrey M. Kelly et al., From Copyright Case to AI Data Crisis: How The New York Times v. OpenAI Reshapes Companies’ Data Governance and eDiscovery Strategy, Nelson Mullins (Jul. 10, 2025), https://www.nelsonmullins.com/insights/blogs/corporate-governance-insights/all/from-copyright-case-to-ai-data-crisis-how-the-new-york-times-v-openai-reshapes-companies-data-governance-and-ediscovery-strategy.
[26] See Xingyi Wang et al., Users’ Mental Models of Generative AI Chatbot Ecosystems, arXiv (Jan. 31, 2025), at 9-10, https://arxiv.org/abs/2501.19211 (discussing the “agent” model).
[27] Id.
[28] See Julia Ive et al., Privacy-Preserving Behaviour of Chatbot Users, arXiv (Aug. 9, 2025), https://arxiv.org/abs/2411.17589.
[29] See Sarah Tran et al., Understanding Privacy Norms Around LLM-Based Chatbots: A Contextual Integrity Perspective, arXiv (Aug. 9, 2025), at 1, https://arxiv.org/abs/2508.06760 (“Our findings reveal a stark disconnect between user concerns and behavior: 82% of respondents rated chatbot conversations as sensitive or highly sensitive—more than email or social media posts—but nearly half reported discussing health topics and over one-third discussed personal finances with ChatGPT.”); see also Helen Nissenbaum, Contextual Integrity Up and Down the Data Food Chain, 20 Theo. Inq. L. 221 (2019).
[30] Tran et al., id. at 1 (“Our results suggest that users apply consistent baseline privacy expectations to chatbot data, prioritizing procedural safeguards over recipient trustworthiness.”).
[31] See Elizabeth Mohn, Information Asymmetry, EBSCO Info. Svcs. (2025), https://www.ebsco.com/research-starters/social-sciences-and-humanities/information-asymmetry (“Information asymmetry refers to an economic event in which one party has more information about an economic transaction than the other party.”). Adverse selection occurs where asymmetric information results in a party to a transaction exploiting undisclosed information to gain disproportionate benefit in a trade.
[32] See George A. Akerloff, The Market for “Lemons”: Quality Uncertainty and the Market Mechanism, 84 Q. J. Econ. 488 (1970); see also Martin Obschonka & Moren Levesque, A Market for Lemons? Strategic Directions for a Vigilant Application of Artificial Intelligence in Entrepreneurship Research (Sep. 2024), https://www.researchgate.net/publication/384057477_A_Market_for_Lemons_Strategic_Directions_for_a_Vigilant_Application_of_Artificial_Intelligence_in_Entrepreneurship_Research.
[33] See CARFAX Vehicle History Reports, Carfax (last accessed Nov. 12, 2025), https://www.carfax.com/vehicle-history-reports.
[34] See, e.g., Lemon Protector, https://www.lemonprotector.com.
[35] The technology may mitigate the verification problem to a limited degree. Conversational AI systems can reduce user confusion by delivering data-use explanations in a more accessible, contextual way than traditional privacy policies or terms of service. For example, an AI system might explain how user data is handled in response to a direct question during an interaction. This kind of “in-flow” transparency can improve comprehension and salience, but it does not solve the core verification problem: Users may better understand stated commitments yet still lack independent means to confirm that those commitments are actually followed.
[36] See Runhuan Feng et al., Robo-Advisors Beyond Automation: Principles and Roadmap for AI-Driven Financial Planning, arXiv (Sep. 12, 2025), https://arxiv.org/html/2509.09922; Daniel Schwarcz, Tom Baker, & Kyle Logue, Regulating Robo-Advisors in an Age of Generative Artificial Intelligence, 82 Wash. & Lee L. Rev. 775 (2025).
[37] Ruo (Alex) Jia, Martin Eling, & Tianyang Wang, Gen AI Risks for Businesses: Exploring the role for insurance, at 30, Geneva Ass’n (Oct. 2025), available at https://www.genevaassociation.org/sites/default/files/2025-10/gen_ai_report_0110.pdf.
[38] Anas Alfaris, The Rise of the Cognitive Enterprise: Why Agentic AI Platforms Are the Next Great Business Revolution, World Econ. Forum (Jun. 25, 2025), https://www.weforum.org/stories/2025/06/cognitive-enterprise-agentic-business-revolution.
[39] See, e.g., Jonas Wanner, Lukas-Valentin Herm, Kai Heinrich, & Christian Janiesch, The Effect of Transparency and Trust on Intelligent System Acceptance: Evidence from a User-Based Study, 32 Electron. Mark. 2079, 2094 (2022), https://link.springer.com/article/10.1007/s12525-022-00593-5 (“[W]e find that [system transparency] poses a strong influential factor concerning the attitude and intention to use an intelligent system…”); Steve Fineberg et al., In the Gen AI Economy, Consumers Want Innovation They Can Trust, Deloitte (Sep. 25, 2025), https://www.deloitte.com/us/en/insights/industry/telecommunications/connectivity-mobile-trends-survey.html (explaining that Deloitte’s sixth Connected Consumer study, which surveyed 3,500 US consumers in June 2025 about their digital lives, found “[r]espondents are more satisfied and spend more with tech companies that lead on both innovation and data responsibility”).
[40] The FTC has long required clear and conspicuous disclosures for advertisements and endorsements. See 16 CFR §255.05(f) (“’[C]lear and conspicuous’ means that a disclosure is difficult to miss (i.e., easily noticeable) and easily understandable by ordinary consumers.”); see also .Com Disclosures: How to Make Effective Disclosures in Digital Advertising, Fed. Trade Comm’n. (Mar. 2013), available at https://www.ftc.gov/sites/default/files/attachments/press-releases/ftc-staff-revises-online-advertising-disclosure-guidelines/130312dotcomdisclosures.pdf; Commission Enforcement Policy Statement in Regard to Clear and Conspicuous Disclosure in Television Advertising, Fed. Trade Comm’n. (Oct. 21, 1970), available at https://www.ftc.gov/system/files/documents/public_statements/288851/701021tvad-pr.pdf.
[41] See 2023 Privacy and Data Security Update Fed. Trade Comm’n (2023), at 23-24, available at https://www.ftc.gov/system/files/ftc_gov/pdf/2024.03.21-PrivacyandDataSecurityUpdate-508.pdf.
[42] See, e.g., Oliver Roberts, EU AI Act’s Burdensome Regulations Could Impair AI Innovation, Bloomberg Law (Feb. 21, 2025), https://news.bloomberglaw.com/us-law-week/eu-ai-acts-burdensome-regulations-could-impair-ai-innovation (“The EU has now stunted its own AI development and cemented itself behind the US and China. Instead of focusing on AI investment and growth, the EU jumped straight to regulation—an ill-advised move in a nascent and rapidly evolving industry.”).
[43] See, e.g., Dual-Use Foundation Models with Widely Available Model Weights, Nat’l Telecomm. & Info. Admin. (Jul. 2024), available at https://www.ntia.gov/sites/default/files/publications/ntia-ai-open-model-report.pdf.
[44] Id. at 3-4.
[45] EU AI Act Overview, https://www.euaiact.com/blog/high-risk-ai-systems-under-the-eu-ai-act (last updated Aug. 1, 2024).
[46] See, e.g., Forum on Information & Democracy, A Voluntary Certification Mechanism for Public Interest AI (Sep. 2024), available at https://informationdemocracy.org/wp-content/uploads/2024/09/FID-Public-Interest-AI-Sept-2024.pdf.
[47] Some generative AI services aimed at health-care professionals attempt to comply with, for example, the Health Insurance Portability and Accountability Act (HIPAA). See, e.g., What Is BastionGPT?, BastionGPT (last accessed Nov. 24, 2025), https://bastiongpt.com/health (“BastionGPT is a private and compliant AI specially designed for healthcare professionals. It uses specially trained versions of the leading AI models including ChatGPT… Claude and Gemini to deliver the highest quality AI documentation and assistance in the market, and all while keeping personal data private and HIPAA compliant.”).
[48] For example, a Thomson Reuters Legal blog post notes that: “AI systems lacking adequate security measures risk exposing this data to hostile actors and putting the law firm in violation of its obligations. Security requirements make determination of a potential AI solution’s data protection capabilities essential.” Building Trust in AI to Keep Firm and Client Data Safe, Thomson Reuters (Nov. 3, 2025), https://legal.thomsonreuters.com/blog/how-to-use-ai-and-keep-law-firm-and-client-data-safe.
[49] Children’s Online Privacy Protection Rule, 90 Fed. Reg. 16918, 16950 (Apr. 22, 2025), (“Disclosures of a child’s personal information to third parties for monetary or other consideration, for advertising purposes, or to train or otherwise develop artificial intelligence technologies, are not integral to the website or online service and would require consent…”).
[50] 15 U.S.C. §§ 6801 et seq.
[51] 15 U.S.C. § 6802.
[52] 15 U.S.C. § 6803; § 6802.
[53] 42 U.S.C. §§ 1320d et seq.; 45 CFR §§ 164.500 et seq.
[54] 45 CFR § 164.502.
[55] 42 U.S.C. § 1320d-6.
[56] See Ans Baig & Muhammad Faisal Sattar, Do Not Track: Everything You Need To Know, securiti (last updated Nov. 23, 2024), https://securiti.ai/what-is-do-not-track-dnt.
[57] Id.
[58] See, e.g., Global Privacy Control, securiti (last accessed Nov. 24, 2025), https://securiti.ai/global-privacy-control.
[59] See California Civil Code §1798.120; Trishla Ostwal, Sephora to Pay $1.2 Million for Violating California’s Privacy Law, Adweek (Aug. 24, 2022), https://www.adweek.com/programmatic/sephora-violates-california-privacy-law (“Sephora failed to process people’s requests to opt-out of the sale of information to third-party companies.”).
[60] Disclosure requirements should be narrowly tailored. Their purpose is to make firms’ data-use commitments clear and enforceable—not to force the disclosure of trade secrets, sensitive security information, or internal system designs.