CyLab researchers to present at ACM FAccT 2026

CyLab Security and Privacy Institute researchers are set to present seven papers at the ninth annual ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT 2026).

The conference will take place in Montréal from June 25th to the 28th, bringing together a diverse community of scholars from computer science, law, social sciences, and humanities to investigate and tackle issues in the emerging area of algorithmic systems filtering, sorting, scoring, recommending, personalizing, and otherwise shaping human experience.

Below, we’ve compiled a list of papers presented by CyLab researchers at this year’s event.

Barriers to Evidence in AI-Related Cases and the Privatization of Proof

Authors: Sarah H. Cen, Carnegie Mellon University; Hannah Ismael, University of California, Berkeley; Lucia Zheng, Stanford University

Abstract: Evidence lies at the core of litigation, but it is increasingly difficult to obtain in AI-related disputes. Even when a claimant’s position has merit, cases are often settled or dismissed because decisive facts are hidden inside proprietary models, platform logs, and protected databases. Grounding our discussion in past and ongoing cases, we investigate how asymmetries in access, resources, and expertise can create functionally insurmountable barriers to producing AI evidence. We show how developers and deployers resist disclosure through various strategies, including trade secrecy claims and claims of technology irrelevance. From these patterns we identify six recurring sources of asymmetry—access to models, data, compute, infrastructure, expertise, and logs—that create uneven evidentiary burdens at multiple stages of litigation. We argue that these recurring burdens reflect a broader pattern that we call the privatization of proof: a shift in effective authority away from courts and toward private actors via control over evidence. We propose a three-part test for determining when proof privatization occurs and when it prevents a claimant from accessing information that ordinary litigation should be able to surface, drawing on concepts such as proportionality and feasible alternatives.

From Concepts to Practice: Challenges of Risk-Benefit Assessment in AI Research & Applications

Authors: Motahhare Eslami, Shixian Xie, Niloofar Nikookar, Mai Lee Chang, Cindy Peng, Julia Kim, John Zimmerman, and Hoda Heidari, Carnegie Mellon University

Abstract: Responsible AI research has largely emphasized identifying risks and harms, with comparatively less attention to how such risks should be weighed against potential benefits. As a result, AI researchers often lack structured support for engaging in risk–benefit reasoning about their own work. We present a two-phase study examining how AI researchers understand and apply concepts intended to support ethical evaluation of AI benefits and risks. In Phase 1, we introduced AI researchers to an ethical framework designed to support structured reasoning about benefits and risks, grounded in the capability approach and AI failure modes, and examined whether participants could apply it to their own projects. While participants demonstrated an understanding of the framework’s concepts and their relevance, they struggled to use them to evaluate the benefits and risks of their own research. To investigate the sources of this difficulty, Phase 2 involved interviews with 25 AI researchers, focusing on how they conceptualize AI research, research value, and the relationship between foundational and applied work. We find that norms privileging foundational research, ambiguity around what counts as AI research, and the diffusion of responsibility for societal outcomes constrain researchers’ engagement in risk–benefit reasoning. We argue that these challenges reflect broader features of AI research culture rather than shortcomings of individual frameworks, and we introduce a conceptual articulation intended to serve as a boundary object for examining how responsibility for assessing benefits and risks is understood and distributed across different forms of AI research.

Disclosure or Marketing? Analyzing the Efficacy of Vendor Self-reports for Vetting Public-sector AI

Authors: Blaine Kuehnert and Nari Johnson, Carnegie Mellon University; Ravit Dotan, TechBetter; Hoda Heidari, Carnegie Mellon University

Abstract: Documentation-based disclosure has become a central governance strategy for responsible AI, particularly in public-sector procurement. Tools such as model cards, datasheets, and AI FactSheets are increasingly expected to support accountability, risk assessment, and informed decision-making across organizational boundaries. Yet there is limited empirical evidence about how these artifacts are produced, interpreted, and used in practice. In this paper, we present a qualitative study of the GovAI Coalition FactSheet, a widely adopted transparency document designed to support AI procurement and governance in government contexts. Drawing on semi-structured interviews with vendors and public-sector practitioners, alongside a systematic analysis of completed FactSheets, we examine how FactSheets are used, what information they surface, and where they fall short. We find that FactSheets are asked to serve multiple and conflicting purposes simultaneously: showcasing vendor offerings, supporting evaluation and due diligence, and facilitating early-stage dialogue between vendors and agencies. These competing expectations, combined with the structural constraints of voluntary and public self-disclosure, limit the ability of FactSheets to function as standalone evaluation or risk-assessment tools. At the same time, our findings suggest that when understood as relational artifacts used to establish trust, shared understanding, and ongoing dialogue, FactSheets can help create conditions that support more meaningful disclosure and governance over time.

Evaluating AI-Generated Images of Cultural Artifacts with Community-Informed Rubrics

Authors: Nari Johnson, Microsoft Research and Carnegie Mellon University; Deepthi Sudharsan, RiskSpan; Hamna and Samantha Dalal, Microsoft Research; Theo Holroyd, The Stephen Perse Foundation; Anja Thieme, Microsoft Research; Hoda Heidari, Carnegie Mellon University; Daniela Massiceti, Jennifer Wortman Vaughan, and Cecily Morrison; Microsoft Research

Abstract: Responsible AI research has largely emphasized identifying risks and harms, with comparatively less attention to how such risks should be weighed against potential benefits. As a result, AI researchers often lack structured support for engaging in risk–benefit reasoning about their own work. We present a two-phase study examining how AI researchers understand and apply concepts intended to support ethical evaluation of AI benefits and risks. In Phase 1, we introduced AI researchers to an ethical framework designed to support structured reasoning about benefits and risks, grounded in the capability approach and AI failure modes, and examined whether participants could apply it to their own projects. While participants demonstrated an understanding of the framework’s concepts and their relevance, they struggled to use them to evaluate the benefits and risks of their own research. To investigate the sources of this difficulty, Phase 2 involved interviews with 25 AI researchers, focusing on how they conceptualize AI research, research value, and the relationship between foundational and applied work. We find that norms privileging foundational research, ambiguity around what counts as AI research, and the diffusion of responsibility for societal outcomes constrain researchers’ engagement in risk–benefit reasoning. We argue that these challenges reflect broader features of AI research culture rather than shortcomings of individual frameworks, and we introduce a conceptual articulation intended to serve as a boundary object for examining how responsibility for assessing benefits and risks is understood and distributed across different forms of AI research.

Beyond the Single Turn: Reframing Refusals as Dynamic Experiences Embedded in the Context of Mental Health Support Interactions with LLMs

Authors: Ningjing Tang, Alice Qian, and Qiaosi Wang, Carnegie Mellon University; Esther Howe, University of Washington School of Medicine; Blake Bullwinkel, Microsoft; Paola Pedrelli, Harvard Medical School; Jina Suh, University of Washington; Hoda Heidari and Hong Shen, Carnegie Mellon University

Abstract: *Content Warning: This paper contains participant quotes and discussions related to mental health challenges, emotional distress, and suicidal ideation.* Large language models (LLMs) are increasingly used for mental health support, yet the model safeguards—particularly refusals to engage with sensitive content—remain poorly understood from the perspectives of users and mental health professionals (MHPs) and have been reported to cause real-world harms. This paper presents findings from a sequential mixed-methods study examining how LLM refusals are experienced and interpreted in mental health support interactions. Through surveys (N=53) and in-depth interviews (N=16) with individuals using LLMs for mental health support and MHPs, we reveal that refusals are not isolated, single-turn system behaviors but rather constitute dynamic, multi-phase experiences: pre-refusal expectation formation, refusal triggering and encounter, refusal message framing, resource referral provision, and post-refusal outcomes. We contribute a multi-phase framework for evaluating refusals beyond binary policy compliance accuracy and design recommendations for future refusal mechanisms. These findings suggest that understanding LLM refusals requires moving beyond single-turn interactions toward recognizing them as holistic experiences embedded within users' support-seeking trajectories and the broader LLM design pipeline.

Investigating the Role of AI in Emergency Management: Use Cases, Challenges, and Opportunities

Authors: Samsara Foubert, Minjung Park, Kanad Shrikar Pardeshi, and Thomas Manzini, Carnegie Mellon University; Robert McDaniel and David Merrick, Florida State University; Robin Murphy, Aarti Singh, and Hoda Heidari, Carnegie Mellon University

Abstract: Though natural disasters and other hazards remain a persistent threat, Emergency Managers (EMs) face substantial barriers to effectively completing their life-saving job functions. Pervasive staffing challenges and diminishing funding sources complicate a profession tasked with timely, critical decision-making under uncertainty. Technology presents a significant opportunity to complement EMs' workflows; however, commercial products are costly and serve narrow use cases. To understand how EMs use and envision using technology, and in particular AI, we conducted in-depth interviews with 32 professionals spanning different geographic regions within the US. Through iterative thematic analysis and affinity mapping, we identified 7 key categories of AI technology use cases for EMs' core job functions. In the context of these use cases, we describe the limitations of current technologies and EMs' imaginings of how AI could complement their work. Reflecting on these findings, we provide guidance on EMs' design needs and highlight opportunities for computing research to support the development of future EM technologies.

Toward Third-Party Assurance of AI Systems: Design Requirements, Prototype, and Early Testing

Authors: Rachel Minyoung Kim, Blaine Kuehnert, Alice Lai, Ken Holstein, Hoda Heidari, and Rayid Ghani, Carnegie Mellon University

Abstract: As Artificial Intelligence (AI) systems proliferate, the need for systematic, transparent, and actionable processes for evaluating them is growing. While many resources exist to support AI evaluation, they have several limitations. Few address both the process of designing, developing, and deploying an AI system and the outcomes it produces. Furthermore, few are end-to-end and operational, give actionable guidance, or present evidence of usability or effectiveness in practice. In this paper, we introduce a third-party AI assurance framework that addresses these gaps. We focus on third-party assurance to prevent conflict of interest and ensure credibility and accountability of the process. We begin by distinguishing assurance from audits in several key dimensions. Then, following design principles, we reflect on the shortcomings of existing resources to identify a set of design requirements for AI assurance. We then construct a prototype of an assurance process, that consists of (1) a responsibility assignment matrix to determine the different levels of involvement each stakeholder has at each stage of the AI lifecycle, (2) an interview protocol for each stakeholder of an AI system, (3) a maturity matrix to assess AI systems’ adherence to best practices, and (4) a template for an assurance report that draws from more mature assurance practices in business accounting. We conduct early validation of our AI assurance framework by applying the framework to two distinct AI use cases—a business document tagging tool for downstream processing in a large private firm, and a housing resource allocation tool in a public agency—and conducting six expert validation interviews. Our findings show early evidence that our AI assurance framework is sound and comprehensive, usable across different organizational contexts, and effective at identifying bespoke issues with AI systems.