AI Division Publications
• Collection
Publisher
Software Engineering Institute
Collection Items
Concept-ROT: Poisoning Concepts in Large Language Models with Model Editing
• SEI Report
By Keltin Grimes, Marco Christiani, David Shriver, Marissa Connor
This work introduces Concept-ROT, a method for inserting trojans into LLMs that trigger on high-level concepts, bypassing safety and enabling harmful behaviors.
Read
Physics-Informed Deep B-Spline Networks for Dynamical Systems
• SEI Report
By Zhuoyuan Wang (Carnegie Mellon University, Department of Electrical and Computer Engineering), Raffaele Romagnoli (Duquesne University), Jasmine Ratchford, Yorie Nakahira (Carnegie Mellon University, Department of Electrical and Computer Engineering)
In this work, we integrate B-spline functions and physics informed learning to form physics-informed deep B-spline networks that can efficiently learn parameterized PDEs with varying initial and boundary conditions.
Read
Red-Teaming for Generative AI: Silver Bullet or Security Theater?
• SEI Report
By Michael Feffer, Anusha Sinha, Wesley H. Deng (Carnegie Mellon University), Zachary C. Lipton (Carnegie Mellon University), Hoda Heidari
In this work, we identify cases of red-teaming activities in the AI industry and conduct a survey to characterize the scope, structure, and criteria for AI red-teaming practices.
Read
Building Hybrid B-Spline And Neural Network Operators
• SEI Report
By Raffaele Romagnoli (Duquesne University), Jasmine Ratchford, Mark H. Klein
This paper proposes a B-spline neural operator for real-time CPS safety, combining neural networks with inductive bias to predict system behavior on a quadrotor.
Read
Transparency in the Wild: Navigating Transparency in a Deployed AI System to Broaden Need-Finding Approaches
• SEI Report
By Violet Turri, Katelyn Morrison (Carnegie Mellon University), Katherine-Marie Robinson, Collin Abidi, Adam Perer (Carnegie Mellon University), Jodi Forlizzi (Carnegie Mellon University), Rachel Dzombak
This case study focuses on incorporating various data sources and connecting with a broad ecosystem of stakeholders to support our analysis.
Read
Assessing LLMs for High Stakes Applications
• SEI Report
By Shannon Gallagher, Jasmine Ratchford, Tyler Brooks, Bryan Brown, Eric Heim, Bill Nichols, Scott McMillan, Swati Rallapalli, Carol J. Smith, Nathan M. VanHoudnos, Nick Winski, Andrew O. Mellinger
This work explores LLM deployment in intelligence reporting, highlighting key challenges in data, scaling, and assessment.
Read
Gone but Not Forgotten: Improved Benchmarks for Machine Unlearning
• SEI Report
By Keltin Grimes, Collin Abidi, Cole Frank, Shannon Gallagher
This paper describes and proposes new methods to evaluate unlearning algorithms, revealing key limitations through experiments across state-of-the-art models and vision datasets.
Read
Trustworthy by Design
• SEI Report
By Carol J. Smith
The rise of generative AI sparks demand for human-centered, trustworthy systems. Decades of HCI can guide responsible design for dynamic AI challenges.
Read
Tales from the Wild West: Crafting Scenarios to Audit Bias in LLMs
• SEI Report
By Katherine-Marie Robinson, Violet Turri, Carol J. Smith, Shannon Gallagher
This work introduces a scenario-based audit using RPG-style prompts where LLMs role-play characters to reveal bias in descriptions of individuals around them.
Read
Deep Operator Learning-Based Surrogate Models for Aerothermodynamic Analysis of AEDC Hypersonic Waverider
• SEI Report
By Khemraj Shukla (Brown University, Applied Mathematics Department), Jasmine Ratchford, Luis Bravo (DEVCOM Army Research Laboratory), Vivek Oommen (Brown University, Applied Mathematics Department), Nicholas Plewacki (DEVCOM Army Research Laboratory), Anindya Ghoshal (DEVCOM Army Research Laboratory), George Karniadakis (Brown University, Applied Mathematics Department)
In this work, we built a DeepONet-based surrogate model for 3D flow, showing the two-step method improves shock prediction and interpretability vs. baseline models.
Read
An Analytic Solution to Covariance Propagation in Neural Networks
• SEI Report
By Oren Wright, Yorie Nakahira (Carnegie Mellon University, Department of Electrical and Computer Engineering), José Moura (Carnegie Mellon University, Department of Electrical and Computer Engineering)
This paper presents an analytic moment propagation technique to accurately characterize the input-output distributions of deep neural networks.
Read
Augmenting Intelligence: Ethical Challenges in the Age of AI
• SEI Report
By Carol J. Smith
As AI evolves rapidly, this paper offers a framework to guide responsible review, integration, and management of emerging technologies in organizations.
Read
Towards Better Understanding of Domain Shift on Linear-Probed Visual Foundation Models
• SEI Report
By Eric Heim
This study finds some visual foundation models fail domain transfer, as linear probes on shifted data often show low training accuracy and poor transfer.
Read