Award

• [2025] JailbreakV won Second Prize of SafeBench Competition
• [2024] USENIX Security Distinguished Paper Award.
• [2024] Schmidt Sciences AI2050 Early Career Fellow
• [2024] ACM Gordon Bell Final list
• [2024] My PhD student Xiaogeng Won NVIDIA Fellowship on Security track
• [2024] Selected in Stanford/Elsevier Top 2% Scientists List 2024
• [2023] ACM Gordon Bell Special Prize for HPC-Based COVID-19 Research
• [2023] Impact Award from Argonne National Laboratory.
• [2021] International Conference on Embedded Wireless Systems and Networks(EWSN) Best Paper Award
• [2014] MobiCom Best Paper Award

Recent Invited talks

• [12/2025] I will be at NeurIPS and lead a discussion session at Alignment workshop@SD on How to Secure AI Agents
• [6/2025] Invited talk at The 5th Workshop of Adversarial Machine Learning on Computer Vision: Foundation Models + X at CVPR
• [5/2025] Invited talk at Secure Generative AI Agents Workshop at IEEE Symposium on Security and Privacy
• [4/2025] Invited talk at International Symposium on Trustworthy Foundation Models
• [12/2024] Invited talk at SFU@NeurIPS
• [12/2024] Invited talk at LLM and Agent Safety Competition@NeurIPS.
• [10/2024] Keynote at CCS Workshop on Large AI Systems and Models with Privacy and Safety Analysis.
• [10/2024] Invited talk at Trillion Parameter Consortium (TPC) on LLM safety and security
• [10/2024] Invited talk at NSF Workshop on Large Language Models for Network Security
• [06/2024] Invited talk at CVPR Workshop of Adversarial Machine Learning on Computer Vision: Robustness of Foundation Models
• [06/2024] Talk at NAACL tutorial on Combating Security and Privacy Issues in the Era of Large Language Models
• [05/2024] Invited talk at ICLR Secure and Trustworthy Large Language Models
• [12/2023] Invited Talk at NeurIPS TDW workshop

Recent News

• package
• package
• [3/2026] Senior Area Chair of ACL, EMNLP and NeurIPS.
• [1/2025] We have 9 papers at ICLR and 3 papers at ACL on Model Safety and Security. Congratulations to all authors.
• [12/2024] We got the Fall Research Competition Award at UW-Madison. Thank you UW-Madison and OVCR.
• [11/2024] Our group recently received funding and donations. Thank you Amazon and Apple.
• [11/2024] Our lab will have a winter break this Dec. Lab members will enjoy some well-deserved vacation time, with their families and loved ones.
• [09/2024] We have four papers at NeurIPS regular track.
• [09/2024]Our study on safety of RLFH Alignment is accepted to S&P (Oakland) 2025
• [07/2024] MultiModal jailbreak benchmark is accepted to COLM. It is from the interns in my group.
• [07/2024] 4/4 papers are accepted to ECCV on the topic of trustworthy VLM and driving. Two of them are from interns in my group.
• [06/2024] Senior Area Chair for NeurIPS Benchmark track
• [06/2024] I am currently serving as AC for the NeurIPS regular track
• [05/2024] Our jailbreak paper is accepted to USENIX Security. Congratulations, Zhiyuan!
• [03/2024] Five papers at NAACL on LLM security (4 main and 1 finding): two on the backdoor attack, one on backdoor defense, one on jailbreak attacks, and one on model fingerprint. Stay tuned on these exciting fields
• [03/2024] PreDa for personalized federated learning is accepted at CVPR 2024.
• [01/2024] Three papers at ICLR.
• [01/2024] Two papers at TMLR
• [12/2023] Invited Talk at NeurIPS TDW workshop
• [10/2023]Our paper MoleculeSTM has been accepted to Nature Machine Intelligence. MoleculeSTM aims to align the nature language and molecule representation into the same representation space.
More
[10/2023] Three papers at EMNLP and one paper at NeurIPS. For our NeurIPS paper, we study a new threat of the instruction tuning of LLMs by injecting the Ads. This is the first work that views the LLMs as the generative model and aims to attack the generative property of LLMs.
[10/2023] Our tutorial on Security and Privacy in the Era of Large Language Models is accepted to NAACL.
[05/2023] One paper at ACL. Congratulations to zhuofeng and jiazhao. We propose an attention-based method to defend against NLP backdoor attacks
[04/2023] Two papers at ICML. Congratulations to Jiachen and Zhiyuan. We propose the first benchmark for code copyright of code generation models.
[02/2023] Two papers at CVPR. Congratulations to Yiming and Xiaogeng. Xiaogeng is an intern from my group at ASU.
[02/2023] I will give a tutorial at CVPR 2023 on the topic of trustworthiness in the era of Foundation Models. Stay tuned!
[01/2023] Impact Award from Argonne National Laboratory.
[01/2023] One paper got accepted to USENIX Security 2023.
[1/2023] Three papers are accepted to ICLR 2023 [a]: We explain why and how to use diffusion model to improve adversarial robustness and design DensePure which leverages the pretrained diffusion model and classifier to provide the state-of-the-art certified robustness. [b]:This is our first attemp on retrieval-based framework and AI for drug discovery. We will recently release more work in this research line. Stay tuned!
[12/2022] Our team won the ACM Gordon Bell Special Prize for COVID-19 Research.
[09/2022] One papers got accepted to USENIX Security 2023.
[09/2022] Two papers got accepted to NeurIPS 2022.
[09/2022] Our paper RobustTraj has been accepted to CORL for oral presentations. We explore to train a robust Trajectory Prediction Model against adversarial attacks.
[08/2022] I will be giving a talk in virtual seminar series on Challenges and Opportunities for Security & Privacy in Machine Learning.
[07/2022] One survey paper to discuss the challenge and opportunity of machine learning security got accepted to ACM Computing Survey 2022.
[07/2022] Two papers got accepted to ECCV 2022.
[05/2022] Two papers got accepted to ICML 2022. Thanks for all of my collaborators.
[3/2022] I will be giving a talk in AAAI 2022 1st International Workshop on Practical Deep Learning in the Wild.
[3/2022] I will be giving a talk in AAAI 2022 workshop on Adversarial Machine Learning and Beyond.
[2/2022] One paper is accepted to ICLR.

Selected Publications ([Full List])

(* represents equal contribution)

[Google Scholar]

Model/Agent Safety and Robustness Evaluation(Red-teaming)

• AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs

  Xiaogeng Liu*, Peiran Li*, G. Edward Suh, Yevgeniy Vorobeychik, Zhuoqing Mao, Somesh Jha, Patrick McDaniel, Huan Sun, Bo Li, Chaowei Xiao

  ICLR 2024
• AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models.

  Xiaogeng Liu, Nan Xu, Muhao Chen, Chaowei Xiao.

  ICLR 2024.
• DecodingTrust-Agent Platform (DTap): A Controllable and Interactive Red-Teaming Platform for AI Agents

  Zhaorun Chen, Xun Liu, Haibo Tong, Chengquan Guo, Yuzhou Nie, Jiawei Zhang, Mintong Kang, Chejian Xu, Qichang Liu, Xiaogeng Liu, Tianneng Shi, Chaowei Xiao, Sanmi Koyejo, Percy Liang, Wenbo Guo, Dawn Song, Bo Li

  .
• A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems.

  Fangzhou Wu, Ning Zhang, Somesh Jha, Patrick McDaniel, Chaowei Xiao

  [pdf]
• Do Not Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models.

  Zhiyuan Yu, Xiaogeng Liu, Shuning Liang, Zach Cameron,Chaowei Xiao, Ning Zhang.

  USENIX Security 2024. [pdf]

Safety Alignment/Mitigation

• Safety Mid-Training: Internalizing LLM Safety as a Foundational Capability

  Zhengyue Zhao, Liwei Jiang,Yejin Choi, Chaowei Xiao

  [pdf]

• ARMOR: Aligning Secure and Safe Large Language Models via Meticulous Reasoning

  Zhengyue Zhao, Yingzi Ma, Somesh Jha, Marco Pavone, Patrick McDaniel, Chaowei Xiao.

  ICLR 2026. [pdf]

• Mitigating Fine-tuning Jailbreak Attack with Backdoor Enhanced Alignment.

  Jiongxiao Wang, Jiazhao Li, Yiquan Li, Xiangyu Qi, Muhao Chen, Junjie Hu, Yixuan Li, Bo Li, Chaowei Xiao

  NeurIPS 2024

Agent Security via System-level Solutions

• DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agents

  Hao Li, Xiaogeng Liu, Hung-Chun Chiu, Dianqi Li, Ning Zhang, Chaowei Xiao.

  NeurIPS 2025. [pdf]

• AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection.

  Weidi Luo, Shenghong Dai, Xiaogeng Liu, Suman Banerjee, Huan Sun, Muhao Chen, Chaowei Xiao.

  ACL 2025. [pdf]

• PIGuard: Prompt Injection Guardrail via Mitigating Overdefense for Free.

  Hao Li, Xiaogeng Liu, Ning Zhang, Chaowei Xiao.

  ACL 2025. [pdf]

• System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspective

  Fangzhou Wu, Ethan Cecchetti, Chaowei Xiao

  [pdf]

Ai4Science (Bio and Math)

• ProteinDT: A Text-guided Protein Design Framework

  Shengchao Liu, Yanjing Li, Zhuoxinran Li, Anthony Gitter, Yutao Zhu, Jiarui Lu, Zhao Xu, Weili Nie, Arvind Ramanathan, Chaowei Xiao*, Jian Tang*, Hongyu Guo*, Anima Anandkumar*

  Nature Machine Intelligence 2025 [pdf]

• Multi-modal molecule structure–text model for text-based retrieval and editing

  Shengchao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang*, Chaowei Xiao*, Animashree Anandkumar*.

  Nature Machine Intelligence. [pdf]

• ChatDrug: ChatGPT-powered Conversational Drug Editing Using Retrieval and Domain Feedback

  Shengchao Liu, Jiongxiao Wang, Yijin Yang, Chengpeng Wang, Ling Liu, Hongyu Guo, Chaowei Xiao

  ICLR 2024. [pdf]

• LeanAgent: Lifelong Learning for Formal Theorem Proving.

  Adarsh Kumarappan, Mo Tiwari, Peiyang Song, Robert Joseph George, Chaowei Xiao, Anima Anandkumar.

  ICLR 2024

Foundation Models, Agents, Test-time Training

• Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models

  Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, Chaowei Xiao

  NeurIPS 2022. [pdf]

• Dolphins: Multimodal Language Model for Driving.

  Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, Chaowei Xiao

  ECCV 2024
• Voyager: An Open-Ended Embodied Agent with Large Language Models.

  Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, Anima Anandkumar

  TMLR 2024
• Prismer: A Vision-Language Model with Multi-Task Experts

  Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, Anima Anandkumar.

  TLMR 2024. [pdf]

• Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study

  Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, Anima Anandkumar, Bryan Catanzaro

  EMNLP 2023. [pdf]

• Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning

  Zhuolin Yang, Wei Ping, Zihan Liu, Vijay Anand Korthikanti, Weili Nie, De-An Huang, Linxi Fan, Zhiding Yu, Shiyi Lan, Bo Li, Mohammad Shoeybi, Ming-Yu Liu, Yuke Zhu, Bryan Catanzaro, Chaowei Xiao*, Anima Anandkumar*

  EMNLP 2023. [pdf]

Trustworthy LLMs

• [Watermark]Can Watermarks be Used to Detect LLM IP Infringement For Free?

  Zhengyue Zhao, Xiaogeng Liu, Somesh Jha, Patrick McDaniel, Bo Li, Chaowei Xiao

  ICLR 2024
• [Hallucination]HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detection.

  Xuefeng Du, Chaowei Xiao, Yixuan Li

  NeurIPS 2024 (spotlight)
• [Fingerprinting] Instructional Fingerprinting of Large Language Models.

  Jiashu Xu, Fei Wang, Mingyu Derek Ma, Pang Wei Koh, Chaowei Xiao, Muhao Chen.

  NAACL 2024. [pdf]

• [Memory Poisoning]AgentPoison: Red-teaming LLM Agents via Memory or Knowledge Base Backdoor Poisoning

  Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, Bo Li. .

  NeurIPS 2024. [pdf]

• [Training-time threats] On the Exploitability of Instruction Tuning

  Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao*, Tom Goldstein*

  NeurIPS 2023. [pdf]

Adversarial Machine Learning

• [Defense] Diffusion Models for Adversarial Purification

  Weili Nie, Brandon Guo, Yujia Huang,Chaowei Xiao, Arash Vahdat, Anima Anandkumar.

  ICML 2022[pdf] [code]

• [Certification] DensePure: Understanding Diffusion Models towards Adversarial Robustness.

  Chaowei Xiao*, Zhongzhu Chen*, Kun Jin*, Jiongxiao Wang*, Weili Nie, Mingyan Liu, Anima Anandkumar, Bo Li, Dawn Song

  ICLR 2023. [pdf] [code]

• [Physical Attacks] Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attacks.

  Yulong Cao*, Ningfei Wang*,Chaowei Xiao*, Dawei Yang*, Jin Fang, RuigangYang, Qi Alfred Chen, Mingyan Liu, Bo Li.

  IEEE Symposium on Security and Privacy (Oakland) 2021

• [Attacks] Spatially Transformed Adversarial Examples

  Chaowei Xiao*, Jun-Yan Zhu*, Bo Li, Warren He, Mingyan Liu and Dawn Song

  In International Conference on Learning Representations (ICLR), 2018 [pdf]

• [Attacks] Generating Adversarial Examples with Adversarial Networks

  Chaowei Xiao, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu and Dawn Song

  In International Joint Conference on Artificial Intelligence (IJCAI), 2018. [pdf]

• [Physical Attacks] Robust Physical-World Attacks on Machine Learning Models

  Kevin Eykholt*, Ivan Evtimov*, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno and Dawn Song

  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 [pdf]