Evaluating the Adversarial Robustness of Deployed AI Models

Deep learning vision models are known to be vulnerable to adversarial examples, where small, optimized input perturbations can cause confident mispredictions. A common objection is that such attacks work only in the digital domain and do not survive real-world conditions. In this talk, we show that gradient-based white-box attacks extend to deployed systems in three domains. First, we demonstrate physical attacks on vision models that remain effective under real-world variation in viewpoint, lighting, and distance, including perturbations that are difficult for humans to perceive. Second, we adapt the methodology to large language models, producing jailbreak and prompt injection attacks that elicit harmful content despite safety alignment. Third, we show that an adversarial image displayed on screen can hijack computer-use agents into executing attacker-chosen commands. These results show that deployed AI models can be reliably compromised by adversaries with gradient access, with consequences that grow as models move from perception to autonomous action. We present an automated framework for systematic adversarial evaluation, which we argue should become a standard part of AI deployment.
Co-sponsored by: Santa Clara University
Speaker(s): , Dr. Nageen Himayat
Room: 1302, Bldg: SCDI, 500 El Camino Real , Santa Clara , California, United States, 95053