Robust & trustworthy AI
Adversarial robustness, jailbreak defense, hallucination mitigation, and secure, reliable multimodal alignment.
Md Zarif Hossain
PhD Candidate in Computer Science · SPEED Lab, Florida Atlantic University
Doctoral Fellow Awardee · Ex-Gen AI Research Intern, AT&T Labs
I am a PhD candidate and Graduate Research Assistant at Florida Atlantic University, where I conduct research in the SPEED Lab under the supervision of Dr. Ahmed Imteaj. My research focuses on developing robust, efficient, and trustworthy AI systems, with particular emphasis on large vision-language models (LVLMs), AI safety and alignment, and federated learning.
A central theme of my research is understanding the interplay between robustness, visual grounding, and safety in multimodal AI systems, particularly how improvements in one dimension can influence other aspects of model reliability. More recently, I have been exploring the underlying mechanisms of safety and alignment in LVLMs. My work has appeared in leading conferences and journals, including NeurIPS, CVPR, EMNLP, ICCV, AAAI, IEEE BigData, ICDCS, and IEEE Transactions.
- Office
- EE 307, Florida Atlantic University, 777 Glades Road, Boca Raton, FL 33431
20peer-reviewed papers
5A* conference papers
15students mentored
Research interests
Multimodal AI alignment
Aligning vision and language in LVLMs and multimodal LLMs: closing the modality gap, grounding outputs, and safe, reliable multimodal behavior.
Scalable & on-device AI
Federated and meta-learning, resource-efficient VLMs, edge deployment and distributed optimization.
Control theory of LLMs
Control-theoretic approaches to steering LLMs and LVLMs, and to stable multimodal reasoning.
News
Paper “Do Robust LVLMs Hallucinate More? Uncovering Robustness-Induced Hallucination in Large Vision-Language Models” accepted to the NeurIPS 2026 Main Track.
Award Received the NeurIPS 2026 Travel Grant.
Paper Two papers accepted at EMNLP 2026: “Do LVLMs Know When They Are Wrong?” (Main Track) and “From Benchmarks to Real-World Use” on chart-understanding robustness (Industry Track).
Service Serving as a reviewer for NeurIPS 2026, CVPR 2026, ICLR 2026, ECCV 2026, ICRA 2026, EMNLP 2026 and ACL 2027.
Paper “When Data is Scarce, Learn to Adapt: Robust Federated Learning via Adversarial Meta-Optimization” accepted to the Findings track of CVPR 2026. Project page →
Talk Project presentation and research demo accepted at the 2026 NAIRR Annual Meeting in Arlington, VA. Grateful to NSF for travel support.
Service Reviewed for IEEE Transactions on Information Forensics & Security and IEEE Transactions on Vehicular Technology.
Position Joined Florida Atlantic University as a Graduate Research Assistant in the SPEED Lab.
Service Reviewed for IEEE Transactions on Consumer Electronics.
Position Started as an Advanced Research Intern at AT&T Labs, Bedminster, NJ.
Paper Blockchain-empowered federated learning paper published in IEEE Transactions on Artificial Intelligence.
Paper Paper accepted at the CVPR 2025 Workshop on Fair, Data-Efficient and Trusted Computer Vision.
Paper SLADE accepted to the CVPR 2025 main conference (22.1% acceptance rate). Received the CVPR 2025 Travel Grant.
Paper One paper accepted at the AAAI 2025 Spring Symposium.
Talk Presented our robust VLM encoder work at IEEE BigData 2024.
Paper “Securing Vision-Language Models Against Jailbreak and Adversarial Attacks” accepted at IEEE BigData 2024.
Award Awarded the SIU Doctoral Fellowship, the sole recipient from the Computer Science department.
Paper Three papers accepted at IEEE COMPSAC 2024.
Paper Flamingo accepted at the IEEE ICDCS 2024 Workshops.
Award Received an NSF Travel Grant to attend MobiHoc 2023.
Paper One paper accepted at IEEE SMARTCOMP 2023.
Position Began my PhD as a Graduate Research Assistant in the SPEED Lab, Southern Illinois University.
Selected research
HALS
Do Robust LVLMs Hallucinate More? Uncovering Robustness-Induced Hallucination in Large Vision-Language Models
Swapping in an adversarially robust vision encoder makes LVLMs hallucinate more: robust training homogenizes visual tokens until patches look alike. HALS steers the decoder’s hidden states back toward grounded outputs, training-free and with the encoder untouched.
- −31.3
- CHAIRS points
- +2.6
- POPE accuracy
- −30.6%
- AMBER hallucination
FAML
When Data is Scarce, Learn to Adapt: Robust Federated Learning via Adversarial Meta-Optimization
Train one client adversarially and the whole federation becomes robust. FAML meta-learns a shared initialization so robustness learned on a single client transfers to all, while drift regularizers keep every client aligned with the global model.
- +6.26
- PGD-20 robust acc.
- +8.46
- AutoAttack robust acc.
- 2.1×
- faster per round
Trajectory-Lens
Do LVLMs Know When They Are Wrong? Trajectory-Based Correctness Estimation from a Single Forward Pass
Correctness lives in the trajectory, not in any single layer. Trajectory-Lens follows language, vision and projection signals across the model’s depth and fuses them into one score that flags wrong answers, all from the answer’s own forward pass.
- +16.44
- AUROC
- +15.61
- AUPR
- 4.6 ms
- per sample
SLADE
Shielding against Dual Exploits in Large Vision-Language Models
Adversarial images and jailbreak prompts both enter an LVLM through its CLIP vision encoder. SLADE fine-tunes CLIP with patch- and global-level contrastive losses on adversarial views, with no labels and no LVLM training, giving a drop-in encoder that shields against both exploits without losing detail.
- 0 / 125
- targeted attacks succeed
- 87.5
- CIDEr under targeted attack
- 13.7%
- toxicity under VisualAdv
Selected publications
Full list →Bold marks my name; marks A* conferences (CORE).
2026
-
NeurIPS ★ A* conference
Do Robust LVLMs Hallucinate More? Uncovering Robustness-Induced Hallucination in Large Vision-Language Models
The Fortieth Annual Conference on Neural Information Processing Systems (NeurIPS), Main Track, 2026
-
EMNLP ★ A* conference
Do LVLMs Know When They Are Wrong? Trajectory-Based Correctness Estimation from a Single Forward Pass
Conference on Empirical Methods in Natural Language Processing (EMNLP), Main Track, 2026
-
EMNLP ★ A* conference
From Benchmarks to Real-World Use: Evaluating the Robustness of Vision-Language Models for Chart Understanding
Conference on Empirical Methods in Natural Language Processing (EMNLP), Industry Track, 2026
-
CVPR ★ A* conference
When Data is Scarce, Learn to Adapt: Robust Federated Learning via Adversarial Meta-Optimization
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Findings Track, 2026
-
IJCNN
Sim-CLIP: Unsupervised Siamese Adversarial Fine-Tuning for Robust and Semantically-Rich Vision-Language Models
International Joint Conference on Neural Networks (IJCNN), 2026
2025
-
CVPR ★ A* conference
SLADE: Shielding against Dual Exploits in Large Vision-Language Models
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025 22.1% acceptance
-
TAI
Blockchain-Empowered Cyber-Secure Federated Learning for Trustworthy Edge Computing
IEEE Transactions on Artificial Intelligence, 2025 Q1 journal
-
ICCVW
Pose to Protect: Federated Skeleton-Based Anomaly Detection for Privacy-Conscious Video Surveillance
International Conference on Computer Vision Workshops (ICCVW), 2025
-
CVPRW
Towards Trustworthy Autonomous Vehicles with Vision-Language Models Under Targeted and Untargeted Adversarial Attacks
IEEE/CVF CVPR Workshops, 2025
2024
-
BigData
Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks
IEEE International Conference on Big Data (BigData), pp. 6250–6259, 2024
Highlights
Experience
-
Aug 2025 – Present
Graduate Research Assistant, SPEED Lab
Florida Atlantic University · Boca Raton, FL
- Research on adversarial robustness and hallucination mitigation in large vision-language models.
- Mentor undergraduate and master's students on multimodal and federated learning projects.
- Contribute to federally funded research on trustworthy multimodal systems.
-
Summer 2025
Advanced Research Intern
AT&T Labs · Bedminster, NJ
- Built a Transformer-based template recommendation system for site-maintenance workflows and an ML ticket-approval automation system.
- Automated preprocessing for 100K+ heterogeneous maintenance logs, enabling scalable, interpretable clustering.
- Delivered an automation framework with projected annual savings above $250K.
-
Jan 2023 – Aug 2025
Graduate Research Assistant, SPEED Lab
Southern Illinois University · Carbondale, IL
- Robust federated learning against adversarial attacks; jailbreak vulnerabilities in LVLMs.
- First-author papers at CVPR 2025, IEEE BigData 2024 and IEEE TAI.
- Contributed to NSF- and DoD-funded proposals, including a successful NSF CRII award ($163K).
Projects
Ongoing
RoadSentry.ai
AI-Powered Roadside Construction Safety Intelligence
An AI safety platform for roadside work zones that combines computer vision, multi-object tracking and vision-language models to monitor workers, vehicles and equipment, flag hazards and near-misses, and raise real-time alerts.
A spatial safety layer reasons about restricted-zone entry, worker–equipment proximity and movement, while a central dashboard surfaces AI-generated incident summaries, risk levels, visual evidence and safety trends.
Ongoing
AutoScene
End-to-end synthetic edge-case generation for autonomous driving. Couples the CARLA simulator with an LLM to synthesize rare, safety-critical scenes, then uses reinforcement learning to align vision-language and control models.
Ongoing
Bridging the multimodal gap
Reducing the modality gap between visual and textual embeddings to improve alignment and semantic consistency in VLM representations.
Ongoing
Self-correcting multimodal LLMs
Techniques that let an LVLM notice when it starts hallucinating and correct itself, producing accurate, grounded responses without external intervention or human feedback, to make LVLMs more reliable in real-world deployment.
VLMs for autonomous driving
Combining visual input with contextual language understanding to improve vehicle perception and real-time decision-making.
Semi-federated crime detection with UAVs
A semi-federated framework for real-time crime detection from UAV surveillance, using YOLO for weapon detection and LSTM pose estimation. IEEE ICDCS 2024 poster.
Drone swarming & distributed learning
Orchestrating a fleet of DJI Tello drones that stream observations to each other and learn collaboratively from a shared knowledge base.
