Generative Models · Multimodal LLMs · Computer Vision

Haojie Zhang | 张浩杰

I am currently a Senior Algorithm Researcher at Alibaba ATH, working in Token Foundry. I received my M.Sc. in Information and Communication Engineering from South China University of Technology (SCUT) in 2026, advised by Prof. Kui Jia and Dr. Xun Xu, and my B.Eng. in Information Engineering from SCUT in 2023. I previously visited the Department of Automation at Tsinghua University, where I worked with Prof. Jianhua Tao.

My research explores generative models, multimodal large language models, and foundation vision systems. My long-term goal is to bridge perception, understanding, and imagination in intelligent systems.

Email: eehaojiezhang@gmail.com Google Scholar GitHub

Hangzhou, China

News

Aug. 2026I joined Alibaba ATH as part of Token Foundry.
Jul. 2026The technical report for LingBot-World 2.0 was released.
May 2026MuSS was accepted to ACM MM 2026 as an oral presentation.
Apr. 2026LetsTalk was accepted for publication in IEEE Transactions on Multimedia.
Jan. 2026PaDT was accepted to ICLR 2026.

Selected Publications

* Equal contribution. Selected work is ordered by recent publication milestones.

LingBot-World 2.0 teaser showing diverse interactive worlds

Technical Report · 2026

Infinite Worlds with Versatile Interactions

Zelin Gao, Qiuyu Wang, Jiapeng Zhu, Jingye Chen, Zichen Liu, et al., Haojie Zhang, et al.

An interactive video world model supporting real-time, persistent, and controllable virtual worlds.

arXiv Project Code

ACM MM · Oral · 2026

MuSS: A Large-Scale Dataset and Cinematic Narrative Benchmark for Multi-Shot Subject-to-Video Generation

Haojie Zhang, Di Wu, Bingyan Liu, Linjie Zhong, Yuancheng Wei, Xingsong Ye, Nanqing Liu, Yaling Liang

A large-scale cinematic dataset and benchmark for narrative coherence and identity consistency in multi-shot video generation.

arXiv Project Code

Under Review · 2026

DiffCap-Bench: A Comprehensive, Challenging, Robust Benchmark for Image Difference Captioning

Yuancheng Wei*, Haojie Zhang*, Linli Yao, Lei Li, Jiali Chen, et al.

A challenging benchmark and LLM-as-a-Judge protocol for evaluating image difference captioning.

arXiv Code

IEEE TMM · 2026

Multimodal Diffusion Transformer with Memory Bank for Scalable Long-Duration Talking Video Generation

Haojie Zhang, Zhihao Liang, Ruibo Fu, Bingyan Liu, Zhengqi Wen, Xuefei Liu, Jianhua Tao, Yaling Liang

LetsTalk combines a diffusion transformer with a memory bank for efficient and consistent long-duration talking video generation.

arXiv Project Code

ICLR · 2026

Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs

Yongyi Su*, Haojie Zhang*, Shijie Li, Nanqing Liu, Jingyi Liao, et al.

PaDT introduces Visual Reference Tokens that enable multimodal language models to generate diverse dense visual predictions.

arXiv Code

TPAMI · Under Review

Improving the Generalization of Segmentation Foundation Models via Weakly-Supervised and Unsupervised Adaptation

Haojie Zhang, Yongyi Su, Nanqing Liu, Shijie Li, Xulei Yang, Xiangyu Yue, Kui Jia, Xun Xu

WeSAM++ extends foundation-model adaptation with patch-level contrastive learning and masked image modeling.

Preprint Code

CVPR · 2024

Improving the Generalization of Segmentation Foundation Model under Distribution Shift via Weakly Supervised Adaptation

Haojie Zhang, Yongyi Su, Xun Xu, Kui Jia

WeSAM adapts segmentation foundation models to distribution shifts through weakly supervised self-training and efficient regularization.

Paper arXiv Project Code

Education & Experience

Education

2024.04 – 2024.11

Tsinghua University

Visiting Student, Department of Automation
Advisor: Prof. Jianhua Tao

2023.09 – 2026.06

South China University of Technology

M.Sc. in Information and Communication Engineering
Advised by Prof. Kui Jia and Dr. Xun Xu

2019.09 – 2023.06

South China University of Technology

B.Eng. in Information Engineering (Innovation Class)
Graduated with distinction · GPA 3.64/4.0

Experience

2026.08 – Present

Alibaba ATH

Token Foundry · Hangzhou

2026.04 – 2026.07

Ant Research Institute

Lingbot-World Team · Hangzhou

2025.10 – 2026.04

Tencent TEG

Hunyuan Team · Shenzhen

2025.04 – 2025.10

Tencent WXG

WeChat Vision · Shenzhen

2024.12 – 2025.03

Tencent IEG

LIGHTSPEED · Shenzhen

Open-source Projects

Research code and project pages accompanying selected work.

LetsTalkLong-duration talking video generation MuSSMulti-shot subject-to-video benchmark PaDTUnified visual tasks in MLLMs WeSAMAdaptation of segmentation foundation models