Towards the vehicular metaverse: exploring distributed inference with transformer-based diffusion model

Generative artificial intelligence (GAI) is emerging as a promising solution for the vehicular metaverse due to its adaptable, high-quality, and multi-modal content generation capabilities. Particularly noteworthy is the recent introduction of the Sora model, a Transformer-based diffusion model, which exhibits exceptional performance in visual scenarios. However, diffusion vision transformer (DViT) models face limitations in terms of device resources, inference latency, and personalized requirements at the edge, despite their practical effectiveness in clouds. In response, we propose a DViT-enabled system to enhance vehicular metaverse services. Our approach involves a distributed DViT inference mechanism where road-side units (RSUs) and vehicles collaborate to execute the diffusion process and generate personalized content within vehicles using local prompts. Additionally, we address users' latency-sensitive service demands by formulating a distributed latency optimization problem that considers bandwidth, computation power, and dynamic positioning of heterogeneous devices. We then propose a value iteration-based distributed inference algorithm capable of adaptively determining optimal inference strategies within resource-constrained vehicular networks. Numerical simulations demonstrate that our approach achieves superior performance in reducing latency and enhancing success rates for inference tasks.