With the rapid advancement of autonomous vehicles, there is a growing demand for infotainment services that require high-quality and delay-sensitive video content. This paper proposes a multi-agent deep reinforcement learning (MADRL) approach for video cache replacement and delivery in mobility-aware vehicular networks. Unlike previous studies, our work focuses on videos of finite lengths and incorporates dynamic cache replacement, optimizing this alongside the delivery of individual video chunks. Considering the challenge of obtaining complete network state information at a central unit (e.g., macro base station), we adopt a MADRL framework to enable roadside units (RSUs) to autonomously decide on video caching and delivery strategies, leveraging partial information from neighboring RSUs. We evaluate the proposed method using various quality-of-service (QoS) metrics. Extensive simulation results demonstrate that our scheme consistently delivers high average video quality while reducing playback stalls, replacement costs, and backhaul usage.