A collision prediction framework integrating scenario-based approach with data-driven approach is proposed to enhance the safety of autonomous driving vehicles as well as advanced driver assistance systems. No matter howthe autonomous driving is intelligent, it is inevitable to consider malfunction or faults of sensors, actuators, and processors, thus resulting in the collision. To address these issues, several studies have been proposed to improve performance based on model-based or data-driven approaches. However, there are several challenges in terms of the scarcity of accident data and the lack of explainability of deep neural networks. To overcome the limits of both approaches, an integrated framework that includes trajectory prediction, threat assessment, and decision-making based on convolutional neural network (CNN) for collision prediction is introduced. For more detail, both trajectory prediction based on Kalman filter and probabilistic threat metric are added in the form of a simplified bird’s eye view (SBEV), which is the input to the network. In the development of the proposed algorithm, pre-crash simulation data and experimental data have been employed. A comparative study shows that the proposed algorithm outperforms the model-based algorithm on simulation data containing safety-critical scenarios. Furthermore, it outperforms the data-driven algorithm on experimental data.