In this paper, the strategy decision algorithm for automatic evasive steering (AES) integrating a convolution neural network (CNN) with a physics-based threat assessment is proposed. Five collision avoidance or mitigation strategies, including evasive steering, lane change, and steering to shoulder stop are considered for the strategy decision. Although there are many model-based or data-driven approaches for collision avoidance in the literature, a new decision method integrating data-driven classification based on CNN with both threat assessment and prediction techniques is proposed to improve reliability as well as accuracy. First, a set of abstracted images in a bird eye’s view including the threat assessment and trajectory prediction information are generated. More specifically, a few collision indexes and interaction multiple model-unscented Kalman filter are used respectively for threat assessment and prediction. Once a stack of the images so called predicted semantic map corresponding to each collision avoidance strategy are generated, the decision classification based on CNN follows to choose an appropriate strategy for AES. Finally, the proposed decision algorithm is trained and validated through typical safe scenario data coming from field operation tests (FOT) and safety-critical scenario data via simulations.