This study proposes an automated three-dimensional (3D)-modeling method that combines convolutional neural networks (CNNs) with unmanned aerial vehicle (UAV) technology for the efficient management of thermal anomalies in building exteriors. Conventional 3D-modeling methods for thermal imaging management either require the processing of large volumes of data due to the use of thermal distribution information from entire image regions or involve increased processing time when architectural drawings are unavailable. In this study, RGB and infrared (IR) thermal images collected via UAVs were used to automatically detect windows and thermal anomalies using a CNN-based object detection model (YOLOv5). Subsequently, Global Navigation Satellite System (GNSS)-based coordinate data and image metadata were used to convert the resolution coordinates into actual spatial coordinates, which were then vectorized to automatically generate a 3D model. The resulting 3D model demonstrated high similarity to the actual building, accurately representing the locations of thermal anomalies. This method enabled faster, more objective, and more cost-effective maintenance compared to conventional methods, making it especially beneficial for efficiently managing difficult-to-access high-rise buildings.