Building maintenance plays an increasingly important role as buildings age. During maintenance, it is necessary to analyze building defects and record their locations when performing exterior inspections. Hence, this study proposes an automatic three-dimensional (3D) modeling method based on image analysis using unmanned aerial vehicle (UAV) flights and convolutional neural networks. A geographic information system is used to acquire geographic coordinate points (GCPs) for the geometry of the building, and a UAV is flown to collect the GCPs and images, which provide location information on the building elements and defects. Comparisons revealed that the generated 3D models were similar to the actual buildings. Next, the recorded locations of the building defects and the actual locations were examined, and the results confirmed that the defects were generated correctly. Our findings indicated that the proposed method can improve building maintenance. However, it has several limitations, which provide directions for future research.