Over the past decade, extensive research has been conducted on employing deep learning techniques to detect visual defects in structural facades during inspection. Although these models have shown accuracy in identifying defects from visual data, they encounter limitations in practical applications. This includes uncertainty with data that fall outside the trained distribution and their lack of explanation of detection results. In addition, owing to their extensive parameters, these models require substantial computational resources, which is impractical for visual inspection. These limitations impede immediate defect checking and misjudgment of the deep learning model. The study aims to address these challenges by optimizing a deep-learning-based defect recognition model using a selective layer attention network (SAN). This utilizes a selective feature extraction method to provide essential visual defect information through feature maps within a deep learning model. SAN can effectively represent defect information from building surface images across each layer using the gradient-weighted class activation-mapping visualization technique. These findings demonstrate that the SAN-based model offers clear visual information while significantly reducing the usage of computational resources by 90% compared with the original network, maintaining an equivalent performance level.