INTRODUCTION: Fatal fall from height accidents, especially on construction sites, persist, underscoring the importance of monitoring and managing worker behaviors to enhance safety. Deep learning showed the possibility of substituting the manual work of safety managers. However, applying detection results to determine compliance with safety regulations has limitations. METHOD: This study estimated the actual working height depending on the height of the object detection bounding box by specifying the consistent hinge part as a target marker based on ladder manufacturing regulations. Furthermore, an attempt was made to improve the separation between workers, coworkers, and persons unconnected to ladder activities by applying an optimized loss function alongside an attention mechanism. RESULTS: The experimental results showed that an average precision increased from 87.60% to 90.44%. The performance of the monitoring unsafe behavior of ladder worker following the Korea Occupational Safety and Health Agency (KOSHA) guide was evaluated by 91.40 F1-Score, which accumulated sorted according to the working height. CONCLUSIONS: Experimental results show the feasibility of the real-time automate safety monitoring in ladder work. PRACTICAL APPLICATIONS: By linking the estimated working height and deep learning multi-detection results to established safety regulations, the proposed method shows the potential to automatically monitoring unsafe behaviors in construction site.