We present the submission of the AIP team to the WMT 2025 Unified MT Evaluation Shared Task, focusing on the span-level error detection subtask.Our system emphasizes responseformat design to better harness the capabilities of OpenAI's o3, the state-of-the-art reasoning LLM.To this end, we introduce Tagged Span Annotation (TSA), an annotation scheme designed to more accurately extract span-level information from the LLM.On our refined version of WMT24 ESA dataset, our referencefree method achieves an F1 score of approximately 27 for character-level label prediction, outperforming the reference-based XCOMET-XXL at approximately 17.

Jaehyeok Lee, Keisuke Sakaguchi, JinYeong Bak. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

INTRODUCTION: Fatal fall from height accidents, especially on construction sites, persist, underscoring the importance of monitoring and managing worker behaviors to enhance safety. Deep learning showed the possibility of substituting the manual work of safety managers. However, applying detection results to determine compliance with safety regulations has limitations. METHOD: This study estimated the actual working height depending on the height of the object detection bounding box by specifying the consistent hinge part as a target marker based on ladder manufacturing regulations. Furthermore, an attempt was made to improve the separation between workers, coworkers, and persons unconnected to ladder activities by applying an optimized loss function alongside an attention mechanism. RESULTS: The experimental results showed that an average precision increased from 87.60% to 90.44%. The performance of the monitoring unsafe behavior of ladder worker following the Korea Occupational Safety and Health Agency (KOSHA) guide was evaluated by 91.40 F1-Score, which accumulated sorted according to the working height. CONCLUSIONS: Experimental results show the feasibility of the real-time automate safety monitoring in ladder work. PRACTICAL APPLICATIONS: By linking the estimated working height and deep learning multi-detection results to established safety regulations, the proposed method shows the potential to automatically monitoring unsafe behaviors in construction site.

BACKGROUND Harmful suicide content on the internet poses significant risks, as it can induce suicidal thoughts and behaviors, particularly among vulnerable populations. Despite global efforts, existing moderation approaches remain insufficient, especially in high-risk regions like South Korea, which has the highest suicide rate among OECD countries. Previous research has primarily focused on assessing the suicide risk of individuals rather than the harmfulness of content itself, highlighting a gap in automated detection systems for harmful suicide content. OBJECTIVE In this study, we aimed to develop an AI-driven system for classifying online suicide-related content into five levels: illegal, harmful, potentially harmful, harmless, and non-suicide-related. Additionally, the researchers construct a multi-modal bench- mark dataset with expert annotations to improve content moderation and assist AI models in detecting and regulating harmful content more effectively. METHODS We collected 43,244 user-generated posts from various online sources, including social media, Q&A platforms, and online communities. To reduce the workload on human annotators, GPT-4 was used for pre-annotation, filtering and categorizing content before manual review by medical professionals. A task description document ensured consistency in classification. Ultimately, a benchmark dataset of 452 manually labeled entries was developed, including both Korean and English versions, to support AI-based moderation. The study also evaluated zero-shot and few-shot learning to determine the best AI approach for detecting harmful content. RESULTS The multi-modal benchmark dataset showed that GPT-4 achieved the highest F1 scores (66.46 for illegal and 77.09 for harmful content detection). Image descriptions improved classification accuracy, while directly using raw images slightly decreased performance. Few-shot learning significantly enhanced detection, demonstrating that small but high-quality datasets could improve AI-driven moderation. However, translation challenges were observed, particularly in suicide-related slang and abbreviations, which were sometimes inaccurately conveyed in the English benchmark. CONCLUSIONS This study provides a high-quality benchmark for AI-based suicide content detection, proving that LLMs can effectively assist in content moderation while reducing the burden on human moderators. Future work will focus on enhancing real-time detection and improving the handling of subtle or disguised harmful content.

The application scope of Large Language Models (LLMs) continues to expand, leading to increasing interest in personalized LLMs that align with human values. However, aligning these models with individual values raises significant safety concerns, as certain values may correlate with harmful information. In this paper, we identify specific safety risks associated with value-aligned LLMs and investigate the psychological principles behind these challenges. Our findings reveal two key insights. (1) Value-aligned LLMs are more prone to harmful behavior compared to non-fine-tuned models and exhibit slightly higher risks in traditional safety evaluations than other fine-tuned models. (2) These safety issues arise because value-aligned LLMs genuinely generate text according to the aligned values, which can amplify harmful outcomes. Using a dataset with detailed safety categories, we find significant correlations between value alignment and safety risks, supported by psychological hypotheses. This study offers insights into the "black box" of value alignment and proposes in-context alignment methods to enhance the safety of value-aligned LLMs.

Background: Harmful suicide content on the internet poses significant risks, as it can induce suicidal thoughts and behaviors, particularly among vulnerable populations. Despite global efforts, existing moderation approaches remain insufficient, especially in high-risk regions such as South Korea, which has the highest suicide rate among Organisation for Economic Co-operation and Development countries. Previous research has primarily focused on assessing the suicide risk of the authors who wrote the content rather than the harmfulness of content itself which potentially leads the readers to self-harm or suicide, highlighting a critical gap in current approaches. Our study addresses this gap by shifting the focus from assessing the suicide risk of content authors to evaluating the harmfulness of the content itself and its potential to induce suicide risk among readers. Objective: This study aimed to develop an artificial intelligence (AI)-driven system for classifying online suicide-related content into 5 levels: illegal, harmful, potentially harmful, harmless, and non-suicide-related. In addition, the researchers construct a multimodal benchmark dataset with expert annotations to improve content moderation and assist AI models in detecting and regulating harmful content more effectively. Methods: We collected 43,244 user-generated posts from various online sources, including social media, question and answer (Q&A) platforms, and online communities. To reduce the workload on human annotators, GPT-4 was used for preannotation, filtering, and categorizing content before manual review by medical professionals. A task description document ensured consistency in classification. Ultimately, a benchmark dataset of 452 manually labeled entries was developed, including both Korean and English versions, to support AI-based moderation. The study also evaluated zero-shot and few-shot learning to determine the best AI approach for detecting harmful content. Results: The multimodal benchmark dataset showed that GPT-4 achieved the highest F1-scores (66.46 for illegal and 77.09 for harmful content detection). Image descriptions improved classification accuracy, while directly using raw images slightly decreased performance. Few-shot learning significantly enhanced detection, demonstrating that small but high-quality datasets could improve AI-driven moderation. However, translation challenges were observed, particularly in suicide-related slang and abbreviations, which were sometimes inaccurately conveyed in the English benchmark. Conclusions: This study provides a high-quality benchmark for AI-based suicide content detection, proving that large language models can effectively assist in content moderation while reducing the burden on human moderators. Future work will focus on enhancing real-time detection and improving the handling of subtle or disguised harmful content.

We present the submission of the AIP team to the WMT 2025 Unified MT Evaluation Shared Task, focusing on the span-level error detection subtask.Our system emphasizes responseformat design to better harness the capabilities of OpenAI's o3, the state-of-the-art reasoning LLM.To this end, we introduce Tagged Span Annotation (TSA), an annotation scheme designed to more accurately extract span-level information from the LLM.On our refined version of WMT24 ESA dataset, our referencefree method achieves an F1 score of approximately 27 for character-level label prediction, outperforming the reference-based XCOMET-XXL at approximately 17.

Jaehyeok Lee, Keisuke Sakaguchi, JinYeong Bak. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.