Under Frequency Load Shedding (UFLS) is a critical last-resort measure used to stabilize system frequency in power systems following a major loss of generation. Traditional UFLS methods are primarily based on stability criteria, which may overlook the potential unintended negative impacts on certain loads. To address this issue, we propose a novel Reinforcement Learning (RL)-based UFLS scheme that combines local frequency measurements with high-resolution load criticality functions. By leveraging data-driven models, the proposed approach learns optimal load-shedding strategies tailored to specific emergency conditions. Numerical results obtained in the IEEE 39-Bus system show that this data-augmented approach can successfully shift load shedding to resilient regions while maintaining system stability and ensuring a robust response to under-frequency events.