A passive keyless entry and start (PKES) system is an electronic lock for an automobile that provides the great convenience of opening the door when the user is in proximity. However, the system suffers from relay attacks. Recent studies revealed that relayed signals result in valid packets that are sufficient to unlock doors. In particular, the adversary causes proximity errors by injecting a certain time delay before relaying to manipulate the phase rotation in the response signal. To this end, we present a novel relay-resilient proximity detection solution, <i>BackProx</i>, which uses pseudo-random frequency hopping with the assistance of a reference backscattering device. Since the relay adversary transmits the relayed signals from the key fob at long distances, the signals should propagate over longer distances, resulting in inevitable significant phase rotation with different frequencies. Inspired by this finding, <i>BackProx</i> uses an additional backscattering device to ensure the proximity of the key fob using the invariant characteristics of radio frequency signals in the physical layer (i.e., phase rotation). Our evaluation demonstrates the effectiveness of <i>BackProx</i> in resisting three types of relay attacks. The results show that it achieved a 98% true positive rate at close range and a 0.3% false positive rate at long range.