Radio-frequency identification (RFID) system, successfully adopted in many industrial applications, suffers from security issues due to the inherent weakness of wireless communication, such as eavesdropping, replay attack, impersonation attack, and traceability issues. A lot of research efforts based on cryptographic primitives have been conducted in a decade, however, most of the existing security protocols depending on cryptosystems are not feasible to be applied due to the minimalist design of passive tags. A lightweight cryptographic authentication is one of the practical solutions, but it has traceability issues from physical layer information. The other approach is to use the properties of the physical layer of RFID systems. However, since the physical-layer characteristics cannot be intentionally updated, they are vulnerable to situations where an adversary actively obtains authentic data for traceability attacks or replay attacks. Therefore, to resist the security threats, we propose Tagora, a cross-layer authentication protocol, which is the first integration work of two-layer approaches that harness the unpredictable properties of tag's collision responses at both the physical and application layers. Our protocol design is composed of a collision recovery algorithm with a random offset scheme and phase encryption in the physical layer, and authentication process based on a challenge-response mechanism in the application layer. We evaluate Tagora in terms of the untraceability and reliability, and also provide security analysis on how Tagora can defend against plausible attacks while meeting security requirements.