Estimating contact force for a robot manipulator without an force/torque (F/T) sensor poses challenges due to uncertain torques, such as backlash and flexibility. To address this limitation, this article proposes a data-driven uncertain torque model and an overall gray-box structured approach. The contributions of this article are threefold. First, a joint domain unified neural networks (DUNNs)-based model is proposed to compensate for the uncertain torques. This model effectively captures uncertain torques beyond studies focusing solely on individual uncertainty. Second, the DUNNs model receives dynamic and joint domain information, enabling a single DUNNs model to estimate all joint uncertain torques through joint domain knowledge. This approach reduces the model size while maintaining performance. Third, the structure in which the DUNNs model works with conventional static friction models is introduced. This structure improves contact force estimation performance and enhances robustness against untrained data compared with the black-box model. Experimental results verify the method's effectiveness.