We propose a phase-assisted camera-in-the-loop optimization technique for holographic displays to enhance the quality of phase representation. Phase-only hologram is one of the most popular methods for three-dimensional hologram reproduction. However, the optical noise resulting from imperfections in the optical configuration, including the light source, the relay optics, and the spatial light modulator, affects the reconstructed hologram quality. We focus on constant noise terms that arise during the phase representation process in a phase-only spatial light modulator (PSLM). To estimate these noise terms, we employ a polarization-based 4-f system to acquire the constant complex term at the PSLM surface and the Fourier spectrum. Moreover, we use a random phase at iteration to effectively isolate and extract common noise components. Based on the extracted noise, we optimize the initial hologram using the modified angular spectrum propagator, taking into account the actual noise presented in the PSLM. Experimental results validate the feasibility of the proposed method, demonstrating that a single noise estimation significantly reduces speckle and improves image quality in multi-depth holograms.