In this paper, a deep phase retrieval algorithm for speech signals based on the dual Alternating Direction Method of Multipliers (ADMM) incorporating a deep prior network that exploits the sparsity of speech signals is presented. The proposed network, named DADMM-net, unfolds the dual ADMM for the -regularized non-convex optimization problem of phase retrieval with several two-dimensional convolutional neural networks (2D-CNNs). In order to efficiently optimize the deep unfolding network for high-dimensional parameter vectors, a novel updating scheme referred to as soft coordinate descent (soft-CD) is proposed, where dual parameter updates are determined through interpolation between the current values and the updated values coordinate-wise with respect to the weights computed by deep networks in each layer. Numerical simulations on a publicly available dataset confirm the state-of-the-art performance of the proposed method in terms of perceptual evaluation of speech quality and short-time objective intelligibility with a significantly faster convergence speed compared to existing methods.