Latent factor models can perform spatio-temporal forecasting (i.e., predicting future responses at unmeasured as well as measured locations) by modeling temporal dependence using latent factors and considering spatial dependence using a spatial prior on factor loadings. However, they may fail to capture complex spatio-temporal dependence because the latent factors are typically assumed to follow a classical linear time series model, such as a vector autoregressive model. In this article, we propose a deep latent factor model for spatio-temporal forecasting that can model complex spatio-temporal dependence more flexibly by leveraging the high expressive power of a deep neural network. Specifically, the latent factors are modeled using a recurrent neural network and the factor loadings are modeled using a distance-based Gaussian process. The proposed model allows the number of latent factors to be inferred from the data using a beta-Bernoulli process, which enables computationally more efficient implementation compared to previous methods. We derive a stochastic variational inference algorithm for scalable inference of the proposed model and validate the model using simulated and real data examples.