The tumor microenvironment (TME) is associated with tumor prognosis, immunotherapy response, and prognosis in patients. Here, we hypothesized that the entire TME in pathology image is associated with the survival time prediction. To address this hypothesis, we utilize the entire TME on pathology image of early-stage lung adenocarcinoma (esLUAD), which is the most common histological subtype of lung cancer. Notably, we investigated whether machine learning models can predict individual survival time from pathology images without region-level annotation and solely based on patient-level survival data. In particular, we proposed a pathology image-based predictive model in a cascaded learning system to predict the individual survival time of esLUAD patients in two independent cohorts (National Lung Screening Trial (NLST) and Cancer Genome Atlas Program (TCGA)). Besides that, we estimate a mean absolute error (MAE) score and a C-index score that are strongly associated with the survival time prediction. Our method achieved (361.90 MAE - 0.70 C-index) and (365.67 MAE - 0.58 C-index) in early-stage NLST and early-stage TCGA cohorts, respectively. Together, the presented results highlight the importance of computation pathology algorithms in predicting survival time using the entire TME information in pathology images and support the use of computational methods to improve the efficiency of clinical trial studies.