This study aims to compare the potential environmental impact of the manufacture and production of recycled and by-product aggregates based on a life cycle assessment and to evaluate the environmental impact and cost when they are used as aggregates in concrete. To this end, the six potential environmental impacts (i.e., abiotic depletion potential, global warming potential, ozone-layer depletion potential, acidification potential, photochemical ozone creation potential, and eutrophication potential) of the manufacture and production of natural sand, natural gravel, recycled aggregate, slag aggregate, bottom ash aggregate, and waste glass aggregate were compared using information from life cycle inventory databases. Additionally, the environmental impacts and cost were evaluated when these aggregates were used to replace 30% of the fine and coarse aggregates in concrete with a design strength of 24 MPa. The environmental impact of concrete that incorporated slag aggregate as the fine aggregates or bottom ash aggregate as the coarse aggregates were lower than that of concrete that incorporated natural aggregate. However, concrete that incorporated bottom ash aggregate as the fine aggregates demonstrated relatively high environmental impacts. Based on these environmental impacts, the environmental cost was found to range from 5.88 to 8.79 USD/m3.