As the population ages, Alzheimer’s disease (AD) and Parkinson’s disease (PD) are increasingly common neurodegenerative diseases among the elderly. Human voice signals contain various characteristics, and the voice recording signals with time-series properties include key information such as pitch, tremor, and breathing cycle. Therefore, this study aims to propose an algorithm to classify normal individuals, Alzheimer’s patients, and Parkinson’s patients using these voice signal characteristics. The study subjects consist of a total of 700 individuals, who provided data by uttering 40 predetermined sentences. To extract the main characteristics of the recorded voices, a Mel–spectrogram was used, and these features were analyzed using a Convolutional Neural Network (CNN). The analysis results showed that the classification based on DenseNet exhibited the best performance. This study suggests the potential for classification of cognitive impairment through voice signal analysis.