Physical frailty may weaken structural brain integrity (SBI), which may lead to cognitive impairments. In this cross-sectional study, the number of final study participants was 2057 (700 men, 1357 women). The proportion of the frail men was 5.3% (n = 37), and the proportion of the frail women was 6.6% (n = 90). We used the Fried Frailty Phenotypes (FFP) to assess the physical frailty associated with SBI regarding cognitive impairments by sex. We performed multivariate linear and logistic regression to assess the associations between the FFP and SBI markers and probable mild cognitive impairment. The false discovery rate test using the Benjamini-Hochberg procedure was applied to fully adjusted models. Frail women had large white matter hyperintensities, low fractional anisotropy, and high mean diffusivity (β = 0.30, 95% Confidence Interval: 0.09 to 0.50; β = -0.28, 95% CI: -0.46 to -0.11; β = 0.31, 95% CI: 0.16 to 0.47). Frail men had a smaller thalamus and hippocampus (β = -0.36, 95% CI: -0.60 to -0.12; β = -0.47, 95% CI: -0.78 to -0.17). Compared with the opposite sex, white matter integrity was low in frail women, whereas gray matter integrity was low in frail men.