Background and Objectives The course and pattern of cognitive decline in ischemic cerebral small vessel disease remain poorly characterized. We analyzed the trajectory pattern of cognitive decline from age 25 to 75 years in cerebral autosomal dominant arteriopathy with subcortical infarcts and leukoencephalopathy (CADASIL). Methods We applied latent process mixed models to data obtained from patients with CADASIL who were repeatedly scored during their follow-up using 16 selected clinical scales or cognitive tests. Results The modeled evolutions of these scores obtained from 1,243 observations in 265 patients recruited at the French National Referral Centre (50.1 years on average and 45.3% men) showed wide and heterogeneous variations in amplitude along the age-related progression of the disease. Although the Backward Digit Span remained essentially stable, a linear deterioration of scores obtained using the Symbol Digit Numbers or Number of Errors of Trail Making Test B was detected from 25 to 75 years. By contrast, the largest score changes were observed at midlife using the Digit Cancellation Task. All other tests related to executive functions, memory performances, or global cognitive efficiency showed a rate of change accelerating especially at the advanced stage of the disease. Male gender and the presence of gait disorders or of some disability at baseline were found to predict earlier or large changes of 4 scores (Index of Sensitivity to Cueing, Delayed Total Recall, Initiation/Perseveration, and Barthel Index) in a subgroup of individuals distinct from the rest of the sample. Discussion Cognitive alterations develop heterogeneously during the progression of CADASIL and vary largely according to the stage of the disease. These results suggest that not only the target population and study duration but also the stage of disease progression should be considered in preparing future clinical trials aimed at reducing cognitive decline in any such condition.