Background: Colorectal cancer (CRC) survivors generally have a higher healthcare utilization (HCU) than the general population due to cancer burden. However, it is unclear which factors are associated with this increased uptake. Our study aimed to (1) compare CRC-related and non-CRC visits to general practitioners (GPs) and medical specialists (MSs) by comorbidities, and (2) assess whether HCU differs by demographic, clinical, and psychological factors. Methods: We used data from a German population-based cohort of 1,718 survivors of stage I–III CRC diagnosed in 2003 through 2010 who provided information on HCU at 5-year follow-up. Multivariable linear regression was used to calculate least-square means of CRC-related and non-CRC HCU according to the Charlson comorbidity index and comorbidity cluster, adjusting for relevant demographic, clinical, and psychological characteristics. Results: A higher comorbidity level was associated with more CRC-related MS visits and non-CRC GP visits. In addition to being strongly associated with non-CRC GP visits, comorbidity clusters were associated with CRC-related GP and MS visits, but their association varied by specific cardiometabolic comorbidities. HCU was less dependent on prognostic factors for CRC, such as age and tumor stage, but was strongly associated with disease recurrence, depression, and emotional functioning. Conclusions: Comorbidities, rather than age or tumor stage, were related to HCU, suggesting that CRC survivors use healthcare mainly for reasons other than cancer 5 years postdiagnosis. Improved communication between primary and tertiary healthcare providers could enhance the medical care of cancer survivors with complex health needs and thereby also reduce healthcare costs.