Background: Serum CEA has been widely used to screen for potential recurrent disease after resection in rectal cancer. However, the influence of baseline CEA on the performance of CEA in recurrence surveillance needs to be investigated. Patients and Methods: This longitudinal cohort study included 484 patients with nonmetastatic rectal cancer from 18,013 patients in a prospectively enrolled institutional database program of colorectal disease. Baseline CEA levels were determined before treatment, and CEA-based follow-up tests and examinations were applied in the surveillance after treatment. Results: A total of 62.6% (62/99) overall, 53.5% (23/43) local, and 64.9% (50/77) distant recurrences were seen in patients who had similar CEA levels with their baseline statuses. The sensitivity of elevated CEA levels during surveillance for overall recurrence was significantly lower in patients with negative baseline CEA than in those with elevated baseline CEA levels (41.3% vs 69.4%; P =.007). Moreover, similar results were observed in the surveillance for local (50% vs 61.5%; P =.048) and distant (39.6% vs 72.4%; P =.005) recurrences between these 2 patient groups. However, CEA had comparable and excellent specificity during surveillance for recurrent disease in these groups. The addition of CA19-9 to the CEA assay significantly improved the sensitivity in recurrence surveillance for patients with negative baseline CEA (49.2% vs 41.3%; P =.037). Finally, we identified a subgroup of CEA-turn recurrences characterized by negative CEA at baseline, elevated CEA at recurrence, and worse survival outcomes after recurrence (hazard ratio, 1.88; 95% CI, 1.07–3.30; P =.026). Conclusions: In patients with rectal cancer with negative baseline CEA, serum CEA had insufficient sensitivity in recurrence surveillance after treatment, and additional surveillance may improve oncologic outcomes. Baseline CEA should be considered before CEA-based surveillance can be applied in the follow-up trials.