As colorectal cancers have a long latency period, their origins may lie early in life. Therefore childhood body mass index (BMI; kg/m2) and height may be associated with adult colorectal cancer. Using a cohort design, 257,623 children from The Copenhagen School Health Records Register born from 1930 to 1972 with measured heights and weights at ages 7 to 13 years were followed for adult colon and rectal adenocarcinomas by linkage to the Danish Cancer Registry. Hazard ratios (HRs) with 95% confidence intervals (CI) were estimated by Cox proportional hazard regressions. During follow-up, 2676 colon and 1681 rectal adenocarcinomas were diagnosed. No sex differences were observed in the associations between child BMI or height and adult colon or rectal cancers. Childhood BMI and height were positively associated with colon cancer; at age 13 years the HRs were 1.09 (95% CI 1.04–1.14) and 1.14 (95% CI 1.09–1.19) per z-score, respectively. Children who were persistently taller or heavier than average, had increased risk of colon cancer. Similarly, growing taller or gaining more weight than average was positively associated with colon cancer. No associations were observed between BMI or height and rectal cancer. Childhood BMI and height, along with above average change during childhood are significantly and positively associated with adult colon cancers, but not with rectal cancer, suggesting different etiologies.