The Census of 1790: 800K white males 16 and over, 800K males under 16, 1,556K white females. The photo below is of 4 of the 15 million mostly European immigrants who arrived between 1880 and 1910.

1790 persons subjected to forced migration and slavery: 700K first and later generations of Africans. 

Residents subjected to genocide and disease caused deaths: Native Americans were not counted in the Census until 1860. I cannot find an estimate of the number in the late 18th Century. California Indians declined by 90% during the 19th century, from 250,000 + in the early 19th century to about 15,000 at the end of the century, mostly due to disease.