Most people on Reddit would have hated Polk, if you think Trump is racist you're going to have a hard time with pretty much every president pre-civil rights (including Lincoln). Polk himself was a lifelong slave owner and was supportive of the concept of slavery. Lincoln believed that coexistence was impossible between blacks and whites, and he advocated a post slavery resettlement of the black population (away from the United States). This was an ideal he shared with Thomas Jefferson. To quote Thomas Jefferson:

It will probably be asked, Why not retain and incorporate the blacks into the state, and thus save the expence of supplying, by importation of white settlers, the vacancies they will leave? Deep rooted prejudices entertained by the whites; ten thousand recollections, by the blacks, of the injuries they have sustained; new provocations; the real distinctions which nature has made; and many other circumstances, will divide us into parties, and produce convulsions which will probably never end but in the extermination of the one or the other race. -- To these objections, which are political, may be added others, which are physical and moral. The first difference which strikes us is that of colour. Whether the black of the negro resides in the reticular membrane between the skin and scarf-skin, or in the scarf-skin itself; whether it proceeds from the colour of the blood, the colour of the bile, or from that of some other secretion, the difference is fixed in nature, and is as real as if its seat and cause were better known to us. And is this difference of no importance? Is it not the foundation of a greater or less share of beauty in the two races? Are not the fine mixtures of red and white, the expressions of every passion by greater or less suffusions of colour in the one, preferable to that eternal monotony, which reigns in the countenances, that immoveable veil of black which covers all the emotions of the other race? Add to these, flowing hair, a more elegant symmetry of form, their own judgment in favour of the whites, declared by their preference of them, as uniformly as is the preference of the Oranootan for the black women over those of his own species. The circumstance of superior beauty, is thought worthy attention in the propagation of our horses, dogs, and other domestic animals; why not in that of man?