It is not an undeniably racist term because it was created by native tribes as a way to identify different groups of people, to include the term "whiteskin". The Smithsonian archives have letters and other correspondence between tribal leaders and colonial leaders where those terms were used by both parties because it was common usage at that time.

It was never a pejorative term at its inception and some sources indicate that natives even used the term "red" in the pre-Colombian era. In the 17th century it was common usage between natives and colonists though it became less and less common between the 18th and 19th century, where both tribes and colonial folks stop using color as a commonplace identifier.

Here's a publication from the American Historical Review that discusses the origin.