I am actually half way through book 1 right now, and I don't think that particular part is all that important, since only a few people actually treat him differently for his appearance, and it's a very passing remark that takes up about a paragraph. What I do think is important is that it's a major plot point that it's unheard of for a person from further away than a town or two over to come through the town, which makes it a huge deal when Moiraine and Lan arrive, and it then becomes an even bigger point of emphasis when Moiraine reveals the history of the town's founding by Aemon after the trolloc attack. How could this town have been home to the same people of the same bloodline for thousands of years and there are still completely distinct races? That would be gone in like two generations of intermarriage.