In the context of this conversation, Canada, and First Nations people of Canada, then yes, most everyone who isn't first nations is a colonizer.

Most black people in Canada did not come here as slaves, they moved here to escape slavery in the south, or they moved here from Africa or Jamaica. Same with the many asian people living here. Just because you're escaping slavery doesn't mean you get to partake in stealing someone elses land. It's not about "going back", it's about not moving to Canada in 1850 when first nations were being stolen from.

Finally, there were white people brought to the americas as slaves, and they are still colonizers by your definition. So it's not loopy.