Colonialism appeared long before the arrival of anti-colonialism. No justification was needed, in fact, it was seen as beneficial. Individual incidents of brutality were protested, often, but not the concept as a whole.
We have a hard time seeing things through the lens of an utterly different ethical framework. How would you react to someone coming to you from the far future, in a time when society accepts that some of the things we consider normal today, even beneficial, were in fact horrible social ills, and thank goodness they were stopped in the 32nd century!
But in this time, colonialism was nigh universal. It wasn't a matter of the Europeans doing it to all others, it was the more advanced and powerful doing it to the less advanced and less powerful. The civilizations of the world had conquered and reconquered territories from each other since the beginning of recorded history. Only recently have people really started to question the morality of that. Bear in mind, the two leading empires in the Americas at the time were themselves highly imperialist, as were Africa's Zulus.
