I mean, it may be racist but that racism has actual grounding in the fact that yes, they attacked, no, they didn't come in peace. That tends to taint your thoughts in regards to a race. They attacked without provocation. It's logical that you fear further attacks, which is why you generally want them out. And killing "only" 300-400 when even a couple being killed nowadays is considered a major tragedy is absolutely underselling the point, they massacred people essentially. Fact is, they gave those racist thoughts actual grounding in reality and it's not the "you are inferior because of your skin colour, so you don't belong" but "your people actually killed our people, so you don't belong".