Our hominid forebears generally survived in bands of about 25 individuals. A lot of the dynamics of why that worked for them is baked into our DNA and brainstems. Everyone "in" is good, "out" is bad. It's the root of all our divisiveness, racism, religious schism, etc.
This is it. Will they outnumber us or will we outnumber them? Teach your children well.