Telling someone to “go back to their country” is always racist. We can remove politics and gender and it’s still racist.
If you were sitting at a board meeting and a brown man suggested an idea, and someone said “go back to your country” — is that racist? Of course it is. Is it still racist if everyone agrees the idea was dumb? Yes. Telling someone to go back to “their” country is always racist. And if that brown man was born in the USA and you tell him to go back to his country, the racism becomes even more blatant.