I always headcanon black and brown people into the different races. i mean, gondor had colonies and settlements in umbar and the coasts of harad for a while, that’s why the corsairs were willing to attack gondor. it only makes sense that gondor, especially, would have people of color.