I had a similar experience with an extremely conservative co-worker who was on a rant about "illegals." I asked him how he would feel about his son mowing lawns and his daughter cleaning toilets. He got all indignant that I would think his children would do demeaning work.

They don't see people of color who perform menial work. They only see POC if they are working alongside them and competing for raises and promotions. Then POC are visible and a problem.