I agree, and feel like an unfortunate splash of cold water falls on this plot bunny in the form of the fact that Jeeves, bless his heart, is hidebound, reactionary, and a part of a strict hierarchy. I feel like there's a significant likelihood that he wouldn't even consider engaging in a relationship of any sort with a black liftman.