Seems to me he wears white gloves when he's serving formal dinners (Brinkley Court) or when Bertie has luncheon or dinner guests at the flat. This is TV canon, but in every uppercrust movie I've seen, the people waiting on tables wear white gloves. You don't see Bertie eating much at home in the TV series (breakfast in bed), and I haven't seen specific references to gloves in the canon stories. But I will admit there are plenty of people better-versed in book canon than I am.