Nick i agree. a pair of baggy trousers over plus fours would look ridiculously bulky.
here's my theory based on no evidence whatsoever; the trousers in the previous photo were maybe the grey jogging bottoms (U.S. sweat pants) of their day. it does mention that those chaps were between races. maybe it was originally a comfy trouser to pull on over athletic / rowing shorts while waiting about ? that would also explain the 'cozy' fleece fabric. (Two Types, while i agree it is some sort of cheap wool i don't think that amount of even all-over wear would be feasible).
people saw them and reproduced the look in smarter fabrics; the rest is history. that's my version anyway.