For me its some of the more "preppy/trad" items like boat shoes, tasseled loafers (or any loafers really), madras shorts, red trousers, etc.
They just bring up too many bad memories of preppies in high school, and the loafers remind me too much of some of the more smarmy people I've worked with in government or business. Seems like there is this unwritten rule that if you are a dishonest person in a white colar job you have to wear tasselled loafers. Note: there are great people who do wear them (I am not trying to insult Loungers who like them), but almost all the disagreable people I've worked with do wear them. This has remained true everywhere I've lived in the US, from Washington, DC, to Honolulu and points in-between.
"His modest resources meant that he could dress no more than reasonably well, but he did so with a kind of faded elegance that ignored the dictates of fashion... the overall effect was of someone frozen in time, indifferent to the new fashions of the agitated age he was living through. The truth is that he took pleasure in this, for obscure reasons that perhaps even he could not have explained."
I try and keep some semblance of casual at all times, and can't ever see myself wearing bowties, ascots, cummerbunds, or pocket squares. Business casual is as formal as I go.
Outside of certain settings (that is to say, re-enactment, photoshoots, etc) I also wouldn't wear a bowler; though for slightly different reasons to those above (as it would be rather hypocritical of me). I just think it's gone from a piece of semi-formal headwear to almost "comic national British dress". Wearing a bowler about strikes me as rather akin to a Scotsman actually wandering around in a kilt and woad, a Bavarian in lederhosen, or an Irishman in a green suit with a buckle on his hat - too stereotypical, and almost in the realms of a send up of Englishness.
formal wear holds zero interest for me; white tie, tails, top hat, wing collars etc.
Your comment made me think of when a friend and I attended the Summer Ball at university. A third friend refused to attend on the grounds that he would be expected to wear a dinner suit, which he described as "the uniform of reaction" (it was the 1980s).