Most workers change before leaving the workplace. The only people I can see wearing them casually are really old people. But they are very few.
For example, you won't see a firefighter in full uniform strolling around.
Work clothes are just that clothes for work [emoji16] at least in France.
Just look around instagram the only people wearing les bleus casually are foreigners, French people wearing them are either at work or they're going to paint a room or doing some crafts.
I take your points entirely, and can offer only this rationale.
It's always been the case that the clothes of one country are taken up for a different purpose or reason by the citizens of another. In the UK, the 1950s saw the adoption of American jeans as style rather than workwear, the 60s Afghan coats for cold weather in Helmand worn to summer rock concerts in England, and so on. The only USAAF D-1, A-2 and B-3 jackets seen here used to be ones given away during the war, but now they are popular and made throughout the world, and I guess the number of actual aviators wearing them is tiny (I do, and I'm not).
For me, bleus de travail are the epitome of French style and quality when applied to everyday working clothes (rather than high fashion), and unfortunately that's not something found in most UK clothes of this sort. Therein lies some of the attraction, and, being foreigners, the customs and constraints that apply to clothes worn in their country of origin, don't do so here. Please consider it an "hommage".