In the "old days", men wore their fur felts for everything from chopping wood and hunting, to church and theatre, but usually had a "beater" and a nice hat. If I am at the beach, I will usually wear a "Tilley" type hat to protect from the sun, so I don't care if it gets crushed in the sand or wet.
For other casual outings, say fishing, hiking, etc, I think a fur felt is fine, even with jeans and t-shirt. On those occasions however I will wear a hat I don't mind getting wet or dirty. A Fed or Campdraft is great for that, or even a beat-up cowboy hat. I reserve my "good" hats (nice vintage hats) for work, church, eating out, etc.