No real consensus on this. Some folks do, some don't. Most of our vintage hats are rabbit or rabbit blends....not many are 100% beaver. Most of my hats get worn regularly....for dress and or for kicking around. I do have a few beater hats that I wear if working/sweating and a few that are purely for dress but those are new custom made. The only vintage hats I own that get babied are my 2 1950's pristine Borsalinos. The rest have to earn their keep.
One other aspect to keep in mind is that good fur felt is very forgiving.
Decent fur felt hats can be used/abused AND refurbished back to almost new. I follow a number of western custom hat makers that also do refurbs. Some will post before/after pics of them transforming a hat that was dumpster material and bringing it back to like new.....it borders on the miraculous. My thought is; wear them, enjoy them and when/if the time comes send them out for a refresh.