You (like all of us) are paying a lot more attention to how a hat looks on you than anyone else is. If people seem to looking or staring a little bit, the odds are they just see a guy wearing a hat not the brim width unless it's really big or something. Some people are looking to try to figure out what kind of hat that is. Many people are a little surprised to see someone wearing something other than a ball cap. When I wear hats out in public, especially ones I am a little hesitant about, someone will usually say that they like my hat.