This may be the most important reason. If you’re writing for an audience, you’ll quickly figure out what topics appeal to your readers. Write more about those. You’ll also find out which topics your readers couldn’t care less about. Write about those less.
This makes sense if you’re a content writer. But if not, then reader feedback just increases the time it takes to figure out who we are as a writer. A vegetarian has to lose her voice to speak at a beef-lovers convention. You know?
That said — ditching the fear is the biggest and most important thing and you nailed it. So many of us have that fear and diminishing it is the best favor we can do ourselves.