I am a fair bit above-average in my 'musicality' although I've never performed any instrument to a significant level: that is, I hear what is going-on in music, and it affects me quite powerfully.
This is a plus when it comes to the appreciation of good music, well-performed - but is, under normal circumstances, more of a curse.
For example, a powerful memory of childhood (still with power to alter my mood) is the leaden misery of attending church; mainly due to the miserable hymns, miserably performed.
Even as a young child, I only liked a few hymns, and I only liked these when they were performed well - by which I meant with sincerity and dedication...
To hear bad hymns, or to hear any kind of hymn drearily intoned at about half the proper tempo and with a further rallentando at the end of each verse, and with an indeterminate-number-of-beats gap between verses... To hear a sluggish and rasping church organ, or fluting and over-articulating boys choir...
All this for me this was not-merely not-enjoyable, but actively unpleasant: a kind of moment-by-moment torment.
Even nowadays - music is seldom 'background' for me; in the sense that when I dislike it, it makes me feel worse.
In sum, I find it all-but impossible to ignore music; and since there is a lot more bad music than good, and a lot more bad performance than good; this does considerable net-harm to my mood.
People who don't really appreciate music, people whom music does not-much affect, will find this pretentious or feeble-minded; but so far as I can tell it is hard-wired - and I am stuck with it.
And, quite likely, it is an impulse towards my greater than usual desire for solitude: by avoiding other people, I avoid their music and the almost-inevitable depressive effect that has on me.