Fact: People were once locked in asylums for this.

About 3 years ago I read an article in Smithsonian Magazine about how some people associate colors with things like numbers, sounds, or physical pain. A composer named Christopher Rouse has composed music based on the colors he sees in his mind when he hears certain melodies and chords. One woman thinks of the color orange when something hurts her.
I associate colors with numbers, letters, and sometimes words and the things those words denote. Up until I read this article, I had no idea that not everybody did this. I'd assumed it was part of everyone's experience of the world, and took it for granted to such an extent that the only conversation I remember having about it was with my mom, wherein we disagreed about which colors went with which items. (I say 7 is yellow, she says it's purple. I don't know what's wrong with her.)
For those of you who are wondering what on earth I'm talking about, the best way I can describe it is to say it's like Sesame Street. (If you haven't seen Sesame St., then I can't help you.) When they introduce a letter or number, they put it on the screen, and it's always a color. "S!" they'll say, and there will be a big yellow S on the screen. It's like I have a screen in my head, and when I think of S, it's always yellow. I would be frustrated if somebody go me something pretty with my initials on it, and the colors were "wrong." H is light blue, K is light pink, and S is yellow. Anything else is incongruous to me.
Does anybody else out there do this?