Dave;It's not as rare as people think.
Going back to my photography days but IIRC about 1 in 5 men have a form of colour blindness, meaning they have difficultly recognising different colours, as opposed to complete colour non-recognition.
I know it's more than 10% in the countries where people get tested for this sort of thing.
Growing up someone points out this blob of color to you and says "Red". When asked, and you see a similar blob of color you say "Red" like you're taught, even if you don't see it the same as the other person. No different than growing up knowing a different language I'd guess.
Makes me wonder and fasinate upon what the colors actually look like, as opposed to how they look to me.
Of course, this also goes for Synesthesia, which, like colorblindness can go undiagnosed for some time. You just associate something with the word your given and that's normal for you.