Synesthesia is a brain condition that creates just this sort of sensory crossover. Some synthesthetes “smell” colors or “taste” words. For these people, sounds can be cold or damp or spiked, and scents may have distinct textures.
It's a condition that runs in families, and might affect as many as one in every two dozen people. The crossover is not universal (not everyone tastes green as mint, but there is some overlap from person to person...) Oh -- and the condition sometimes visits the brain with help from traumatic brain injury, seizure disorder, psychedelic drugs.