You're approaching this from a bias and privilege of a normal vision person. People who are color blind have adapted to vision by putting just as much importance on texture (if not more) than normal vision people.

Think about it like if you were accommodating legally blind people (like someone who could see light, but basically could not see otherwise). Your solution right now is basically to try and do different shades of light and exaggerated colors for someone who is legally blind to be able to differentiate instead of trying to accommodate them through modalities that they CAN "see" and have already adapted to which make more sense (i.e. sound and touch).

By ignoring textures/symbols, you are ignoring a major modality in how to convey and communicate information.