It stops the brick from "breathing", so any cracks in the paint allow water to get into the brick, but not evaporate out as easily. Not only do you start to get moisture problems in the walls, but during winter, the freeze-thaw cycles cause the moisture to form ice crystals inside the brick which splits the brick and often causes the face of the brick to spall off. And it's pretty much irreversible, as it us very difficult to get all of the paint out of the pores of the brick. If you pressure wash it to remove it you usually also end up removing the hard "crust" face of the brick that gives it its durability. There are some products you paint on and then peel off along with the paint, but I haven't seen one that really gets all the paint out of the pores.

Also, it is brick. Why change the color of what the architect intended to be brick? Brick is nice. If you don't like brick, don't buy a brick house or building.