That is actually a very good question, basically it is because there is experimental evidence that gravity obeys the weak equivalence principle. Loosely speaking that means that we can do a geometrical transformation that locally "removes" gravity for ALL objects. For other forces each object reacts differently to them, so there isn't for example a boost that can set the electromagnetic force locally to zero for EVERY object.
Having said that, all the other forces can be modeled using the curvature of a manifold, but it is NOT the Minkowski (meaning space-time). Electromagnetism can be thought in terms of curvature of a U(1) Lie group manifold over the Minkwoski space time.