I think the 'Royale with cheese' scene in Pulp Fiction popularized the kind of casual, meandering dialogue that became typical for a lot of Hollywood movies during (and after?) the nineties.

I think it could be partially considered as an aesthetic innovation or fad, where someone came up with a new form of poetic artifice and, owing to its efficacy, it caught on. It could also be considered as an artifact of the shifting values and attitudes of our culture. In the past, stoicism in males was more valued. Now, it is exuberance and emotional expressivity which are prized. Characters speaking in a more relaxed and less artificial (but also less impressive) manner could reflect this.

A similar disjunction is evident in action heroes then versus now, I think. Compare Clint Eastwood and Sean Connery with Bruce Willis and Daniel Craig, for instance. The former tend to be laconic, stern, collected, etc., while the latter are emotional, extravagant, even histrionic at times. Often it is the case where a modern action hero experiences an emotional breakdown over the course of a movie. This seems to fit into the same theme as dialogue becoming more casual.