The defense of the four-ness of Gospels comes in the context of his attack on other, rival groups, each of which use either fewer or more Gospels than Irenaeus is comfortable with.
The Montanists privilege direct experience above written evidence, and trust (new) prophecies more than scripture (for Irenaeus, this is effectively the same as having zero Gospels, although the actual Montanists might not have felt that way).
The Ebionites (Torah-observant Jewish Christians) use only Matthew.
Marcion and his followers (Gentile Christians who reject the Old Testament) use only a stripped-down version of Luke (presumably edited to exclude any reference to the OT).
An unnamed group of Docetists relies on Mark, and maintains that (human) Jesus died on the cross while (divine) Christ survived.
Valentinus and a group of gnostics cite John to prove their points, as well as the Gospel of Truth.
At the end of the passage Irenaeus makes a sort of Goldilocks argument: zero through three Gospels is too few. Five or more is too many (confusingly, it seems like he may be accusing Valentinus of using two and that being too many because one of them, the Gospel of Truth, was the wrong kind of Gospel; but never mind). His language:
These things being so, all who destroy the form of the Gospel are vain, unlearned, and also audacious; those, [I mean, ] who represent the aspects of the Gospel as being either more in number than as aforesaid, or, on the other hand, fewer. The former class [do so], that they may seem to have discovered more than is of the truth; the latter, that they may set the dispensations of God aside. For Marcion, rejecting the entire Gospel, yea rather, cutting himself off from the Gospel, boasts that he has part in the [blessings of] the Gospel. Others, again (the Montanists), that they may set at nought the gift of the Spirit, which in the latter times has been, by the good pleasure of the Father, poured out upon the human race, do not admit that aspect [of the evangelical dispensation] presented by John's Gospel, in which the Lord promised that He would send the Paraclete; but set aside at once both the Gospel and the prophetic Spirit. Wretched men indeed!
Four Gospels, for Irenaeus, is just right. So long as it's the right four, anyway.