I would first quote a 14th century Orthodox bishop of Thessalonica, Nilus Cabasilas, in saying,

As long as the pope observes due order and remains in the truth, he preserves the first place which belongs to him by right; he is the head of the Church and supreme pontiff, the successor of Peter and of all the apostles, all must obey him and treat him with complete respect.

So, the issue isn't one of whether the Pope is the primus, so much as it is whether he is observing due order, and remaining in the truth. To quote another Orthodox bishop, Nicetas of Nicomedia, from the 12th century,

My dearest brother, we do not deny to the Roman Church the primacy amongst the five sister patriarchates; and we recognize her right to the most honorable seat at an ecumenical council. But she has separated herself from us by her own deeds when through pride she assumed a monarchy which does not belong to her office. . . . How shall we accept from her decrees that have been issued without consulting us and even without our knowledge? If the Roman pontiff, seated on the lofty throne of his glory, wishes to thunder at us and, so t speak, hurl his mandates at us from on high, and if wishes to judge us and even to rule us and our churches, not by taking counsel with us but as his own arbitrary pleasure, what kind of brotherhood, or even what kind of parenthood can this be? We should be the slaves, not the sons, of such a church, and the Roman see would not be the pious mother of sons but a hard and imperious mistress of slaves.

These two quotes, I think, outline the perspective we should address this question from. It, to me, is a matter of administration and of the good order of the Church, much more than it is one of heresy.

(I found these quotes in Metropolitan Kallistos' essay Primacy, Collegiality, and the People of God from the anthology Primacy in the Church: The Office of Primate and the Authority of Councils edited by John Chryssavgis. It contains many excellent essays from Orthodox, Catholic, and Anglican authors, if you are interested.)