My problem, every time I entertain the idea of Djokovic being the GOAT, is the age / era overlap.

Federer has spent almost all of Djokovic, Nadal and Murray's peak years as a 30+ year old with kids, titles and plaudits. Djokovic, a few months younger than Federer was (29.5 from Jan 2011) during Djokovic's 2011 break-out year, went on a 2 year career lull unlike anything Federer had at that age (although at 32 he had those problems with his back for 6 months). Murray was crocked and Nadal from 28.5 - 30.5 years old had the following results at slams:

Absent (A), QF, QF, 2R, 2R, 1R, 3R, A, 4R

I think there's also legacy, longevity records (oldest world number 1), playing style, consistency etc to consider.

At any rate, it'll be interesting to see how the next few years play out.