I wonder if it has something to do with genetic affinity. From what I remember of biology class it is not necessarily the father's genetics that go into offspring, but those of the grandparents.

Therefore if a man has a child, and he didn't get on with his father himself, AND he also doesn't particularly like his in-laws, he pretty much has a 1 in 4 chance of getting an offspring he might actually like.

However when it comes to his grandchildren finally his own genes could find expression, so it is not unusual for grandparents to feel greater fondness for their grandchildren than their own children.

This of course assumes that his wife/partner has been faithful and not cuckolded him. Maybe somewhere lurking in a father's subconscious, he suspects children he feels no affinity for could be the result of infidelity.

Mothers may also lack an affinity with one of their children, but they tend to have a greater maternalism, and they know for absolute certainty their offspring are their own.