any handling offense like this is stopping a promising attack (yellow card)

It is not. A GK cannot receive a card for a handling offence in their PA (except if it's a double touch form a restart).

The goalkeeper has the same restrictions on handling the ball as any other player outside the penalty area. If the goalkeeper handles the ball inside their penalty area when not permitted to do so, an indirect free kick is awarded but there is no disciplinary sanction. However, if the offence is playing the ball a second time (with or without the hand/arm) after a restart before it touches another player, the goalkeeper must be sanctioned if the offence stops a promising attack or denies an opponent or the opposing team a goal or an obvious goal-scoring opportunity.