I think a lot of these "YTA"s are very hypocritical. Regardless if this is a real story or not, I think you were being honest and making a harmless joke.
The baby is a few weeks old, it doesn't understand nor will remember what you said. As other redditors have mentioned, most babies ARE ugly. So what's the problem? People seem to be a snowflake and can't handle the fact that not everyone likes looking at a wrinkly bald 10 pound sack of human flesh. When is it okay to say someone's child is ugly or not? In a few months? A few years?
Your friend is also not TA, it's 100% normal for her to be protective over her baby and getting offended. She felt uncomfortable with your comment and they asked you to leave. You complied. Maybe you should apologize for saying something that might be perceived as "rude" to the parents of the child, but you are not an "asshole".