The dragon balls light up when Goten made that wish.

Yet Shenlon never appears, and we've never seen either Shenlon or Polunga simply grant a wish without being called out first. So that aspect there in itself makes no sense whatsoever.

Even if Shenlon did really grant a wish despite not being summoned, then given how vague Goten's wish was, it could have just allowed Goku to appear there in a spiritual form.

Gohan said they needed more power with Goku confirming that statement by saying that Gohan and Goten can't win alone and that Broly is his to finish off. Why would Goku make such a selfish statement if he was only there only in spirit?

Believing they were fighting alongside their father would certainly help them to dig deeper into their strength to try and force Broly's blast away, especially with him urging them to search deep within and give it their all.

We saw how crucial Goku's reassurance was when it came to Gohan's Kamehameha struggle against Cell, where thanks to Goku's words, he was able to find those reserves that he otherwise didn't feel he had.

Broly says "Kakarot" when Goku appears for the family Kamehameha.

That still only means that he was there in some sense. If it was a purely spiritual sense, visible and audible to give the illusion of being there, but otherwise unable to interact with anything, he'd still be able to potentially be seen by Broly as well.

If it was a situation of it being a clear cut, "He was there" situation, then it wouldn't make sense for the narrator to indicate that he may not have "really" been there. For his dialogue to make any sense, then it has to be a situation of regardless of what all was shown indicating that he was there, we have to accept that it's very possible that he wasn't there too.