There's quite a few scenes where Fudge wasn't obviously acccompanied by aurors in the movies too. I think it's implied that they're nearby but letting the minister have some space. Just because they're not standing there being obvious does not mean they're not present. You don't always see Secret Service members being immediately obvious when the president is around either, but I guarantee you they're nearby. You don't have to literally stand right on top of someone to protect them, particularly in a space, which presumably has decent security where they should be fairly safe anyway.

Not to mention, Harry is an auror, Ron is a former auror and Hermione is perfectly capable of defending herself. All 3 of them are heroes in the wizarding world and your Average Joe Wizard who might be disgruntled and want to hurt them would have no shot.