Look, religious people can get offended by literally everything, so even if you find a way to decline that you deem respectful, there will always be people to tell you you're disrespectful. At this point it's just up to you to decide if you prefer to be true to yourself, even if it might spark controversy, or if you prefer to avoid any tension, in which case the safest option is to accept to pray. You can't always have everything you want, sometimes you have to make some decisions even if both options have downsides.

If it can be of any help, personally if I am forced to attend a religious event, I just don't pray, don't kneel down, don't sing religious songs. I'm quite sure that, when Christians visist a Mosque or a Shinto temple, they don't start praying to the Islamic or Shinto deities, as it would surely feel like betraying their own faiths. I don't see why the same would suddenly be unacceptable for agnostics.