Well in that case maybe don't sign a lease that clearly states the property can be visited by the owner/agent, under reasonable notice? I'm sure it's possible/negotiable, if that's the level of privacy your require. But if you do willingly sign such lease agreement, then you have to comply.

I'm 99.99% sure OP's lease contains such clause, especially being from a real estate agency. In the current case it seems the agency has made a very short notice request to OP due to the circumstances (landlord going overseas). I don't see why people think it's such a problem for them to ask OP? There might be scenarios where OP is fine to make the short notice work (why wouldn't he, if he's at home anyway, unless you've got something to hide?), and everyone's happy.

OP feels like he can't say no anyway because he will be evicted

But that's not true a all. OP says he feels if he says "no" the landlord might start an eviction process. This doesn't seem reasonable at all - unless there are some past issues with the landlord that OP forgot to mention. For all we know OP could be in a different state or abroad on a holiday, it's completely fine to say no if that's the case, or if he simply doesn't feel comfortable.