Hmmm.

I don't think I would "rifle" through drawers. Rifling suggests I am searching for something - in comparison to "this thing is on display and I see it and look at it". Also, drawers have a sense of being closed. So, I don't think I, personally, would consider them part of the unspoken "you have been given access to this" contract. Drawers would require an additional "ask for permission".

If someone invited me into their room and their diary was out - I wouldn't pick it up and read it if it was closed. If it was open at a specific page, I would read that specific page, yes. I wouldn't hide that I was doing it, because I'm not sneaking - I'm reading something clearly on display. I would assume that the diary page fell into the realm of "you can look". After all, they invited me into that room, and they left a diary page open. But I wouldn't turn to other pages and read them without asking.

So, I suppose my line is "do I have to do something extra to see the thing?" - e.g. open a drawer, a cupboard, a page, an envelope? If I do, then it requires an extra ask. If I don't, and the item is in an area I've been given access to - then I assume it is OK. I suppose that's my line - sounds like other people have other lines of what is acceptable.