when he sighs I feel like I might have actually upset him, he's just not telling me what I did wrong.

He needs to stop doing this. I don't understand people who will give obvious non-verbal signs that they're annoyed, and then deny it when you ask. They're not fooling anyone and if they really weren't annoyed then they wouldn't be giving annoyed signals.

You need to get to a place where he establishes a new pattern of always telling you, verbally and directly, when he's annoyed and why so that when he's not, you are able to trust him and break the cycle you describe. If he lies to you during the times when he is, then of course you can never know if he is or isn't.