In all fairness, it sounds like you were unhappy for a very long time and did not say a word to him about it before coming home one day and suddenly exploding on him. Many guys are not mind readers - they do not know if there is an issue unless you say so. Considering your sudden explosion and introduction of the topic of divorce, I do not think it is unreasonable to ask if you cheated. It is a very valid concern and pretty straight forward question (we have all heard or read stories like this that involve cheating).

He also did not immediately jump to "did you cheat?" - as much as it is being presented that way, it seems apparent that there was more back and forth conversation between you two about your unhappiness before he asked if you were considering divorce. You likely already had his attention when you told him before that you had no desire to spend your life with him.

Now.. instead of simply saying no to the cheating question, your immediate response was to be insulted, in disbelief, etc. I imagine you had several words to say and probably spoke at length about your modicum of integrity. Not only that, you could not let it go and several days later brought it up again while acting what some would consider to be overly offended/defensive about the topic. I will lean towards believing you when you say you did not cheat, but in another story your behavior sure sounds like that of that person who we later find out had in fact cheated...

All this to say that I would probably have asked you if you cheated as well. I'd also probably interpret your response and being a bit extra and suspicious ("The lady doth protest too much, methinks").

On a final note, he has only asked you once in 4 years if you cheated and never came to you with any accusations of cheating before (or since). I'm not sure how this is so terrible...