First, as a gender fluid person living a (somewhat) male existence, let me say that he was not the most intelligent or intuitive of men to answer the way he did. Even though you asked for an honest answer, the fact is he should have embellished a bit, or at least begged out of answering at all.
But since he did, let's go over this:
Is this on a 1-10 scale? So he considered you "average"? That's better than "unattractive" in his view, I'm sure. Perhaps he reserves 6-10 for supermodels and celebrities with personal trainers and he considers 5 to be "good for a regular person"? It's really hard to understand what the number 5 really means in his mind, so don't assign your meaning to it. Ask him to be specific about its meaning.
Now let me say that YES! Feeling attractive is vital to your own libido! This is true no matter what your gender! If you feel unattractive you won't perform well, sexually. Your sexual desire will wane. What he said was damaging to your relationship! That's why I deemed him insensitive.
I consider my fiancee a 10 to me. What she is to anybody else doesn't concern me, because to me she's the most beautiful person in the world. I've been with her for five years, and I've known her for over 20 years, and I still see her as incredibly sexy and beautiful. Even if she got really fat, or suffered a horrible accident that damaged her body or face, I'd still consider her a 10 in my book.
That's not bullshit, either. If you're head-over-heels in love, then the person you love should be the most beautiful person in the world to your eyes. Flaws and all. The "flaws" become part of their beauty, and you start to see people who don't have those same features to be less attractive.
What I'm saying is that there's something deeper at play here... A man who was deeply, madly in love with you would always see a 10. A man who is having doubts or worries-- about himself, about the relationship, about something else-- might be seeing those issues in the way he perceives you.
It's not about him understand where you're coming from. It's about you both understand what brought you to this point in your relationship? Have you been married long enough to become bored? Is there some other issue? You need to communicate. You can't let this fester, nor can you take it all upon yourself as being your problem. It's not. For him to deem you a "5" like that means he has something going on in his mind, too.
Finally, you need to ask yourself something and answer honestly, because believe or not, men have body issues, pride, and libido problems related to how they're perceived, too: How do you rate his body?
If you answer "5", then maybe he's aware and self-conscious, and his answer to you-- calling you a "5"-- was his way of hanging on to you, not letting you know that he thinks you're out of his league so he doesn't lose you. If he says you're a 10, and thinks he's a 5 or worse, then in his mind he's seeing that you're gorgeous and maybe worries that you'll think you deserve a man more attractive than he is.
Men are not emotionless lumps. They worry. They get scared. They do stupid things to hang on to the people they love.
So as I said before: You need to communicate. Talk to him. Find out what's going on here. Leaving it will only make things worse. You need to know, and you need to address it. If he balks because he's not the kind of guy who talks about his emotions, then you have to find more subtle ways of finding out what's going on in his head, and (very important!) letting him know what's going on in yours.
Good luck.