Unbelief can be taken in two ways: simply a lack of faith through ignore and opposition to the truth presented to you.

Having unbelief due to ignorance of divine things is not a sin, but it is a punishment on account of being in Adam, for ignorance of divine things is one of the results of Original Sin.

We know this kind of unbelief through ignorance is not a sin for the Lord Jesus said: "if I had not come and spoken to them they would not have sin." Therefore if your unbelief is due to ignorance of divine things then it does not have sin (think an isolated tribe that has never yet heard the Gospel). If these people are damned it will not be due to their unbelief, but other sins.

Unbelief that arises from opposing the divine things presented to you, or refusing to hear it, and like things, is a sin. This is known from the above quote of the Lord: He came and spoke to them, and they refused to believe Him, so they now had sin. Another saying of the Lord Jesus confirms this, speaking of the preaching of the truth He sent His Apostles to do:

"Whoever does not welcome you or listen to your words, shake off the dust from your feet as you are going out of that house or that town. Truly I say to you, it will be more bearable for the region of Sodom and Gomorrah on the day of judgment than for that town!"

So it is known that they have sin and will be condemned. If these people are damned it will be due to their unbelief (and other sins if they have it). Remembering also, "whoever is not for me is against me," that the Lord said.