I have three points:
You are under no obligation to play with people that make your experience worse. A lot of people seem to be defending playing with this individual, since not doing so is wrong and/or bigoted. I disagree; the activity they are playing is directly related to their handicap. You shouldn't feel guilty for not wanting to play a game that requires attention to detail and social interaction with someone who lacks attention to detail and social interaction skills.
Just as you don't know the stories of others, they don't know yours. I have played a lot of games with autistic and other handicapped indivisuals, and quite frankly it's not a good experience. I did it because I am a decent human being who is willing to help others. But you know what? I won't lie; I feel like I've paid my dues. There was a time in which I didn't have a game night where I actually enjoyed myself the whole time for months because of a few individuals. I don't see how this can simply be ignored, or how it can be "my fault" because I'm not willing to volunteer having a miserable time. Some people are Ill suited to play board games. That may be harsh but it's the truth, and when it's turning people away from the hobbyist not something we can ignore.
On international tabletop day, I brought games to my local library. The group that assembled included an autistic kid. After four hours, I cut the day short and took my games home because I couldn't take it anymore.
The reason? Two preteens were being unbearably obnoxious. The autistic person was awesome and loved every minute of it, and I loved playing with him.
So...you shouldn't immediately dismiss playing with someone who is autistic. But if it is a negative experience for you, there should be no guilt in saying no.