NTA. You were defending your girlfriend from assault. It had nothing to do with his disability in the same way it wouldn’t have anything to do with his race if you were white and he black/Asian/Hispanic. You tried everything you could before you took that step, including leaving the area. You are not trained in safely restraining people (I assume), so you just did the best you could to stop the situation.

I am kind of concerned for the man though- I know that many people with Downs are able to live independently, but it sounds like this person may need some extra support (maybe there’s another learning disability at play here too?).

(Also, I’m reading your post as “I wouldn’t have questioned what I did had the man not had a learning disability” and I do hope that I’m right, and you would still have taken the same steps to avoid having to hit someone even if that person didn’t have a learning disability...)