Roots has some very sensory friendly sweatpants and also shorts that are almost the same as the sweatpants but in short form, although they definitely aren't dressy.
I've bought some really nice clothing from American Eagle, as well, although more so shirts and things than shorts and pants. I do have some nice jeans and jean shorts from them that are pretty much the only jeans or jean shorts I can wear for longer than three seconds. I just had to try on a lot of the non sensory friendly pants and shorts before I found them.
I believe I have also had some sensory friendly clothes from the Gap, although I'm trying to remember as it has been a while. That may be another store to check out, though. I hope this helps!