people struggling to find work

I think a lot of it ties into the last point you brought up. There's a lot of people here with poor attitudes, no ambition, and a lot of complaints. People constantly complain about high unemployment and "no jobs", but unemployment is actually at a low right now and many companies are desperate to hire competent qualified people. I've heard from people in my network that employers get tons of resumes in Fredericton and Moncton, but barely get any decent applications here in Saint John. Another big part of the problem is that many companies aren't willing to pay salaries that compete with other provinces so skilled workers leave. There's work here, but you have to compromise by taking a lower salary or by taking a less desirable job. That or you have to be ambitious to set yourself above the crowd. Good workers have no trouble finding work here, but they can usually find better work outside NB. That's just the tradeoff of living in a smaller city with such a low cost of living. Less people, less jobs, less pay.

negative comments about moving there from other parts of Canada

This is definitely a problem in Saint John. It's a minor pet peeve I have with locals, but it is really isolated to certain crowds and their platforms. Usually the people complaining have no context to base their opinions on and aren't doing anything to change their own situation for the better. I'm pretty young and all my friends who complain about the city are people who have never lived anywhere else, have barely traveled outside NB, and have very little ambition. They'll complain about no jobs and nothing to do, but all they do is sit at home all day watching Netflix and they never even try to apply for jobs. There is a Facebook group called News Chasers of something like that and I've heard it is dominated by that negative attitude (as well as a lot of negativity in general) so I avoid it. Don Darling (our Mayor) could be out shoveling their sidewalk himself and those people would complain that he didn't use enough salt. At least that's the impression I've gotten from friends who follow that group. If you surround yourself with those people, then you are going to have a bad time in SJ. They will influence your perception and ruin your experience.

At work and at local events I've never gotten comments like that. Sheltered bitter locals and young people from Fredericton are the only people I hear it from. Saint John isn't perfect by any means, but the people who are making the best of the city will have more valid and informed complaints than "Why'd you move to Saint John? It sucks! THERE"S NOTHING HERE!!!!", and they aren't going to start throwing shade at the slightest mention of moving here.

excluded from social activities/community

I've personally never experienced this. Friends groups can be pretty long-established and keep to themselves, but local groups are very welcoming and friendly. I've had great experiences playing table-top games, video games, mountain biking, and meeting people at the pub. Growing a friend group can be hard, but that's difficult in any new city.

This subreddit and the r/newbrunswick subreddit have a more positive outlook on the city in general. Search "moving to Saint John" and similar phrases and you will get tons of great advice and insight from previous threads. NB and Saint John definitely have problems, but IMHO you'll get a more balanced view of the situation by reading around and considering all the opposing opinions. There's truth in the negativity, but there's also a lot of ridiculous exaggerations.