As soon as i've read "there's not much here" about fort Joy, i knew that something was a miss.

This game can be challenging, even if your first run is on Classic mode, DOS2 can do that sometimes.

I can only recommend you to talk to everybody, like...everybody.
Some quest are intertwined, but really you gotta think "outside the box" regarding to quest and leveling.
If you see this more like an action RPG and not a Tabletop game, you'll miss out on many thing.
Also if you don't have the pet pal perks, you can pass by some interesting quests & fun.

Only in fort joy, you can fight toads, crocodiles, guards, fight in a arena, help various people try to escape, and escape with you, find companions, explore caves, encounter royalty, (even snailed one...) , find equipment, heal some people, kill some more, look for oranges, play hide & seek with children, help souls escape the realm of reality, find some psycko dudes, steal everything that isn't nailed to the floor, pickpocket vendors or people ...

And this is quite scratching only the surface.

It's really dense. And that's only the beginning.
You're in Act 1 , more or less at the beginning... i'm 150 hours ahead of you on this save and only in the beginning of Act 3.

If you've having a hard time fighting, don't hesitate to lower the difficulty if it's really that difficult for you, everyone is different, and play a certain way.
You'll be able to respect on the end of Act 1 as much as you'll like after.

I'll also recommend to play with a party of 4, (you and 3 IA companions, or only 1 with both the Lone Wolf perk )

Good luck !