The plan wasn't merely to rescue Han but to also deal with Jabba as well who is a huge crime boss on what is basically Luke's home planet.

Luke knew he couldn't simply storm Jabba's palace and take everyone out by force, not to mention whenever a crime boss was wiped out like that someone else would move in on their former territory keeping the cycle going.

Leia and Chewie were never meant to be successful, they were supposed to get caught in order to lower Jabba's guard and bolster his ego making him a far easier target. Luke was always the one to set everything off.

Getting Jabba away from his palace ensured not only his death but also everyone that was important in his inner circle too. Luke pretty much had Jabba gather everyone important up together and made it super easy to kill them all by blowing up the barge instead of killing them all one at a time or in just giant fire fight.

Jabba thought everything was stacked in his favor. Luke had no weapons, his Jedi powers were useless on him, and all of his allies had been captured and/or turned to Jabba's side.

And of course this plays into the trope of villains gloating their victory to the point of allowing the hero to win. Instead of just offing Luke or imprisoning him straight away like he did with the others he made a show out of it. Tried to feed him to the Rancor and that didn't work so let's haul him out to the Sarlac pit and make him really suffer. Jabba setup a grand stage to execute them and instead was killed himself.

News of that would travel fast and certainly would deter others from potentially trying to take over.