Explain to your friend (outside of work) that work is work and for the sake of the job, for both of you, professionalism needs to be maintained.

Tell him: "we are here to make money and get a job done. The fact we are friends does not change this fact and we should not compromise our friendship because we work together, but likewise, we should not compromise our jobs because we are friends. The job needs to get done, bills need to be paid, and we socialization is for outside work."

Or something to that effect.

When I DO put my foot down, he gets all mad and acts like "Where did THAT come from?!"

You need to be consistent too. If you allow certain behaviors and randomly put your foot down, it is causing the problem.