I am an attorney, and I work in family law. This doesn't constitute legal advice, this is for informational purposes only, and does not constitute legal representation. If you want legal representation, then you should find a licensed attorney in your jurisdiction who can help you. In fact, depending on your income, you may be able to take advantage of legal aid/community legal resources that can help you. Look for "low bono" resources. On that note, you may not like the first part of what I have to say.

The main problem that I see, and it may just be your phrasing, is that you said you let him convince you not to pursue child support. The reason this could be a problem is that if you told him, whether verbally or in writing, that you would not pursue child support, you may have actually forfeited your right to pursue it later. This is because of a doctrine called laches). Laches means that if you state that you will not pursue a legal right, then you may be barred from pursuing it at a later time. This becomes more difficult to get out of if the father has, and can show, reliance on your statement in some way. However, it is entirely possible that your state has a law saying that you cannot ever forfeit your right to child support. So I would look into this, and hope he doesn't have something in writing saying that you agree not to pursue child support.

In pursuing child support, it is fairly straightforward. You'll need to get an order from the court establishing the amount of child support he owes you. Typically, you'll file a petition for child support. Assuming laches or some other theory does not bar your complaint (and he would have to argue this for the court to consider it), it should be fairly straightforward. You'll each show your income, resources, etc., and the court will order a monthly amount. Most states have a set range based on the cumulative amount of income the parents have, with deviations allowed if the parents make too much or too little. The court will also address the issue of back child support, and state how much he is to pay. Once the order is in place, and if he fails to pay, you can do a couple of things. You can bring a motion for contempt, you can obtain a judgment against him in the past due amount (which you can then use to file a lien against his vehicle or home, though it sounds like he is judgment proof) and have his wages garnished (I believe you can go through the sheriff for this, IIRC), or you can apply for support enforcement through your state's child support division (probably your best option). Once support is in place, as your incomes change, you can have the amount adjusted. In WA, it's as a matter of right every two years. It may be different where you are, so I'd look into that; it should be in your state's statutes.

As far as the father having access to your son indefinitely, there may be some things you can do. First, I would write down all of the events that occur where your son indicates that his father is not part of his family. Second, catalog any events that occur that indicate the father should not have visitation. This could be showing up to exchanges drunk, not showing up at all, being verbally and emotionally abusive, etc. But general bad parenting, such as letting the kid stay up late watching TV will almost never work; courts tend to view it as a waste of time, and will give you the side eye. Third, talk to your son about how he feels. Monitor his behavior in daycare. Does he start acting out? Does he appear to be having more volatile emotions since his dad is back in his life? Things like this can be useful, because you may be able to go back to the court and say, "This isn't working, he shouldn't be able to have this much visitation." In WA, we have what are called .191 restrictions, which you can use to severely limit a parent's access to the child. One factor here is willful abandonment, and another is lack of an emotional bond between parent and child. You will want to look into this. In the aggregate, the fact that the dad walked away for several years, and the child is not bonded with the parent, plus any negative factors on the parent's part (not showing up to exchanges, causing conflict, using conflict to get his way, etc.), might be enough to eliminate all visitation or restrict it further. There is always the chance that this can backfire, and the court can actually order more visitation.

Again, your best bet is to contact an attorney. Check with your state's bar association, and look for attorneys with sliding scales or who offer "low bono" services. There may be community organizations that provide legal services as well. If you cannot afford or find an attorney, and as much as I hate it to recommend it, take advantage of the free consultations most family law attorneys offer. Feel free to PM me if you would like me to direct you to more resources.