Child Support and getting Sole Custody?

I have an open child support case through my state and have had one for six years, and have barely received anything. My son is six years old now, I am his mother and have been his primary gaurdian. His father left when he was an infant, basically he is a deadbeat dad and has been in and out of jail/prison. I was never married to him but his name is on the birth certificate. I never thought I would ever have to proceed with anything thru the Courts because his dad never wanted anything to do with my son in the first place. I am worried now though that because he paid a little bit of child support thru the years that now he could get visitation rights, but he never made any effort to do that so far. I want Sole Custody of my son, his father and the life he lives is no good. Would it be wise for me to proceed and ask for Sole Custody or would the judge just tell me he can get visitation rights now because he paid me a little bit of child support that he pretty much earned all while he was in prison? Even though he is a bad person?