As someone who went through something like this (a couple of different times by different people), SPEAK UP. SAY SOMETHING. From what you are describing, there are too many red flags to just ignore this.
Just reading through this list of things you noticed, the uncle seems fixated on "Sammy". It's likely because "Sammy" isn't as wont to object. I would assess that the uncle has attempted to molest "Connor" as well, but Connor's probably not as easy a target.
Red flags for me:
Here's my assessment as someone who was molested at 5 or 6 years of age, now in late 40's. Going through this list of your observances, The uncle is likely molesting the 2 boys, though he seems fixated more on "Sammy"... "Connor" is probably objecting or resistant more. I'm convinced the mothers know, but are in denial (at best), but they love their brother & also don't want to have to face this major strife... but it's not about them. It's about the safety, health, and well-being of these 2 kids.
As for the uncle slipping his hand down the back of Sammy's pants... yeah, that's not normal AT ALL. I was also an only child to a single mother, and I was picked on ALOT at school. I remember enjoying affection, but I didn't have a clear idea of what was appropriate, and what wasn't. The hand down Sammy's pants.... yeah, that's exactly what I'm talking about. You witnessed first-hand inappropriate behavior.
You KNOW there is a reason you are still thinking about this 3 years later, and you know there IS still something you can do about it. No, it's not too late. For whatever reason, the adults in these 2 boys lives are failing them. Based solely on what you wrote, I suspect the uncle is molesting them and/or grooming them, but also more. I would recommend reporting it to the police, or a social worker, telling them everything you wrote about in this post. Do not confront the uncle directly. Be the adult that these kids seem to need help from, but don't seem to be getting yet.