Alright. So what has the show revealed about Negan?
They've shown that what he does is extort communities for half of their supplies. Which means what? He has a formidable enough force to feel confident that he can force people into this deal.
Jesus tells the group that they have never seen Negan. They have only seen The Saviors "with groups about as big as 20." Which means what? This "Negan" apparently has enough people to send out groups almost half the size of Alexandria, simply for scouting.
Rick and the group stormed The Saviors' compound and killed a lot of people. Let's say a modest 30. Couple that with the about 8 that Daryl killed with the RPG and you are already nearing Alexandria's population. (Which is 54 by the way if you missed it. Maggie says so in 610, so you don't have to cry spoilers again.) Now, they killed about 40 people and there was still a guy left over plus the woman on the radio who captured Carol and Maggie. One can easily infer, they have a lot of people.
The guy on the bike. Who's bike was that? Oh yeah, Daryl's. When was the last time we saw it? Episode 606 when it, along with Daryl's crossbow, was stolen by that man and woman. What was the premise of that episode again? Daryl gets separated and finds three people running away from their group. In the episode this group finds them and tries to take them back, but Daryl saves them and they take his stuff. Now, the next time we see the bike is when this guy at the compound is trying to escape on it. Which means, either The Saviors found the guy that took Daryl's bike and took it from him or The Saviors and the group that guy was from are the same people. If the former was true then it's all just a coincidence. But we can infer that it's not because they focused on the bike when the guy at the compound was trying to escape. They wanted you to realize it was Daryl's bike and that the guy who stole it is related to The Saviors in some way. The main point of this part is that That group from 606 sent out a lot of people, as shown in the episode, to retrieve one man and two women. Then they show in 612 that those two incidents are related. They keep showing you the amount of people this group has. Which at this point is well over 54.
After the guy tried to escape the compound, a woman called in over his radio telling Rick "we have a Carol and a Maggie." Emphasis on "we" because there's more than one. Which means what? Either The rest of The Saviors were conveniently out while Rick murdered their people or they simply have more people.
The show keeps reinforcing that The Saviors have a lot of people. All my examples are taken from the show. So, yeah, if you have been paying attention you would notice.
Now I do apologize if what I said spoiled you. But I honestly believe that I did not say anything that can't be infered from only watching the show.