I think that you are vastly overestimating the popularity of this genre. I doubt that the rate of loli hentai consumption is much more than 5% of the population, which is supposedly the rate of pedophiles. But I would like to challenge the strongest version of your position, so I will accept for the sake of argument that loli hentai is widely consumed and essentially normalized in Japan.
There are still two fatal flaws in your reasoning, though:
“But if they developed a taste for it,” you are saying, “they must have had the tendencies deep down the entire time!” Maybe, but most people encounter actual children much earlier, and much more frequently, than they ever encounter loli hentai. If actual children don’t awaken these tendencies, but loli hentai does, it seems to me that their desire is for loli hentai, not children.
I hope that I have changed your view by showing that the consumption of “loli” material is clear evidence of pedophilic tendencies.