It isn't just the fact he wrote the novel. It's a collection of various facts and observations that when combined form a compelling case, of which his novel is only one piece. On it's own the novel means nothing, but pretending that people think he's a pedophile just because of the novel is disingenuous.

Take for example this bit of information;

Donald Barr was headmaster of the Dalton School from 1964 to 1974. During his time as Dalton's headmaster, Barr is alleged to have had a role in hiring Jeffrey Epstein as a math teacher despite the fact that Epstein had dropped out of college and was only 21 years old at the time.

How many other 21 year old college drop-outs do you suppose are being brought on to teach math and physics to kids at prestigious college preparatory schools?