I think you can be honest and say you don't feel you are compatible. You could provide reason (writing style, original world and character dynamics, etc), but if you worry about that hurting someone you don't have to. You could also mention that you saw that the sample and the response seemed almost identical despite it not really connecting with the story. Since you mentioned them having trouble finding partners, offering specific reasons for why you are no longer interested might help them moving forward.
It can be hard and frustrating to put so much effort in and have to drop it, but the sooner and more upfront you are with your desire to part ways the better.