When you use them, don't be incredibly specific. Be a little vague in the chapter title so they don't automatically dismiss it but still gives the video structure.

I used to be paranoid about chapters for the same reason but if anything, views have been up because viewers come back knowing the video is more organized and less of a headache to find information. My videos generally are more informative though. I think for some videos, it's better to not have them for watch time. Like, if you know most people only are going to care about a small segment of the video. If you have "Top 10 Rare Cats" in the title and the thumbnail has a picture of an alien cat with 3 eyes, people are going to possibly skip through checking for it.

They're also awesome for paid promotions. You can just set a chapter for the ad and I feel like it gives viewers a lot less to complain about.