I've done a bit of targeted ad work before, and the conclusion I've come to is much scarier. Google doesn't need to listen to our conversations. Tracking everything we do online is much more likely to give much more relevant results. The targeting algorithms they use are much more accurate and ever-present than listening to our conversations though microphones could ever be. Companies don't want to pay for ads to be displayed for things a phone might have possibly overheard someone say, they want to pay for things that they know someone is interested in based on their web history.

It seems like they're listening to our conversations because these targeting algorithms are so scarily accurate they can sometimes know things relevant to you before you even know it.