The difference between one or a few samples VS the next successfully received sample could create little poping or clicking sounds in your speakers/headphones as but nothing more than that (Otherwise it's just silence). Losing heaps every second you'd hear more popping and crackling as your speakers are effectively jumped to the next sample. Long enough it would just be full gaps of silence in the audio stream to your ears.

This is easily seen in Audacity, especially when looping a sample with no silence and trying to match the db from the end with the start to eliminate the 'pop' jump between two different amplitudes at the start and end samples.

It's not all that complicated. But is is an issue to look for in any sound related work.