There are multiple explanations, though it's not clear which is the correct one.
One answer is that it's a stylised 8, as the early silver "pieces of 8" which were originally called dollars included 8 pesos. The hand-written 8 eventually developed over time to represent $.
An alternative explanation is that the dollar was valued at 8 shillings in certain parts of New England. This was abbreviated 8s or 8/, which then also stylised and developed into $. As one author points out, the double vertical line was originally only found in places where the 8/ abbreviation was used. Presumably this credits one vertical as representing the slash, and the other is just the vertical developed as it had in the first theory of the symbols origin.
These two theories come from the first source at the bottom. The second source is in response to the first, some years later.
I'm quoting because the explanation is already as concise as it can be. "Thaler" in this quote refers to a German coin and the etymological origin of the English word "dollar".
Early in the sixteenth century the count who ruled over the Joachimsthal, a region in Bohemia containing rich silver mines, began the striking of heavy silver pieces. Because of its place of origin, the Germans soon gave the name Joachimsthaler to this coin. The obverse of the Thaler bore the Bohemian lion rampant; on the reverse there appeared St. Joachim between the letters S and I, evidently the initials of the saint's name in Latin. By the superposition of the letters S and I (or J), a symbol was apparently devised that was used first for the Joachimsthaler and later for all other Thaler.
It's uncertain which one of these is the actual origin, assuming there is a single origin story. In modern typefaces, the difference between the single and double vertical lines is largely stylistic and more generally based on readability or aesthetics. The same thing happens with the ¥ symbol when used for Chinese currency.
Sources: