That's exactly what you do. It's 12v battery about the size and weight of a brick. It takes a long time to charge but you can get about 12 hours off a full charge depending. I bring a $600 finder with me on a ten foot sit-in. Over the top? Perhaps, but it's cool as hell. There are various ways to configure this and mount the transducer which have been discussed here before, but you don't need a specific type of depth finder for a kayak.

What you do need is a way to mount or carry the battery. The set up I have, which is from Humminbird is a base which holds the battery and screen in a bag, so it's all in a self contained unit. I leave the transducer mounted behind the seat. So, when I go to launch i just set the bag between my shins, unzip it and plug in the transducer. Throw in the rest of my gear and paddle away. The transducer is mounted with Duc-seal and shoots through the hull. No drilling or epoxy, removable, super easy, and works perfectly except perhaps a few degrees off on water temp.

Now let's be clear, this will definitely go over your budget. However, I don't see why you couldn't rig up something homemade. Scotty makes some good detachable mounts, and I've seen battery boxes for these go for around $40. The Duc-seal can be purchased out Home Depot or the like roughly $3. The battery costs around $30.

Just do your homework and you can probably pull this off for relatively short money. You need the right cables, which may have to be bought separately but I've managed to make mine work so far with what comes included. Ideally you want a 3 amp fuse in-line but I've skipped that before and haven't experienced any problems.

Here's the battery You can get these elsewhere for cheaper.

Here's the battery and case system I use

There are a lot of other options too, but this is what works best for me.