I love driving too. I live in Seoul and don't own a car because the transportation system here is balls to the wall amazing in every way (including price), however I do rent cars whenever I can just because I enjoy the freedom of driving. I also owned a motorcycle for a few years so I could feel the fresh air and not have to stand in sardine cans like city buses and subways.
That said, of all the drivers I see on the road every single day, half of them have no business behind the wheel. If you polled them, most would probably agree with this assessment and even agree that they were subpar drivers. The reason they drive and the reason most people drive is because they have to, not because they want to. Our cities and infrastructure are all based around the car and public transportation just doesn't cut it, even in cities that have great public transportation like this one. In my home town in Canada, not owning a car is pretty much social suicide.
Automobile accidents are what, the #2 or 3 leading cause of death in the developed world? How many people needlessly lose their lives in car accidents? Don't even get me started on the developing world (traffic deaths/injuries are just a part of everyday life in places like Peru and Vietnam). The sooner we can start automating driving on public roads, the sooner we can start saving scores of human lives all around the world and the sooner we can cheaply scale this technology and benefit from it economically, environmentally, socially etc. Not having to drive will be more liberating for humanity than having the liberty to drive.
Countries in North America, East Asia (mainly Japan), and Europe have a strong hobbyist/enthusiast subcultures centred around cars and motorcycles. Ideally what will happen is that people who don't want to drive will have options ranging from owning their own autonomous car to subscribing to an autonomous car service and people who love driving can continue to do so, requiring autonomous vehicles to work around this.
What will inevitably happen, however, is that autonomous vehicles will start driving faster and faster in order to be more convenient (think 180km/h on highways with no distance between cars, no traffic jams and no accidents). When this happens you'll probably have the option to drive on designated roads where people are still allowed to drive manually. I suspect some people will turn to VR simulators and other adrenaline/adventure-seeking pursuits.
People who love driving won't give up without a fight and since many power holders within our society fit into this camp, I think people who love driving will be catered to for years and years even after autonomous vehicles become the norm. You might just have to sacrifice some things.