your description of what the car will do sounds good when there are little deviations, and crazy situations are relatively rare, but that is where it is really important. If we ignored all of the relatively rare occurances then self driving cars would be far simpler, but when there are 1,000,000 very rare things that could happen and you ignore all of them, they can add up to a fairly high likelihood of happening.
Here is an example that happened to me. I was on an 2lane interstate driving in the left lane. I was passing a car in the right lane. as I was a little over half way overtaking the car, it suddenly jerks towards me. Braking wouldn't have slowed me down fast enough for her to swerve ahead of me, accelerating would have been too slow to finish getting ahead of her. I could have just let her hit me but with a rear wheel car and her hitting the back half of my car, she could have spun me out, and that at 70mph didn't seem too fun. There was a small shoulder on the left side and then a large flat grass median. I started off pulling away from her since there was the shoulder, but she kept coming over. I ended up with my right wheels nearly off the shoulder and my left wheel far into the grass. she finally jerked back over into her lane and continued on. I was able to smoothly pull back onto the shoulder and then back into my lane so luckily there was no damage. I caught back up to her and when I saw a chance, I quickly passed her because she was an extremely old woman and she continued to swerve around while looking at her phone, but this ins't about ranting about phone use or elderly drivers.
Long story short, I took a risk of possibly losing control by driving onto grass on the interstate in order to avoid someone hitting me.
It doesn't matter if the automated car notices the car drift 9mm off center, the speed that she jerked into my lane would not have given any time to brake or accelerate out of the way. The two options would have been to get hit, or go into the grass. No matter how fine the level of decision making is broken down to, those little decisions would have added up to one of those choices. It would have seen the distance continue to decrease regardless of steering adjustment until there is no shoulder left. At that point, the computer will see the data of zero clearance to grass as well as incoming vehicle. What it its next fine level command.
Long Story Short: When people bring up the trolley problem, it is usually not literally saying that car AI needs to solve the trolley problem. They are saying that no matter how good and how safe cars get, they will have to be programmed with some general guidelines which will translate to determine what we would consider to be the car's ethics. For the decision making algorithms to function, values have to be given to different actions. should the car swerve an inch to avoid a pothole? should the car swerve 2 feet to avoid a pothole even if that means crossing over a solid yellow line as long as there are no other cars? Should the car drive off the road to avoid a collision? what if the oncoming car is a huge loaded down semi that will crush you? seems like driving into a shallow ditch would be a good option. What if the oncoming car is a guy on a motorcycle who will surely die if it. should the same car damaging maneuver be performed to save his life as it was done to save yours? what if his bike is the same brand as your car and the manufactuers wants to save the life of one of its customers, or advertised specifically to those buying that bike the added safety feature of its brand's cars will actively avoid them. would that be allowed?
It doesn't matter how fine you say the operating level is, all operations on any computer can be broken down to granular if/then statements really, the overarching software design will have to specifically include methods for dealing with these situations, even if that method is a default "do nothing and continue driving according to traffic laws"