The bus had the right of way. With that said, there seems to be a misunderstanding about Ontario's yield to bus law. Under the Highway Traffic Act, you must yield if the bus stopped at a bus bay, unless it is unsafe to do so:

142.1 (1) Every driver of a vehicle in the lane of traffic adjacent to a bus bay shall yield the right of way to the driver of a bus who has indicated his or her intention, as prescribed, to re-enter that lane from the bus bay.

What's a bus bay? The definition is found in regulation 393/02:

“bus bay” means that portion of the highway beside a bus stop sign that is used by buses for the boarding and alighting of passengers, the use of which portion of the highway requires buses to exit from and subsequently re-enter an adjacent lane of traffic.

Strictly speaking, it looks something like this. But the regulation's definition is broad enough to include bus stops in the curb lane if the driver has to pull into the curb lane and then back to the adjacent lane to continue along the road, for instance because the curb lane is used for parking beyond the bus stop. Or if the bus stop is in the right turn lane.

That's when you must yield. In all other situations, it's merely a courtesy. In this case, the bus was stopped at a bus stop in the right turn lane. He had to re-enter an adjacent lane of traffic to continue along his route. He had the right of way.