Statistics helps too. Rarely are objects in life manufactured as just pure hues. Green objects are the most likely to be a pure hue. Red and orange objects tend to be darker (brown is just dark orange). So each shade of grey tends to be a certain hue of color.

Here's the set of Addams Family, entirely greyscale, processed thru an A.I. colorizer - https://imgur.com/ql5mqt0.png . Notice the pink walls - a strange choice for that shade of grey. But if you look it up it's true https://www.boredpanda.com/the-addams-familys-living-room-was-actually-pink/ . How did it know? Because red was a popular color on hollywood sets. Directors knew reddish hues show up darker on b&w film.