It's not even that it's not the worst, it just really doesn't matter.

Skill damage scaling is heavily multiplicative between Atk, Atk%, Elemental Damage, CR, and CD. Each multiplier makes the others better. When you don't scale the multipliers, small increases to one of them just doesn't matter. I don't care about crit rolls on my EM artifacts.

To give an example, compare how much damage is gained by 1 max CR substat roll on a level 90 Venti with R1 Stringless level 9 talent, 100% Atk, 70/140 crit, and Anemo damage goblet to a level 1 talent full EM build.

In the first case, your average skill damage before resists/defenses is 773 x 469% x (1 + 100%) x 1.706 x (1 + 0.7 x 1.4) = 24503. Adding 3.9% CR increases this to 25178, a difference of 675.

In the second case with no Atk%, CR, or CD, and base talent level, your baseline average skill damage is 2623. Adding one CR roll is increases that to 2673--a difference of merely 50 damage.

The more of each of the multipliers you have, the more important it is to get more of them. On builds that are starting with none (full EM VV builds) they don't matter. I treat CD and Atk% as basically dead stats on EM VV pieces because they basically are. The amount of Atk/CR/CD substats you'd have to roll for them to be at all important is so high that it's just not worth worrying about. For all intents and purposes the only substats with any meaningful value in EM VV builds is ER and/or CR (for Fav weapons). All the other DPS stats don't matter.