I don't like counting "successes" ... Rolling 2 or more 10s upgrades your roll by one step, and rolling 3 or more 1s downgrades your roll by one step.

This is counting (critical) successes and failures.

Also, a highly skilled character is more likely to roll both upgrades and downgrades. A character who rolls two dice can never critically fail, so when you gain your third rank your odds of critical failure actually go up (from 0 to 0.1%). At rank 4, it goes to 0.36%, and so on, until eventually reaching 2.3% for rank 7.

You didn't explain what a step is, so I'm going to assume it bumps your check from one difficulty category to the next (e.g. hard to expert). If this is the case, why not just say rolling double 10s gives +5 and triple 1s a -5 to the final result? Unless steps means something different.

how do I calculate the last part to check probabilities?

This is the problem. When even designers themselves don't understand the mathematical implications of their own mechanics, things start to get very wonky. (Example: In ASoiAFRP, even Jaime Lannister in his prime could never land a hit on a Braavosi fencer. Or in older RuneQuest, a battle involving thousands of axemen would leave hundreds of them dead from self-decapitation). The mechanic you have chosen (variable dice pool summed versus target number, PLUS per-die result counting to modify the roll) is one that has extremely complicated probabilities. Extremely. At the risk of sounding elitist, if you want to make this work you will need at least a stats and probability 101 course and intimate familiarity with Excel (or R).