Believe it or not, some cultures actually do use this sort of technique. For example, the French learn a fixed-do system where each pitch has a permanent number, so C = 0, C# = 1, and so on up to Bb = t (for ten) and B = e (for eleven). There are pros and cons to our system and their system, as follows.

Movable Do (7 names for 12 notes) is very easy to transpose into all twelve keys, but then pitches are 'spelled' differently enharmonically, as in F#/Gb. Its also easier to talk about raised or lowered pitches, especially when teach how major and minor chords and scales differ from one another.

Fixed Do (12 numbers for 12 notes) is relatively difficult to transpose, as you have to understand that 047 is the same thing as 6t1 (major triads of CEG and F# A# C#) just by looking at the numbers, but the notes don't ever change their spellings. In addition, this method of note naming is becoming the standard method of analyzing 20th century literature (set theory and dodecaphonic music, if you're interested - Schoenberg's Pierrot Lunaire is often considered the epitome of this sort of literature), so students analyzing 20th century works have an advantage if they were initially taught with numbers as opposed to letters.

Edit: after rereading your post, I'll answer your questions more directly. Yes, that system does in fact work, like the French I mentioned. The 7-note name system is so prevalent because of Guido d'Arezzo, mentioned multiple times in this thread, and Western tradition has created over a millennia's worth of music in this notation. Also, the problem with notation is that it's an attempt to convey sound through a visual medium. As you could probably guess, that's pretty damn hard, especially if you want to talk about pitch, volume, timbre, attack, style, and all the other musical characteristics that go into, well, music. So what you're used to is pretty much the best we've come up with that works for most Western music (besides the 20th century stuff I mentioned).