OP, you're right in saying that harmonic relationships are mathematical, but you're missing a key piece of the puzzle — the way humans interpret those mathematical relationships is not absolute. Psychology plays just as important a role in the way music is perceived, interpreted and experienced as mathematics.
The perception of harmony is based on consonance, which is inherently a psychological phenomenon, and therefore subjective. In fact, the perception of consonance in Western music has changed over time. Prior to the 1600s, unlead 7th chords were considered extremely dissonant, which is why you simply didn't hear them in music before that era. Today, jazz is anchored by the unlead 7th. Our very definition of what is consonant and dissonant has changed, therefore it's not fixed or set in stone.
The interpretation of consonance and dissonance also has a lot to do with expectations. The chords and progressions we're most familiar with sound most consonant to us through sheer repetition, while more unusual voicings may sound dissonant, though they may not actually, mathematically, be any more or less dissonant, than others. Of course, there are exceptions. Major and minor triads, which 20th and 21st century pop music is based largely upon, are interpreted as highly consonant because they have perfect 4th or 5th intervals and no 2nd intervals, creating a very simple overtone structure that isn't subject to a lot of interpretation. But as soon as you start voicing more complex, sophisticated chords, you create opportunities for people to hear them differently.
Musical harmonics is not a perfect science. If it were, then harmonic estimation wouldn't be called "estimation." While you could perform a Fourier harmonic analysis on a series of chords and get a consistent result, the meaning of that result will differ for different people. There are dozens of scholarly examples that support this — just Google it.
Even withing the realm of notation there are different ways of interpreting the same music. Let's use a single concrete example. Look at this chord and tell me what it is. If you called it B7#9b13 you're right. But if you called it B7#5#9 you'd also be technically right. Mathematically, they're the same thing, but semantically they're quite different. By saying this B7 contains a b13, rather than a #5, you're suggesting that it's an inversion, and likely part of a chord sequence that plays off of the b13 in a higher register. The presence of a #5 implies a #4, though a #4 doesn't necessarily imply a #5, but interpreted as a b13, that suggests that there would be either a natural 5, or a #11 and no 5. Similarly, Eb+maj7(#11) and Gadd9(b13) use exactly the same notes, but they have different contexts, different purposes. This isn't just pedantic chord grammar. This variety in interpretation has profound implications on what chords may come before and after.
Furthermore, there's a whole cultural aspect that pokes holes in your argument. You write from the perspective of the Western tradition of music, with 12 equally-spaced tones each octave, which is just the mark of each frequency doubling, but that's not the only way music is played. People who come from other musical traditions, like Chinese and sub-Saharan African music, for an example, divide octaves very differently, and people from those traditions hear music quite differently that Western ears. That doesn't make them wrong, but it does show that a cultural bias has a strong influence on how music is interpreted.
TL;DR: the mathematics of frequency relationships are absolute math, but the psychology of music is open to interpretation, and subject to our cultural expectations.
