Yeap exactly. I'm sort of in the field of Cognitive Science and I can't even begin to explain how flimsy IQ tests are. It was a neat idea that isn't really recognized as valid anymore. One way I can explain it simply is this:

There is a guy called 3Blue1Brown on YouTube who does math videos, he thinks extremely visually, a big part of his conceptual understanding comes from geometric/spatial thinking. If you watch his videos, and look at his education history, he is clearly pretty smart.

My math professor, also pretty smart and good at math and with good education history, does not think visually. He thinks in systematic terms, he said this himself. He thinks in the world of formulas and puzzles and how these semi-verbal instruments interlock with each other. His thinking is symbolic and systematic, not visual. He again said himself that he could not ever come up with the stuff 3Blue1Brown does.

Yet both are very good at math. Is one smarter than the other? No. They just employ different methods of intelligence.

Another example from me, I was really good at physics and space geometry, because I also think pretty visually and I have an intuitive understanding of how objects interact with each other. Compared to these subjects, I am absolutely terrible at arithmetics. Am I suddenly stupid when I take an arithmetics test? Am I suddenly smart when I take a physics test? Notice that both are mathematical fields that rely on pattern recognition and problem solving.

The truth is we cannot accurately assess how smart someone is from a simple test, you'd have to get very close into how that person thinks. Even then, the issue could be with a lack of skill in expressing what's in their mind rather than an issue with their mind.

IQ has died out relatively recently so its popularity continues, but I wish it didn't give people self esteem issues. The brain, cognition, and the mind are much much more complex.