Innappropriate would in this case refer to it being racist. Something is not inappropriate by itself, a factor makes it innappropriate. Since the use of the swastika is related to racism, my rebuttal for the symbol by itself, is referring to the (possibly) racist nature of the swastika in western culture.

In parts of eastern culture this is simply not the case. The swastika was already used for centuries and was not innappropriate.

Since the swastika can be perceived both as negative/ innappropriate and positive, there is no 100% consensus on whether it is innappropriate, thus banneable. Therefore, whether a swastika is innappropriate or not all comes down to the observer.