I'm not suggesting a houserule. A sorcerer with subtle spell with material components isn't "doing anything weird." All anyone would detect is that someone is holding an object, assuming they aren't covering it like closing their fist around it. Technically, it's not clear you have to even touch it, just have "access" to it, so an amulet may work. Furthermore, the effects of a spell are not detectable unless they say they are (e.g. Charm Person) or it manifests an obvious effect (e.g. most Evocation spells).

Verbal and somatic are nearly impossible to hide, but detectable material components are not a requirement to cast; detectable components are a requirement to detect.