I think the add factor is what makes the difference here. Occasional amphetamine use by people with normal focus is pretty easy to recognize as beneficial. But I think one of the factors that define nootropics (if were going with the original definition) are compounds a normal person can take every day indefinitely without ill effect.

Obviously people with add can do this (I myself was very against amps until I saw studies of add patients brain images on amphetamine over time- the longer add patients take them, the more normal the scans become). But amphetamines have negative effects over time with daily use. Compulsions, obsessions, excessively verbouse, hyperfocusing on the wrong thing, anxiety, depression and excessive alternating with non-existent confidence.

All of that is worth not having add, because add is profoundly disruptive to living a normal life. If someone takes amps daily, they lose the euphoric effect within a month are so, then it becomes a question of the benefits and costs. A healthy person won't see those benefits, so I think you're right.

Patients in severe pain or anxiety see significant cognitive improvements on narcotics or bemzodiapzapines, but I'd hesitate to call oxy or xanax nootropics.