Aspartame, fluoride and MSG are all damaging to the body's endocrine system. They're a hazard if you care about your hormone production, but they're not going to poison you within the levels they're sold, they're "safe" in the regard that they are non-lethal, but they are not totally benign.

Aspartame and MSG in particular are excitotoxins. MSG binds into the NMDA and AMPA receptors of the brain, causing the receptors to uptake too much calcium ions, which overexcites the receptors, and eventually damages them. The oversynaptic activation of the NMDA receptors shuts down some of the cAMP (cyclic adenosine monophospate) activity in the brain, which serves as a signaling molecule between the brain and the balls.

Animal studies have shown a clear link between lowered testosterone levels, central nervous system damage, and excitotoxin intake in male rodents (study, study, study).

Fluoride displaces iodine from cells and causes oxidative damage in the testicles. Fluoride anions that come in contact with calcium ions in the body form a compound called calcium fluoride which can mess with the function of many neuroendocrine glands, causing calcification and impaired protein signaling. Multiple rodent studies show a dose-dependent impact on testosterone production. Human males in high fluoride water areas have 40% lower serum testosterone on average than those in low fluoride areas. Dosages of 2-13 mg have also been proven to negatively impact testosterone production and fluoride is particularly troublesome because it accumulates in the body, so regular low doses can be just as damaging as occasional high doses.

The truth lies somewhere between.