Here is proof:
" What Are the Long-Term Effects of Hallucinogens?
LSD users quickly develop a high degree of tolerance to the drug’s effects, such that repeated use requires increasingly larger doses to produce similar effects. Use of hallucinogenic drugs also produces tolerance to other drugs in this class, including psilocybin and peyote. Use of classic hallucinogens does not, however, produce tolerance to drugs that do not act directly on the same brain cell receptors. In other words, there is no cross-tolerance to drugs that act on other neurotransmitter systems, such as marijuana, amphetamines, or PCP, among others. Furthermore, tolerance for hallucinogenic drugs is short-lived—it is lost if the user stops taking the drugs for several days—and physical withdrawal symptoms are not typically experienced when chronic use is stopped.
The long-term residual psychological and cognitive effects of peyote remain poorly understood. Although one study found no evidence of psychological or cognitive deficits among Native Americans who use peyote regularly in a religious setting, those findings may not generalize to those who repeatedly abuse the drug for recreational purposes (Halpern, 2005). Peyote users may also experience hallucinogen persisting perception disorder (HPPD)—also often referred to as flashbacks. The active ingredient mescaline has also been associated, in at least one report, to fetal abnormalities (Gilmore, 2001).
Long-term effects of DMT use and abuse and addiction liability are currently unknown. Unlike most other hallucinogens, DMT does not appear to induce tolerance (Winstock, 2013).
As with some other hallucinogens, there is little information to suggest that ayahuasca use creates lasting physiological or neurological deficits, especially among those using the brew for religious activities.
Overall, two long-term effects—persistent psychosis and HPPD—have been associated with use of classic hallucinogens (see text box below). Although occurrence of either is rare, it is also unpredictable and may happen more often than previously thought, and sometimes both conditions occur together. While the exact causes are not known, both conditions are more often seen in individuals with a history of psychological problems but can happen to anyone, even after a single exposure. There is no established treatment for HPPD, in which flashbacks may occur spontaneously and repeatedly although less intensely than their initial occurrence. Some antidepressant and antipsychotic drugs can be prescribed to help improve mood and treat psychoses, however. Psychotherapy may also help patients cope with fear or confusion associated with visual disturbances or other consequences of long-term LSD use. More research on the causes, incidence, and long-term effects of both disorders is being conducted."
tl;dr: LSD affects the wiring in your brain and can lead to persistent psychosis and HPPD
for those that don't know HPPD: Hallucinogen persisting perception disorder (HPPD) is a disorder characterized by a continual presence of sensory disturbances, most commonly visual, that are reminiscent of those generated by the use of hallucinogenic substances.