I have a degree in neuropsychopharmacology, which is a part of of neurobiology, physiology, and behavior science. I studied precisely this subject (although not only for illegal drugs, also all the legal and prescription ones).

I can assure you that LSD causes almost no damage at all, if there is even any damage at all. Of course, you will be able to find the occasional study claiming that LSD causes rupture of the neurons at the synaptic cleft, but these studies are usually just poorly done and don't really give us useful information. And btw, nearly identical "studies" can be found about a multitude of other drugs, and are almost always done with ridiculous doses of the drug being administered to the neurons until the cell suffers some terrible fate.

I can tell you, however, that psilocybin mushrooms are, in fact, harmful to the body. Fungi rely on chemicals for defense, and psilocybin is not the only chemical manufactured by the mushroom. If you eat enough of them, there will start to be damage to the body, but I can't remember if it was the kidneys, or the liver. Anyway, the damage is caused not only by the initial exposure to certain chemicals, but also done by the metabolites, or by interfering with other metabolic pathways (say, the breakdown of another chemical that would otherwise be considered "safe").

Believe it or not, most drugs that people use/abuse for recreation harm the body very little on their own. Some drugs can actually be beneficial. For example, small amounts of nicotine improve memory retention and attentiveness for up to two weeks after cessation of the drug. Most of the time, the things that cause damage to the body of drug users are the lifestyle changes, and the vehicle of drug administration. For example, smoking is bad, regardless of what you smoke; injection causes scars, infections secondary to the actual drug effects, and it bypasses first-pass liver filtration, so even more impurities get into the body; smoking drugs like meth deposits acids directly onto the teeth and gums, and causes dry mouth and subsequent tooth decay. On top of things like that, people abusing drugs tend to not sleep correctly, not eat correctly, tend to ignore proper hygiene and other health habits, and tend to expose themselves to chemicals that healthier people are not exposing themselves to.

Anyway, LSD probably isn't going to do much harm, but you might want to be careful ingesting lots of mushrooms.