Unfortunately, pot is like any other activity: the more you do it the "better" you get at it, and this includes being able to control your mindset and keep calm.
When I first started smoking pot, probably 4 out of 5 times would be great, euphoric, intense, absolutely wonderful. Food tasted better, I could feel music in my body as if my nerves had been tethered into an audio-jack, I could feel every hair on my body and my clothes against my skin as if my entire body had turned into the tip of my penis. And sex, sex was fucking awesome.
But, I had trips like yours. Horrible, horrible experiences. Once I got trapped in this feeling of deja vu that lasted for 6 hours. 6 fucking hours.
Deja vu might not sound too awful, but when it is placed in the context that you are only moments away from a horrible death and you feel as if every moment you experience has been scripted out before you to the extent that you are essentially riding through time in a little minecart on a track, watching time and space unfold before you, inching closer and closer to your untimely death while "seeing" everything that is about to happen just before it does...
Jesus Christ, I thought I was going to have a heart attack from that one.
That was almost a decade ago. When I do it now, I occasionally get a little paranoid, but even then I'm able to chill myself out and enjoy myself after a few minutes.
With that said, it's nothing so spectacular that you aren't just fine without it. Not to mention, it affects different people differently, and isn't for everyone. I've got a few anxiety disorders, so pot's a mixed bag for me. It is great at relieving mild anxiety for me, but if I'm in the middle of a panic attack I know better now because it rides not the pot and then gets amplified, and this is just my own personal body chemistry.
I've got friends who have experiences like yours every time they smoke pot, and they refuse to touch it anymore.
It's all good though, there's enough beauty, happiness, and joy in the world that nobody needs it if it isn't contributing to those things. If you can't enjoy it, it's nothing to make yourself miserable over, trying to experience what everyone else does.