Your weapon & spell damage scale off of your max stats, though. So if you run dual wield on a magicka character, where your stamina pool is small, then your weapon damage isn't going to be great. Also, the dual wield skills are mostly (if not all) stamina attacks, and since you'd be running on a small pool of stamina, you would run out quickly. So you wouldn't be able to get a great DPS output, since you would have lower weapon damage and less ability to utilize skills.

As for how much exactly you would lose, without knowing everything about your build, but yes, if you switch to dual wield/2H on a pure magicka player you are most likely going to lose some damage output.