Article II, Section 3 is a very complicated part of the constitution. Here is an annotated explanation of Section 3 from the Congressional Research Service at the LoC (make sure to click next to read the entire section). The interplay with Section 2 is also important.

tl;dr: There are lots of issues and politics at play here.

Now, as far as whether it is "illegal" to invite a foreign dignitary to speak in front of Congress...

There is a strong argument that the Curtis-Wright decision would be controlling in any actual dispute between Congress and the President over actual powers, according to the National Constitution Center. Theoretically, I imagine if the President didn't want, let's say, Fidel Castro, to speak to Congress, he could keep him out.

Interestingly, a joint session of Congress to listen to a foreign dignatary is an "unofficial" session, according to the Senate's website. On a now deleted page, Congress shows how it contorts to allow this.

This brings up some interesting questions. If Congress isn't in an official session, then it could be argued that it is a private assembly of citizens who can host whomever it wants under the 1st Amendment. And doesn't everyone, citizen or not, have a right to petition the government?

tl;dr2: Congress has hosted dozens of foreign dignitaries in joint session. Why is it such a big deal? Politics, clearly.