Yeah, you're right: it's all arbitrary, and it's not the best system. Kurt Vonnegut invented a nice term for arbitrary groups like the nation-state, the "granfalloon," in Cat's Cradle. However, local self-determination gets severely screwed over when you don't believe in locality.
Most people want a higher standard of living within their own locality. They don't want to have to come to America. We should focus on that. And if people really want higher standards of living, within their native locality - as opposed to doing away with localities (be they the nation-state or states or municipalities) - then we should concentrate on making the global economy better for the world than fulfilling some pollyanna notions about anarcho-anti-statism or wherever direction you're coming from.
Didn't nineteenth century immigration enrich the "nation state"?
Yes, but there's a incredible systemic difference between a 21st century economy and 19th century economy, where there was need for surplus low-skilled labor. In today's job market, low-skilled peopled suffer from a lot of challenges. If the only preventative against the world migrating to the USA were the cost of the transportation, then that'd wreak havoc on our legal system and local (nation-state) self-determination.
Even then, what does preserving the "nation state" have to do with building a just society?
People have a right to have their society's preferences, as expressed in their government, match their locality. This is justice. Justices, globally, may conflict, but this is a pretty basic justice. We have the right, as other people do, to have our locality provide the policies we want. We need boundaries - which are, perforce, arbitrary - to simply say where one locality begins and ends. This is necessary if we're going to accept any legal system at all. For example, one society may wish to legalize a particular drug, for recreational or spiritual purposes, whereas another does not; this distinction does not hold if instead of borders we had amorphous blobs. By extension, should a locality wish to modulate residency and entry requirements for their locality, that's their prerogative.
Historical legacy doesn't make this fair, but neither would a panglossian borderless world really be fair to a locality in implementation because it undermines self-determination.