Am I being terrible and selfish?

No.

Can a marriage work without sexual activity?

Yes, of course.

What challenges were presented when you opened up your relationship?

Ooof, this is complicated. I think every couple is different. The main problem is what happens if one guy meets someone "better"? Do you break up? But from what you've written, it sounds like your relationship would survive it.

How do I start this conversation in a loving, non-threatening way?

With a therapist involved.

However, I also feel that, if you are so inclined, you are within your rights to open up the relationship unilaterally and let the chips fall where they may.