Two things for you to consider:
People do exactly what they want to do, every single time; they put their time, effort, and energy into those things that are important to them, and they put less (or none) into things that are less (or not) important.
Every action you take -- every single one, even if it feels like habit -- is a choice you have made. You might be making that choice subconsciously, but you are still choosing.
So.
I just want to listen to the person I love.
If you wanted to be listening to him, you would be.
So the question you need to ask yourself is:
"Why do I not want to listen to the person I love?"
Or, put another way:
"What is it that, in the moment the person I love is speaking to me, I want to be doing more than listening to him?"
Or, put another another way:
"What need inside of me is fulfilled by doing something other than listening to the person I love?"
That's the first thing you need to figure out. Because if you wanted (really, truly, actually wanted) to choose to listen to him...you would. But you don't. So there's something else that is more important to you (and you may not even be aware of it) that is causing you to choose not to listen.
Understanding the need that drives that choice is the only way to resolve it, because as long as that need exists, you will find ways to serve it, and until you know what it is, the ways you subconsciously find to serve it will cause you to do things that you *think" you "don't want to do".
So. The first step is to figure out what that underlying need is that is driving your choice not to listen to him when he speaks.
Once you do that (and it's possible that working with an outside person who doesn't have your blind spots about your own behavior, such as a counselor or therapist, can be helpful in that regard), then the next steps are:
Figure out different, more constructive ways to service that need inside of you, so that it is taken care of and is not driving your subconscious choices in a way that does not serve you and your relationship, and then
Make a conscious choice to actively listen to him when he speaks. Right now the choice you are making not to listen is unconscious, somewhere down at the level of habit; you have formed pathways in your mind that cause you to automatically tune him out, like rainwater cuts a streambed in the land, so that the next time it rains, water will follow the same path.
What you have to do is carve a new path for that rainwater to follow. It's possible, but it won't happen passively or by accident; you actually have to do the work of digging that new pathway in your mind, and the way you to that is, when he speaks, stop whatever you are doing, put your phone down, shut off anything that might be distracting, look him in the eye, and consciously choose to pay attention to what he is saying. If it helps, respond to what he's saying; nod if he says something of note. Ask questions or make other responses, not only to show him that you are listening, but to force your mind to stay focused on this one thing that you are doing.
At first it will feel awkward and weird and onerous. But if you stick with it, then over time (just as water flowing down a new path you have dug for it) things will settle out, and you will find that your new habit is to do the thing that you have been consciously choosing to do: listening to him when he speaks will become automatic, and you won't have to put effort into doing so any more.
That whole process will take time; it usually takes somewhere around 90 days for a new, deliberately-chosen path of action to become habit.
So figure out what is driving your need not to listen to him (with outside help if necessary), figure out how to address that need in ways that are constructive, choose to change your behavior in a conscious and active way, and then stick with that changed behavior for as long as it takes for that new pattern to become habit.