Chords are so lovely in this song.

I think your pulse is getting lost too much. It's fine to add rubato and be gently and sway-y, but you shouldn't ever loose the pulse completely or have it jump. Can you play it fine to a metronome? Even if your end goal isn't to play it "metronomically", you should be able to, and you should learn it that way. Once you can play it that way confidently - then try adding in the slows and pauses to give it more feeling. If you start doing that before you can play it in time, you risk cementing and 'normalising' your pauses and hesitations that are actually a result of not playing the timing properly as opposed to tastefully added interpretation.