The issue here is the fence line, not the tree.

Tell the neighboor that, “I am confused” regarding their apparent assertion that your fence is on their property. Tell them that you think a survey is in order, but that you don’t have any money. They’ll probably pay for it.

Before that however, I’d find my official lot dimensions and measure the fence lines with a very long tape measure. Get one that’s long enough to do it in one stretch and pull it tight. This will give you an idea if you’re missing or have too much property for your official lot size.

Be careful with this neighbor, they cone off as a touch passive aggressive.