Is it your fence or their fence? A good way to tell is the way the fence is structured. If the rails face towards your house (meaning that the nicer side faces away towards the neighbors), then it's probably your fence. And of course, vise versa. This is not always true, but is the way a reputable fence installer would build the fence.

Anyway, this is how my fence guy explained it to me when I was trying to figure out how to get a fence repaired when the neighbor was being uncooperative. Additionally, most fencing companies suggest that a fence be placed slightly within property lines in order to deter problems like you are having now. IANAL, but I think that if it is directly on the property line, then it has to become a joint decision on what to do with the fence, rather than just being your responsibility to deal with.