Keep in mind that we will not advise you wether you should or should not have surgery in this subreddit. There is no way to properly diagnose a patient over the internet and doing so would be totally wrong. Keep also in mind that I'm not a doctor (yet) and in no way familiar with metatarsal fractures.
Nevertheless I'll try to give you some more information so you can come to an own decision. A short search on pubmed turned up several studies. As it seems non-operative treatment is the preferred one for less complicated fractures with good results*[1,2] . This fits with the way your fracture was treated.
However, apparently many (33%) patients keep experiencing pain up to one year after fracture*[3] (at which time this study cuts off) with most (>90%) patients having no limitation in their activities.
According to this*[4] study most fractures healed well within 33 months of injury, so prolonged pain after 4 years might still be a reason for concern.
In general I'd say prolonged pain after a fracture would indicate a problem and I would be careful to ignore it. Small bones in hand and foot have the problem that they are hard to fixate while healing and sometimes (scaphoid bone in the hand is notorious for this) a so called "pseudarthrosis" occurs. The two parts of a bone do not fuse together after fracture due to continuous strain and poor support by blood vessels and instead for a "joint" between them. I couldnt find anything on wether these happen in metatarsals too. As the 5th and the 1st metatarsal bear most of the weight it seems plausible though.
I'm not sure if we have any orthopaedic surgeons around here but if we do they may give you a better overview. I hope I could still be somewhat helpful.
[1] http://www.ncbi.nlm.nih.gov/pubmed/23908388
[2] http://www.ncbi.nlm.nih.gov/pubmed/23594755