Amalgam fillings are often the most significant mercury exposure in populations, providing the greatest single source exposure for millions of people. Check out health Canada or the FDA's sites, they both confirm that.
Also inorganic mercury from fillings is actually apparently more toxic than organic mercury as recent inquiry indicates..
http://m.phys.org/news/2015-12-links-inorganic-mercury-exposure-cell.html
Finally there are many variables at play. The dose makes the poison, true but there are highly variant doses here dependent on factors like bruxism or gum chewing which can significantly alter the amount of mercury that's offgassed.
Then you have to consider an individual's health profile. Mercury from fillings has been show to bioaccumulate in tissue, brain, kidney's, etc (see fda's page). This accumulation is of course variant depending on an individual's ability to excrete the mercury. A recent university of Washington study highlights the fact that there is significant genetic variability among humans in ability to excrete toxins like mercury, leaving some more vulnerable than others.
And so while I'm sure you've done the right math and made a logical conclusion from it you're conclusion does not reflect the actual ambiguity of this issue. While it is true that the average daily mercury dose for those with amalgams is low, the risk provided by bioaccumulation over often decades of daily 24 hour exposure, even to small amounts, is not negligible. There is also risk for sensitive populations like those with kidney disease for instance (health canada). Emerging evidence indicates what should be common sense, that implanting a potent neurotoxin into the mouth as a chewing surface is a poor idea indeed.
http://www.mcclatchydc.com/news/nation-world/national/article53118775.html
http://www.hc-sc.gc.ca/dhp-mps/pubs/md-im/dent_amalgam-eng.php