No, you should not be taking them off. It is not easy for you to have your fillings taken off. Whereas Allah commands as follows in the verses related to -wudu- ablution and ghusl:
“Allah does not desire to put on you any difficulty.” (Al-Ma’ida/ The Feast 5:6)
In a different verse:
“He has not laid upon you any difficulty in religion.” (Al-Hajj/ The Pilgrimage 22:78)
Besides, Shafii and Maliki madhabs count the inside of mount and nose as inner side of body, not the outer. Therefore, washing the mouth and nose is not obligatory -fard-, but sunnah according to them. You can also behave accordingly.