According to the Midrash, he knew there were rumours that he had pocketed some of the people’s donations of silver and gold and had spirited away enough to live in luxury for the rest of his life. He was a man of integrity, and there was no way he would give in to dishonesty.
Nonetheless he must have been tempted to lose his temper and tell the Israelites, “If you don’t trust me, finish the job yourselves – I resign!” But his instinct told him that there will always be malicious mutterings, and nothing should deter him from completing the mission with which God had entrusted him.
Leadership is never easy – but every leader has to hold their head high and persevere with their responsibility. There might not be any votes of thanks; maybe people will quickly forget all that the leader has put into the work – but the one whose approbation really matters is the Divine Employer, and the leader has to hope that He will say yeyasher kochacha, “Well done!”
At the end of the Chumash, as reported by Rashi, God gives Moses a yeyasher kochacha; hopefully every leader will win the same mark of approval.