Look at its context and you understand the verse. Israelite worship had to avoid all forms of idolatry. Authentic Jewish worship had to be structured according to the pattern of priests, Levites and Temple.
Why were the Levites such an important group? Maimonides explains at the beginning of his Hil’chot Avodah Zarah that they were the real spiritual leaders of the community. Not that the kohanim were not essential, but it was the Levites who facilitated the work of the priests.
The name Levi probably comes from a root that means “to accompany” (Gen. 29:34; Num. 18:24), because accompanying the priests and working with them was the Levites’ God-given role. They were like the shammash charged with the task of making sure that the menorah would shed light.
They had a physical role in the sanctuary equipment and its rituals, but they were also the singers who aroused the spiritual and emotional feelings of the people.
And they were an intellectual repository of the traditions of Israel: people whose minds were struggling could come to them for guidance and direction.
The tribe of Levi merited this distinction because they were not involved in the sin of the golden calf.
Why were the people of Israel told not to abandon the Levites? Because the Levites did not abandon them.