This is a kal vachomer, an argument from minor to major: if X applies, how much more will Y apply.
Its logic is undermined because of the special situation of the Israelites.
Why do they not listen? Because they are worn out by slavery. They are just too tired.
Why does Pharaoh not listen? In his case it is sheer obstinacy.
Moses himself realises his kal vachomer is faulty. He does not blame the people at all. If they don’t listen, the fault is not in them but in him. They don’t listen because he is not patient enough with them; he does not explain things properly.
A good leader does not blame the people but himself.