The actual text is hayah ro’eh et echav batzon v’hu na’ar – “He was shepherding his brothers with the sheep, and he was a lad”.
There is no need to read the verse as if Joseph was doing the physical work of feeding the flock. It may be that he was the overseer of the work and the others did the work; they were labour and he was management.
No wonder his brothers resented him. To add insult to injury, he was only a lad and they were grown ups. Presumably Jacob trusted him more than the brothers and expected him to report back about anything they did or failed to do.
That their jealousy of him was due to their antagonism towards a younger brother being placed in authority over them is more likely than that they were jealous of his coat of many colours.