Presumably the answer is that when Potiphar’s wife tried to entice him he had the moral courage to say “No” (Gen. 39:8-9).
In this sense he is a paradigm of the Jewish people as a whole – often tempted to abandon their faith and ethics, but strong enough to refuse. Righteousness is the ability to resist temptation.
We ask what impels a tzaddik to do the right thing. Is it the fear of discovery, of consequences, of what people will say?
This is all part of the answer, but Joseph’s own words add an extra dimension when he says that if he succumbs it will be a sin against God.
Bringing God into the equation is what moves you into the “tzaddik” category. It’s not just what other people will say, not even what my own conscience will think, but how God will judge me.