Prior to Kol Nidrei there is a declaration that the Yeshivah On High and the Yeshivah Below give permission to pray with avaryanim, “transgressors”.
The theory that it refers to Marranos who had transgressed by pretending to be Christians has no basis.
These avaryanim were in cherem for breaking a law of the Torah or even a communal ordinance. Normally they could not pray in the synagogue, and if they did the congregation had to keep 4 cubits away.
On Yom Kippur the 4-cubit rule was hard to maintain in a packed synagogue, and in any case no-one was free of sin and beyond suspicion.
The yeshivot in the formula are not Talmudical colleges but court sessions (from yashav, to sit). The formula is said three times for emphasis.