They don't have to have your passport with them to arrange an exit permit. All they need to arrange an exit permit is your passport number, residence permit number and date of travel. And they can do it online...they don't need your passport anymore. They 'attach' the exit permit electronically to your passport. So that is not a reason for them holding it.