It is not "yours" it is the property of the government that issued it to you. No one (no employer or other government) has the right to retain it legally.

As for the exit permit argument. Great, if there is someone available to issue the exit permit whether it be a personal emergency or something more sinister.

Say for example there was some sort of natural disaster or other crisis and an exit permit could not be issued - how will you prove to your embassy or anyone else who you are and what your nationality is?

I'm glad you have confidence in your employer under those circumstances, I don't have the same in mine.