Actually it does matter Kareena... they are NOT allowed to hold your passport.. so you should demand it to be given to you... and if they say it is for security reasons - then tell them you are a grown up man and you can take care of your passport security by yourself.

The only time you should give them passport is just for a week to make any immigration arrangements... SO GO GET YOUR PASSPORT.. and dont listen to anyone who tells you it is a common practice.. it is NOT.