Say "go on then, send me back". If he is serious he will send you back and if he is not serious he will stop saying it. Either way you will not have to worry about your boss threatening you any more.