It feels selfish, but you need to do it. Explain if you want, but you have to do what you have to do, and at least one person is sure to be unable to understand. It will all smooth over soon enough.