Took paternity leave and my job wants me back early

I'm 3 weeks into 8 weeks and they're already asking when I'm coming back. Baby was only 3 days old when I got the email. I feel guilty staying home but I also feel guilty for them. This is supposed to be protected time. Anyone have advice on not caving to the pressure?