Why do I feel guilty for wanting alone time?

Legit question. My kids are great, I love them, but I just spent all Saturday afternoon at a coffee shop alone while my wife watched them and I felt GUILTY the entire time. Like I was abandoning my family or something. The moment I got home my daughter asked why I left and suddenly I'm the bad guy in my own head. Anyone else deal with this?