Partner doesn't understand why I need "alone time for my mental health"

So I've been trying to carve out like 30 mins a few times a week where I just go sit at a coffee shop by myself. No kids, no work emails, just me and a cup of coffee. My wife keeps making comments about it like "oh here comes dad's special me time" in this tone that makes me feel guilty for wanting space. I'm not avoiding the family, I work all day and come home and parenting happens. But apparently asking for a little decompression time makes me selfish? Anyone else dealing with this dynamic?