Went from 'freedom' to 'I panic when I'm alone'

This is probably weird to admit but when I get a rare evening to myself now I kind of hate it. Used to dream about having the house to myself, just me and whatever I wanted. Now when my wife takes the kids somewhere I get anxious after like an hour. Feel like I'm missing something or someone needs me. I've become someone's whole world and now I can't function without that responsibility. Miss the independence I guess but gained something that actually matters.