Took paternity leave and it's complicated

Had 4 weeks off when my son was born. First two weeks were amazing. By week 3 I was losing my mind and my wife could tell. By week 4 I was checking work emails constantly and couldn't wait to get back. Feel like a failure as a dad for that. Everyone says cherish it but my brain needs the structure and adult interaction. Anyone else feel this way or am I just broken?