Taking paternity leave and worried about being seen as less serious

Got the greenlight for 6 weeks paternity leave when baby arrives next month. My boss says it's fine but the vibe at work is weird. Nobody in my department has actually taken the full time. I don't want to mess up my shot at the promotion next year but I also don't want to miss this time with my kid. This shouldn't be this stressful. Is anyone else dealing with this unspoken pressure?