Manager just made a comment about my 'limited availability'

I leave at 4:30pm every day to pick up my kids. It's in my employment agreement and everything - I don't work late or weekends to compensate. Not ideal hours but I deliver results.

New manager said in a one-on-one that he's noticed I'm 'not as available as some of the other team members in the evenings.' This is my second week with him.

Should I be worried? Should I defend myself? Should I start staying later? I feel like I'm at a crossroads between my kids and my career growth, and I hate it.