Ex won't stick to custody schedule and I'm losing it

My ex has moved the kids' pickup time THREE times this month without asking. I plan my whole week around having them Wed-Fri and she just texts me an hour before saying she needs them earlier. When I try to talk about it she says I'm being rigid and the kids don't need a "schedule" but honestly I think they do. Am I wrong to push back on this? I don't want to be that guy but I also have work commitments I'm arranging around our agreement.