Ex wants to change custody agreement last minute

My ex is now saying she needs to swap weekends because her new boyfriend wants to go on a trip. I have my kids every other weekend and honestly that's the only real time I get with them since the divorce. This is the third time she's asked to switch in the last two months. Do I have to let her? My lawyer said technically no but I don't want to be the bad guy. What would you guys do?