Our son arrived 3 days ago and I'm terrified I'm going to break him

He's so tiny and fragile and every time I hold him my heart rate spikes because I'm convinced I'm doing something wrong. How am I supposed to know what he needs when he cries? My wife is way more calm about this than I am. Does this feeling go away or am I just gonna be nervous forever?