My son won't sleep unless I'm holding him

So my boy is 3 weeks old and I can't put him down without him screaming. Wife is exhausted and keeps saying I need to let him cry it out but he's only 3 weeks old, that seems too early right? I've been getting like 2 hours of sleep a night because he'll only nap on my chest. Anyone else deal with this? Does it get better or am I gonna be a human pacifier forever?