Makes me wonder if Jimmy just didn't get those bonding chemicals that are talked about with a natural pregnancy. Jag put in the work, but that seems like the kind of oversight he could make.

The bonding chemicals are working against 60 years of semi-sociopathy. They got their work cut out with Jimmy.

As Jag said, Jimmy doesn't actually want to hurt his daughter, but he also can't help being himself. He lashes out when he's stressed, he exerted massive self-control by not taking it out on the helpless infant. That's Jimmy's way of showing he does care, by not acting like a psycho.