The chance of a corporation mismanaging & accidentally letting their key get leaked is not a technical problem, like you pose. The HSM is fine. No one's going to crack your encryption.
But omgosh the Conways law implications of securing your nuclear waste, oops, i mean your hsm, are incredibly complex & long lived challenges. The odds of any given company accidentally messing up some month or another are quite high. If you have certificate pinning, you literally cannot escape your own mistakes. The ability to respond to mistakes should probably be taken as a necessary for most security footings of most organizations, and the whole point of certificate pinning is that response is impossible, that a cert is pinned in.
Certificate pinning makes declaring an incident where you’ve had your private key stolen effectively impossible.
Which means you’re going to end up sacrificing user security when it inevitably ends up in that situation.