I'm using linux for some time and I installed tons of software without my package manager (thats unavoidable because not every package archive has every software).

In the end its all about trust. If you trust some web domain you can also trust their software. If that software is compromised you're out of luck. No package manager or walled apple garden can help you with that.