TBH, as the Dockerfile based Lambda layers have a 20GB file size limit, I'm not sure the size matters as much.

It's certainly easier to figure out what's going on in a smaller container though! I've had to debug some nasty situation with layers in Python Lambdas before and it's not fun...