Milk that enters the food supply from a diary farm to retailers should absolutely be pasteurized, but farms shouldn't be raided by state police if someone wants to accept the risks and come purchase raw milk for whatever reason.
When it comes to getting sick yourself, sure that's reasonable enough. Right now though we have H1N1 in the milk supply, so no I'm not really willing to accept another pandemic. Your freedoms and my freedoms are balanced in tension against each other, and something like a pandemic virus that you can get from raw milk and then spread tips the scales away from your freedoms.