Employ HTTP 5xx retries on the client-side (well, they are a must-have anyway).
One minor word of caution on this one. While this is a very good idea you also have to be careful how this is implemented. The application needs to be aware of these retries or you can submit the same data multiple times and jobs can run-amuck. This can get expensive if the jobs are financial in nature. Some proxies can re-post data on 5xx's but you have to be sure that the receiving side can detect duplicate submissions or your PITA network timeouts can turn into very expensive job transactions. I'm sure all the developers here already understand this. I am posting this for the folks that implement middle-proxies that can re-post data such as haproxy, F5, netscaler, etc...
Overall good article. I have run into these same problems many times.
One minor word of caution on this one. While this is a very good idea you also have to be careful how this is implemented. The application needs to be aware of these retries or you can submit the same data multiple times and jobs can run-amuck. This can get expensive if the jobs are financial in nature. Some proxies can re-post data on 5xx's but you have to be sure that the receiving side can detect duplicate submissions or your PITA network timeouts can turn into very expensive job transactions. I'm sure all the developers here already understand this. I am posting this for the folks that implement middle-proxies that can re-post data such as haproxy, F5, netscaler, etc...
Overall good article. I have run into these same problems many times.