TCP ACKs back up are needed even for downloads and can get crowded if the bandwidth in that direction full or crowded. If those tiny ACKs don't get through downstream may stall. Linux QoS can help, but only if your router is Linux and you can run the appropriate commands on it.