Have you looked into HSA architecture that helps to remove this latency? I think this is the direction Intel will move to in a few years.
We are actively looking into this.