Interesting. That doesn't seem to be specified in the RFC1122 standard or the Linux ip(7) docs, but it's an explicit special case in the kernel (ip_route_output_key_hash_rcu):
if (!fl4->daddr) {
fl4->daddr = fl4->saddr;
if (!fl4->daddr)
fl4->daddr = fl4->saddr = htonl(INADDR_LOOPBACK);
...