Even though you lose the filesystem-based security, you can still use SO_PEERCRED or getpeereid and validate the caller's UID is what you expect, something which Linux doesn't support on localhost TCP sockets. Requiring the client's UID (and maybe GID too) to be the same as your own is a sane default for services intended for per-user usage.