Can you point me to some example of generated text this model produced? Something similar in quality to that unicorn story from GPT-2?

but does it bring any new SOTA?

Looking at their tables, seems so. The code is open source, and there's demo at https://mosaickg.apps.allenai.org/