Ah, so this is why I suddenly got a bunch of email.

Hey all, site owner here. Thanks for the visits and all the fun stories! I really miss this era of computing. Feel free to let me know if you have something that should be added to the site.

Here's some site meta-history too:

https://telnet.org/history/

Cool site! I especially like the list of RFCs.

Just fyi, towel.blinkenlights.nl:23 still works for me, though I think maybe that's an IPv6 version, there's a note about ipv6 at the start that I was too slow to read. Maybe it should be re-listed? :)

Thanks! I did notice there are mixed stories about it working, and I got some email about it too. I'll check it out and make an update.

it's possible one of those emails might have been from me ;)

There's a lot of potential here. I'm hopeful this turns out well for the community of Abilene.

Writing code that runs down hole or otherwise connects back to the real world would be fun. Maybe I should pickup firmware skills. Good luck with your hiring!

It is a ton of fun! I've been hooked for years!

For Vespa there's a managed version hosted by the Vespa company in their cloud environment, and then the open source version is easily run locally or in any environment of your choosing. It takes some attention to detail, but it's quite flexible. I have a long running single node instance on an Intel NUC, but I've also run more complex cluster variations across different cloud environments.

This is a good place to start:

* https://hamel.dev/blog/posts/evals/#level-1-unit-tests

And more broadly:

* https://applied-llms.org/

Here's one I've used:

* https://www.together.ai/

Here are all the models:

* https://docs.together.ai/docs/chat-models

* https://docs.together.ai/docs/language-and-code-models

Unrelated to the core topic, I really enjoy the aesthetic of their website. Another similar one is from Fixie.ai (also, interestingly, one of their customers).

This was my first thought too, after reading through their blog. This feels like a no-frills website made by an engineer, who makes things that just work.

The documentation is great, I really appreciate them putting the roadmap front and centre.

Yes, I like the turboxyz123 animation and contrast to the minimalist website (reminds me of the zen garden with a single rock). I think people forget nowadays in their haste to add the latest and greatest react animation, that too much noise is a thing.

This website rocks

what does fixie do these days?

They pivoted, but will probably pivot back to their original quest.

Nah, we're pretty happy with the new trajectory. :)

Yeah! fast, clean, cool, unique.

This will get you the first 80%:

* Any solid search engine (bm25 + embeddings and hnsw)

* Any api to a model (gpt3.5, gpt4, claude, etc)

* Some middleware to call search then build the prompt

Then the remaining:

* Create an eval dataset, then tune the search and the prompt as needed

For me, it was a bit different, and it comes from a perspective that's a blend of cognitive science and computer science:

Complex systems can be created through the composition of simple processes which are easily explained or modeled. Sometimes there are mysterious emergent properties in the overall system, even when we can explain the components. Other times, through investigation / science / engineering, we are finally able to explain the entire system. It might lose a little of the magic or mystery as a result, but the system itself didn't change. Instead our perspective and understanding changed.

On that note, until we can fully explain some of the workings of our own minds, I'm reluctant to write off "just predicting the next token" as an unimportant process. It's one way to explain LLM inference simply, but it doesn't eliminate the importance. It also doesn't account for as-yet unexplained things which may be happening as a part of training.

In terms of "the moment", I would imagine it happened during development inside Google (Lamda) or OpenAI (GPT2/3).

More technically, here's one of the key papers discussing the topic (from google):

* https://arxiv.org/abs/2206.07682

Emergent Abilities of Large Language Models

Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities of large language models. We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. The existence of such emergence implies that additional scaling could further expand the range of capabilities of language models.

Version history (for relevant dates):

   [v1] Wed, 15 Jun 2022 17:32:01 UTC (59 KB)
   [v2] Wed, 26 Oct 2022 05:06:24 UTC (88 KB)