That is not what the article argues.

Haha, very true. Exactly as described in the article.

This is true with one caveat.

In most cases, e.g. with regular ML, evals are easy and not doing them results in inferior performance. With LLMs, especially frontier LLMs, this has flipped. Not doing them will likely give you alight performance and at the same time proper benchmarks are tricky to implement.

This is a very good point. When I came in, the founder did a lot of evaluation based on a few prompts and with manual evaluation, exactly as described. Showing the results helped me underline the fact that "works for me" (tm) does not match the actual data in many cases.

Doesn't this depend a lot on private vs company usage? There's no way I could spend more than a few hundreds alone, but when you run prompts on 1M entities in some corporate use case, this will incur costs, no matter how cheap the model usage.

It's not you, it's the HN hug of death. There's so much load on the server, I'm barely able to download the redis image I need for caching...

Thanks. Will take a look.

Depends on your remaining budget ;)

That is absolutely right. :)

I've skipped that in the article, but absolutely!

Fixed, thanks. Not a native speaker.