I'm wondering: how can understanding gradient descent help in building AI systems on top of LLMs? To mee it feels like the skills of building "AI" are almost orthogonal to skills of building on top of "AI"
I take your point in that they are mostly orthogonal in practice, but with that being said, I think understanding how these AI's were created is still helpful.
For example, I believe that if we were to ask the average developer about why LLM's behave randomly, they would not be able to answer. This to me exposes a fundamental hole in their knowledge of AI. Obviously one shouldn't feel bad about not knowing the answer, but I think we'd benefit from understanding the basic mathematical and statistical underpinnings on these things.
You can still understand that quite well without understanding backprop, though.
All you need is:
- Basic understanding of how a Markov chain can generate text (generating each word using corpus statistics on the previous few words).
- Understanding that you can then replace the Markov chain with a neural model which gives you more context length and more flexibility (words are now in a continuous space so you don't need to find literally the same words, you can exploit synonyms, similarity, etc., plus massive training data also helps).
- Finally, you add the instruction tuning (among all the plausible continuations the model could choose, teach it to prefer the ones human prefer - e.g. answering a question rather than continuing with a list of similar questions. You give the model cookies or slaps so it learns to prefer the answers humans prefer).
- But the core is still like in the Markov chain (generating each word using corpus statistics on the previous words).
I often give dissemination talks on LLMs to the general public and I have the feeling that with this mental model, you can basically know everything a lay user needs to know about how they work (you can explain things like hallucinations, stochastic nature, relevance of training data, relevance of instruction tuning, dispelling myths like "they always choose the most likely word", etc.) without any calculus at all; although of course this is subjective and maybe some people will think that explaining it in this way is heresy.
Sure, but it'd be similar to being a software developer and not understanding roughly what a compiler does. In a world full of neural network based technology, it'd be a bit lame for a technologist not to at least have a rudimentary understanding of how it works.
Nowadays, fine tuning LLMs is becoming quite mainstream, so even if you are not training neural nets of any kind from scratch, if you don't understand how gradients are used in the training (& fine tuning) process, then that is going to limit your ability to fully work with the technology.
Irs super complicated to use.
And you need an existing Telegram account to actually handle that cryptocurrency to buy these pseudo-numbers outside of the telephone namespace.
Guess what you need to register those. An actual working phone number.
> There's no technical reason, other than thinness, for the way Dell soldered the RAM down, for instance.
That's a sentiment that's been repeated a lot, and it's not fully true.
One important property that soldered-on RAM has is increased security. There's been a demonstrably practical way to break full-disk encryption with physical access to a turned on or sleeping computer by re-attaching the memory quickly to another computer. The keys then can be read from the memory.
That's not a vector you have anymore if the memory is not removable.
It’s impactful but in what direction? How many repairable or upgradeable laptops actually got repairs and upgrades? Or was it just wasted material and energy in production?
I don’t have the answer but these things can be surprising.
Some of the reparable, upgradable laptops get upgraded and/or repaired.
Every corporate IT organization I've ever talked to wants to be able to swap batteries, RAM and disks, and if they're easy enough, keyboards. They'd really prefer if the user could be sent a screwdriver and a replacement for batteries and RAM.
The more important reason this is not true is that there is technically feasible DIMM form factor for LPDDR4/4x/5/5x RAM. And there are actual power savings (~20% reduction in J/bit transferred) to be had for using LPDDR RAM as opposed to similar generation DDR RAM. Laptops that support replaceable memory have to use DDR4/DDR5 and therefore cannot take advantage of the power savings you get from LPDDR4/4x memory.
No amount of security or performance benefits can outweigh even the environmental/eWaste disaster alone that non-upgradable RAM chips/storage and lack of user replaceable batteries are responsible for.
Then there’s the cost thing. Have 8GB of RAM in your MacBook and want 16GB? Whoa, boy; that’ll cost you. That old computer’s pretty useless now, and we go right back to the eWaste problem.
Soldered RAM/storage/batteries are the epitome of greed in technology.
They prioritized the customer purchasing a whole new product over simply upgrading one component; creating a literal never-ending shit-stream of eWaste and products increasingly designed to stay properly functional for less time.
The law chases them; and instead of getting the message and smartening up; they find loopholes to get out of it.
There’s no excuse. It’s greed.
EDIT: More importantly - can we please stop making excuses for these greedy assholes, and start collectively working on change?
We need to start focusing on the planet, and this is a super easy start.
The problem is that because they can't be upgraded, they are no longer 'perfectly good'.
I recently was donated a 2010 MacBook Air with 4GB of RAM, and a paltry 128GB SSD.
What the hell is anyone supposed to do with that?! If the RAM wasn't soldered to the board I could probably at least upgrade it to 8GB and someone could at least use it for email and Facebook.
So I don't really get your point.
My original point is that these laptops are not perfectly good anymore - but if I could pop that 2010 Air with 8GB of RAM and a 512GB SSD, it could be useful for someone. A student, maybe. Boot it with Windows and you'd probably even be able to run a more recent OS.
I have an old Dell Laptop from 2010 (Core 2 Duo 2.66Ghz) with 4GB of RAM that ran Windows 10 as a Plex Server for years. With Macs of that era, you could stick a Windows 10 DVD and install it without any issues.
Pre-Covid (2019), my mom was using my old 2006 era Mac Mini with 1.25 GB of RAM running Windows 7 when she use to tutor. It was good enough to run Chrome and access Google Docs.
A 2010 laptop isn’t “perfectly good”. If the only reason you needed to upgrade to a Mac that was running a supported operating system was because you needed more RAM, you could get money back by selling it. The resell value of a 2010 MacBook - or any other computer is practically nil.
I wonder if MDN would be better served by branching out of Mozilla's grasp and becoming its own independent MDN Web Fondation or something. Mozilla seems to have a poor track record of maintaining anything as we saw when they fired most of the people working on MDN and other engineers despite being able to sell licenses to special editions of Firefox or other things like email, the whole non-profit Mozilla vs for profit Mozilla structure is just awful.
MDN is its only gem left outside of Firefox itself, it's good they're doing this though, but only if all proceeds go primarily towards MDN itself. I can foresee people unsubscribing over funding issues if they find out Mozilla foundation is just pocketing profits from this for things outside of MDN.
>despite being able to sell licenses to special editions of Firefox or other things like email
In 2021, there is not a chance in hell that any significant number of people are going to pay money for their web browser. At best (and this is very optimistic) it might make a few million dollars from dedicated HN-types, which is nowhere close to what Google gives them.
I actually would pay money for a Mozilla-branded email service, but running an email service isn't a walk in the park, and I expect all of the typical people would be complaining about Mozilla spending time and money on something that isn't Firefox, regardless of whether it's profitable (see also: all the complaining about Pocket).
Honestly, I wish that Mozilla had acquired Scroll (rather than Twitter) and started pushing it harder. That would be very in-line with their goals as an organization, and might eventually become a significant revenue source.
Scroll probably would have worked well with Pocket because the former could have increased user engagement (and, in turn, help better/newer stories end up in the top posts/popular category) while in the latter support for some paywalled content-providers could be added under a single subscription system.
Weird idea - since these big companies all voluntarily stopped documenting in favor of MDN, perhaps they should pay Mozilla for this (it's saving them money, after all).
If Mozilla is still having to do a paid option - MDN Plus - they are obviously not getting enough. Looks like around the price of one employee for the top funders.
Unless Mozilla is suddenly operating for profit and following the ol' "why get some of the money when you can get all of the money" business model.
You're thinking of the Mozilla Foundation, which is not-for-profit. The Foundation owns the Mozilla Corporation, which is "for-profit" and develops Firefox and MDN.
Money flows from the Corporation to the Foundation every year.
That's not the entire story. Google still has web.dev, which blurs the line between "web platform" features and Google-specific or "experimental" features. Neither of the two actively promote MDN either (web.dev doesn't mention it even where it would make sense).
This doesn't seem as much of an intentional decision of the two corporations to promote MDN as an independent resource but more likely the result of developers working for these companies refusing to compete with MDN.
Question from someone relatively clueless: Does that mean that NFTs also will use negligible amount of electricity once that's completely gone through?
Basically. Also, it's already possible to do NFTs with negligible amount using Immutable X layer 2 scaling solution on Ethereum - it's just that they seem to be doing it only for Gods Unchained right now. There will be even more other NFT scaling solutions.
Literally the first link in "Discover our content selection" leads to an expired domain name [1].
I'm not sure what to make of it but it does seem like a somewhat of a telltale for what's going to happen with my videos should I upload them to one of the PeerTube hosts.
