Nothing to add other than that I agree, and this is a easy to understand, correct way of framing the problem. But the framing probably won't convince exuberant AI folks. These tools are incredibly powerful, but they can only take you faster in the direction you were already going. Using them responsibly requires discipline, and not everyone has it, and we're about to undertake a grand social experiment in seeing how it plays out.
Unnecessary complexity was already a big problem in software prior to LLMs, and I think we're about to see it become, unbelievably, a much much bigger problem than before. My personal belief is that there are a lot of exponential harms that stack up as complexity increases:
1) Difficulty of understandinga codebase rises expontentially with code size. A 100kLOC codebase is more than 10x harder to understand than a 10kLOC codebase.
2) Large codebases get larger faster, classic snowball effect. Implementing a given feature against a larger codebase requires a larger change, even if it's the ideal, perfect change. Plus, ast shortcuts and bad abstractions make future shortcuts and bad abstractions much more likely.
3) Team size increases with codebase size, larger team size triggers Brooks law: number of communication paths grows quadratically with team size.
Prior to LLMs, the world already had large codebases that had essentially become emergent organisms that became so complex, the organizations that created them completely lost control of the ability to work in them effectively. Visual Studio and MSBuild, or Microsoft Teams come to mind. But historically, it was very expensive to get to this point. Now it's going to be much easier
The section coding.pdf has their code style guidelines, colloquially known as Cutler Normal Form, CNF for short. I'm conflicted on it. Definitely overly verbose, but you can't argue with the results of the NT team. Such a rigid style guide almost feels like the technical version of a dress code. And there's an idea called "enclothed cognition" which is like, if you wear a business suit to work, it exerts a subconscious influence that results in you taking the work more seriously, focusing your attention, etc: https://en.wikipedia.org/wiki/Enclothed_cognition
It's also important to remember that a ton of things we take for granted now simply didn't exist (source code control was in its infancy, merging was shit, syntax highlighting was minimal at best, compiling took time, etc).
I agree that in general people are influenced by their perception of themselves. For example I always pretend I'm a kernel programmer when hacking on small kernels. This did lend me a lot of patience to debug and figure things out, which I do not have for my work.
I imagine it would be frustrating to be the windows shell dev who has to investigate the torrent of bizarre memory corruption bugs that inevitably occur on Windhawk users’ machines after major OS updates. There’s really no avoiding it when you detour unstable “implementation detail” sort of functions across the taskbar/systray/start/etc. especially now that c++ coroutines are in widespread usage therein.
But to be fair, I understand the demand for products like this, because of several painful feature takebacks between 10 -> 11. It would be nice if cleaner approaches like wholesale shell replacement were still as straightforward as they were prior to windows 8. The “immersive shell” infra running in explorer + the opaque nature of enumerating installed UWPs + a bunch of other things make that almost impossible today.
> I imagine it would be frustrating to be the windows shell dev who has to investigate the torrent of bizarre memory corruption bugs that inevitably occur on Windhawk users’ machines after major OS updates.
What makes you think they will be bothered to investigate other's people bugs, when they don't investigate even their own ?
From what I've learned: stuff like this makes up a not insignificant portion of the crash reports that come through. This results in crash dumps that are useless at best because they just look like memory corruption or badly written malware. In my discussions with folks about this, an annoying number of people who run this sort of software either a) do not care that it makes developing Windows harder for the devs or b) actively want the usable signal for the Windows development teams to be low.
Using the UIA tree as the currency for LLMs to reason over always made more sense to me than computer vision, screenshot based approaches. It’s true that not all software exposes itself correctly via UIA, but almost all the important stuff does. VS code is one notable exception (but you can turn on accessibility support in the settings)
I’ve been working hard on our new component implementation (Vue/TS) to include accessibility for components that aren’t just native reskins, like combo and list boxes, and keyboard interactivity is a real pain. One of my engineers had it half-working on her dropdown and threw in the towel for MVP because there’s a lot of little state edge cases to watch out for.
Thankfully the spec as provided by MDN for minimal functionality is well spelled out and our company values meeting accessibility requirements, so we will revisit and flesh out what we’re missing.
Also I wanna give props (ha) to the Storybook team for bringing accessibility testing into their ecosystem as it really does help to have something checking against our implementations.
Agreed. I've noticed ChatGPT when parsing screenshots writes out some Python code to parse it, and at least in the tests I've done (with things like, "what is the RGB value of the bullet points in the list" or similar) it ends up writing and rewriting the script five or so times and then gives up. I haven't tried others so I don't know if their approach is unique or not, but it definitely feels really fragile and slow to me
I noticed something similar. I asked it extract a guid from an image and it wrote a python script to run ocr against it...and got it wrong. Prompting a bit more seemed to finally trigger it to use it's native image analysis but I'm not sure what the trick was.
I recently tried using Qwen VL or Moondream to see if off-the-shelf they would be able to accurately detect most of the interesting UI elements on the screen, either in the browser or your average desktop app.
It was a somewhat naive attempt, but it didn't look like they performed well without perhaps much additional work. I wonder if there are models that do much better, maybe whatever OpenAI uses internally for operator, but I'm not clear how bulletproof that one is either.
These models weren't trained specifically for UI object detection and grounding, so, it's plausible that if they were trained on just UI long enough, they would actually be quite good. Curious if others have insight into this.
Unnecessary complexity was already a big problem in software prior to LLMs, and I think we're about to see it become, unbelievably, a much much bigger problem than before. My personal belief is that there are a lot of exponential harms that stack up as complexity increases:
1) Difficulty of understandinga codebase rises expontentially with code size. A 100kLOC codebase is more than 10x harder to understand than a 10kLOC codebase. 2) Large codebases get larger faster, classic snowball effect. Implementing a given feature against a larger codebase requires a larger change, even if it's the ideal, perfect change. Plus, ast shortcuts and bad abstractions make future shortcuts and bad abstractions much more likely. 3) Team size increases with codebase size, larger team size triggers Brooks law: number of communication paths grows quadratically with team size.
Prior to LLMs, the world already had large codebases that had essentially become emergent organisms that became so complex, the organizations that created them completely lost control of the ability to work in them effectively. Visual Studio and MSBuild, or Microsoft Teams come to mind. But historically, it was very expensive to get to this point. Now it's going to be much easier
reply