It would be slower, but I'd disagree that it's too slow for that to work. In fact, I bet it could write something like autohotkey scripts to accomplish what it needs to do. You wouldn't have to have video and slowly move your mouse across the screen. You could get a screenshot, figure out where to move the mouse, then move the mouse to those coordinates and press left mouse button, take a screenshot to confirm the app is open, etc.

Having said that, anything that can be accomplished by opening a terminal should just be done there as it would be faster. In the short term though, there's lots of applications that are designed for humans that it would be great for LLM's to be able to interface with. Maybe in the long term they'll just write their own applications to accomplish something we'd normally need a gui for. Maybe there will be interfaces that have a human viewable component but most of the controls will gone. Like imagine a 3D modelling application that just has a viewer with just a few buttons to move the view around (It'll be easier to just spin the object to an angle yourself then say it.) But you'll have pointing and painting tools to help collaborate with the AI. ::draw a circle around a part of the mesh:: Make this area a little rougher. ::point to a leg, then draw a line coming out in a curve:: Have a tooth-like spike come out right here. Etc.

It'll be neat to see where this all goes, I suspect that UIs will radically change but in the near-term I'm sure there will be stop-gaps using current tech, too.