GitHub Copilot can now interact with desktop apps with computer use

Computer use is now available in public preview in GitHub Copilot CLI and the GitHub Copilot app on macOS and Windows.

Copilot can interact with desktop applications on your behalf (e.g., reading accessible app content and visual context, clicking controls, entering and editing text, pressing keys, scrolling, dragging, and navigating workflows across applications). This expands the tasks Copilot can help automate, including workflows in legacy and GUI-only software that do not provide an API, command-line interface, or MCP integration.

Copilot uses computer-use tools to navigate an expense-report workflow in Safari.

You remain in control. Copilot asks for approval before controlling an app, and you can review or reset apps that you have chosen to always allow. On macOS, computer use also guides you through the required Accessibility and Screen Recording permissions. Organization-managed settings can disable the feature.

To get started:

• In Copilot CLI, run /computer on. Use /computer show to check its status and /computer off to disable it.
• In the GitHub Copilot app, open Settings, select Computer Use, and turn on Enable Computer Use. You can also use /computer on.

Computer use works best when you describe the outcome you want, the applications involved, and any important constraints. For example, you can ask Copilot to summarize notifications in a browser, update content in a presentation, or move information through a workflow in a desktop application.

Learn more about computer use in GitHub Copilot CLI and the GitHub Copilot app

The post GitHub Copilot can now interact with desktop apps with computer use appeared first on The GitHub Blog.