	In Browser
	StumbleUpon
	del.icio.us
	Google
	Google Buzz
	reddit
	LinkedIn

	Facebook
	Twitter
	Linkedin
	E-Mail

Generative AI > Anthropic Claude API > Claude Computer Use API

Claude Computer Use API

Author: Venkata Sudhakar

Claude's Computer Use API (currently in beta) lets Claude control a computer by taking screenshots, moving the mouse, clicking buttons, and typing text. For ShopMax India's internal operations team, this enables automating repetitive desktop tasks like updating product listings in legacy systems, filling supplier order forms, or navigating internal admin portals that have no API.

The Computer Use API provides three built-in tools: computer (screenshot, mouse, keyboard), text_editor (view and edit files), and bash (run shell commands). Claude receives a screenshot of the current screen state, decides what action to take, issues a tool_use block with the action, your code executes it, takes a new screenshot, and the loop continues until the task is done. You control the actual execution environment - Claude only decides WHAT to do.

The example below simulates ShopMax India using Computer Use to automate a product price update task. It demonstrates the action loop pattern with screenshot-action cycles. Note: this requires the anthropic[computer-use] package and a real display environment in production.

import anthropic
import base64

client = anthropic.Anthropic()

computer_tools = [
    {
        "type": "computer_20241022",
        "name": "computer",
        "display_width_px": 1280,
        "display_height_px": 800,
        "display_number": 1
    }
]

def take_screenshot():
    # In production: use pyautogui or scrot to capture the real screen
    # Returning a placeholder base64 PNG for demonstration
    placeholder_png = (
        "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk"
        "YPhfDwAChwGA60e6kgAAAABJRU5ErkJggg=="
    )
    return placeholder_png

def execute_action(action_type, params):
    # In production: use pyautogui to perform the actual action
    if action_type == "screenshot":
        return take_screenshot()
    elif action_type == "left_click":
        print("Clicking at:", params.get("coordinate"))
    elif action_type == "type":
        print("Typing:", params.get("text"))
    elif action_type == "key":
        print("Key press:", params.get("text"))
    return None

task = "Update price of Sony 65-inch TV to Rs 87999 in the ShopMax admin portal"
messages = [{"role": "user", "content": task}]

for step in range(3):
    screenshot_b64 = take_screenshot()
    if messages[-1]["role"] == "user" and step > 0:
        messages[-1]["content"] = [{
            "type": "tool_result",
            "tool_use_id": last_tool_id,
            "content": [{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": screenshot_b64}}]
        }]

response = client.messages.create(
        model="claude-opus-4-7",
        max_tokens=1024,
        tools=computer_tools,
        messages=messages
    )

print("Step", step + 1, "- Stop reason:", response.stop_reason)

if response.stop_reason == "end_turn":
        for block in response.content:
            if hasattr(block, "text"):
                print("Claude says:", block.text)
        break

for block in response.content:
        if block.type == "tool_use" and block.name == "computer":
            last_tool_id = block.id
            action = block.input.get("action")
            print("Action:", action, block.input)
            execute_action(action, block.input)

messages.append({"role": "assistant", "content": response.content})
    messages.append({"role": "user", "content": ""})

print("Computer use loop complete")

It gives the following output,

Step 1 - Stop reason: tool_use
Action: screenshot {}
Step 2 - Stop reason: tool_use
Action: left_click {'coordinate': [640, 400]}
Clicking at: [640, 400]
Step 3 - Stop reason: end_turn
Claude says: I can see the admin portal. I need a real screenshot to proceed with updating the price to Rs 87999.
Computer use loop complete

In production at ShopMax India, run Computer Use inside a sandboxed Docker container with a virtual display (Xvfb) to prevent Claude from accessing unintended system resources. Always define a maximum step count (typically 20-50) to prevent infinite loops. Review screenshots at each step before sending them back to detect if Claude has navigated to an unexpected page. Computer Use works best for repetitive, well-defined tasks with stable UI layouts - avoid it for complex decision-making workflows that should use tool-based APIs instead.

Send your comments, suggestions or queries regarding this site to [email protected].