> ## Documentation Index
> Fetch the complete documentation index at: https://docs.julep.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Browser Use

> Learn how to use Julep browser automation capabilities

## Overview

This tutorial demonstrates how to:

* Set up browser automation with Julep
* Navigate web pages programmatically
* Execute browser actions like clicking and typing
* Process visual feedback through screenshots
* Create goal-oriented browser automation tasks

<div style={{position: 'relative', paddingBottom: '62.5%', height: '0'}}>
  <iframe src="https://www.loom.com/embed/d957274286664eeea426e9b7162b6365?sid=97234925-c4d9-42b7-b87c-4aea7f01587d" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen style={{position: 'absolute', top: 0, left: 0, width: '100%', height: '100%'}} />
</div>

## Task Structure

Let's break down the task into its core components:

### 1. Input Schema

First, we define what inputs our task expects:

```yaml theme={"dark"}
input_schema:
  type: object
  properties:
    goal:
      type: string
    agent_id:
      type: string
      description: The id of the agent to use for the browser automation
  required:
    - goal
    - agent_id
```

This schema specifies that our task expects a goal string describing what the browser automation should accomplish.

### 2. Tools Configuration

Next, we define the external tools our task will use:

```yaml [expandable] theme={"dark"}
tools:
- name: create_browserbase_session
  type: integration
  integration:
    provider: browserbase
    method: create_session
    setup:
      api_key: "YOUR_BROWSERBASE_API_KEY"
      project_id: YOUR_PROJECT_ID

- name: get_session_view_urls
  type: integration
  integration:
    provider: browserbase
    method: get_live_urls

- name: perform_browser_action
  type: integration
  integration:
    provider: remote_browser
    method: perform_action
    setup:
      width: 1024
      height: 768

- name: create_julep_session
  type: system
  system:
    resource: session
    operation: create

- name: session_chat
  type: system
  system:
    resource: session
    operation: chat
```

### 3. Main Workflow Steps

<Steps>
  <Step title="Create Julep Session">
    ```yaml theme={"dark"}
    - tool: create_julep_session
      arguments:
        agent: $ str(agent.id)
        situation: "The environment is a browser"
        recall: 'False'
    ```

    This step initializes a new Julep session for the AI agent. The session serves as a container for the conversation history and enables the agent to maintain context throughout the interaction.
  </Step>

  <Step title="Store Session ID">
    ```yaml theme={"dark"}
    - evaluate:
        julep_session_id: $ _.id
    ```

    After creating the session, we store its unique identifier for future reference.
  </Step>

  <Step title="Create Browser Session">
    ```yaml theme={"dark"}
    - tool: create_browserbase_session
      arguments:
        project_id: YOUR_PROJECT_ID
    ```

    This step establishes a new browser session using BrowserBase. It creates an isolated, headless Chrome browser instance that the agent can control.
  </Step>

  <Step title="Store Browser Session Info">
    ```yaml theme={"dark"}
    - evaluate:
        browser_session_id: $ _.id
        connect_url: $ _.connect_url
    ```

    We store both the browser session ID and connect URL in a single evaluation step.
  </Step>

  <Step title="Get Session View URLs">
    ```yaml theme={"dark"}
    - tool: get_session_view_urls
      arguments:
        id: $ _.browser_session_id
    ```

    This step retrieves various URLs associated with the browser session, including debugging interfaces and live view URLs.
  </Step>

  <Step title="Store Debugger URL">
    ```yaml theme={"dark"}
    - evaluate:
        debugger_url: $ _.urls.debuggerUrl
    ```

    We specifically store the debugger URL, which provides access to Chrome DevTools Protocol debugging interface.
  </Step>

  <Step title="Initial Navigation">
    ```yaml theme={"dark"}
    - tool: perform_browser_action
      arguments:
        connect_url: $ steps[3].output.connect_url
        action: "navigate"
        text: "https://www.google.com"
    ```

    This step navigates to Google's homepage to avoid sending a blank screenshot when computer use starts.
  </Step>

  <Step title="Start Browser Workflow">
    ```yaml theme={"dark"}
    - workflow: run_browser
      arguments:
        julep_session_id: $ steps[1].output.julep_session_id
        cdp_url: $ steps[3].output.connect_url
        messages:
        - role: "user"
          content: |-
            $ f"""
            <SYSTEM_CAPABILITY>
            * You are utilising a headless chrome browser to interact with the internet.
            * You can use the computer tool to interact with the browser.
            * You have access to only the browser.
            * You are already inside the browser.
            * You can't open new tabs or windows.
            * For now, rely on screenshots as the only way to see the browser.
            * You can't don't have access to the browser's UI.
            * YOU CANNOT WRITE TO THE SEARCH BAR OF THE BROWSER.
            </SYSTEM_CAPABILITY>
            <GOAL>
            * + {steps[0].input.goal} + NEWLINE + </GOAL> """
    ```

    Finally, we initiate the interactive browser workflow with system capabilities and user goal.
  </Step>
</Steps>

### 4. Run Browser Subworkflow

The `run_browser` subworkflow is a crucial component that handles the interactive browser automation. It consists of three main parts:

<Steps>
  <Step title="Agent Interaction">
    ```yaml theme={"dark"}
    - tool: session_chat
      arguments:
        session_id: $ _.julep_session_id
        messages: $ _.messages
        recall: $ False

    - evaluate:
        content: $ _.choices[0].message.content
        tool_calls: |-
          $ [ 
            { 
              'tool_call_id': tool_call.id, 
              'action': load_json(tool_call.function.arguments)['action'], 
              'text': load_json(tool_call.function.arguments).get('text'), 
              'coordinate': load_json(tool_call.function.arguments).get('coordinate') 
            } 
            for tool_call in _.choices[0].message.tool_calls or [] if tool_call.type == 'function']
    ```

    This step engages the AI agent in conversation, allowing it to:

    * Process and understand the user's goal
    * Plan appropriate browser actions
    * Generate responses based on the current browser state
    * Make decisions about next steps
  </Step>

  <Step title="Action Execution">
    ```yaml theme={"dark"}
    - foreach:
        in: $ _.tool_calls
        do:
          tool: perform_browser_action
          arguments:
            connect_url: $ steps[0].input.cdp_url
            action: $ _.action
            text: $ _.get('text')
            coordinate: $ _.get('coordinate')
    ```

    This component:

    * Iterates through planned actions sequentially
    * Executes browser commands (navigation, clicking, typing)
    * Handles different types of interactions (text input, mouse clicks)
    * Captures screenshots for visual feedback
  </Step>

  <Step title="Goal Evaluation">
    ```yaml [expandable] theme={"dark"}
    - evaluate:
        contents: >-
          $ [ \
            { \
              'type': 'image_url', \
              'image_url': { \
                'url': result['base64_image'], \
              } \
            } if result['base64_image'] is not None else \
            { \
              'type': 'text', \
              'text': result['output'] if result['output'] is not None else 'done' \
            } \
            for result in _]

    - evaluate:
        messages: "$ [{'content': [_.contents[i]], 'role': 'tool', 'name': 'computer', 'tool_call_id': steps[1].output.tool_calls[i].tool_call_id} for i in range(len(_.contents))]"

    - workflow: check_goal_status
      arguments:
        messages: $ _.messages
        julep_session_id: $ steps[0].input.julep_session_id
        cdp_url: $ steps[0].input.cdp_url
    ```

    This final part:

    * Assesses progress toward the user's goal
    * Determines if additional actions are needed
    * Maintains conversation context
    * Decides whether to continue or conclude the workflow
  </Step>
</Steps>

### 5. Check Goal Status Subworkflow

The `check_goal_status` subworkflow is a recursive component that ensures continuous operation until the goal is achieved:

<Steps>
  <Step title="Check Goal Status">
    ```yaml theme={"dark"}
    check_goal_status:
    - if: $ len(_.messages) > 0
      then:
        workflow: run_browser
        arguments:
          messages: $ _.messages
          julep_session_id: $ _.julep_session_id
          cdp_url: $ _.cdp_url
    ```
  </Step>
</Steps>

This workflow:

* Checks if there are any messages to process (`len(_.messages) > 0`)
* If messages exist, recursively calls the `run_browser` workflow
* Passes along the current session context and connection details
* Maintains the conversation flow until the goal is achieved
* Automatically terminates when no more messages need processing

This recursive pattern ensures that the browser automation continues until either:

* The goal is successfully achieved
* No more actions are needed
* An error occurs that prevents further progress

<Accordion title="Complete Task YAML" icon="code">
  ```yaml YAML [expandable] theme={"dark"}
  # yaml-language-server: $schema=https://raw.githubusercontent.com/julep-ai/julep/refs/heads/dev/src/schemas/create_task_request.json
  name: Julep Browser Use Task
  description: A Julep agent that can use the computer tool to interact with the browser.

  ########################################################
  ################### INPUT SCHEMA #######################
  ########################################################

  input_schema:
    type: object
    properties:
      goal:
        type: string
    required:
      - goal

  ########################################################
  ####################### TOOLS ##########################
  ########################################################

  tools:
  - name: create_browserbase_session
    type: integration
    integration:
      provider: browserbase
      method: create_session
      setup:
        api_key: "YOUR_BROWSERBASE_API_KEY"
        project_id: "YOUR_BROWSERBASE_PROJECT_ID"

  - name: get_session_view_urls
    type: integration
    integration:
      provider: browserbase
      method: get_live_urls
      setup:
        api_key: "YOUR_BROWSERBASE_API_KEY"
        project_id: "YOUR_BROWSERBASE_PROJECT_ID"

  - name: perform_browser_action
    type: integration
    integration:
      provider: remote_browser
      method: perform_action
      setup:
        width: 1024
        height: 768

  - name: create_julep_session
    type: system
    system:
      resource: session
      operation: create

  - name: session_chat
    type: system
    system:
      resource: session
      operation: chat

  ########################################################
  ################### MAIN WORKFLOW ######################
  ########################################################

  main:

  # Step #0 - Create Julep Session
  - tool: create_julep_session
    arguments:
      agent: $ str(agent.id)
      situation: "Juelp Browser Use Agent"
      recall: 'False'

  # Step #1 - Store Julep Session ID
  - evaluate:
      julep_session_id: $ _.id

  # Step #2 - Create Browserbase Session
  - tool: create_browserbase_session
    arguments:
      project_id: "c35ee022-883e-4070-9f3c-89607393214b"

  # Step #3 - Store Browserbase Session Info
  - evaluate:
      browser_session_id: $ _.id
      connect_url: $ _.connect_url

  # Step #4 - Get Session View URLs
  - tool: get_session_view_urls
    arguments:
      id: $ _.browser_session_id

  # Step #5 - Store Debugger URL
  - evaluate:
      debugger_url: $ _.urls.debuggerUrl

  # Step #6 - Navigate to Google
  # Navigate to google to avoid sending a blank 
  # screenshot when computer use starts
  - tool: perform_browser_action
    arguments:
      connect_url: $ steps[3].output.connect_url
      action: "navigate"
      text: "https://www.google.com"

  # Step #7 - Run Browser Workflow
  - workflow: run_browser
    arguments:
      julep_session_id: $ steps[1].output.julep_session_id
      cdp_url: $ steps[3].output.connect_url
      messages:
      - role: "user"
        content: |-
          $ f"""
          <SYSTEM_CAPABILITY>
          * You are utilising a headless chrome browser to interact with the internet.
          * You can use the computer tool to interact with the browser.
          * You have access to only the browser.
          * You are already inside the browser.
          * You can't open new tabs or windows.
          * For now, rely on screenshots as the only way to see the browser.
          * You don't have access to the browser's UI.
          * YOU CANNOT WRITE TO THE SEARCH BAR OF THE BROWSER.
          </SYSTEM_CAPABILITY>
          <GOAL>
          * + {steps[0].input.goal} + NEWLINE + </GOAL> """


  ########################################################
  ################# RUN BROWSER SUBWORKFLOW #################
  ########################################################

  run_browser:

  # Step #0 - Agent Interaction
  - tool: session_chat
    arguments:
      session_id: $ _.julep_session_id
      messages: $ _.messages
      recall: $ False

  # Step #1 - Evaluate the response from the agent
  - evaluate:
      content: $ _.choices[0].message.content
      tool_calls: |-
        $ [ 
          { 
            'tool_call_id': tool_call.id, 
            'action': load_json(tool_call.function.arguments)['action'], 
            'text': load_json(tool_call.function.arguments).get('text'), 
            'coordinate': load_json(tool_call.function.arguments).get('coordinate') 
          } 
          for tool_call in _.choices[0].message.tool_calls or [] if tool_call.type == 'function']

  # Step #2 - Perform the actions requested by the agent
  - foreach:
      in: $ _.tool_calls
      do:
        tool: perform_browser_action
        arguments:
          connect_url: $ steps[0].input.cdp_url
          action: $ _.action if not (str(_.get('text', '')).startswith('http') and _.action == 'type') else 'navigate'
          text: $ _.get('text')
          coordinate: $ _.get('coordinate')

  # Step #3 - Convert the result of the actions into a chat message
  - evaluate:
      contents: >-
        $ [ \
          { \
            'type': 'image_url', \
            'image_url': { \
              'url': result['base64_image'], \
            } \
          } if result['base64_image'] is not None else \
          { \
            'type': 'text', \
            'text': result['output'] if result['output'] is not None else 'done' \
          } \
          for result in _]

  # Step #4 - Convert the result of the actions into a chat message
  - evaluate:
      messages: "$ [{'content': [_.contents[i]], 'role': 'tool', 'name': 'computer', 'tool_call_id': steps[1].output.tool_calls[i].tool_call_id} for i in range(len(_.contents))]"

  # Step #5 - Check if the goal is achieved and recursively run the browser
  - workflow: check_goal_status
    arguments:
      messages: $ _.messages
      julep_session_id: $ steps[0].input.julep_session_id
      cdp_url: $ steps[0].input.cdp_url


  ########################################################
  ############## CHECK GOAL STATUS SUBWORKFLOW ##############
  ########################################################

  check_goal_status:

  # Step #0 - Check if the goal is achieved and recursively run the browser
  - if: $ len(_.messages) > 0
    then:
      workflow: run_browser
      arguments:
        messages: $ _.messages
        julep_session_id: $ _.julep_session_id
        cdp_url: $ _.cdp_url
  ```
</Accordion>

## Usage

Here's how to use this task with the Julep SDK:

<CodeGroup>
  ```python Python [expandable] theme={"dark"}
  from julep import Client
  import yaml
  import time

  # Initialize the client
  client = Client(api_key=JULEP_API_KEY)

  # Create the agent
  agent = client.agents.create(
    name="Julep Browser Use Agent",
    about="A Julep agent that can use the computer tool to interact with the browser.",
  )

  # Load the task definition
  with open('browser_task.yaml', 'r') as file:
    task_definition = yaml.safe_load(file)

  # Create the task
  task = client.tasks.create(
    agent_id=agent.id,
    **task_definition
  )

  # Create the execution
  execution = client.executions.create(
      task_id=task.id,
      input={
          "agent_id": agent.id,
          "goal": "Search for recent news about artificial intelligence"
      }
  )

  # Wait for the execution to complete
  while (result := client.executions.get(execution.id)).status not in ['succeeded', 'failed']:
      print(result.status)
      time.sleep(1)

  # Print the result
  if result.status == "succeeded":
      print(result.output)
  else:
      print(f"Error: {result.error}")
  ```

  ```js Node.js [expandable] theme={"dark"}
  import { Julep } from '@julep/sdk';
  import yaml from 'yaml';
  import fs from 'fs';

  // Initialize the client
  const client = new Julep({
    apiKey: 'your_julep_api_key'
  });

  // Create the agent
  const agent = await client.agents.create({
    name: "Julep Browser Use Agent",
    about: "A Julep agent that can use the computer tool to interact with the browser.",
  });

  // Parse the task definition
  const taskDefinition = yaml.parse(fs.readFileSync('browser_task.yaml', 'utf8'));

  // Create the task
  const task = await client.tasks.create(
    agent.id,
    taskDefinition
  );

  // Create the execution
  const execution = await client.executions.create(
    task.id,
    {
      input: { 
        "agent_id": agent.id,
        "goal": "Search for recent news about artificial intelligence"
      }
    }
  );

  // Wait for the execution to complete
  let result;
  while (true) {
    result = await client.executions.get(execution.id);
    if (result.status === 'succeeded' || result.status === 'failed') break;
    console.log(result.status);
    await new Promise(resolve => setTimeout(resolve, 1000));
  }

  // Print the result
  if (result.status === 'succeeded') {
    console.log(result.output);
  } else {
    console.error(`Error: ${result.error}`);
  }
  ```
</CodeGroup>

## Key Features

* **Browser Automation**: Performs web interactions like navigation, clicking, and typing
* **Visual Feedback**: Captures screenshots to verify actions and understand page state
* **Goal-Oriented**: Continues executing actions until the user's goal is achieved
* **Secure Sessions**: Uses BrowserBase for isolated browser instances
* **Interactive Workflow**: Uses run\_browser subworkflow for continuous interaction

## Next Steps

* Try this task yourself, check out the full example, see the [browser-use cookbook](https://github.com/julep-ai/julep/blob/main/cookbooks/advanced/06-browser-use.ipynb).
* To learn more about the integrations used in this task, check out the [integrations](/integrations/supported-integrations) page.

## Related Concepts

* [Agents](/concepts/agents)
* [Tasks](/concepts/tasks)
* [Tools](/concepts/tools)