How Capable are Web Agents at Solving Common Knowledge Work Tasks? (2024)

3.1 WorkArena Tasks

In WorkArena, each task is coupled with a natural language goal that provides instructions to the agent. Each goal is automatically generated from a human-designed template filled with pre-defined values (menu name, field value, item specifications etc.), and explicitely provides all the information required to solve the task (examples in Figure2). Other key components of tasks include: i) validation functions and ii) oracle functions, a unique feature that distinguishes WorkArena from previous work. The validation functions offer real-time feedback to agents, identifying errors ranging from minor (e.g., unfilled mandatory fields) to critical (such as pushing invalid data to the database).The oracle functions are hand-crafted solutions that automatically complete the tasks using Playwright browser automation(Microsoft, 2023).They serve three purposes: (i)they ensure the feasibility of each task, (ii)they act as ground truth for agents that have learning capabilities, and (iii)they help maintain the benchmark’s longevity by making it easier to identify and adjust tasks impacted by future updates to the ServiceNow platform.Below, we outline the main categories of tasks included in the benchmark.

Lists: We consider 12 list-based tasks, which can be grouped into two categories: filtering and sorting. The former consists of using the UI to construct a complex filter with 1 to 5 conditions. The latter consists of using the UI to sort the list based on up to 3 columns. In both cases, the interaction with the UI is non-trivial and requires opening a hidden menu, adding the right number of conditions, and filling them out accordingly. There are 6 tasks of each type, each corresponding to a different data table (e.g., users, incidents). In both cases, client-side validation is used to verify that the resulting list satisfies the expected conditions. Together, these tasks yield 6,900 instances.

Forms: We consider 5 form-based tasks, which each consist of creating a new entry in a given data table. These tasks vary in complexity based on the number of fields that must be filled (from 1 to 26) and on the intricate properties of each form’s UI. For example, some forms require navigating through a set of tabs to expose hidden fields. Others use dynamic auto-completions fields, which require careful handling (see Figure3 for an example). In all cases, validation proceeds by querying the database to retrieve entries created by the agent and verifying that their values are as expected. Together, these tasks yield 5,000 instances.

Knowledge bases: The benchmark includes an information retrieval task that consists of searching the platform’s knowledge base to answer a question. Concretely, this requires searching with appropriate keywords and browsing the resulting articles to find specific information. These tasks are constructed by starting from a list of facts, generating articles containing each fact with GPT-4(OpenAI, 2023), and generating a series of questions that unambiguously ask for this fact. Then, validation proceeds by verifying if the answer returned by the agent is within a set of acceptable answers. For example, if the question is “What is the level of customer satisfaction?” and the answer is “8.5/10”, alternative answers such as “85%” or “8.5 out of 10” would be accepted. In total, this task yields 1,000 instances. Details on article generation and validation are given in §A.3.

Service catalogs: The benchmark includes 9 tasks that require navigating a catalog of products and ordering items with given specifications. Such tasks vary in complexity based on the number of item configuration options. In all cases, validation is done by querying the database to verify that the order request created by the agent includes the right items in the right amounts, with the expected specifications. Together, these tasks yield 3,550 instances.

Dashboards: The benchmark includes 4 retrieval tasks that require extracting numerical data from charts and (optionally) performing simple reasoning over them to answer a question. The complexity of such tasks depends on i) whether there are one or many charts on the page and ii) the reasoning to be done (e.g., extract the minimum).In all cases, validation consists of verifying if the agent’s response contains key numbers and labels referring to chart elements.Together, these tasks yield 1,862 instances.

Menus: We consider 2 menu-based tasks: i) navigating via the “All” menu and ii) impersonating users. The first consists of using the platform’s main menu to navigate to a given application. In this case, validation simply verifies that the agent has arrived at the expected location. The second consists of impersonating a user, a task commonly performed by IT support agents, where the agent logs into the platform as a given user to diagnose an issue. In this case, validation verifies that the expected user is logged in. Together, these tasks yield 1,600 instances.

A detailed list of all the tasks implemented in WorkArena is available in §A.1 Table6.

3.2 Challenges: the World Wild Web of Work

The ServiceNow platform poses a specific set of challenges for UI assistants, which we believe make WorkArena a complementary and meaningful benchmark for the community.

Non-standard, dynamic UIs: First, the web pages are heavily dynamic, and exhibit complex UI elements and ways of interacting with them. For example, the form to create an incident contains specific rules that can make some fields required or hidden, depending on the value of other fields (e.g., setting an incident’s status to “Resolved” requires filling its “Resolution notes”). Or, on some pages, the right-click can be overloaded to display a dynamic menu in certain areas. While these UI behaviors are not necessarily standard or best practices in web development, they are fairly common and representative of real-world enterprise software, which is not always designed with user accessibility in mind.

Non-standard, exotic HTML: Second, the ServiceNow platform relies on a complex combination of web technologies to implement its web pages, with nested iFrames, shadow DOMs, and proprietary Javascript APIs and HTML tags that do not necessarily adhere to existing web standards.¹¹1https://www.w3.org/standards/ This specificity would require strong out-of-distribution generalization for a UI assistant to successfully solve a task.

Large HTML: Third, the Document Object Model (DOM) of rendered web pages in the ServiceNow platform can be prohibitively large even for state-of-the-art language models, with a flat HTML text size that ranges between 40k and 500k tokens, even after a basic cleaning (removing scripts, styles and empty elements). Thus, even a conceptually simple task, such as finding the next element to click on the current page requires long context understanding, which is an active area of research in language models.

3.3 Availability

WorkArena is open-source and designed to be easily extended by the scientific community.²²2https://github.com/ServiceNow/WorkArenaTasks are executed on Personal Developer Instances, which are real cloud-based instances of the ServiceNow product, accessible for free through their developer program.It is important to note that although our benchmark is built on top of ServiceNow’s platform, it operates independently of any proprietary code.

4.1 Capabilities

BrowserGym implements the following capabilities.

Chat-based user interaction: One of the interaction modalities is a chat interface where the user and the web agent can exchange messages. In WorkArena, the goal of each task is provided as the initial user message, to which the agent can reply at any time. This allows for information retrieval tasks where a specific answer is expected from the agent, but also sequential tasks where user instructions change over time and are delivered sequentially in a way that mimics real-world use cases.

Augmented attributes: For every element on the current page BrowserGym provides a unique identifier (bid), its bounding box coordinates (left,top,right,bottom), and flags indicating whether it is visible and clickable. These attributes both provide a crude summary of the visual rendering of the UI and allow for unambiguous interaction with individual elements using their identifiers.

Rich observation space: at each time step, the observation space contains the content of the chat (list of messages), the currently open pages (list of URLs), the error message from the last action if any (stack trace), and a multi-modal view of the active web page: its HTML DOM snapshot (structured object), its accessibility tree or AXTree (structured object) as originally proposed byZhou etal. (2023), and a viewport screenshot (image). Both DOM and AXTree are obtained through Chrome’s CDP, and are enriched with augmented attributes (bid, coords, visible, clickable). These structured objects can typically be rendered as text for processing by a language model and can be combined with the screenshot for a vision-language model.

Rich action space: the action space is customizable and includes Python code, which can be restricted to specific set of pre-defined high-level primitives, such as bid-based actions (click(bid), type(bid,text), …), and coord-based actions (mouse_click(x,y), keyboard_type(text), …). Alternatively, the action space can allow for the execution of arbitrary Python code, including the entire Playwright API which gives the web agent maximum flexibility in interacting with the browser.For the complete list of high-level primitives available in Browsergym, refer to §B, Table10.

Multi-page navigation: BrowserGym natively supports web tasks that require multiple open pages (tabs, popups) and is also robust to web pages that employ nested encapsulation techniques such as iFrames and shadow DOMs. This robustness is essential to handle the heterogeneity of real-world websites and is missing in existing web environments.

4.2 An Ideal Experimental Framework

Flexible agent design: BrowserGym offers an extensive list of features but does not impose any restriction on how web agents should be implemented. The agent is responsible for using the provided observations or not (HTML, AXTree, screenshot, error message), deciding how to handle the history (past observations and actions), or deciding which action space it should be using (python, bid, coord, coord+bid). As such, with BrowserGym, researchers can easily experiment with new ideas and evaluate and compare a wide variety of web agents on the same set of tasks, such as text-only agents, vision-augmented agents, memory-augmented agents, and so on.

Minimal task design: BrowserGym reduces the burden of creating new benchmarks to a minimum. Implementing a new task in BrowserGym boils down to implementing four functions: setup(), teardown(), validate() and cheat() (an optional oracle function). The setup() function is responsible for initializing anything the task needs beforehand, such as creating database entries, navigating to the starting URL, authenticating, etc. Likewise, teardown() is responsible for cleaning up any resource that might have been created during the task’s execution. validate() is responsible for checking if the task’s goal was fulfilled, which can involve operations such as querying a database, validating the URL and the content of the current page, or looking at the messages sent by the agent in the chat. The method returns a reward, an optional user message for the chat, and a done flag indicating the end of the task. Finally, each task can optionally implement an oracle function, cheat(), that automatically solves the task using a hard-coded Playwright solution. This can notably be used to assess the feasibility of tasks.

Extensibility:BrowserGymis easily extensible to additional benchmarks. We currently support MiniWoB §B.2), WebArena (Zhou etal., 2023) and WorkArena. We encourage the community to add new benchmarks or new agents to this platform. We also offer support to reduce the friction of adding new benchmarks to the platform. Please reach out on GitHub for assistance.

5.1 Agent Design

We implement a simple web agent with chain-of-thought prompting(Wei etal., 2022b), and evaluate its performance across two axes: (1) the underlying LLM, and (2) the use of BrowserGym features. To study the effect of different design choices in our agents, we use flags to activate or deactivate certain features, such as use_thinking for chain-of-thought.

Observation space: Our observation space is composed of the goal, the current page’s HTML and/or AXTree,⁶⁶6On WebArena and WorkArena we only use AXTrees because HTML is prohibitively large. On MiniWoB we use both AXTree and HTML as it consistently gave the best performance. the currently focused element, and the error from the previous action if any. We activate / deactivate these features with flags use_last_error and use_focused_element, and we decide whether or not to augment each element with extra information using flags coords=center or coords=box for screen coordinates, and extract_visible_tag and extract_clickable_tag for whether elements are visible and/or clickable. Last, the flag filter_visible_only allow us to only include elements that are visible on the screen.

Action space: We leverage the flexibility of BrowserGym with a multi_action flag, which either allows agents to execute multiple actions per step (e.g., click(12) click(52)) or not. The action_set=bid flag only permits primitives that use element identifiers, while action_set=bid+coord also includes primitives operating with 2D coordinates (x,y). This is useful in MiniWoB, where certain tasks require clicking at specific positions in SVG images. Finally, we also experiment with the textual description of the action space which can either include a short or long description for each primitive (long_description), as well as examples of valid function calls or not (individual_examples).

History: To extend the horizon window of our agents, we experiment with use_action_history which includes in the prompt the history of actions since the beginning of the episode, use_error_history which includes all previous error messages, and use_think_history which re-injects the previous chain-of-though outputs into the prompt, thus creating an artificial memory that gives agents a chance to recall their previous thoughs.

Zero-shot examples: In the prompt, we provide a single generic example of how the chain of thought and action outputs should be formatted. This contrasts with other methods(Kim etal., 2023) where task-specific few-shot examples are provided, yet aligns with our objective of developing zero-shot agents able to solve a large range of new tasks.

Parse and retry: Once the LLM provides an answer, we have a parsing loop that can re-prompt the agent up to 4 times to make it aware of a parsing mistake. This can save the agent from making basic mistakes and is mainly useful for less capable LLMs such as GPT-3.5. Once parsed, the action is executed via BrowserGym, which moves to the next step.

Language models: Our study distinguishes between closed- and open-source LLMs. For the closed-source segment, we evaluate GPT-3.5 (gpt-3.5-turbo-1106, 16K context) and GPT-4o(OpenAI, 2023) (gpt-4o-2024-05-13, 128K context), through OpenAI’s API. In the realm of open-source LLMs, we sought a model that 1) understands code and HTML, 2) can manage a substantial context size, and 3) is instruction-finetuned. Our choice fell on Llama3-70b(Meta, 2024) (meta-llama-3-70B-instruct, 8K context), a recently released model that exhibits performances close to GPT-4. This model was deployed using Hugging Face’s Text Generation Inference (TGI) library on 4 A100 GPUs. We also explore the effect of providing the screenshot of the page using GPT-4o vision and Set-of-Mark(Yang etal., 2023) as proposed in WebVoyager(He etal., 2024).

Prompt truncation: We use a maximum prompt length of 40K tokens for GPT-4o, 15K for GPT-3.5 and 8K for Llama3. When the prompt is too large (Figure4), we progressively truncate the HTML and AXTree from the end until it fits the maximum allowed number of tokens.

5.2 Experimental Protocol

Standard Error: To be able to run a range of experiments under a fixed budget, we limit the number of seeds to 10 per task for MiniWoB and WorkArena, and 1 per task for WebArena. After averaging results over each benchmark (or subset of it), we usually observe a sufficiently low standard error to draw the needed conclusions. We use stratified bootstrapto obtain 1,000 samples of the mean and report the average and standard deviation of these means as success rate and standard error.

Max step: In all tasks we give our agents a maximum of 15 steps per episode, as in He etal. (2024). This ensures a low-performing agent will not wander for too long if they are incapable of solving the task. Note that 15 steps is considered sufficient for MiniWoB, but on WorkArena some tasks might require more than 15 steps to complete unless the agent runs in multi-action mode (see §A.1 Table6 for an upper bound).

Model selection: We evaluate all LLMs using a common agent code-base that we tune by activating / deactivating flags (see §5.1). For each LLM we find the best agent configuration via random search on MiniWoB and WorkArena. We then fix the final configuration (one per LLM), and re-run a full evaluation on all benchmarks with a different seed.⁷⁷7WebArena being deterministic, we don’t use it for tuning. Final configurations are reported in Table1.

5.3 Results

We report the final performance of our agents on MiniWoB(Liu etal., 2018), WebArena(Zhou etal., 2023) and WorkArena in Table2. We emphasize our key findings below.

GPT-4o’s Superiority: The data unequivocally demonstrate GPT-4o’s dominance over GPT-3.5 and Llama3, on all benchmarks. The performance gap is particularly striking on WorkArena, where GPT-4o achieves 43% success, in stark contrast to GPT-3.5’s 6.1% and Llama3’s 17.9%. The performance disparity between LLMs is significantly more pronounced in WorkArena and WebArena than in MiniWoB, aligning with findings on the emergent properties of AI systems(Wei etal., 2022a). As the complexity of task increases, the necessity for a more advanced LLM to achieve any score becomes apparent, with noticeable improvements in performance correlating with the enhancement of capabilities.

WorkArena poses a great challenge: Consistent with our expectations, our newly proposed benchmark, WorkArena, proves to be a significant challenge for current LLMs. While the tasks remain simple at a high-level (navigating menus, filling forms) and mostly require general knowledge, the difficulty arises primarily from the use of complex user interfaces from real-world software environments, resulting in long contexts and non-trivial interactions. Consequently, all agents exhibit low performance levels, with no agent achieving 100% success in one specific task. Notably, the list-based tasks which require interacting with a non-standard HTML widget (§A.2 Figure5) prove to be the most challenging ones with a clear 0% success rate for all LLMs, while being fairly simple for a human to complete.

State-of-the-art on MiniWoB and WebArena: Our GPT-4o agent demonstrates notably high performance on MiniWoB, achieving significantly greater success compared to other agents. The outcomes on the full benchmark and the WebGum subset (82.9%) surpass those of prior studies on zero-shot web agents (Assouel etal., 2023; Zeng etal., 2023), underscoring the effectiveness of both our agent design and the features offered by BrowserGym. On WebArena, while the score obtained by our GPT-4o agent is only 23.5%, it constitutes the best zero-shot performance reported so far, way above the 14.4% succes rate obtained using GPT-4 in the original paper(Zhou etal., 2023).

Open-source LLMs: While our Llama3 agent still performs poorer than a closed-source GPT-4o, its performance is remarkably high compared to a GPT-3.5 agent despite its shorter context window (8K vs 16K). Most notably, in preliminary experiments with a Llama2 model we could not obtain any success (flat 0%) on both WebArena and WorkArena, which makes us hopeful for the future of open-source LLMs. These results also indicate that web agent benchmarks, and WorkArena in particular, offer a great tool for evaluating emerging capabilities in LLMs, with larger performance gaps and more room for improvement compared to regular benchmarks (Touvron etal., 2023).

5.4 Ablation Study

We report in Tables3, 4 and5 an ablation study of each LLM on MiniWoB and WorkArena, for the most important agent flags.⁸⁸8The ablation study was done with a different seed, hence numbers differ slightly from those in Table2. We present our findings below.

Chain-of-though is crucial: In all of our 3 agents, asking the LLM to directly produce the next best action without chain-of-though is very detrimental to performance. This is clearly seen in Table5, with a drop of 10 points for Llama3 on both MiniWoB and WorkArena. This conclusion holds for all three LLMs very clearly.

More is not always better: As we gradually add more features, the prompt becomes longer and seems to overwhelm the LLM. This is mostly observed with GPT-3.5 and Llama3, the weaker LLMs, where seemingly harmless features such as a long_description or individual_examples in the action space description actually degrade performance. We hypothesise that since the AXTree is already large, adding too much non-crucial features is likely to distract the agent instead of helping it. We also note that in WorkArena, more features imply more prompt truncation for GPT-3.5 and Llama3 due to their limited context length (Figure4).

Memory can hurt: The use_think_history flag provides an interesting capacity to our agents, that of remembering their thoughts from previous time steps. We observed in practice that this features works, in the sense that the agents do re-use information about previous pages. However, such a feature is mostly useful when the task requires some form of memory, such as navigating a first page to find some information (e.g., an address), and then going to another page (e.g., Google maps) to type this information. On both WorkArena and MiniWoB the vast majority of tasks does not require navigation, and all the necessary information is always present on the current page. What we observe instead, is that agents with this feature activated tend to stick to decisions decided in early time steps, even erroneous ones, and are somehow less prone to self-correction. This is mostly observed with the GPT agents, Tables3 and4.

Deceiving multimodal performance: We report in Table2 a complimentary experiment with GPT-4o-V, a vision-augmented GPT-4o model, to assess the capacity of multi-modal models at utilizing the screenshot of the current web page for solving web tasks. We augment the screenshot with Set-of-Mark(Yang etal., 2023) as proposed in WebVoyager(He etal., 2024), and we keep the rest of the flags identical to the GPT-4o best performing agent. The results are somewhat deceiving, with very minor performance improvements on MiniWoB and WebArena. Similarly poor multimodal performances are reported by Xie etal. (2024), which could be explained by the lack of screen-related training data in current vision-language models. This leaves room for future development in this area.

2D features do not always help: Several MiniWoB tasks require precise 2D understanding of the UI and interaction with $x,y$ coordinates. While we observe a notable performance gain on this benchmark when giving GPT-4o additional coordinate features and actions (Table3), this does not translate to WorkArena where the tasks seem best solved using bid only. This seems to indicate that, while 2D actions are clearly required in specific cases (i.e., drawing on a web whiteboard), most web interactions actually do not require it, which can greatly simplify the agent design.

A.1 Tasks

This section provides additional details on each type of task included in the benchmark.

A.2 Task User Interface Examples

In this section, we provide an example of the typical user interface encountered for each category of task (Figures5, 6, 7, 9 and8). We omit “form” tasks, as such an example has already been presented in Figure3.

A.3 Knowledge Base Tasks – Additional Details

This task consists of searching a company knowledge base for specific information to answer a given question.Here, we explain how the knowledge base included in WorkArena is generated, how we produce the questions and answers used in each task instance, and how validation is performed.

B.1 Action Space

B.2 MiniWoB

As part of BrowserGym, we provide a port of the MiniWoB benchmark(Shi etal., 2017a; Liu etal., 2018). Whereas in MiniWoB the goal of each task is embedded in the HTML of the web page, in BrowserGym the goal is provided in a separate chat window, accessible to the agent. We therefore run a minimal Javascript snippet in the setup() function of MiniWoB tasks to extract the goal from the web page and place it in the chat instead. Apart from that change and removing the hard time limit per episode, porting MiniWoB to BrowserGym required minimal effort.