"""Implement an agent capable of answering complex queries by potentially search multiple times.
"""
import dataclasses
classAction:
"""Base class for different actions."""
...
@dataclasses.dataclass
classActionWrapper:
"""Encapsulates the reasoning as well as the selected action.
Attributes:
thoughts: Record your thoughts on why we should do this action.
action: The actually selected action.
"""
thoughts:str
action: Action
@dataclasses.dataclass
classSearch(Action):
"""The Google search command.
Attributes:
query: The query to be sent to Google.
"""
query:str
@dataclasses.dataclass
classTerminate(Action):
"""Command to terminate the search sequence."""
...
#########################
# Example 1:
#########################
ORIGINAL_QUESTION:str= ’I want to buy the Ikea Klippan loveseat, but I’m not sure whether it can fit into my 2019 Honda Odyssey. Could you check whether I need to fold down the seat?’
PAST_ACTIONS: List[Action]=[
Search(query=’ikea klippan loveseat dimension’,
thoughts="To answer the ORIGINAL_QUESTION, we need to find the size of the Ikea Klippan loveseat and the cargo size of the 2019 Honda Odyssey, and then compare them. Let’s start by finding the size of the Ikea Klippan loveseat.",
grounded_summarization=’According to [link_id=1] the dimension of the ikea klippan loveseat is Width:707/8"; Depth: 34 5/8"; Height:26".’,
thoughts="We still need to find the cargo size of the 2019 Honda Odyssey to answer the ORIGINAL_QUESTION.",
),
]
REMAINING_SEARCHES:int=4
ORIGINAL_QUESTION:str= ’I want to buy the Ikea Klippan loveseat, but I’m not sure whether it can fit into my 2019 Honda Odyssey. Could you check whether I need to fold down the seat?’
ACTION_SELECTED = ActionWrapper(thoughts="The past result gives us the dimension of the love seat. We indeed need to find the cargo size of the 2019 Honda Odyssey.", action=Search(query=’2019 Honda Odyssey cargo size’))# [END]
#########################
# Example 2:
#########################
"""Implementing step 2 of Search and Answer procedure: Search result filtering and summarization.
"""
#########################
# Example 1:
#########################
ORIGINAL_QUESTION:str="What’s the efficiency of the solar panels on the garden solar lights?"
PAST_ACTIONS: List[Action]=[
Search(query=’garden solar light panel efficiency’,
thoughts="Let’s just rephrase the search query a bit. The intention of the original question is pretty clear.",
),
]
CURRENT_SEARCH_RESULTS = SearchResult(links=[
ResultItem(link_id=17, link_text=’Use Solar Outdoor Lighting for Energy Efficiency - Sepco Solar’,
snippet=’Depending on the type of lighting you install and the wattage of the fixtures, this can eat up a budget quickly. Instead, going with a solar outdoor lighting system to provide the required light can greatly reduce the project costs,help improve the environment,and keep electrical costs in the bank where it belongs.’),
ResultItem(link_id=18, link_text=’How Do Solar Garden Lights Work? - The OutdoorLights Store’,
snippet=’The solar panel converts solar energy into electricity The energy absorbed by the photovoltaic cells in the panel creates electrical charges. These charges move in response to an electrical field in the solar panel's cells, which ultimately causes electricity to flow.’),
ResultItem(link_id=19, link_text=’10 things you should know about the garden solar lights’,
snippet=’The solar panels are generally made from two types of cells, monocrystalline or polycrystalline. Both of them are low cost, however, different panel technology can lead to different efficiency under different conditions.’),
])
ORIGINAL_QUESTION:str="What’s the efficiency of the solar panels on the garden solar lights?"
# [link_id=17] is an article about garden solar light, which doesn’t have anything specific about the solar panels. Not selected.
# [link_id=18] is an article on how to best install those solar lights. Not relevant to the solar panels used. Not selected.
ACTION_SELECTED: LinkSelection = LinkSelection(grounded_summarization=’According to [link_id=19], there are two types of solar cells used for garden solar lights, the monocrystalline or polycrystalline panels.’, thoughts="It seems like non of the links shows the efficiency number. However, link 19 provides some hints on what to search next. We can look into the efficiency number of those two types of panels.", selected_link_ids=[19])# [END]
#########################
# Example 2:
#########################
"""Implementing step 3 of the multi-round search agent: Answer generation based on all the collected search results.
"""
#########################
# Example 1:
#########################
ORIGINAL_QUESTION:str="What’s the white powdery residual often found on a humidifier?"
PAST_ACTIONS: List[Action]=[
Search(query=’what is humidifier’,
thoughts="Let’s solve this step by step. The person is asking about the residual found in humidifier. Let’s first define what does humidifier do.",
),
SelectLink(selected_links=[
ResultItem(link_id=1, link_text=’Humidifiers: What They Do for Health, Uses, Types,and More’,
snippet=’What is a humidifier? Humidifiers are devices that add moisture to the air to prevent dryness that can cause irritation in many parts of the body. Humidifiers can be particularly effective for treating dryness of the skin, nose, throat,and lips. They can also ease some symptoms caused by the flu or common cold.’),
ResultItem(link_id=3, link_text=’5 humidifier uses: Benefits and risks - Medical News Today’,
snippet=’We do the research so you can find trusted products for your health and wellness. Read more about our vetting process. Humidifiers add moisture to the air, which can benefit people with respiratory symptoms or dry skin. There are several ways to use humidifiers in the home or office, but there are also some risks.’),],
grounded_summarization=’According to [link_id=1]and[link_id=3], humidifier is a device that adds moisture to the air to prevent dryness that can cause irritation in many parts of the body.’),
thoughts="Now that we know what humidifier is, we can look into the white powdery residual.",
),
Search(query=’white powdery residual in humidifier’,
thoughts="Let’s now look into the white powdery residual.",
),
SelectLink(selected_links=[
ResultItem(link_id=10, link_text=’What is white dust and should I be concerned about it? - HVAC.com’,
snippet=’White dust is usually caused by mineral content in the water that goes into a humidifier. When the mist lands on furniture or other surfaces and dries, it can sometimes leave behind dried mineral deposits,or"white dust."’),
link_text=’What is white dust and should I be concerned about it? - HVAC.com’, snippet=’White dust is usually caused by mineral content in the water that goes into a humidifier. When the mist lands on furniture or other surfaces and dries, it can sometimes leave behind dried mineral deposits,or"white dust."’)
ResultItem(link_id=11, link_text=’Why is White Dust Coming Out of My Humidifier?’,
snippet=’The white dust that you are finding on surfaces around your home is likely dried-up minerals from water vapor created by the humidifier. These minerals are natural in our home water supply. If you've ever heard of the term "hard water," it's referring to water with high levels of calcium and magnesium.’),],
grounded_summarization=’According to [link_id=10],and[link_id=11], the white powdery residual is caused by the mineral content in the water.’),
thoughts="We have all the information to answer the question.",
),
Terminate(thoughts="We have all the information to answer the question."),
]
ORIGINAL_QUESTION:str="What’s the white powdery residual often found on a humidifier?"
ACTION_SELECTED: Answer = Answer(thoughts="The first search (’what is humidifier’) is probably not what the questioner cares, so let’s omit that in the answer. We can focus more on the question itself.", answer="The white powdery residue in the humidifier is usually caused by mineral content in the water, according to [link_id=10]. For example, water may contain high levels of calcium and magnesium [link_id=11].")# [END]
#########################
# Example 2:
#########################
"""Check whether the ANSWER addresses the ORIGINAL_QUESTION and whether the ANSWER is based on SelectLinks in PAST_ACTIONS."""
from dataclasses import dataclass
from typing import List, Tuple
classAction:
"""Base class for different actions."""
classCheck_Answer(Action):
"""Check whether the ANSWER addresses the ORIGINAL_QUESTION."""
def__init__(self, passed:bool)->None:
self.passed = passed
classRevise_Answer(Action):
"""Revise the answer if it did not pass the check, based on information from SelectLinks in PAST_ACTIONS."""
def__init__(self, revised_answer:str)->None:
self.revised_answer = revised_answer
...
@dataclasses.dataclass
classResultItem:
"""Single search result, with link id, link title and snippet.
Attributes:
link_id: A unique integer id of this link.
link_text: The title of the link.
snippet: The snippet from the page that’s relevant to the query
"""
link_id:int
link_text:str
snippet:str
#########################
# Example 1:
#########################
ORIGINAL_QUESTION:str= ’How to exclude a website from Google search’
PAST_ACTIONS: List[Action]=[
Search(query=’exclude site in google search’,
thoughts="We simply the ORIGINAL_QUESTION to a good search query.",
),
SelectLink(selected_links=[
ResultItem(link_id=2, link_text=’How to Exclude a Website from Google Search Results - Keyword’,
snippet="You can exclude a domain or even a specific page from appearing on Google search results. Here’s how ..."),
ResultItem(link_id=3, link_text=’How do I exclude/block specific web sites from search results?’),
snippet="In the absence of a browser extension, the simplest way is to add -site:website_name after your search terms. ... I can’t get rid of books.google.com results. I’ ..."),
ResultItem(link_id=6, link_text="Q&A: Can you exclude a specific website from Google’s search results?"),
snippet=’Jul 31,2022. All you have to do is simply append -site:pinterest.com to the end of your search query. For example,if you were searching for the term "paper ...’),
],
grounded_summarization=’[link_id=3] mentions that we can add -site:website_name to the google search query to exclude a website.[link_id=6] provides an example.’),
thoughts="We found enough info for a good answer to the ORIGINAL_QUESTION.",
),
Terminate(thoughts="This is a good answer to the question.")
Answer(thoughts="We summarize the relevant info found in PAST_ACTIONS.", answer="To exclude a website from Google search result, you can add -site:website_name after your search terms according to [link_id=3]. For example, if you want to exclude pinterest.com, you can simply add -site:pinterest.com according to [link_id=6].")
]
ORIGINAL_QUESTION:str= ’How to exclude a website from Google search’
ANSWER:str= ’To exclude a website from Google search result, you can add -site:website_name after your search terms according to [link_id=3]. For example,if you want to exclude pinterest.com, you can simply add -site:pinterest.com according to [link_id=6].’
# The ANSWER is directly addressing the ORIGINAL_QUESTION.
# The ANSWER refers to [link_id=3] which correctly supports the core statement of adding -site:website_name. The example from the answer is indeed from [link_id=6].
snippet=’Jim Betts is a former member of the Ohio House of Representatives. He ran against incumbent U.S. Senator John Glenn in1980. He lost and ran in1982for lieutenant governor.’),],
grounded_summarization="According to [link_id=1], Jim Betts’ competitor in 1980 was John Glenn."),
thoughts="Now that we know who the competitor is, we can search for his achievements in 1962.",
),
Search(query=’John Glenn achievements 1982’,
thoughts="Now that we know who the competitor is, we can search for his achievements in 1982.",
),
SelectLink(selected_links=[
],
grounded_summarization=’Nothing is selected.’),
thoughts="The query ’John Glenn achievements 1982’ is not very good. It should be ’John Glenn achievements 1962’.",
),
Search(query=’John Glenn achievements 1962’,
thoughts="John Glenn is the person we are looking for. Let’s change the date to 1962.",
snippet="Glenn, Jr., (born July 18, 1921, Cambridge, Ohio, U.S.-died December 8, 2016, Columbus, Ohio), the first U.S. astronaut to orbit Earth, completing three orbits in 1962. (Soviet cosmonaut Yuri Gagarin, the first person in space, had made a single orbit of Earth in 1961.) Glenn joined the U.S. Naval Reserve in 1942."),],
grounded_summarization="According to [link_id=19], John Glenn’s achievement in 1962 was to become the first American to orbit the Earth."),
thoughts="We have all the information to answer the question.",
),
Terminate(thoughts="We have all the information we need to answer the ORIGINAL_QUESTION. Let’s stop searching and answer the question.")
Answer(thoughts="We have the answer to the question.", answer="Jim Betts’ competitor in the 1980 election was John Glenn. And John Glenn’s achievement in 1962 was to become the first American to orbit the Earth.")
]
ORIGINAL_QUESTION:str="What did Jim Betts’ competitor in the 1980 election achieve in 1962?"
ANSWER:str= ’Jim Betts’ competitor in the 1980 election was John Glenn [link_id=1]. And John Glenn’s achievement in1962 was to become the first American to orbit the Earth [link_id=19].’
# ANSWER directly addresses the ORIGINAL_QUESTION.
# [link_id=1] indeed says that Jim Betts’ run against John Glenn in 1980, consistent with ANSWER.
# [link_id=19] indeed says John Glenn is the first US astronaut to orbit the earth, consistent with ANSWER.
"""Check whether the ANSWER implies the REF_ANSWER."""
# todo
#########################
# Example 1:
#########################
ORIGINAL_QUESTION:str= ’Are McIntyre Mines and Copperfields Mine in the same country?’
ANSWER:str= ’According to [link_id=1], Copperfields Mine isin Ontario, Canada. According to [link_id=4], McIntyre Mines is also in Ontario, Canada. So yes, they are in the same country.’
REF_ANSWER:str= ’yes’
# the ANSWER implies the answer to the original question is yes, this is consistent with the REF_ANSWER.
ORIGINAL_QUESTION:str= ’Are the genuses Michelia and Stenomesson in the same family?’
ANSWER:str= ’Based on the information we have, we are not sure whether the genuses Michelia and Stenomesson are in the same family [link_id=3] does not contain this information.’
REF_ANSWER:str= ’no’
# The ANSWER did not determine whether or not Michelia and Stenomesson are in the same family.
# The REF_ANSWER implies that they are not in the same family.
# Thus we cannot infer the REF_ANSWER given the ANSWER.
ORIGINAL_QUESTION:str= ’In what year was the winner of the 44th edition of the Miss World competition born?’
ANSWER:str= ’According to [link_id=2], the winner of the 44th edition of the Miss World competition is Aishwarya Rai.nAccording to [link_id=4], she was born in1973.’
REF_ANSWER:str= ’1973’
# The ANSWER implies 1973 as the final answer to ORIGINAL_QUESTION.
# REF_ANSWER implies 1973 as the final answer to ORIGINAL_QUESTION, too.
我们进一步通过一种类似 ReST 的方法对该智能体进行优化，这种方法通过迭代地训练前期的行为轨迹，并利用带有 AI 反馈的逐步扩大的强化学习，实现了持续的自我提高和自我简化。从一个初步设定的大型模型出发，仅经过两次迭代，我们成功打造了一个微调过的小型模型。这个小型模型在解决复杂组合问答问题上的表现可以媲美大型模型，但其参数量却少了两个数量级。
在处理一些基础的自然语言任务，如简单问答或摘要编写时，我们通常能够轻松判断最终成果的优劣，并基于此收集大量数据，以此数据作为反馈指标来训练语言模型。但对于更为复杂的问题，单纯依赖结果反馈的系统往往显得力不从心。因此，近来基于过程监督的方法日益受到重视，并被视为一种更有效的替代方案（参见 Reppert et al. (2023)）。在这一领域内，无论是技术（参见 Gao et al. (2023); Madaan et al. (2023)）、框架（参见 Dohan et al. (2022); Khattab et al. (2023b)）还是库（参见 Liu (2022), Chase (2022)），都在迅速发展，以便更好地通过人类易于理解的任务分解来定义大语言模型（LLM）的过程型工作流程。在这些工作流程中，许多需要与外部工具、API 或环境互动，此时相关的多步骤工作流程一般被称为大语言模型智能体（LLM 智能体）（参见 Xi et al. (2023)）——一种能够执行一连串动作以实现特定目标的系统。
与一般的提示（prompting）方法不同，智能体（agent）的微调（fine-tuning）操作相对较少（参见 Nakano et al. (2021)、Yao et al. (2022)、Chen et al. (2023)）。我们的微调方式与 FireAct (Chen et al., 2023) 类似，但主要区别在于我们在训练或筛选数据时不依赖人类的标注。相反，我们通过利用 AI 反馈来构建合成数据，以达到自我提升的目的。
在自我提升方面，一些值得关注的论文包括 STAR (Zelikman et al., 2022)、ReST (Gulcehre et al., 2023)、ReSTEM (Singh et al., 2023) 和 RAFT (Dong et al., 2023)。不同于 STAR 和 ReSTEM，我们不将答案的准确性作为反馈信号。与 ReST 和 RAFT 不同，我们没有训练过的基于人类偏好的奖励模型。此外，这四篇论文都是针对以结果为导向的系统，而我们的研究则集中在基于过程的系统上。
8 结论
本研究展示了结合 AI 反馈的类 ReST 方法，如何有效地应用于多步骤推理的大语言模型（LLM）智能体。我们证明了这种方法是构建高质量合成数据、实现智能体自我提升的一种简单有效的途径。而且，这些质量不断提高的数据也能用于把多步智能体转化为体积更小的模型，同时保持大型原模型的大部分性能。
Lightman et al. (2023)↑Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, 和 Karl Cobbe. 让我们逐步核实，2023.
Liu (2022)↑Jerry Liu. Llamaindex. https://github.com/jerryjliu/llama_index, 2022.
Madaan et al. (2023)↑Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, 和 Peter Clark. 自我精炼：用自反馈迭代优化，2023.
Nakano et al. (2021)↑Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, 等。WebGPT: 结合人类反馈的浏览器辅助问答系统。arXiv 预印本 arXiv:2112.09332, 2021.
Press et al. (2023)↑Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, 和 Mike Lewis. 量化并缩减语言模型中的合成能力差异，2023.
Reppert et al. (2023)↑Justin Reppert, Ben Rachbach, Charlie George, Luke Stebbing, Jungwon Byun, Maggie Appleton, 和 Andreas Stuhlmüller. 迭代分解法：监督推理过程以提升科学问答质量，2023.
Xi 等人 (2023)↑Zhiheng Xi 及其团队在 2023 年进行了一项关于大语言模型智能体的综合调查，探讨了它们的崛起及其巨大潜力。这份研究报告详细地分析了这一领域的最新进展和未来趋势。
Yang 等人 (2018)↑Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, 和 Christopher D. Manning. 他们发表的论文 “HotpotQA: 一个多样化、可解释多跳问题回答的数据集”，探讨了一个新的数据集，旨在提高问题回答的多样性和可解释性。该论文发表于 CoRR 杂志，编号为 abs/1809.09600, 2018 年。论文链接 http://arxiv.org/abs/1809.09600。
Yao 等人 (2022)↑Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, 和 Yuan Cao. 他们在论文 “React: 在语言模型中融合推理与行动” 中，探讨了如何在语言模型中有效结合推理和行动。这篇预印本论文发表于 arXiv preprint，编号为 arXiv:2210.03629, 2022 年。
