Author(s): Gowtham Boyina

Originally published on Towards AI.

Why teaching an AI to pick from a menu beats letting it write its own questions

Here is a strange failure that shows up when you train an AI agent to search for answers using reinforcement learning. You ask it to research a question. It writes a search query, gets some results, decides it needs more information, and writes a new query. On paper this looks like exploration. The agent is trying different phrasings, chasing different angles, behaving like a curious researcher.

The article explains how reinforcement-learning “search agents” can suffer from retrieval-equivalence collapse: different rewritten queries often retrieve the same documents, so the agent’s apparent exploration is illusory and the training signal stops being meaningful. It then describes a fix from the paper “Harness-G,” which turns open-ended query generation into a multiple-choice menu of explicit actions (e.g., selecting evidence, looking up connected entities, and answering), enabling true diversity and better, structured credit assignment (including non-myopic credit that rewards steps based on their downstream usefulness). With this menu interface and improved reward signals, Harness-G improves F1 across multiple multi-hop and single-hop benchmarks, trains more stably, generalizes across datasets and domains, and does so efficiently using a programmatic graph rather than LLM-built knowledge graphs. The author concludes with limitations—text-only for now and slightly weaker performance on certain single-hop tasks—and a broader takeaway that the core action space may matter as much as (or more than) reward engineering.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI