STRUCTUREDRAG: JSON RESPONSE FORMATTING WITH
LARGE LANGUAGE MODELS
Connor Shorten
Weaviate
Charles Pierse
Weaviate
Thomas Benjamin Smith
Weaviate
Erika Cardenas
Weaviate
Akanksha Sharma
Weaviate
John Trengrove
Weaviate
Bob van Luijt
Weaviate
ABSTRACT
The ability of Large Language Models (LLMs) to generate structured outputs, such as JSON, is
crucial for their use in Compound AI Systems. However, evaluating and improving this capability
remains challenging. In this work, we introduce StructuredRAG, a benchmark of six tasks designed to
assess LLMs’ proficiency in following response format instructions. We evaluate two state-of-the-art
LLMs, Gemini 1.5 Pro and Llama 3 8B-instruct with 4-bit quantization using two distinct prompting
strategies. We introduce these prompting strategies as f-String and Follow the Format (FF) prompting.
Across 24 experiments, we find an average success rate of 82.55%. We further find a high variance
in performance across tasks, models, and prompting strategies with success rates ranging from 0
to 100%. We find that Llama 3 8B-instruct often performs competitively with Gemini 1.5 Pro.
We observe that task complexity significantly influences performance, with tasks involving lists or
composite object outputs proving more challenging. Our findings highlight the need for further
research into improving the reliability and consistency of structured output generation in LLMs. We
have open-sourced our experimental code and results at github.com/weaviate/structured-rag.
1 Introduction
Large Language Models (LLMs) have become extremely effective at Zero-Shot Learning. Zero-Shot Learning is used
to describe a machine learning model’s ability to perform a task without any training data for the task given in advance.
An emergent area of importance is not only to test LLMs on how well they can perform novel tasks, but also how
well they can structure their output in a particular format. This is a critical requirement for developing Compound AI
Systems [
1
,
2
] that consist of multiple LLM inferences or external computational tools. For example, Multi-Hop RAG
[
3
] is a Compound AI System where an LLM inference first predicts one or multiple search queries for an input and
then sends these queries to a search tool. Another LLM inference then aggregates these search results and the original
question to generate a response. In order for the Multi-Hop RAG system to parse the response from the query writer to
send to the search tool, it is critical that the query writer follows a particular response format such as a JSON with the
key “queries” and a list of strings as the value.
In this work, we seek to measure the ability of LLMs to follow JSON response format instructions with Zero-Shot
Learning. While structured decoding methods, such as DOMINO [
4
], have emerged as a popular solution for ensuring
correct JSON outputs in Compound AI Systems, we seek to better understand the baseline performance of Zero-Shot
Learning. Structured decoding may slow down inference throughput, complicate system integration, and interfere with
the LLM’s prior knowledge and the benefits of prompt optimization [
5
]. To address these concerns, we a construct a novel
benchmark of six RAG-inspired [
6
] structured output tests. These tests explore different typed JSON responses such as
string, integer, or boolean values, as well as outputting a list of strings, denoted as List[string]. Further, we illustrate the
use of composite objects containing more than one type per instance. We present the AnswerWithConfidence composite
object consisting of a string valued answer and an integer valued confidence. We further test the ability to output a
list of AnswerWithConfidence objects, similarly denoted as List[AnswerWithConfidence]. An output from the LLM
passes these tests if it is able to be parsed into the requested JSON response format. This entails that the output jointly
arXiv:2408.11061v1 [cs.CL] 7 Aug 2024
资源评论
豪AI冰
- 粉丝: 73
- 资源: 68
最新资源
- 用QT实现的桌面端聊天室软件，含服务端和客户端，使用经过SSL加密的TCP通
- 一款基于 MATLAB 的 EEG 神经反馈训练系统 在神经反馈实验过程中可实时观察并记录 EEG 信号和神经反馈实验标记
- Java SSM 商户管理系统 客户管理 库存管理 销售报表 项目源码 本商品卖的是源码，合适的地方直接拿来使用，不合适的根据
- 基于Spring boot 的Starter机制提供一个开箱即用的多数据源抽取工具包，计划对RDMS(关系型
- 水泵系统水力计算公式-标准版
- Wesley是一套为经销商量身定制的全业务流程渠道 分销管理系统（手机APP称为经销商管家）
- Adaptive Autosar EM 标准规范
- 鼓谱图片转MuseScore超文本文档实验程序
- 自动驾驶感知动态障碍物算法上车效果 （Xavier jetson&autoware）
- 【实验指导书-2024版】实验一：查验身份证.doc
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
