Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities
of LLMs
Kewei Cheng
1,2
, Jingfeng Yang
2
, Haoming Jiang
2
, Zhengyang Wang
2
, Binxuan Huang
2
Ruirui Li
2
, Shiyang Li
2
, Zheng Li
2
, Yifan Gao
2
, Xian Li
2
, Bing Yin
2
, Yizhou Sun
1
1
University of California, Los Angeles
2
Amazon
{chenkewe, jingfe, jhaoming, zhengywa, binxuan, ruirul,
syangli, amzzhe, yifangao, xianlee, alexbyin}@amazon.com
yzsun@cs.ucla.edu
Abstract
Reasoning encompasses two typical types: de-
ductive reasoning and inductive reasoning. De-
spite extensive research into the reasoning ca-
pabilities of Large Language Models (LLMs),
most studies have failed to rigorously differenti-
ate between inductive and deductive reasoning,
leading to a blending of the two. This raises an
essential question: In LLM reasoning, which
poses a greater challenge - deductive or induc-
tive reasoning? While the deductive reasoning
capabilities of LLMs, (i.e. their capacity to
follow instructions in reasoning tasks), have
received considerable attention, their abilities
in true inductive reasoning remain largely unex-
plored. To investigate the true inductive reason-
ing capabilities of LLMs, we propose a novel
framework, SolverLearner. This framework
enables LLMs to learn the underlying function
(i.e.,
𝑦 = 𝑓
𝑤
(𝑥)
), that maps input data points
(𝑥)
to their corresponding output values
(𝑦)
,
using only in-context examples. By focusing
on inductive reasoning and separating it from
LLM-based deductive reasoning, we can isolate
and investigate inductive reasoning of LLMs in
its pure form via SolverLearner. Our observa-
tions reveal that LLMs demonstrate remarkable
inductive reasoning capabilities through Solver-
Learner, achieving near-perfect performance
with ACC of 1 in most cases. Surprisingly, de-
spite their strong inductive reasoning abilities,
LLMs tend to relatively lack deductive reason-
ing capabilities, particularly in tasks involving
“counterfactual” reasoning.
1 Introduction
Recent years have witnessed notable progress in
Natural Language Processing (NLP) with the de-
velopment of Large Language Models (LLMs) like
GPT-3 (Brown et al., 2020) and ChatGPT (Ope-
nAI, 2023). While these models exhibit impressive
reasoning abilities across various tasks, they face
challenges in certain domains. For example, a re-
cent study (Wu et al., 2023) has shown that while
LLMs excel in conventional tasks (e.g., base-10
arithmetic), they often experience a notable decline
in accuracy when dealing “counterfactual” reason-
ing tasks that deviate from the conventional cases
seen during pre-training (e.g., base-9 arithmetic).
It remains unclear whether they are capable of fun-
damental reasoning, or just approximate retrieval.
In light of this, our paper seeks to investigate
the reasoning capabilities of LLMs. Reasoning
can encompasses two types: deductive reasoning
and inductive reasoning, as depicted in Fig. 1. De-
ductive reasoning starts with a general hypothesis
and proceeds to derive specific conclusions about
individual instances while inductive reasoning in-
volves formulating broad generalizations or princi-
ples from a set of instance observations. Despite
extensive research into the reasoning capabilities of
LLMs, most studies have not clearly differentiated
between inductive and deductive reasoning. For in-
stance, arithmetic reasoning task primarily focuses
on comprehending and applying mathematical con-
cepts to solve arithmetic problems, aligning more
with deductive reasoning. Yet, when employing
in-context learning for arithmetic reasoning tasks,
where the model is prompted with a few
〈
input,
output
〉
examples, the observed improvements are
often attributed to their inductive reasoning capac-
ity. This fusion of reasoning types poses a critical
question: Which is the more significant limita-
tion in LLM reasoning, deductive or inductive
reasoning?
To explore this question, it’s crucial to differen-
tiate between deductive and inductive reasoning.
Current methods that investigate deductive and in-
ductive reasoning often rely on disparate datasets,
making direct comparisons challenging (Xu et al.,
2023a; Tang et al., 2023; Dalvi et al., 2021; Han
et al., 2022; Sinha et al., 2019; Yu et al., 2020). To
overcome this limitation, we have designed a set of
comparative experiments that utilize a consistent
task across different contexts, each emphasizing
arXiv:2408.00114v2 [cs.AI] 7 Aug 2024
剩余17页未读，继续阅读
资源评论
豪AI冰
- 粉丝: 73
- 资源: 68
