韩家龙__Logic Attention Based Neighborhood Aggregation for Inductiv
需积分: 0 12 浏览量
2022-08-03
14:33:04
上传
评论
收藏 767KB PDF 举报
Logic Attention Based Neighborhood Aggregation
for Inductive Knowledge Graph Embedding
Peifeng Wang
1∗
, Jialong Han
2
, Chenliang Li
3
, Rong Pan
1
1
Sun Yat-sen University,
2
Tencent AI Lab,
3
Wuhan University
1
{wangpf3@mail2., panr@}sysu.edu.cn,
2
3
clle[email protected]du.cn
∗
This work was done during the first author’s internship at Tencent AI Lab.
Motivation and Contributions
The paper aims at embedding new entities from
the knowledge graph (KG) inductively by aggre-
gating the neighbors with respect to their un-
ordered and unequal nature.
• We propose three desired properties that de-
cent neighborhood aggregators for KGs should
possess.
• We propose a novel aggregator called Logic
Attention Network (LAN), to facilitate induc-
tive KG embedding.
• We conduct extensive experiments to validate
the superiority of LAN.
Background and Problem Setting
work_as
play_for
live_in
nationality
Emerging Entity
Basketball_Player
Chicago_Bulls
American
News
Article
KG
?
• KG consists of numerous triplet facts like (s, r, o).
• KG embedding represents entities and relations
in low-dimension space.
• Challenge All the entities are required to be seen
during training.
Given a knowledge graph K, we would like to learn a
neighborhood aggregator A which uses an entity e
i
’s
neighborhood in K to embed e
i
as a low-dimensional
vector e
i
.
Framework
Basketball_Player
Michael_Jordan Chicago
live_in
Chicago_Bulls America Illinois Rahm_EmanuelEnglish
play_for nationality work_as language
contains
－
1
mayor
Scoring
Function
Input
Embeddings
Transformed
Embeddings
Output
Embeddings
Transform
by relation
Weighted
Aggregation
Decoder
Encoder
• Encoder
1. A relation-specific transforming function
T
r
(.) applied on a neighbor e
I
j
:
T
r
(e
I
j
) = e
I
j
− w
>
r
e
I
j
w
r
.
2. An entity e
i
’s neighbors are aggregated as:
e
O
i
=
X
(r,e
j
)∈N
K
(i)
(α
Logic
j|i,q
+ α
NN
j|i,q
)T
r
(e
I
j
).
• Decoder
Scoring function on the output embedding:
φ
O
(s, q, o) = −|s
O
+ q − o
O
|
L1
.
Desired Properties of Neighborhood Aggregator for KG
• Permutation Invariant - Neighbors of an entity are naturally unordered. The aggregator
should be irrelevant to potential permutations of the neighbors.
• Redundancy Aware - Facts in KGs tend to depend on each other. It is beneficial to exploit
the redundancy in an entity’s neighborhood.
• Query Relation Aware - An aggregator may take advantage the query relation in concern
to concentrate on relevant facts in the neighborhood.
Logic Rule Mechanism
1. The confidence of the implication r
1
⇒ r
2
:
P(r
1
⇒ r
2
) =
P
e∈E
1
(r
1
∈ N
R
(e) ∧ r
2
∈ N
R
(e))
P
e∈E
1
(r
1
∈ N
R
(e))
.
2. The logic rule mechanism of measuring neigh-
bors’ importance:
α
Logic
j|i,q
=
P(r ⇒ q)
max({P(r
0
⇒ r)|r
0
∈ N
R
(e
i
) ∧ r
0
6= r})
.
• The numerator promotes relations r strongly
implying q.
• The denominator demotes those implied by
some other relation in the same neighborhood.
Neural Network Mechanism
The importance of an entity e
i
’s neighbor e
j
is
measured by:
α
NN
j|i,q
= softmax(α
0
j|i,q
) =
exp(α
0
j|i,q
)
P
j
0
∈N
E
(i)
exp(α
0
j
0
|i,q
)
.
The unnormalized attention weight α
0
j|i,q
is
given by an attention neural network as
α
0
j|i,q
= u
>
a
· tanh(W
a
· [z
q
; T
r
(e
I
j
)]),
where u
a
and W
a
∈ R
d×2d
are global attention
parameters, while z
q
is a relation-specific atten-
tion parameter for the query relation q.
Training and Experiments
Task 1: Link Prediction Task 2: Triplet Classification
Subject-10 Object-10 Subject Object Both
Model MR MRR Hits@10 @3 @1 MR MRR Hits@10 @3 @1 1K 3K 5K 1K 3K 5K 1K 3K 5K
MEAN 293 0.310 48.0 34.8 22.2 353 0.251 41.0 28.0 17.1 87.3 84.3 83.3 84.0 75.2 69.2 83.0 73.3 68.2
LSTM 353 0.254 42.9 29.6 16.2 504 0.219 37.3 24.6 14.3 87.0 83.5 81.8 82.9 71.4 63.1 78.5 71.6 65.8
LAN 263 0.394 56.6 44.6 30.2 461 0.314 48.2 35.7 22.7 88.8 85.2 84.2 84.7 78.8 74.3 83.3 76.9 70.6
• Training
We apply a margin-based ranking loss on each
triplet (s, q, o) as
l
O
(s, q, o) = [γ − φ
O
(s, q, o) + φ
O
(s
0
, q, o
0
)]
+
.
• Subtask:
Scoring function on the input embedding:
φ
I
(s, q, o) = −|s
I
+ q − o
I
|
L1
.
Then a similar margin-based ranking loss
l
I
(s, q, o) is defined for the subtask.
• Training objective:
min
X
(s,q,o)∈∆
X
(s
0
,q,o
0
)∈∆
0
(s,q,o)
[l
O
(s, q, o)+l
I
(s, q, o)].
• Dataset
Based on WordNet and Freebase, we construct
datasets whose test sets contain new entities un-
seen during training.
0
5
10
15
20
25
30
Percentage (%)
5 10 15 20 25
Dataset: Subject-R (%)
40
45
50
55
60
65
Hits@10 (%)
MEAN
LSTM
LAN
Unseen Entities
0
5
10
15
20
25
30
Percentage (%)
5 10 15 20 25
Dataset: Object-R (%)
30
35
40
45
50
55
60
Hits@10 (%)
MEAN
LSTM
LAN
Unseen Entities
Model MRR Hits@10 Hit@3 Hits@1
MEAN 0.310 48.0 34.8 22.2
Global-Attention 0.331 49.7 37.7 24.0
Query-Attention 0.355 51.9 39.5 27.0
Logic Rules Only 0.375 54.7 42.7 28.0
LAN 0.394 56.6 44.6 30.2
Subject and Query Neighbors ranked by LAN Predicted Object from LAN and MEAN
Jared_Drake_Bell
query: origin
place_lived -> Orange_County
breed_origin -> Santa_Ana
website_owner -> Universal_Records
perform_film -> High_Fidelity
friend -> Corbin_Bleu_Reivers
gender -> Male
LAN: Orange_County, Santa_Ana, Laguna_Beach,
City_Orange, Fullerton, Huntington_Beach, Costa_Mesa,
Greenwich_Village, Newport_Beach, Anaheim
MEAN: Costa_Mesa, Santa_Ana, Southern_California,
Berkeley, Oslo, Stuttgart,Newport_Beach,
Miami, Surrey, San_Jose
Georg_Hegel
query: profession
influenced_by -> Aristotle
interest -> Metaphysics
interest -> Aesthetics
interset -> Logic
interest -> Epistenmology
employment
-1
-> Humboldt_University
ethnicity -> Germans
gender -> Male
LAN: Philosopher, Economist, Librarian,
Psychiatrist, Psychologist, Priest,
Scientist, Historian, Pediatrics, Designer
MEAN: Physicist, Aristotle, Karl_Marx,
Gottfried_Leibniz, John_FRS, Immanuel_Kant,
Philosopher, Economist, Architect, Plato
Stephen_Joseph_Harper
query: place_lived
institution -> University_of_Calgary
politician -> Conservative_Party_of_Canda
appointed_by -> Senate_of_Canada
position -> Prime_Minister_of_Canda
religion -> Evangelicalism
profession -> Economist
gender -> Male
LAN: Nunavut, Yukon, Saskatchewan, Alberta,
Connecticut, British_Coulumbia, Nova_Scotia,
Calgary, Oklahoma, Edmonton
MEAN: Yukon, Nunavut, Alberta, Prince_Edward_Island,
Senate_Of_Canda, Nova_Scotia, British_Columbia,
Montana, Quebec, Alaska
无声远望
- 粉丝: 52
- 资源: 298
评论0