Appendix
1 Experimental Details
For both token and character-level translation tasks, we used a maximum area size of 5 for the two layers
for Transformer Tiny and Small, and a maximum area size of 4 for the first layer of Transformer Base with
Eq.3, and for the first two layers of Transformer Base with Eq.9. For Transformer Big EN-DE, we used a
maximum area size of 4 for the first layer with Eq.3, and a maximum area size of 3 for Eq.9. For Transformer
Big EN-FR, we used a maximum area size of 3 for the first two layers with Eq.3, and a maximum area size of
4 for the first layer with Eq.9.
For character-level translation tasks, we here used the same dataset, and the experimental strategies as the
token-level experiments (see Section 4.1.1) with a few differences to reduce the experiment time because it is
significantly slower to train than token-level translation. In particular, we trained all the Transformer Big for
300,000 steps. For Transformer Big EN-DE, we could use a larger batch size that amounts to approximately
32,000 characters. All the LSTM models use the same batch size that amounts to 164,000 characters for
50,000 steps.
2 Additional Experimental Results
We evaluated the approach of area feature combination (Eq.9) on character-level translation tasks as well.
It performed on par with the basic form of area attention (Eq.3). In particular, it outperformed the basic
form (see details in Table 3), with statistical signficance, on Transformer Tiny EN-FR (
BLEU
= 12
.
91) and
Transformer Small EN-FR (
BLEU
= 21
.
93) and EN-DE (
BLEU
= 14
.
5), which seem to imply that when
the basic area attention helps, the method of area feature combination could bring further improvements.
We also explored the approach of using normalized sigmoid (Shen Lee, 2016; Rei Søgaard, 2018) as
the activation function of multi-head attention in Transformer. The quick experiments with Transformer
Tiny and Small by replacing softmax with normalized sigmoid led to poor results, which deserves further
investigation.
3 Related Algorithmic Details for Integral Images
Summed area table is based on an integral image,
I
, which can be efficiently computed in a single pass of
the memory (see Equation 1). Here let us focus on the area value calculation for a 2-dimensional memory
because a 1-dimensional memory is just a special case with the height of the memory grid as 1.
I
x,y
= v
x,y
+ I
x,y−1
+ I
x−1,y
− I
x−1,y−1
(1)
where
x
and
y
are the coordinates of the item in the memory. With the integral image, we can calculate
the key and value of each area in constant time. The sum of all the vectors in a rectangular area can be
easily computed as the following (Equation 2).
v
x1,y1,x2,y2
= I
x2,y2
+ I
x1,y1
− I
x2,y1
− I
x1,y2
(2)
where
v
x1,y1,x2,y2
is the value for the area located with the top-left corner at (
x
1
, y
1
) and the bottom-right
corner at (
x
2
, y
2
). By dividing
v
x1,y1,x2,y2
with the size of the area, we can easily compute
µ
x1,y1,x2,y2
. Based
on the summed area table,
σ
2
x1,y1,x2,y2
(thus
σ
x1,y1,x2,y2
) can also be computed at constant time for each
基鑫阁
- 粉丝: 736
- 资源: 358
最新资源
- 制造业实战：如何用DeepSeek优化供应链预测（附设备维护案例）.pdf
- 制造业实战：基于DeepSeek的智能质检模型私有化部署与产线数据训练指南.pdf
- 制造业极简指南：DeepSeek+传感器数据实现设备故障预警.pdf
- 证券行业：DeepSeek研报情绪分析模型的Prompt优化秘籍.pdf
- 制造业实战：DeepSeek智能质检模型部署与产线数据训练避坑手册.pdf
- 制造业私有化部署实战：DeepSeek低成本训练产线数据，实现智能质检方案落地.pdf
- 制造业效率革命：DeepSeek+工业数据分析实现故障预警90%准确率.pdf
- 制造业数字化转型：基于DeepSeek的自动化质检系统搭建全解析.pdf
- 制造业效率革命：基于DeepSeek-7B的工业质检模型训练指南（RTX3090显卡部署实录）.pdf
- 制造业预测性维护：DeepSeek时序数据分析模型训练全流程.pdf
- 制造业预测性维护：DeepSeek时序数据分析实战，从传感器数据到设备寿命预测模型全链路解析.pdf
- 制造业知识库：用RAG技术构建设备维护问答系统.pdf
- 制造业知识库升级：三一重工工程师分享设备维修QA系统训练数据集构建.pdf
- 制造业知识图谱：DeepSeek在设备维修知识库中的迁移学习实践.pdf
- 制造业质量检测优化：DeepSeek视觉模型在缺陷识别中的迁移学习.pdf
- 制造业质量追溯：DeepSeek与PLC数据对接的异常检测模型构建.pdf
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
评论0