1
Lessons learned developing and using a machine learning
model to automatically transcribe 2.3 million handwritten
occupation codes
Bjørn-Richard Pedersen
1
, Einar Holsbø
2
, Trygve Andersen
1
, Nikita Shvetsov
2
, Johan Ravn
3
,
Hilde Leikny Sommerseth
1
, Lars Ailo Bongo
2*
1
Norwegian Historical Data Centre, UiT The Arctic University of Norway
2
Department of Computer Science, UiT The Arctic University of Norway
3
Medsensio AS, Tromsø, Norway
*
Corresponding author: lars.ailo.bongo@uit.no
Abstract
Machine learning approaches achieve high accuracy for text recognition and are therefore
increasingly used for the transcription of handwritten historical sources. However, using machine
learning in production requires a streamlined end-to-end pipeline that scales to the dataset size
and a model that achieves high accuracy with few manual transcriptions. The correctness of the
model results must also be verified. This paper describes our lessons learned developing, tuning
and using the Occode end-to-end machine learning pipeline for transcribing 2.3 million
handwritten occupation codes from the Norwegian 1950 population census. We achieve an
accuracy of 97% for the automatically transcribed codes, and we send 3% of the codes for manual
verification. We verify that the occupation code distribution found in our results matches the
distribution found in our training data, which should be representative for the census as a whole.
We believe our approach and lessons learned may be useful for other transcription projects that
plan to use machine learning in production. The source code is available at:
https://github.com/uit-hdl/rhd-codes
Introduction
Over the last few decades, we have witnessed a boom in the digitization of historical documents,
and many national archives are developing services where the public can easily access their
cultural heritage. For example, in the Norwegian state budget for 2019, 140 million NOK were
allocated to the National Archive of Norway. About half of the budget was assigned for further
development of the Digital Archive, which stores and distributes digitized historical documents.
This will increase the number and availability of digitized historical documents. However, there is
an increasing demand from the research community to have these documents in a data format
suitable for research based on data analysis. This demand has yielded the development of more
time and cost efficient systems in recent decades. Of particular interest for our project is the
development of automatic text recognition for population data, typically characterized as
handwritten documents with a tabular structure. These sources form the basis in the construction
of the Norwegian Historical Population Register (HPR; http://www.rhd.uit.no/nhdc/hpr.html).
The HPR will include the records of the 9.7 million people who lived in Norway in the period from
1801 to 1964. We are building life trajectories across multiple generations by linking individual
剩余19页未读，继续阅读
资源评论
