DefConvs_HTR

Boosting modern and historical handwritten text recognition with deformable convolutions (ICPR20, IJDAR)

This repo contains the source code for the papers:

• Cojocaru, Iulian, et al. "Watch your strokes: Improving handwritten text recognition with deformable convolutions." 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021.
• Cascianelli, Silvia, et al. "Boosting modern and historical handwritten text recognition with deformable convolutions." International Journal on Document Analysis and Recognition (IJDAR) (2022): 1-11.

Abstract

Handwritten Text Recognition (HTR) in free-layout pages is a challenging image understanding task that can provide a relevant boost to the digitization of handwritten documents and reuse of their content. The task becomes even more challenging when dealing with historical documents due to the variability of the writing style and degradation of the page quality.

State-of-the-art HTR approaches usually encode input images with Convolutional Neural Networks, whose kernels are typically defined on a fixed grid and focus on all input pixels independently. Since convolutional kernels are defined on fixed grids and focus on all input pixels independently while moving over the input image, this strategy, this is in contrast with the sparse nature of handwritten pages, in which only pixels representing the ink of the writing are useful for the recognition task. Furthermore, the standard convolution operator is not explicitly designed to take into account the great variability in shape, scale, and orientation of handwritten characters.

To cope with these specific HTR difficulties, we propose to adopt deformable convolutions, which can deform depending on the input at hand and better adapt to the geometric variations of the text.

The considered architectures are inspired by those presented in Are Multidimensional Recurrent Layers Really Necessary for Handwritten Text Recognition? (1D-LSTM) and An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition..

Dependencies

Please refer to the environment.yml file

References

If you find this repo useful, please consider citing:

@article{cascianelli2022boosting,
  title={Boosting modern and historical handwritten text recognition with deformable convolutions},
  author={Cascianelli, Silvia and Cornia, Marcella and Baraldi, Lorenzo and Cucchiara, Rita},
  journal={International Journal on Document Analysis and Recognition (IJDAR)},
  pages={1--11},
  year={2022},
  publisher={Springer}
}

@inproceedings{cojocaru2021watch,
  title={Watch your strokes: Improving handwritten text recognition with deformable convolutions},
  author={Cojocaru, Iulian and Cascianelli, Silvia and Baraldi, Lorenzo and Corsini, Massimiliano and Cucchiara, Rita},
  booktitle={2020 25th International Conference on Pattern Recognition (ICPR)},
  pages={6096--6103},
  year={2021},
  organization={IEEE}
}