Skip to main content
SpringerLink
Log in

Learning to Rank Words: Optimizing Ranking Metrics for Word Spotting

  • Conference paper
  • First Online:
Document Analysis and Recognition – ICDAR 2021 (ICDAR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12822))

Included in the following conference series:

Abstract

In this paper, we explore and evaluate the use of ranking-based objective functions for learning simultaneously a word string and a word image encoder. We consider retrieval frameworks in which the user expects a retrieval list ranked according to a defined relevance score. In the context of a word spotting problem, the relevance score has been set according to the string edit distance from the query string. We experimentally demonstrate the competitive performance of the proposed model on query-by-string word spotting for both, handwritten and real scene word images. We also provide the results for query-by-example word spotting, although it is not the main focus of this work.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    George Washington Papers at the Library of Congress from 1741–1799, Series 2, Letterbook 1, pages 270–279 and 300–309, https://www.loc.gov/collections/george-washington-papers/about-this-collection/.

References

  1. Aldavert, D., Rusiñol, M., Toledo, R., Lladós, J.: Integrating visual and textual cues for query-by-string word spotting. In: Proceedings of the International Conference on Document Analysis and Recognition, pp. 511–515 (2013)

    Google Scholar 

  2. Almazán, J., Gordo, A., Fornés, A., Valveny, E.: Segmentation-free word spotting with exemplar SVMs. Pattern Recogn. 47(12), 3967–3978 (2014)

    Article  Google Scholar 

  3. Almazán, J., Gordo, A., Fornés, A., Valveny, E.: Word spotting and recognition with embedded attributes. IEEE Trans. Pattern Anal. Mach. Intell. 36(12), 2552–2566 (2014)

    Article  Google Scholar 

  4. Brown, A., Xie, W., Kalogeiton, V., Zisserman, A.: Smooth-AP: smoothing the path towards large-scale image retrieval. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12354, pp. 677–694. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58545-7_39

    Chapter  Google Scholar 

  5. Chung, J., Gulcehre, C., Cho, K., Bengio, Y.: Empirical evaluation of gated recurrent neural networks on sequence modeling. In: Proceedings of the NeurIPS Workshop on Deep Learning (2014)

    Google Scholar 

  6. Deerwester, S., Dumais, S., Furnas, G., Landauer, T., Harshman, R.: Indexing by latent semantic analysis. J. Am. Soc. Inf. Sci. 41, 391–407 (1990)

    Article  Google Scholar 

  7. Frinken, V., Fischer, A., Manmatha, R., Bunke, H.: A novel word spotting method based on recurrent neural networks. IEEE Trans. Pattern Anal. Mach. Intell. 34(2), 211–224 (2011)

    Article  Google Scholar 

  8. Gómez, L., Rusinol, M., Karatzas, D.: LSDE: Levenshtein space deep embedding for query-by-string word spotting. In: Proceedings of the International Conference on Document Analysis and Recognition, vol. 1, pp. 499–504 (2017)

    Google Scholar 

  9. He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1904–1916 (2015)

    Article  Google Scholar 

  10. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  11. Jaderberg, M., Simonyan, K., Vedaldi, A., Zisserman, A.: Synthetic data and artificial neural networks for natural scene text recognition. arXiv preprint 1406.2227 (2014)

    Google Scholar 

  12. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  13. Krishnan, P., Dutta, K., Jawahar, C.: Deep feature embedding for accurate recognition and retrieval of handwritten text. In: Proceedings of the International Conference on Frontiers in Handwriting Recognition, pp. 289–294 (2016)

    Google Scholar 

  14. Krishnan, P., Dutta, K., Jawahar, C.: Word spotting and recognition using deep embedding. In: Proceedings of the International Workshop on Document Analysis Systems, pp. 1–6 (2018)

    Google Scholar 

  15. Krishnan, P., Jawahar, C.V.: Matching handwritten document images. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 766–782. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_46

    Chapter  Google Scholar 

  16. Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In: Proceedings of IEEE Conference on Computing Vision Pattern Recognition, vol. 2, pp. 2169–2178 (2006)

    Google Scholar 

  17. Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady 10(8), 707–710 (1966)

    MathSciNet  Google Scholar 

  18. Li, Z., Min, W., Song, J., Zhu, Y., Jiang, S.: Rethinking ranking-based loss functions: only penalizing negative instances before positive ones is enough. arXiv preprint (2021)

    Google Scholar 

  19. Manmatha, R., Han, C., Riseman, E.M.: Word spotting: a new approach to indexing handwriting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 631–637 (1996)

    Google Scholar 

  20. Manmatha, R., Han, C., Riseman, E.M., Croft, W.B.: Indexing handwriting using word matching. In: Proceedings of the ACM International Conference on Digital Libraries, pp. 151–159 (1996)

    Google Scholar 

  21. Mishra, A., Alahari, K., Jawahar, C.V.: Scene text recognition using higher order language priors. In: Proceedings of the British Machine Vision Conference (2012)

    Google Scholar 

  22. Qin, T., Liu, T.Y., Li, H.: A general approximation framework for direct optimization of information retrieval measures. Inf. Retr. 13(4), 375–397 (2010)

    Article  Google Scholar 

  23. Rath, T.M., Manmatha, R.: Word spotting for historical documents. Int. J. Doc. Anal. Recogn. 9(2–4), 139–152 (2007)

    Article  Google Scholar 

  24. Revaud, J., Almazán, J., Rezende, R.S., Souza, C.R.D.: Learning with average precision: training image retrieval with a listwise loss. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5107–5116 (2019)

    Google Scholar 

  25. Rusinol, M., Aldavert, D., Toledo, R., Lladós, J.: Browsing heterogeneous document collections by a segmentation-free word spotting method. In: Proceedings of the International Conference on Document Analysis and Recognition, pp. 63–67 (2011)

    Google Scholar 

  26. Rusiñol, M., Aldavert, D., Toledo, R., Lladós, J.: Efficient segmentation-free keyword spotting in historical document collections. Pattern Recogn. 48(2), 545–555 (2015)

    Article  Google Scholar 

  27. Rusiñol, M., Lladós, J.: A performance evaluation protocol for symbol spotting systems in terms of recognition and location indices. Int. J. Doc. Anal. Recogn. 12(2), 83–96 (2009)

    Article  Google Scholar 

  28. Sudholt, S., Fink, G.A.: PHOCNet: a deep convolutional neural network for word spotting in handwritten documents. In: Proceedings of the International Conference on Frontiers in Handwriting Recognition, pp. 277–282 (2016)

    Google Scholar 

  29. Sudholt, S., Fink, G.A.: Evaluating word string embeddings and loss functions for CNN-based word spotting. In: Proceedings of the International Conference on Document Analysis and Recognition, vol. 1, pp. 493–498 (2017)

    Google Scholar 

  30. Valizadegan, H., Jin, R., Zhang, R., Mao, J.: Learning to rank by optimizing NDCG measure. Adv. Neural. Inf. Process. Syst. 22, 1883–1891 (2009)

    Google Scholar 

  31. Weinberger, K.Q., Saul, L.K.: Distance metric learning for large margin nearest neighbor classification. J. Mach. Learn. Res. 10, 207–244 (2009)

    MATH  Google Scholar 

  32. Weinman, J.J., Learned-Miller, E., Hanson, A.: Scene text recognition using similarity and a lexicon with sparse belief propagation. IEEE Trans. Pattern Anal. Mach. Intell. 31(10), 1733–1746 (2009)

    Article  Google Scholar 

  33. Wilkinson, T., Brun, A.: Semantic and verbatim word spotting using deep neural networks. In: Proceedings of the International Conference on Frontiers in Handwriting Recognition, pp. 307–312 (2016)

    Google Scholar 

Download references

Acknowledgment

This work has been partially supported by the Spanish projects RTI2018-095645-B-C21, and FCT-19-15244, and the Catalan projects 2017-SGR-1783, the Culture Department of the Generalitat de Catalunya, and the CERCA Program/Generalitat de Catalunya.

Author information

Authors and Affiliations

  1. Computer Vision Center and Computer Science Department, Universitat Autònoma de Barcelona, Catalunya, Spain

    Pau Riba, Adrià Molina, Lluis Gomez, Oriol Ramos-Terrades & Josep Lladós

Authors
  1. Pau Riba
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adrià Molina
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lluis Gomez
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Oriol Ramos-Terrades
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Josep Lladós
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pau Riba .

Editor information

Editors and Affiliations

  1. Universitat Autònoma de Barcelona, Barcelona, Spain

    Josep Lladós

  2. Lehigh University, Bethlehem, PA, USA

    Daniel Lopresti

  3. Kyushu University, Fukuoka-shi, Japan

    Seiichi Uchida

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Riba, P., Molina, A., Gomez, L., Ramos-Terrades, O., Lladós, J. (2021). Learning to Rank Words: Optimizing Ranking Metrics for Word Spotting. In: Lladós, J., Lopresti, D., Uchida, S. (eds) Document Analysis and Recognition – ICDAR 2021. ICDAR 2021. Lecture Notes in Computer Science(), vol 12822. Springer, Cham. https://doi.org/10.1007/978-3-030-86331-9_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-86331-9_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-86330-2

  • Online ISBN: 978-3-030-86331-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation