Skip to main content
Log in

A study of Bag-of-Visual-Words representations for handwritten keyword spotting

International Journal on Document Analysis and Recognition (IJDAR) Aims and scope Submit manuscript

Abstract

The Bag-of-Visual-Words (BoVW) framework has gained popularity among the document image analysis community, specifically as a representation of handwritten words for recognition or spotting purposes. Although in the computer vision field the BoVW method has been greatly improved, most of the approaches in the document image analysis domain still rely on the basic implementation of the BoVW method disregarding such latest refinements. In this paper, we present a review of those improvements and its application to the keyword spotting task. We thoroughly evaluate their impact against a baseline system in the well-known George Washington dataset and compare the obtained results against nine state-of-the-art keyword spotting methods. In addition, we also compare both the baseline and improved systems with the methods presented at the Handwritten Keyword Spotting Competition 2014.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Notes

  1. H-KWS 2014 competition homepage: http://vc.ee.duth.gr/h-kws2014/.

    Table 5 Comparison of the performance attained by the system using the baseline and enhanced BoVW configurations with the methods that participated in the H-KWS 2014 competition

References

  1. Adamek, T., O’Connor, N., Smeaton, A.: Word matching using single closed contours for indexing handwritten historical documents. Int. J. Doc. Anal. Recognit. 9(2–4), 153–165 (2007)

    Article  Google Scholar 

  2. Aldavert, D., Rusiñol, M., Toledo, R., Lladós, J.: Integrating visual and textual cues for query-by-string word spotting. In: Proceedings of the International Conference on Document Analysis and Recognition, pp. 511–515 (2013)

  3. Almazán, J., Gordo, A., Fornés, A., Valveny, E.: Efficient exemplar word spotting. In: Proceedings of the British Machine Vision Conference, pp. 67.1–67.11 (2012)

  4. Almazán, J., Gordo, A., Fornés, A., Valveny, E.: Handwritten word spotting with corrected attributes. In: Proceedings of the International Conference on Computer Vision, pp. 1017–1024 (2013)

  5. Ataer, E., Duygulu, P.: Matching ottoman words: an image retrieval approach to historical document indexing. In: Proceedings of the International Conference on Image and Video Retrieval, pp. 341–347 (2007)

  6. Chatfield, K., Lempitsky, V., Vedaldi, A., Zisserman, A.: The devil is in the details: an evaluation of recent feature encoding methods. In: Proceedings of the British Machine Vision Conference, pp. 76.1–76.12 (2011)

  7. Cinbis, R., Verbeek, J., Schmid, C.: Image categorization using fisher kernels of non-iid image models. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 2184–2191 (2012)

  8. Csurka, G., Dance, C., Fan, L., Willamowski, J., Bray, C.: Visual categorization with bags of keypoints. In: Proceedings of the ECCV International Workshop on Statistical Learning in Computer Vision, pp. 1–22 (2004)

  9. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 886–893 (2005)

  10. Dovgalecs, V., Burnett, A., Tranouez, P., Nicolas, S., Heutte, L.: Spot it! finding words and patterns in historical documents. In: Proceedings of the International Conference on Document Analysis and Recognition, pp. 1039–1043 (2013)

  11. Fernández, D., Lladós, J., Fornés, A.: Handwritten word spotting in old manuscript images using a pseudo-structural descriptor organized in a hash structure. Pattern Recognition and Image Analysis, Lecture Notes on Computer Science, vol. 6669, 628–635 (2011)

  12. Fischer, A., Keller, A., Frinken, V., Bunke, H.: Lexicon-free handwritten word spotting using character HMMs. Pattern Recognit. Lett. 33(7), 934–942 (2012)

    Article  Google Scholar 

  13. Frinken, V., Fischer, A., Manmatha, R., Bunke, H.: A novel word spotting method based on recurrent neural networks. IEEE Trans. Pattern Anal. Mach. Intell. 34(2), 211–224 (2012)

    Article  Google Scholar 

  14. Fulkerson, B., Vedaldi, A., Soatto, S.: Localizing objects with smart dictionaries. In: Proceedings of the European Conference on Computer Vision, Lecture Notes in Computer Science, vol. 5302, pp. 179–192 (2008)

  15. Howe, N.: Part-structured inkball models for one-shot handwritten word spotting. In: Proceedings of the International Conference on Documents Analysis and Recognition, pp. 582–586 (2013)

  16. Howe, N., Rath, T., Manmatha, R.: Boosted decision trees for word recognition in handwritten document retrieval. In: Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 377–383 (2005)

  17. Impedovo, S., Mangini, F., Pirlo, G.: A new adaptive zoning technique for handwritten digit recognition. In: Proceedings of the International Conference on Image Analysis and Processing, pp. 91–100 (2013)

  18. Khurshid, K., Faureb, C., Vincent, N.: Word spotting in historical printed documents using shape and sequence comparisons. Pattern Recognit. 45(7), 2598–2609 (2012)

    Article  Google Scholar 

  19. Kołcz, A., Alspector, J., Augusteijn, M., Carlson, R., Popescu, G.: A line-oriented approach to word spotting in handwritten documents. Pattern Anal. Appl. 3(2), 153–168 (2000)

    Article  MATH  Google Scholar 

  20. Konidaris, T., Gatos, B., Ntzios, K., Pratikakis, I., Theodoridis, S., Perantonis, S.: Keyword-guided word spotting in historical printed documents using synthetic data and user feedback. Int. J. Doc. Anal. Recognit. 9(2–4), 167–177 (2007)

    Article  Google Scholar 

  21. Kovalchuk, A., Wolf, L., Dershowitz, N.: A simple and fast word spotting method. In: Proceedings of the International Conference on Frontiers in Handwriting Recognition (2014)

  22. Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 2169–2178 (2006)

  23. Leydier, Y., Ouji, A., LeBourgeois, F., Emptoz, H.: Towards an omnilingual word retrieval system for ancient manuscripts. Pattern Recognit. 42(9), 2089–2105 (2009)

    Article  MATH  Google Scholar 

  24. Liang, Y., Fairhurst, M., Guest, R.: A synthesised word approach to word retrieval in handwritten documents. Pattern Recognit. 45(12), 4224–4236 (2012)

    Article  Google Scholar 

  25. Likforman-Sulem, L., Zahour, A., Taconet, B.: Text line segmentation of historical documents: a survey. Int. J. Doc. Anal. Recognit. 9(2–4), 123–138 (2007)

    Article  Google Scholar 

  26. Lladós, J., Rusiñol, M., Fornés, A., Fernández, D., Dutta, A.: On the influence of word representations for handwritten word spotting in historical documents. Int. J. Pattern Recognit. Artif. Intell. 26(5), 1263,002.1–1263,002.25 (2012)

    Article  Google Scholar 

  27. Lladós, J., Sánchez, G.: Indexing historical documents by word shape signatures. In: Proceedings of the International Conference on Document Analysis and Recognition, pp. 362–366 (2007)

  28. Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  29. Madhvanath, S., Govindaraju, V.: The role of holistic paradigms in handwritten word recognition. IEEE Trans. Pattern Anal. Mach. Intell. 23(2), 149–164 (2001)

    Article  Google Scholar 

  30. Manmatha, R., Han, C., Riseman, E.: Word spotting: a new approach to indexing handwriting. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 631–637 (1996)

  31. Manmatha, R., Rothfeder, J.: A scale space approach for automatically segmenting words from historical handwritten documents. IEEE Trans. Pattern Anal. Mach. Intell. 27(8), 1212–1225 (2005)

    Article  Google Scholar 

  32. Nowak, E., Jurie, F., Triggs, B.: Sampling strategies for bag-of-features image classification. In: Proceedings of the European Conference on Computer Vision, Lecture Notes in Computer Science, vol. 3954, pp. 490–503 (2006)

  33. Papandreou, A., Gatos, B., Louloudis, G.: An adaptive zoning technique for efficient word retrieval using dynamic time warping. In: Proceedings of the First International Conference on Digital Access to Textual Cultural Heritage, pp. 147–152 (2014)

  34. Perronnin, F., Sanchez, J., Mensink, T.: Improving the fisher kernel for large-scale image classification. In: Proceedings of the European Conference on Computer Vision, Lecture Notes in Computer Science, vol. 6314, pp. 143–156 (2010)

  35. Plamondon, R., Srihari, S.: Online and off-line handwriting recognition: a comprehensive survey. IEEE Trans. Pattern Anal. Mach. Intell. 22(1), 63–84 (2000)

    Article  Google Scholar 

  36. Pratikakis, I., Zagoris, K., Gatos, B., Louloudis, G., Stamatopoulos, N.: ICFHR 2014 competition on handwritten keyword spotting (H-KWS 2014). In: Proceedings of the International Conference on Frontiers in Handwriting Recognition, pp. 814–819 (2014)

  37. Rath, T., Manmatha, R.: Word image matching using dynamic time warping. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 521–527 (2003)

  38. Rath, T., Manmatha, R.: Word spotting for historical documents. Int. J. Doc. Anal. Recognit. 9(2–4), 139–152 (2007)

    Article  Google Scholar 

  39. Rath, T., Manmatha, R., Lavrenko, V.: A search engine for historical manuscript images. In: Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 369–376 (2004)

  40. Rodríguez-Serrano, J., Perronnin, F.: Local gradient histogram features for word spotting in unconstrained handwritten documents. In: Proceedings of the International Conference on Frontiers in Handwriting Recognition, pp. 7–12 (2008)

  41. Rodríguez-Serrano, J., Perronnin, F.: Handwritten word-spotting using hidden Markov models and universal vocabularies. Pattern Recognit. 42(9), 2106–2116 (2009)

    Article  MATH  Google Scholar 

  42. Rodriguez-Serrano, J., Perronnin, F.: A model-based sequence similarity with application to handwritten word-spotting. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2108–2120 (2012)

    Article  Google Scholar 

  43. Rodríguez-Serrano, J., Perronnin, F., Sánchez, G., Lladós, J.: Unsupervised writer adaptation of whole-word HMMs with application to word-spotting. Pattern Recognit. Lett. 31(8), 742–749 (2010)

    Article  Google Scholar 

  44. Rothacker, L., Rusiñol, M., Fink, G.: Bag-of-features hmms for segmentation-free word spotting in handwritten documents. In: Proceedings of the International Conference on Document Analysis and Recognition, pp. 1305–1309 (2013)

  45. Rothfeder, J., Feng, S., Rath, T.: Using corner feature correspondences to rank word images by similarity. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshop, p. 30 (2003)

  46. Rusiñol, M., Aldavert, D., Toledo, R., Lladós, J.: Efficient segmentation-free keyword spotting in historical document collections. Pattern Recognit. 48(2), 545–555 (2015)

    Article  Google Scholar 

  47. Rusiñol, M., Aldavert, D., Toledo, R., Lladós, J.: Browsing heterogeneous document collections by a segmentation-free word spotting method. In: Proceedings of the International Conference on Document Analysis and Recognition, pp. 63–67 (2011)

  48. Rusiñol, M., Lladós, J.: Word and symbol spotting using spatial organization of local descriptors. In: Proceedings of the IAPR Workshop on Document Analysis System, pp. 489–496 (2008)

  49. Sankar, P., Jawahar, C., Manmatha, R.: Nearest neighbor based collection ocr. In: Proceedings of the IAPR Workshop on Document Analysis Systems, pp. 207–214 (2010)

  50. Sankar, P., Manmatha, R., Jawahar, C.: Large scale document image retrieval by automatic word annotation. Int. J. Doc. Anal. Recognit. 17(1), 1–17 (2014)

    Article  Google Scholar 

  51. Shekhar, R., Jawahar, C.: Word image retrieval using bag of visual words. In: Proceedings of the IAPR Workshop on Document Analysis Systems, pp. 297–301 (2012)

  52. Shekhar, R., Jawahar, C.: Word image retrieval using bag of visual words. In: Proceedings of the Document Analysis Systems Workshop, pp. 297–301 (2013)

  53. Sivic, J., Zisserman, A.: Video google: a text retrieval approach to object matching in videos. In: Proceedings of the International Conference on Computer Vision, pp. 1470–1477 (2003)

  54. Srihari, S., Ball, G.: Language independent word spotting in scanned documents. In: Digital Libraries: Universal and Ubiquitous Access to Information, Lecture Notes on Computer Science, vol. 5362, pp. 134–143 (2008)

  55. Wang, J., Yang, J., Yu, K., Lv, F., Huang, T., Gong, Y.: Locality-constrained linear coding for image classification. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 3360–3367 (2010)

  56. Wang, P., Eglin, V., Largeron, C., Lladós, J., Fornés, A., Garcia, C.: A novel learning-free word spotting approach based on graph representation. In: Proceedings of the IAPR Workshop on Document Analysis System (2014)

  57. van der Zant, T., Shoemaker, L., Haak, K.: Handwritten-word spotting using biologically inspired features. IEEE Trans. Pattern Anal. Mach. Intell. 30(11), 1945–1957 (2008)

    Article  Google Scholar 

  58. Zhang, X., Tan, C.: Segmentation-free keyword spotting for handwritten documents based on heat kernel signature. In: Proceedings of the International Conference on Document Analysis and Recognition, pp. 827–831 (2013)

  59. Zhu, Q., Yeh, M.C., Cheng, K.T., Avidan, S.: Fast human detection using a cascade of histograms of oriented gradients. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 1491–1498 (2006)

Download references

Acknowledgments

This work has been partially supported by the Spanish Ministry of Education and Science under projects TIN2011-25606 (SiMeVé), and TIN2012-37475-C02-02, by the People Programme (Marie Curie Actions) of the Seventh Framework Programme of the European Union (FP7/2007-2013) under REA Grant agreement No. 600388, and by the Agency of Competitiveness for Companies of the Government of Catalonia, ACCIÓ.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Marçal Rusiñol.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aldavert, D., Rusiñol, M., Toledo, R. et al. A study of Bag-of-Visual-Words representations for handwritten keyword spotting. IJDAR 18, 223–234 (2015). https://doi.org/10.1007/s10032-015-0245-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10032-015-0245-z

Keywords

Navigation