Skip to main content
SpringerLink
Log in

Classification of aesthetic natural scene images using statistical and semantic features

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Aesthetic image analysis is essential for improving the performance of multimedia image retrieval systems, especially from a repository of social media and multimedia content stored on mobile devices. This paper presents a novel method for classifying aesthetic natural scene images by studying the naturalness of image content using statistical features, and reading text in the images using semantic features. Unlike existing methods that focus only on image quality with human information, the proposed approach focuses on image features as well as text-based semantic features without human intervention to reduce the gap between subjectivity and objectivity in the classification. The aesthetic classes considered in this work are (i) Very Pleasant, (ii) Pleasant, (iii) Normal and (iv) Unpleasant. The naturalness is represented by features of focus, defocus, perceived brightness, perceived contrast, blurriness and noisiness, while semantics are represented by text recognition, description of the images and labels of images, profile pictures, and banner images. Furthermore, a deep learning model is proposed in a novel way to fuse statistical and semantic features for the classification of aesthetic natural scene images. Experiments on our own dataset and the standard datasets demonstrate that the proposed approach achieves 92.74%, 88.67% and 83.22% average classification rates on our own dataset, AVA dataset and CUHKPQ dataset, respectively. Furthermore, a comparative study of the proposed model with the existing methods shows that the proposed method is effective for the classification of aesthetic social media images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Adak C, Chaudhuri BB, Blumenstein M (2017) Legibility and aesthetic analysis of handwriting, in proc. ICDAR 175-182

  2. Bhattacharya S, Sukthankar R, Shah M (2011) A holistic approach to aesthetic enhancement of photographs. ACM TOMM 78:1–21

    Google Scholar 

  3. Bum J, Wang JJ, Choo H (2021) Sentiment based sub event segmentation and key photo selection. J Vis Commun Image Represent 74:102973

    Article  Google Scholar 

  4. Cui C, Lin P, Nie X, Jian M, Yin Y (2020) Social-sensed image aesthetics assessment. ACM TOMM 16:1–19

    Article  Google Scholar 

  5. Dai Y (2020) Sample-specific repetitive learning for photo aesthetic auto-assessment and highlight elements analysis. Multimed Tools Appl:1387–1402

  6. Dai Y (2022) Exploring metrics to establish an optimal model for image aesthetic assessment and analysis. J Imaging:85

  7. Deng Y, Loy CC, Tang X (2017) Image aesthetic assessment. IEEE Signal Processing Magazine:80–106

  8. Gattupalli V, Chandakkar PS, Li B (2016) A computational approach to relative aesthetic, in proc. ICPR:2446–2451

  9. Gomez R, Gibert J, Gomez L, Karatzas D (2019) “Exploring hate speech detection in multimodal publications”. https://arxiv.org/abs/1910.03814,

  10. Guntuku SC, Lin W, Carpenter J, Ng WK, Ungar LH, Preotiuc-Pietro D (2017) Studying personality through the content of posted and liked images on twitter

  11. Jang H, Lee JS (2021) Analysis of deep features for image aesthetic assessment. IEEE Access 9:29850–29861

    Article  Google Scholar 

  12. Jiang W, Loui AC, Cerosaletti CD (2010) Automatic aesthetic value assessment in photographic images, in proc. ICME:920–925

  13. Jin X, Wu L, Zhao G, Zhou X, Zhang X, Li X (2018) IDEA: a new dataset for image aesthetic scoring. Multimed Tools Appl:14341–14355

  14. Kim WH, Choi JH, Lee JS (2020) Objectivity and subjectivity in aesthetic quality assessment of digital photograph. IEEE Trans AC 11:493–506

    Google Scholar 

  15. Krishanani D, Shivakumara P, Lu T, Pal U, Lopresti D, Kumar GH (2021) A new context-based features for classification of emotions in photographs. Multimed Tools Appl 80:15589–15618

    Article  Google Scholar 

  16. Kuang Q, Jin X, Zhao Q, Zhou B (2020) Deep multimodality learning for UAV video aesthetic quality assessment. IEEE Trans MM:2623–2634

  17. Kucer M, Messinger DW (2018) Aesthetic inferences for smart mobile devices, in proc. WCACV:1764–1773

  18. Li L, Zhu H, Zhao S, Ding G, Lin W (2020) Personality assisted multi task learning for generic and personalized image aesthetic assessment. IEEE Trans IP 29:388–3910

    MATH  Google Scholar 

  19. Li X, Takayama Y, Shimizu N, Mori T (2009) An aesthetic solution for photo collection page layout, in proc. ICKSE:33–38

  20. Lu P, Yu J, Peng X (2018) Deep conditional color harmony model for image aesthetic assessment, in proc. ICPR, pp 2845-2850

  21. Luo P (2022) Social image aesthetic classification and optimization algorithm in machine learning. Neural Computing and Applications

    Google Scholar 

  22. MacAvaney S, Yao HR, Yang E, Russell K, Goharian N, Frieder O (2019) Hate speech detection: challenges and solutions. PLoS ONE

  23. Mittal A, Moorthy AK, Bovik AC (2012) No reference image quality assessment in the spatial domain. IEEE Trans. IP:4695–4708

  24. Mujumdar A, Krishnan P, Jawahar CV (2016) Visual aesthetic analysis for handwritten document images, in proc. ICFHR:423–428

  25. Murray N, Marchesotti L, Perronnin F (2012) AVA: a large-scale database for aesthetic visual analysis. In Proc CVPR:2408–2415

  26. Muszynski M, Kostaoulas T, Lombardo P, Pun T, Chanel G (2018) Aesthetic highlight detection in movie based on synchronization of spectators reactions. ACM TOMM 14

  27. Pfister J, Kobs K, Hotho A (2021) Self-supervised multi-task pretraining improves image aesthetic assessment. In Proc CVPR-Workshop:816–825

  28. Safari RM, Rahmani AM, Alizadeh SH (2019) User behavior mining on social media: a systematic literature review. Multimed Tools Appl 78:33747–33804

    Article  Google Scholar 

  29. Sankarasubramaiam Y, Munnangi K, Banerjee S, Kuchibhotla A (2010) Paper widgets: visually aesthetic “smarts” for documents, in proc. ICIP:2137–2140

  30. Shen L, Fang R, Yao Y, Geng X, Wu D (2019) No reference stereoscopic image quality assessment based on image distortion and stereo perceptual information. IEEE Trans, ETCI:59–72

  31. Shu Y, Li ZQ, Liu S, Xu G (2020) Learning with privileged information for photo aesthetic assessment. Neurocomputing. 304–316.

  32. Tian X, Dong Z, Yang K, Mei T (2015) Query dependent aesthetic mode with deep learning for photo quality assessment. IEEE Trans MM:2035–2048

  33. “Vision AI | Derive Image Insights via ML | Cloud Vision API”, https://cloud.google.com/vision (accessed Mar. 17, 2021)

  34. Zeng H, Cao Z, Zhang L, Bovik AC (2020) A unified probabilistic formulation of image aesthetic assessment. IEEE Trans IP:1548–1561

  35. Zhang X, Gao X, Lu W, He L, Li J (2019) A gated peripheral foveal convolutional neural network for unified image aesthetic prediction. IEEE Trans. MM:2815–2826

  36. Zhang X, Gao X, Lu W, He L, Li J (2021) Beyond vision: a multimodal recurrent attention convolutional neural network for unified image aesthetic prediction tasks. IEEE Trans MM 23:611–623

    Google Scholar 

  37. Zhang Y, Sun X, Yao H, Qin L, Huang Q (2012) Aesthetic composition representation for portrait photographic recommendation, in proc. ICIP:27523–22756

Download references

Acknowledgments

Palaiahnakote Shivakumara received partial support for this work from the Faculty Grant: GPF096A-2020, GPF096B-2020 and GPF096C-2020, University of Malaya, Malaysia. This work is also partly supported by TIH, ISI.

Author information

Authors and Affiliations

  1. Computer Vision and Pattern Recognition Unit, Indian Statistical Institute, Kolkata, India

    Kunal Biswas & Umapada Pal

  2. Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia

    Palaiahnakote Shivakumara

  3. National Key Lab for Novel Software Technology, Nanjing University, Nanjing, China

    Tong Lu

  4. University of Technology Sydney, Sydney, Australia

    Michael Blumenstein

  5. Computer Vision Center, Universitat Autònoma de Barcelona, Barcelona, Spain

    Josep Lladós

Authors
  1. Kunal Biswas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Palaiahnakote Shivakumara
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Umapada Pal
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tong Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michael Blumenstein
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Josep Lladós
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Palaiahnakote Shivakumara.

Ethics declarations

Conflicts of interests/competing interests

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Biswas, K., Shivakumara, P., Pal, U. et al. Classification of aesthetic natural scene images using statistical and semantic features. Multimed Tools Appl 82, 13507–13532 (2023). https://doi.org/10.1007/s11042-022-13924-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13924-7

Keywords

Search

Navigation