3M: Multi-style image caption generation using Multi-modality features under Multi-UPDOWN model

Chengxi Li; Brent Harrison

doi:10.32473/flairs.v34i1.128380

3M: Multi-style image caption generation using Multi-modality features under Multi-UPDOWN model

Chengxi Li, Brent Harrison

Research output: Contribution to journal › Conference article › peer-review

2 Scopus citations

Abstract

In this paper, we build a multi-style generative model for stylish image captioning which uses multimodality image features, ResNeXt features, and text features generated by DenseCap. We propose the 3M model, a Multi-UPDOWN caption model that encodes multi-modality features and decodes them into captions. We demonstrate the effectiveness of our model on generating human-like captions by examining its performance on two datasets, the PERSONALITYCAPTIONS dataset, and the FlickrStyle10K dataset. We compare against a variety of state-of-the-art baselines on various automatic NLP metrics such as BLEU, ROUGE-L, CIDEr, SPICE, etc¹ . A qualitative study has also been done to verify our 3M model can be used for generating different stylized captions.

Original language	English
Journal	Proceedings of the International Florida Artificial Intelligence Research Society Conference, FLAIRS
Volume	34
DOIs	https://doi.org/10.32473/flairs.v34i1.128380
State	Published - 2021
Event	34th International Florida Artificial Intelligence Research Society Conference, FLAIRS-34 2021 - North Miami Beach, United States Duration: May 16 2021 → May 19 2021

Bibliographical note

Publisher Copyright:
© 2021by the authors. All rights reserved.

ASJC Scopus subject areas

Artificial Intelligence
Software

Access to Document

10.32473/flairs.v34i1.128380

Cite this

@article{42d1e7ce75c844d0903d62169ac139c9,

title = "3M: Multi-style image caption generation using Multi-modality features under Multi-UPDOWN model",

abstract = "In this paper, we build a multi-style generative model for stylish image captioning which uses multimodality image features, ResNeXt features, and text features generated by DenseCap. We propose the 3M model, a Multi-UPDOWN caption model that encodes multi-modality features and decodes them into captions. We demonstrate the effectiveness of our model on generating human-like captions by examining its performance on two datasets, the PERSONALITYCAPTIONS dataset, and the FlickrStyle10K dataset. We compare against a variety of state-of-the-art baselines on various automatic NLP metrics such as BLEU, ROUGE-L, CIDEr, SPICE, etc1 . A qualitative study has also been done to verify our 3M model can be used for generating different stylized captions.",

author = "Chengxi Li and Brent Harrison",

note = "Publisher Copyright: {\textcopyright} 2021by the authors. All rights reserved.; 34th International Florida Artificial Intelligence Research Society Conference, FLAIRS-34 2021 ; Conference date: 16-05-2021 Through 19-05-2021",

year = "2021",

doi = "10.32473/flairs.v34i1.128380",

language = "English",

volume = "34",

}

TY - JOUR

T1 - 3M

T2 - 34th International Florida Artificial Intelligence Research Society Conference, FLAIRS-34 2021

AU - Li, Chengxi

AU - Harrison, Brent

PY - 2021

Y1 - 2021

N2 - In this paper, we build a multi-style generative model for stylish image captioning which uses multimodality image features, ResNeXt features, and text features generated by DenseCap. We propose the 3M model, a Multi-UPDOWN caption model that encodes multi-modality features and decodes them into captions. We demonstrate the effectiveness of our model on generating human-like captions by examining its performance on two datasets, the PERSONALITYCAPTIONS dataset, and the FlickrStyle10K dataset. We compare against a variety of state-of-the-art baselines on various automatic NLP metrics such as BLEU, ROUGE-L, CIDEr, SPICE, etc1 . A qualitative study has also been done to verify our 3M model can be used for generating different stylized captions.

AB - In this paper, we build a multi-style generative model for stylish image captioning which uses multimodality image features, ResNeXt features, and text features generated by DenseCap. We propose the 3M model, a Multi-UPDOWN caption model that encodes multi-modality features and decodes them into captions. We demonstrate the effectiveness of our model on generating human-like captions by examining its performance on two datasets, the PERSONALITYCAPTIONS dataset, and the FlickrStyle10K dataset. We compare against a variety of state-of-the-art baselines on various automatic NLP metrics such as BLEU, ROUGE-L, CIDEr, SPICE, etc1 . A qualitative study has also been done to verify our 3M model can be used for generating different stylized captions.

UR - http://www.scopus.com/inward/record.url?scp=85131141648&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85131141648&partnerID=8YFLogxK

U2 - 10.32473/flairs.v34i1.128380

DO - 10.32473/flairs.v34i1.128380

M3 - Conference article

AN - SCOPUS:85131141648

SN - 2334-0754

VL - 34

JO - Proceedings of the International Florida Artificial Intelligence Research Society Conference, FLAIRS

JF - Proceedings of the International Florida Artificial Intelligence Research Society Conference, FLAIRS

Y2 - 16 May 2021 through 19 May 2021

ER -

3M: Multi-style image caption generation using Multi-modality features under Multi-UPDOWN model

Abstract

Bibliographical note

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this