End-to-end let's play commentary generation using multi-modal video representations

Chengxi Li; Sagar Gandhi; Brent Harrison

doi:10.1145/3337722.3341870

End-to-end let's play commentary generation using multi-modal video representations

Chengxi Li, Sagar Gandhi, Brent Harrison

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

In this paper, we explore how multi-modal video representations can be applied in an end-to-end fashion for automatically generating game commentary based on Let's Play videos using deep learning. We introduce a comprehensive pipeline that involves directly taking videos from YouTube and then using a sequence-to-sequence strategy to learn how to generate appropriate commentary. We evaluate our framework using Let's Play commentaries for the game Getting Over It with Bennet Foddy. To test the quality of the commentary generation, we apply perplexity to evaluate our language models using different input video representations to highlight different aspects of gameplay that might influence commentary.

Original language	English
Title of host publication	Proceedings of the 14th International Conference on the Foundations of Digital Games, FDG 2019
Editors	Foaad Khosmood, Johanna Pirker, Thomas Apperley, Sebastian Deterding
ISBN (Electronic)	9781450372176
DOIs	https://doi.org/10.1145/3337722.3341870
State	Published - Aug 26 2019
Event	14th International Conference on the Foundations of Digital Games, FDG 2019 - San Luis Obispo, United States Duration: Aug 26 2019 → Aug 30 2019

Publication series

Name	ACM International Conference Proceeding Series

Conference

Conference	14th International Conference on the Foundations of Digital Games, FDG 2019
Country/Territory	United States
City	San Luis Obispo
Period	8/26/19 → 8/30/19

Bibliographical note

Publisher Copyright:
© 2019 ACM.

Keywords

Commentary generation
Multi-modality
Sequence to sequence

ASJC Scopus subject areas

Software
Human-Computer Interaction
Computer Vision and Pattern Recognition
Computer Networks and Communications

Access to Document

10.1145/3337722.3341870

Cite this

Li, C., Gandhi, S., & Harrison, B. (2019). End-to-end let's play commentary generation using multi-modal video representations. In F. Khosmood, J. Pirker, T. Apperley, & S. Deterding (Eds.), Proceedings of the 14th International Conference on the Foundations of Digital Games, FDG 2019 Article 76 (ACM International Conference Proceeding Series). https://doi.org/10.1145/3337722.3341870

@inproceedings{f60205a3614743e894a069f8a2b0e5b8,

title = "End-to-end let's play commentary generation using multi-modal video representations",

abstract = "In this paper, we explore how multi-modal video representations can be applied in an end-to-end fashion for automatically generating game commentary based on Let's Play videos using deep learning. We introduce a comprehensive pipeline that involves directly taking videos from YouTube and then using a sequence-to-sequence strategy to learn how to generate appropriate commentary. We evaluate our framework using Let's Play commentaries for the game Getting Over It with Bennet Foddy. To test the quality of the commentary generation, we apply perplexity to evaluate our language models using different input video representations to highlight different aspects of gameplay that might influence commentary.",

keywords = "Commentary generation, Multi-modality, Sequence to sequence",

author = "Chengxi Li and Sagar Gandhi and Brent Harrison",

note = "Publisher Copyright: {\textcopyright} 2019 ACM.; 14th International Conference on the Foundations of Digital Games, FDG 2019 ; Conference date: 26-08-2019 Through 30-08-2019",

year = "2019",

month = aug,

day = "26",

doi = "10.1145/3337722.3341870",

language = "English",

series = "ACM International Conference Proceeding Series",

editor = "Foaad Khosmood and Johanna Pirker and Thomas Apperley and Sebastian Deterding",

booktitle = "Proceedings of the 14th International Conference on the Foundations of Digital Games, FDG 2019",

}

Li, C, Gandhi, S & Harrison, B 2019, End-to-end let's play commentary generation using multi-modal video representations. in F Khosmood, J Pirker, T Apperley & S Deterding (eds), Proceedings of the 14th International Conference on the Foundations of Digital Games, FDG 2019., 76, ACM International Conference Proceeding Series, 14th International Conference on the Foundations of Digital Games, FDG 2019, San Luis Obispo, United States, 8/26/19. https://doi.org/10.1145/3337722.3341870

End-to-end let's play commentary generation using multi-modal video representations. / Li, Chengxi; Gandhi, Sagar; Harrison, Brent.
Proceedings of the 14th International Conference on the Foundations of Digital Games, FDG 2019. ed. / Foaad Khosmood; Johanna Pirker; Thomas Apperley; Sebastian Deterding. 2019. 76 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - End-to-end let's play commentary generation using multi-modal video representations

AU - Li, Chengxi

AU - Gandhi, Sagar

AU - Harrison, Brent

PY - 2019/8/26

Y1 - 2019/8/26

N2 - In this paper, we explore how multi-modal video representations can be applied in an end-to-end fashion for automatically generating game commentary based on Let's Play videos using deep learning. We introduce a comprehensive pipeline that involves directly taking videos from YouTube and then using a sequence-to-sequence strategy to learn how to generate appropriate commentary. We evaluate our framework using Let's Play commentaries for the game Getting Over It with Bennet Foddy. To test the quality of the commentary generation, we apply perplexity to evaluate our language models using different input video representations to highlight different aspects of gameplay that might influence commentary.

AB - In this paper, we explore how multi-modal video representations can be applied in an end-to-end fashion for automatically generating game commentary based on Let's Play videos using deep learning. We introduce a comprehensive pipeline that involves directly taking videos from YouTube and then using a sequence-to-sequence strategy to learn how to generate appropriate commentary. We evaluate our framework using Let's Play commentaries for the game Getting Over It with Bennet Foddy. To test the quality of the commentary generation, we apply perplexity to evaluate our language models using different input video representations to highlight different aspects of gameplay that might influence commentary.

KW - Commentary generation

KW - Multi-modality

KW - Sequence to sequence

UR - http://www.scopus.com/inward/record.url?scp=85072828377&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85072828377&partnerID=8YFLogxK

U2 - 10.1145/3337722.3341870

DO - 10.1145/3337722.3341870

M3 - Conference contribution

AN - SCOPUS:85072828377

T3 - ACM International Conference Proceeding Series

BT - Proceedings of the 14th International Conference on the Foundations of Digital Games, FDG 2019

A2 - Khosmood, Foaad

A2 - Pirker, Johanna

A2 - Apperley, Thomas

A2 - Deterding, Sebastian

T2 - 14th International Conference on the Foundations of Digital Games, FDG 2019

Y2 - 26 August 2019 through 30 August 2019

ER -

End-to-end let's play commentary generation using multi-modal video representations

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this