Ressources

Find bellow links to the Powerpoint document, the slides as PDF, but also links of the website of the PhD and the TFTI PG Symposium event as well as the University of York website.

• Pre-recorded presentation (mp4 1080p)
• Pre-recorded presentation (mp3 audio-only)
• Slides (OpenDocument format)
• Slides (PowerPoint format)
• Slides (PDF format)
• Website of the PhD (gametospeech.com)
• TFTI PG Symposum event
• Department of Theatre, Film, Television and Interactive Media at the University of York

Transcript

At the begining of the presentation, I showed the first 30 seconds of this trailer of the game Assassin’s Creed Valhalla. This trailer was one of the first game trailer with audio description. It was published by Ubisoft in 2020.

The title of the presentation was: Real-time audio description for live streamed games: An early Minecraft prototype. I presented my PhD and what I did in my first year.

In my PhD, I want to advance audio description in gaming. My PhD project focus on gaming live streaming and aims to provide access to this media to visually impaired by generating real-time audio description. I stated that I will also present an early Minecraft prototype at the end of the presentation.

Background

Livestreaming is when a gamer is sharing its gameplay to spectators by live streaming it.

On Twitch, there is an average of 30 million daily visitors (source: Twitch, 2021).

While we are into numbers, it is important to know that 21% of the UK population is disabled (source: UK government, 2021).

If we simply start by linking the accessibility topic and live streaming one, we will find this web page: the Twitch Accessibility Statement. It is basially a web page where Twitch states what they are doing in terms of accessibility. An interesting thing is that there is some exceptions. One of them is about audio description.

An exception is WCAG Success Criteria 1.2.5: Audio Description. It’s not feasible for us to create accurate audio descriptions for the millions of hours of video content streamed on our platform daily. We apologize for this inconvenience.

This exception is first citing the WCAG, the Web Content Accessibility Guidelines. It is guidelines published by the World Wide Web Consortium, a private-public organization that published a lot of web standards and guidelines. Thanks to that, we can deduce that my PhD might be about web accessibility. However, this execption is also mentionning audio description and live streaming is a media. We coudl then think that it is about media accessibility. Finally, Twitch is mainly about gaming content, and we rare talking about improving games' accessibility. So with this excepetion, we discover that my PhD might be in the intersection of web, gaming and media a11y.

Audio description for games

Audio description has been proven effective for movies, TV shows, theater plays, museums… and more. For games, it’s really a new topic. We can find some prototypes with full audio description (i.e. 2D puzzle games). We cannot find that in 3D commercial games. Researchers have stated that commercial games may start with cut-scenes as it is similiar to audio description for movies. I need to mention the US Communications and Video Accessibility Act that mandate some accessibility features for in-game communication: that’s why we can find text-to-speech and speech-to-text features of in-game chats in many games.

Research question

My research question is: To what extent could generated audio description be used to make live streamed games accessible to visually impaired people while ensuring a balance between information and entertainment?. In a sense, my PhD aims to remove the Twitch exception I have previously cited (but for all platforms).

Technological challenges

Today, we gonna focus on some technological challenges of my PhD. Here is a schema of a possible solution and its implementation. On the left, we can find a streamer who is playing a game that is streamed to a live streaming service thanks to a streaming software. The live streaming is then delivered to spectators. I am proposing to add another service which will get game events and output audio description to spectators. It is a simple client-server architecture in parallel of the live streaming service.

I want to focus now the issue of exporting game events: How to get game data? I identified four main ways to do that. I will detail the pros and cons for each of them.

1. Send direct requests to a server: it is a low tech solution where game developers add a piece of code in their game which will send web request to a server. Here, game developers are responsible, by that I mean that they need to be in the loop, if they don’t want to spend time on this accessibility feature, we as researchers and players cannot do a thing.
2. Get data via telemetry SDKs (data analysis tools already embedded in many games): it will be easier to implement for game developpers that may led to a wider availability. But here again, game developers are responsible.
3. Retrieve data with public game APIs: this is sometimes possible in eSports games (games played in competitive events often live streamed). This solution would only work for these type of games.
4. Mod an existing game: modding is when players modify a game with code. From Doom, we can find many games that can be modded. It is great becaus not only recent could benefit from audio description! Here, game developers are not involved.

Demo time

I will present a Minecraft prototype. I chose Minecraft becaus it has a huge community of players and modders. I did not build my mod from scratch, but took a pre-eixsting open source mod called AccessibilityPlus published in 2020 by Luis Sanchez. I modified the mod in order to export data out of Minecraft.

I played a demo video. In it, there was a Mincecraft window on the foreground and a web browser on the background. The web browser was connected to my server. It received data from Minecraft and output audio description thanks to a text-to-speech software. On Minecraft I was pointing my character to some objects and some blocks. For now, the mod only read aloud text written on signs and the blocks’ name. In is really only a technical prototype. I also want to state taht the audio description was really coming from the web browser and that the web browser was on a URL that could be openned from anywhere in the world: basically it is online.

To be continued…

There is still a lot to do. I need to investigate these questions: What to describe? How to make the experience great? And about the question of 'how', there is the question of the method.

Method

About the method, I will do user testing with visually impaired gamers, but not on prototypes like the one I show today, but rather on a vertical slice : a small portion representing the overall experience. I will choose a game and get a live stream footage and write the raw audio description data and embed everything into a software that should let the users customize their experience. As closing words, the accessibility community always says “Nothing about us without us”, and I believe that for every tech we create, the end users should be involved in the process. And that is what I will do in my PhD!

Conclusion

Thanks for listening! Questions? Ideas? Funding? Reference list? Contact me: phd@davidlibeau.fr.

Text of the slides

Photosensitive epilepsy warning: some light flashes in this video.

Real-time audio description for live streamed games

An early Minecraft prototype.

Background

• Live streaming

a gamer sharing its gameplay to spectators by live streaming it

30 million daily visitors on Twitch (twitch.tv 2021)

21% of the UK population is disabled (UK gov 2021)

An exception is WCAG Success Criteria 1.2.5: Audio Description. It’s not feasible for us to create accurate audio descriptions for the millions of hours of video content streamed on our platform daily. We apologize for this inconvenience.

• Web, game, media accessibility?
• Web Content Accessibility Guidelines (WCAG).
• Live streaming is a new entertainment media.
• Improving games’ accessibility.

AD for games

• Narrator features
• Some prototypes with full audio description (i.e. 2D puzzle games).
• Commercial games may start with cut-scenes.
• In-game communication (mandatory).

Research question

To what extent could generated audio description be used to make live streamed games accessible to visually impaired people while ensuring a balance between information and entertainment?

Technological challenges

• How to get game data?

Demo time

• Minecraft prototype
• A mod that export data out of Minecraft.
• Based on the AccessibilityPlus mod published in 2020 by Luis Sanchez.

To be continued…

• A lot to do:
• What to describe?
• How to make the experience great?

Method

• User testing with visually impaired gamers.
• Vertical slice of the overall experience.

“Nothing about us without us”
– The accessibility community, always.

Thanks for listening.

Questions? Ideas? Funding? Reference list? Contact me: phd@davidlibeau.fr

Slides: dav.li/TFTIPGS21
Website: gametospeech.com