RAVEN Project

1. Introduction

(We welcome any suggestions! Please provide them by emailing the first author, Xinyun Cao, at [email protected] directly.)
This is a Unity project that showcases the RAVEN, a system that enables BLV users to issue queries and modification prompts to improve the runtime accessibility of 3D virtual scenes. The system is developed in Unity version 2021.3.8f1.

2. Project Structure

The following are the sub-folders in the project's Assets folder and their corresponding contents. _RAVEN_Util: Contains prefabs and scripts that a developer might need to attach to the scene to incorporate RAVEN.
Materials: Contains the Materials used in the RAVEN system and the Example Scene.
Packages: Contains packages used in the project, including TextMeshPro, Text-to-speech, Windows Voice, and Web Socket packages.
Prefabs: Contains prefabs that are used in the system and example scenes, including canvas and the loading icon.
Resources: Contains the audio file for the torch fire sound used in the Example Scene.
Scenes: Contain one scene, Example Scene, which showcases an example setup of RAVEN.
Scripts: The scripts that cover the inner workings of RAVEN. User need not interact with this folder, only need to interact with _RAVEN_Util.

3. Implementation Steps

To implement RAVEN, use the following steps:

Step 1. Add Manager

Add the RAVEN_Managers prefab (in the _RAVEN_Util folder) into your scene. This will include all the inner logic of the system and the UI.

Step 2. Indicate Player

Make sure your player is named “Player”. Add a TextDescription.cs to your Player object. Check the “Meta Obj” and “Is Player” booleans to be true.

Step 3. Tag Important Scene Objects

Now, for each important object in the scene, attach a TextDescription.cs file. Make sure the naming of the game object is semantic (not random gibberish). Add optional AdditionalDescription if you wish (these are any additional semantic, sound, etc. info you want to add).
Make sure “Meta Obj” and “Is Player” tags are accurate.

• Meta Obj: Whether this object has a meaningful 3D location (not meta), or if it is abstract, like sunlight, ambient sound, etc (meta)
• Is Player: pretty self-explanatory

Note on "important objects":

• Important object: Things that are important to your (imaginary) scene/game experience for Blind and Low vision users. A key, a health potion, critical sound sources, etc.
• Less important objects: purely decorative objects, environmental elements less relevant to the experience.

Step 4. Add API keys

The RAVEN system needs 2 API keys.

1. Under RAVEN_Manager in your scene, there is a child object named GPTManagers, which has a Component Script named Custom GPT. It has a field named Api Key. You need to provide your GPT API key in this field.

2. If you want the Text-to-speech component of RAVEN to work, you need to click on another child object of RAVEN_Manager named TEXT_TO_SPEECH. This object should have a Component Script Text To Speech that has a field named Api Key. You need to provide your Google Cloud API key in this field.

4. Acknowledgement

This system is built on top of GROMIT by Nicholas Nennings et. al., open-sourced here: https://github.com/NicholasJJ/GROMIT.