HOLOS-T2 - J.A.R.V.I.S

A Readme file for HOLOS Team 2 - J.A.R.V.I.S for CS 639 Capstone

GitHub Repo: https://github.com/Utkarsh305/Holos.git

Google Drive folder with the assets and project zip: https://drive.google.com/drive/folders/1EEqfqH1D0YkakREJzS_f8ykGt3NKDo0A?usp=sharing

Project Overview

The project is an innovative educational Mixed Reality application utilizing both Virtual Reality (VR) and Augmented Reality (AR) technologies, designed specifically for Formula 1 enthusiasts. Created with Unity and Blender, this application offers users an engaging and interactive way of learning the sport. Users can explore the intricate details of an F1 car, manipulating various components of the car in the augmented world and also navigate around the circuit providing an illusion of a real-life experience. In addition, the application enables the user to communicate and ask questions to the AI by voice and text in the VR world in real time.

Project Setup Instructions

The entire project was developed on Unity and run on the Meta Quest 3.

Steps to set up the project

1. Download and Install Unity from the internet.
2. Download the test.zip file from the above provided Google Drive link
3. Upload the unzipped file to Unity and download the Unity editor version - 2022.3.18f1
4. Connect the Meta Quest 3 Headset to the PC through the Quest cable
5. Launch the Oculus Quest link
6. Run and Build the Unity Project.

Overview of how the project works

The entire Unity project can be divided into five scenes: Intro-Scene, Augmented View, Exploded View, Map View, and Aerodynamic View.

Scene 1 - Intro-Scene

As the name suggests, the "Intro-Scene" is the first and the landing screen of our unity project. It lands the user in an F1 car garage that contains a non-interactable F1 car and a Menu canvas that enables the user to navigate to other screens. The user can also move around the garage.

Scene 2 - Augmented View

This is an Augmented reality scene in which the F1 car spawns in our space. The user can interact with the car using their hands or the controller. The user can grab and expand different parts of the car, such as the wheels, steering wheel, front wing, and rear wing. It also contains a canvas that allows the user to navigate to other scenes

Scene 3 - Exploded view

This is another Augmented Reality scene of our Unity project. In this scene, the same parts mentioned in the previous scene explode/separate from the body of the car on pressing the "A" button on the controller, and those parts reassemble back to their original positions on pressing the "A" button again. This was implemented using AI Navmesh technology in Unity. It also contains a canvas that allows the user to navigate to other scenes.

Scene 4 - Map View

This is the fourth scene of the project, which includes the Bahrain circuit(3D Model), the augmented car from "Scene 2—Augmented View", and the AI agent. The user can navigate and explore the circuit by walking on the entire track. Additionally, there is an AI agent that will allow users to ask any question related to F1 and Text labels attached to each part of the F1 car. It also contains a canvas that allows the user to navigate to other scenes.

Scene 5 - Aerodynamic View

This is the final scene of the project, which replicates the Aerodynamics of the car and how the wind particles interacts with a moving F1 car. It also contains a canvas that allows the user to navigate to other scenes.

Setting up OpenAI

The AI agent in the project uses the OpenAI API to access the functionalities of ChatGPT and WhisperGPT. You need an API key to use the OpenAI API.

Steps to setup

1. Go to https://openai.com/api and create an account
2. Then navigate to account -> API keys
3. Create a new API key and copy it
4. Go to the script ChatGPTWithWisper and replace "use your API key" with the copied key in line X.

NOTE - Your API key is secret and refrain from sharing it with others!!!!.

What works & what doesn’t

As a team, we achieved to implement what we wanted and more and exceeded our expectations. We implemented a total of 5 scenes, all having different functionalities, from an interactable F1 car to navigating an actual F1 circuit. However, due to the competitive nature of the sport and time constraints, we were not able to display/acquire the internal workings of the car, which led to a limited number of interactions with the components of the car. Also, the exploded view lacks educational functionalities compared to what we initially visualized.

What would we work on next

Moving forward, we would like to make the AI agent include eye tracking, which would enable the AI to provide more robust and relevant information. Besides, we want to simulate a real-time lap of an F1 car around the circuit making the application more immersive.