52111845202_e23af0a464_o.jpg

Augmented Reality Interface for NASA Lunar Missions

Problem: How can we give astronauts autonomy to complete their tasks in space without the help of Mission Control?  

Solution: Provide AR tools to help assist and automate the process of major tasks in Space.

Time Frame: 1.5 years

Team: 16-20 Designers & Engineers 

My Role: Project Co-Lead, Design Lead

Client: NASA

Finished Product

AR Interfaces

To assist astronauts during their extravehicular activities

Including navigation on the lunar surface, vitals display, geological sampling tools, search and rescue

Context

52113116739_15643a54f2_o.jpg

Our team's HoloLens Interfaces getting tested by NASA Designer Skye

NASA Spacesuit User Interface Technologies for Students (SUITS) Challenge selected our team to develop Augmented Reality (AR) software on the HoloLens.

With the HoloLens, we can create 3 types of visual elements...

Spacesuit-helmet-with-Iron-Man-HUD_Hypergiant-Industries.webp

Registered in the environment in a set location

Sticky interface registered in the environment following the user's view 

Registered to the hud following the user's view

 

Team & Milestones

Team: 16-20 members

(responsibility changing throughout the challenge)

2 Team Leads: Bowen(Me) & Selena

Design Lead: Bowen(Me)

   4 Design Sub-Leads:

   Geological Sampling

   EVA

   Navigation

   Auxiliary

2 Software-Leads:

   Frontend

   Backend

Research.png

Research

We talked to: 5 AR, voice, navigation designers, 2 NASA Designers, 2 Brown Planetary Geo-science Professors, and 2 Astronauts!

1200px-James_H_Newman.jpeg

James H. Newman

Former NASA astronaut who flew on four Space Shuttle missions.

1200px-Steven_swanson_v2.jpeg

Steve Swanson

Retired NASA astronaut who flew on three missions to the ISS.

283080main_james-head-904.jpeg

James W Head

Worked on previous Apollo Programs and trained astronauts. Geological Science Professor at Brown.

What are some challenges astronauts face on moon?

Difficult to Navigate

3facfe67db3533ec0344de5020ad8031.jpeg

Dangerous terrain with tripping hazards

34347b667bc870feefd4f74b626e876c.jpeg

Environment with no distinctive features for landmark recognition

cacabbd597919c8dd7a423cbd3f00c81_apollo11_traverse.jpeg

No GPS, relied on route memorization and photos

" There are no high mountains or volcano - something distinct that  you could recognize - a landmark would help tremendously." 

Nothing is given to us right now.  We practice memorizing our paths and rely on some pictures. There was one time where astronauts got lost and couldn't find a destination so they had to abandon the tasks there."

Limited Movements

5d25b175a9c4e.jpeg

Whole-body movements are required to accomplish simple tasks

04-7278_custom-30caa5c19bec76e0e2ead7edd51bc30a6df6d920-s1100-c50.jpeg

Small hand gestures require a large amount of effort

Science_lunardust_257757main_as17-134-20425_full.webp

Both hands are required for most tasks

"Glove bulky, no tactile feedback. you are basically in a balloon. Any time you move you are squeezing the balloon."

Limited time 

Each minute in space wasted is a large sum of money lost. Mistakes are costly and there are a lot of tasks to accomplish.

What are their main objectives on the moon?

Navigation

astronaut-walking-on-the-moon.jpeg

Long-Range Point A to Point B

18-nasa-apollo-pictures-moon-landing.jpeg

Short Range Terrain mapping & Obstacle Avoidance

"We bounce around a lot to navigate. The lighting conditions can be extreme and a lot of hazards are hiding in the shadows. We must always pay attention to the ground in front of us."

Geological Sampling

52112885566_8505a34993_o.jpg
main-qimg-f36bd2c7afcf94e7b182cf2ff3f4a390.webp

Taking Samples

Documentations

"We have specific goals when we arrive at a site, we communicate with the geologists at Mission Control and take exact samples. We take notes, measurements, and photos of the site. "

Staying Alive: Vitals & Rescue

what-are-normal-vitals.jpeg

Vitals Display

shanekimbrough.webp

Emergency Navigation & Communication

"If something goes wrong, how quickly can you go back? You always need to know what's wrong, how much oxygen you have left, and what pace you need to travel at under an emergency."

 
 

Design Process

Objective-based User Profiles

separated our team into design sub-teams based on the main mission objectives and conducted 3 scenario-specific user research. 

Navigation

Extravehicular Activities (EVA)

Geological Sampling

Central Profile

jsc2017e108313_alt.jpeg

Anne McClain

Anne is going on the Artemis Mission where she would travel to multiple unexplored locations on the moon for scientific discovery as well as traveling to established sites to update equipment left by previous missions. Show would have a partner traveling with her and another astronaut in the spacecraft.

Anne's Artemis mission mostly consist of...

astronaut-walking-on-the-moon.jpeg

Traveling on foot for around 15-20min intervals

artemis_humans_space_moon_gateway_main.jpeg

Taking samples and documentation from unexplored sites

NASA-Astronauts-Lunar-South-Pole.jpeg

Maintenance and setting up new equipments

Ideation Summery

Based on the challenges & objectives, the teams went through lighting rounds of ideations.

Untitled (1).jpg

💡 Learning to eliminate

We learned to rigorously eliminate ideas and concepts. There were a lot of amazing ideas that would improve the AR experience which we had cut out because we couldn't do everything.  

We have to ask ourselves, is this function/feature necessary to achieve our goals? If not, let's put these aside into our future features bank.

Flow Diagrams

Simplified This...

IMG_4621.HEIC

To This!

Navigation

Untitled (1).jpg

Vitals, Geo-sampling, Emergency Communication

flowchart2.jpg

Design Iterations

7+ Rounds of Iterations

IMG_7996.JPEG
IMG_0821_edited.jpg
Nav - Bigger Map – 11.png
Navigation Instructions-1.png
SUITS FIGMA (Copy)_Page_2.jpg
Screen Shot 2022-06-23 at 4.40.10 PM.png

Iteration Learnings

💡Lessen & attach elements in the environment

Problem: During testing, we observed that users are often confused when there is more than 2 sticky interfaces floating and following them in the environment. It became hard for them to understand where things are especially when they are doing other physical tasks at the same time.

Solution: Minimize the amount of interface registered in the enviorment that follows the user's view to one at all times. Attach elements to each other into a unity to avoid spatial confusion.

Injured3.png
Navigation Instructions-3.png

💡Set separate use case for each plane of interaction

Problem: Users' attention can only focus on 1 plane of interaction at a time. when they are looking at elements registered in the far environment, they miss elements on the floating interface right in front of them, and vice versa.

Solution: Create different use cases for each plane of interaction so users' attention doesn't have to be separated during a flow. We limited all decision-making processes where users have to be focused on the screen to the sticky interfaces floating right in front of them, all guiding tools such as compass and time to the hud, and all elements that are used while users are actively doing something else to the environment, such as the guiding pathway in navigation mode. 

Navigation Instructions.png

Iterations: instead of a sticky arrow, directions are given with a pathway in the environment.


Decisions such as setting a destination and checking vitals are limited to the sticky interfaces close to the user.

SUITS FIGMA (Copy)_Page_4.jpg
SUITS FIGMA (Copy)_Page_3.jpg

💡Automate processes & combine features

Problem: From astronauts, we learned that they have limited movements and small hand gestures require extra effort. To save astronauts' physical energy and attention, we have to minimize the amount of input needed in a flow. 

Solution: On top of designing voice commands, we automated each flow as much as possible. letting one action directly leads to the next, and interfaces open automatically. We combined multiple features into one, and ask ourselves what is the least amount of elements needed to achieve a goal?

Automate Processes

Combining Features

Mini-map's function of illustrating direction could be eliminated by adding a direction indicator on the compass. It's close range function of showing the route could be performed by having a navigational pathway rather than arrows in the environment.

Nav - Feature set View – 2.jpg

Mini-map

Nav - Suit heading matches Nav HDG; FWD arrow (less intrusive) – 1.png
Navigation Instructions-1.png
Navigation System Expanded View.png

Red direction indicator on compass

SUITS FIGMA (Copy)_Page_3.jpg

Pathway in environmental showing approximate closeness to destination with color

💡Clear the field of view,
Test in the HoloLens early!

Problem: From astronauts, we learned that they have limited movements and small hand gestures require extra effort. To save astronauts' physical energy and attention, we have to minimize the amount of input needed in a flow. 

Solution: On top of designing voice commands, we automated each flow as much as possible. letting one action directly leads to the next, and interfaces open automatically. We combined multiple features into one, and ask ourselves what is the least amount of elements needed to achieve a goal?

On Hover of location-2.png
Selena Home Page.png
Navigation System Expanded View.png
When Nav System is Active.png
On Hover of location-5.png
Rescuer 1 first screen-5.png
Navigation System Expanded View.png
 

Final Designs

SUITS FIGMA (Copy)_Page_4.jpg
On pressed of location.png

Finished Products & NASA on-site Testing

Feedbacks from NASA

52112932998_722543eee9_o.jpg

Validations

Speeds up the Current Meeting System

" If you have access to all of this, it will walk you through the process. You would know this is something you have to talk about, and you can set it up right here. "

Opens a conversation up

" I like that you are forcing people to write out stuff about the meeting beforehand, so the conversation is open before we got there. The hardest part is starting the conversation. This makes it easier. "

Reduce Barriers

" This could give people access to help 24/7 without having to go through a supervisor, which is embarrassing if you want to talk about mental issues, and takes forever."

"This would be a less stressful and easier way for someone to reach out and talk about a touchy topic."

Moving Forward...

1

Test the effectiveness for this format of conversations, which involves in-person or video conferencing with three participants, having a structured timeline, etc.

2

Test for the ideal number of people there can be in a conversation intended for learning and understanding

3

If the tests indicate difficulty in conversations, test another concept which involves having a third experienced party to facilitate the conversation

4

Start researching and designing the platform that civilians and civilians in professional fields use to connect with the officers.

Thank You!

Thank you! I am very excited to show you this work and I would love any criticism and welcome any feedback from you!