Category : AR/VR

Designing for Vision Pro: the coming spatial experience

Oct 12,2023 / By Parth Khandekar / No Comments

As the real and virtual worlds continue to converge, Apple has introduced Vision Pro, a device designed to deliver immersive mixed-reality experiences. Vision Pro enables endless possibilities for exploring human-to-system interactions and spatial dynamics. Users can interact with the digital world more naturally using the mixed-reality headset. It overlaps digital content in the real world, allowing users to simultaneously see and interact with both worlds. Vision Pro is slated for limited release to the public next year, but Apple has already given a demo to select developers.

Vision Pro features two high-resolution micro-OLED displays that project images directly onto the user’s retinas, creating a “screenless” (screen without a screen) experience. The headset has 12 built-in cameras for tracking the user’s head, eyes, and hands, a LiDAR scanner for depth perception, and an Apple M2 chip with 10 CPU and 32 GPU cores.

Vision Pro is designed with accessibility at the center, using gestures, voice, and combinations of these input modes as critical enablers. This was showcased in the launch event video, where users could interact with the headset using gestures to browse the web, watch movies, and play games. Designing for Vision Pro requires understanding these input modes and providing solutions for users to have a seamless journey.

Source: MacRumors Apple Vision Pro Gestures

Designing for a spatial experience

Design principles for traditional screens may not work for the future of spatial experiences, which are more immersive and interactive. Designers need to adapt these principles to consider the unique features of spatial experiences. Spatial experiences allow users to interact with the environment more naturally, so designers must create interfaces that are easy to use and understand.

Vision Pro’s spatial capabilities allow designers to create visually captivating and user-friendly interfaces that feel intuitive and natural to navigate. This focus on user-centric design enhances the user experience across various applications. Below are some key aspects to consider while designing for glass-style UI:

Iconography

To create a seamless user experience, Apple has reimagined the feedback mechanism for icons on the home screen. When a user looks at an icon, it expands as if it is being hovered over. Designers design icons with subtle depth by adding specular shadows and highlights.

To design a great icon, start by creating multiple layers. The system uses these flat layers to create a three-dimensional effect. The icon should be converted to 1024×1024 pixels, with transparent foreground layers. Do not use large regions of semi-transparent pixels, as they will blend with the shadow behind them.

Consider these additional recommendations when designing icons:

Maintain a uniform color scheme and style throughout your icons.
Make sure your icons are clear and easy to understand.
Use high-quality images and graphics.
Test your icons on different devices and screen sizes to ensure they look good everywhere.

Source: App icons | Apple Developer Documentation

Glass panels design

Apple has introduced glass material to create a more spatial and lightweight user experience. This material allows users to see what is behind a window, such as other apps or people, without feeling suffocated.

Therefore, when designing an app window, it is essential to avoid using solid colors. Too many opaque windows can make the interface feel heavy. Instead, use a lighter material to bring attention to interactive elements, such as buttons, or a darker material to separate sections of the app.

For example, if you want to design a lock-up with a lighter button, you can place it on top of the glass material. Or, if you’re going to create more contrast, you can use a darker cell behind the button. However, it is crucial to avoid stacking lighter materials on top of each other, as this can impact legibility and reduce contrast.

Consider these bonus tips for designing with glass material:

Use glass material sparingly. Too much of it can make the interface feel cluttered.
Use glass material to create a sense of depth. For example, you can use a darker glass material for the background and a lighter glass material for the foreground.
Use glass material to highlight essential elements. For example, you can use a lighter glass material for buttons or other interactive elements.

Source: Materials | Apple Developer Documentation

Legibility

Typography

The font-weight can be slightly increased to improve text contrast against vibrant materials. For example, on iOS, regular weight for the body text style must be used; on this platform, a medium can be used. And for titles, instead of semi-bold, bold can be used. This makes the text more legible, even when displayed on a vibrant background. System fonts, which are designed for optimized legibility, can also be used.

Vibrancy

Vibrancy is another crucial detail for maintaining legibility. It enhances the foreground content by bringing light and color from the background forward. On this platform, vibrancy updates in real-time to ensure your text is always legible, even when the background constantly changes. Vibrancy can indicate hierarchy for text, symbols, and fills. Primary vibrancy can be used for standard text, and secondary vibrancy can be used for less critical text.

Pointers for using typography and vibrancy:

Use a heavier font weight for text that needs to be legible, such as body text and titles.
Use system fonts or other fonts that are designed for optimized legibility.
Use vibrancy to brighten foreground content and make it stand out from the background.
Use primary vibrancy for standard text and secondary vibrancy for less critical text.
When using custom fonts, make sure they are designed for readability.
Avoid using small or lightweight fonts, which can be challenging to read, especially on large screens.
Consider using a darker shade for the pop-over background to make the text more legible.

Source: Typography | Apple Developer Documentation

Colors

When designing a glass-style UI, use white text and icons on a colored background. This will make the text and icons stand out and be more legible.

Use system colors whenever possible. System colors are designed for legibility and look best on a glass background.
If you need a custom color, use it sparingly and make sure it contrasts nicely with the background.
Avoid using dark colors for text or icons. Dark colors will blend in with the background, making the text and icons challenging to read.

Here are some additional tips for designing glass-style UIs:

Use a light overall color palette. This will help to create a sense of spaciousness and airiness.
Use transparency and blur effects to create a sense of depth.
Use shadows to add dimension to the UI.
Use gradients to add interest and visual interest.

Source: VisionOS – Apple Developer

How Vision Pro can transform businesses

Businesses of all sizes and across some industries are excited about the potential of Apple Vision Pro, which is still in its early stages of development. The technology has the potential to transform operations, improve customer experiences, and boost overall performance. Still, which industries will most successfully adopt it remains to be seen.

Entertainment

Create immersive gaming experiences that blur the line between the virtual and real worlds. To illustrate, game developers can use Apple Vision Pro to create realistic first-person shooter games where players can interact with the environment lifelike.
Create interactive storytelling experiences. Filmmakers can use Apple Vision Pro to create 3D movies, transporting viewers into breathtaking cinematic worlds.
Provide real-time translation of foreign language text. For example, language learners can use Apple Vision Pro to get a real-time translation of foreign language text while traveling or interacting with people from other cultures.

Source: How will Apple Vision Pro VR influence industries | Merge Development

Education and training

Provide students with interactive learning experiences. For instance, teachers can use Apple Vision Pro to take students on virtual field trips to historical sites or to conduct experiments in a safe and controlled environment.
Offer virtual field trips. For example, students can use Apple Vision Pro to visit museums or other educational institutions without leaving their homes.
Provide real-time translation of foreign language text. Concretely, language learners can use Apple Vision Pro to get the real-time translation of foreign language text while taking a class or reading a book.

Healthcare and medical

Provide realistic surgical simulations and training scenarios. In other words, doctors and medical students can use Apple Vision Pro to practice procedures without risking harming a patient.
Offer remote consultations with patients. For example, doctors can use Apple Vision Pro to consult with patients in remote areas.
Visualize and analyze medical data. For instance, researchers can use Apple Vision Pro to visualize and analyze medical images and data to understand diseases better and develop new treatments.

Source: How will Apple Vision Pro VR influence industries | Merge Development

Real estate and architecture

Give potential buyers virtual tours of properties. Real estate agents can use Apple Vision Pro to give potential buyers a 360-degree view of a property without meeting them in person.
Collaborate with clients and stakeholders on 3D design projects. Architects can use Apple Vision Pro to collaborate with clients and stakeholders on 3D design projects in real time.
Visualize furniture and decor in a physical space. For example, interior designers can use Apple Vision Pro to visualize how furniture and decor will look in a room before making a purchase.

Meetings

Join virtual meetings from anywhere. For instance, remote workers can use Apple Vision Pro to join virtual discussions from anywhere.
Collaborate in real-time and share information. For example, participants in virtual meetings can use Apple Vision Pro to collaborate in real time and share information.
Track customer behavior and improve the shopping experience. Concretely, businesses can use Apple Vision Pro to track customer behavior in a retail store and use that data to enhance the shopping experience.

Source: Apple’s Vision Pro: Revolutionizing Industries Through Spatial Computing| ELEKS

Finance and banking

Visualize financial data. For example, financial analysts can use Apple Vision Pro to visualize financial data to understand the market better and make informed investment decisions.
Help clients track their spending. For example, personal finance managers can use Apple Vision Pro to help clients track their spending and reach their financial goals.
Visit virtual bank branches. Concretely, customers can use Apple Vision Pro to visit virtual bank branches to conduct transactions or speak to a banker.

Retail and e-commerce

Try on clothes and accessories before making a purchase. To illustrate, shoppers can use Apple Vision Pro to try on clothes and accessories before purchasing.
Provide personalized shopping recommendations. Retailers can use Apple Vision Pro to provide personalized shopping recommendations based on a shopper’s past purchases and browsing history.
Offer in-store navigation. For instance, businesses can use Apple Vision Pro to offer in-store navigation to help customers find the products they are looking for.

Source: Apple Vision Pro and The Future of ECommerce (codilar.com)

Vision Pro: the flip side

Innovation can be exciting, promising a brighter future with endless possibilities. However, carefully considering the potential consequences of new technologies is essential. While innovation often brings benefits, it can also come with risks, such as privacy concerns, surveillance risks, and impacts on mental health and social isolation. Balancing progress with responsibility when developing and using new technologies is essential.

Comfort and ergonomics

Early reports suggest that the Vision Pro is well-built but slightly uncomfortable to wear for extended periods. The headset is front-heavy due to its metal construction, which could make it difficult to wear for long periods. Additionally, the headset’s weight distribution and heat management could further impact user comfort. If the Vision Pro is not designed to be comfortable, it may limit its appeal to consumers and businesses.

Privacy at stake

The Vision Pro raises fundamental concerns about personal privacy. The headset’s ability to project floating screens onto our vision while observing our environment could collect vast data about us. This data could include our eye movements, facial expressions, and surroundings. The potential for this data to be used to track our movements, monitor our behavior, and even identify us is a serious privacy concern. Establishing robust safeguards and ethical boundaries is essential to protect individuals’ privacy in the digital age.

Mental health and social isolation

While technological advancements can enhance our lives, we must also consider their potential impact on mental well-being and social dynamics. The Vision Pro’s immersive AR experience could be a double-edged sword. On the one hand, it offers captivating virtual overlays of reality that could tempt users to immerse themselves in a captivating digital realm. On the other hand, this allure could come at the risk of isolating individuals from their physical surroundings and authentic human connections. As we increasingly detach from the present moment and substitute genuine interactions with virtual experiences, the potential for social isolation and its associated mental health consequences is serious.

Source: Apple Developer Forums

Apple’s Vision Pro hasn’t even been released yet, but the company is already planning a smaller and lighter version of the headset and is deep into work on follow-up products.

Conclusion

Apple Vision Pro can redefine how we interact with technology in a human-centric way. Its spatial capabilities allow designers to create visually appealing, user-friendly interfaces that feel natural. This focus on human-centric design will enhance and transform businesses of all sizes and industries by enabling immersive, interactive, and personalized experiences in various applications, from gaming and entertainment to education and training. While Vision Pro can be a powerful tool for good, it is essential to remember that it is also a new technology with potential risks. Our collective responsibility is to ensure that Vision Pro is used responsibly in a human-centric way.

AR/VR Artificial Intelligence Tech Talk

How emerging technologies can enable the ‘next gen’ of education

Oct 28,2022 / By Vishnu Maji / No Comments

In 2021, online learning platform Coursera reported 20 million new learners in the year, equal to the total growth of the three years prior. The COVID-19 pandemic triggered an exponential jump in the already upward trajectory of online learning. Work from home, virtual classrooms, and time to pursue learning new skills saw the US record the highest growth in online learning with more than 17 million registered learners followed by India, Mexico, Brazil, and China.

Amid the devastation caused by the pandemic, governments, teachers, students and corporates benefited by accelerating digitalization efforts. For sure, today’s generation of digitally native learners lapped up this transition by educators. And although initially resistant, teachers discovered digital tools to be welcome assistants while managing schedules, keeping parents included, and doling out and marking assignments. The forced adoption of digital technologies accompanied by wider access to smartphones, made online learning accessible and affordable to larger masses globally. The shift to remote working also saw more professionals sign up on digital learning platforms to upskill and keep pace with the evolving demands of the workplace.

With the global EdTech and Smart Classroom market size expected to reach US$ 259.07 billion by 2028, the future outlook for eLearning platforms and EdTech is bright. Touted to be the mainstay of education in the future, smart classrooms will rely on a wide range of teaching tools and technologies to assist the learning experience end to end. Companies, on their part, already heavily invest their learning budgets in online resources for their workforces. A lot depends, however, on how much EdTech companies invest in the right set of technologies that fulfil the expectations of educators and learners. Their solutions must help the teaching community reduce the burden of administration and deliver affordable, quality education.

The top use cases of emerging technologies that will redefine education and the learning journey include:

Modern learning is student centric. It’s about each student getting to choose what and how to learn, anytime/anywhere, at their own pace, receiving personalized feedback, and accessing tailored recommendations based on their interests, capabilities etc. With the education sector finally on board with digitalization, EdTech offers a delightful range of possibilities to make learning experiences student-centric.

Surgent CPA Review, for example, is an AI-driven, adaptive learning exam prep course. Its proprietary algorithm evaluates performance on questions, student learning styles, exam date available study hours etc. to produce tailor-made study plans. Prodigy is an educational math game that’s becoming popular globally because it can customize content that allows for different learning styles to address specific areas that pose learning difficulties.

Edtech as a teaching assistant

Technology that enables adaptive teaching and learning experiences plays a critical role as it can deliver personalized, updated content that is focused on the unique needs and abilities of each learner. It can also assist teachers across all levels of education. AI supported by machine learning can be used to automate daily administrative tasks like grading/assessments, plagiarism checks, report generation thus freeing up time for teachers and trainers to focus on improving core aspects of their course content and teaching methods . For instance, LEAD’s app for teachers comes with customized curriculum, consistent lesson plans across all partner schools, and a handy AI-driven system automatically generating assignment status updates and assessment reports.

LEAD App in Google Play Store

Georgia State uses Jill Watson, a human-like yet affordable AI assistant to respond to student queries round-the-clock. An elementary school in New Jersey uses an AI-based teaching assistant to help teachers figure out problematic areas of learning mathematics and fine-tune learning methods for each young learner.

Learning companions to improve the inclusiveness of education

Assistive technology is increasing in acceptance as educators are able to extend the learning experience to students who are unable to attend regular classroom sessions. Those with special needs require simpler, easy access to educational content and personalized monitoring because of certain developmental challenges. For example, robots are helping preschoolers with autism practice non-verbal communications skills. The biggest advantage offered by these robots is that they can engage each student with the kind of individual attention and assistance required to help ease their learning journey.

Research is also being conducted to use Artificial Intelligence (AI) for improving learning for those with visual and auditory challenges. For example, the National Technical Institute for the Deaf, housed at the Rochester Institute of Technology, has developed an app that turns speech into text to help deaf/hearing impaired persons interact more easily. This was in response to the communication barriers that came up for persons with hearing difficulties when face masks became compulsory during the pandemic.

ASL TigerChat explained

Another use case of AI that can be a game changer in special education is detecting patterns in large amounts of data and applying these insights to identify and define certain disabilities like dyslexia with greater accuracy.

New immersive experiences shaped by technology

Research suggests that learners retain knowledge better when they are taught using multiple modalities and delivery methods. Both formats are likely to coexist in the future.

With video becoming a popular means of consuming content, digital devices and broadcast technologies finally have an opportunity to converge. OTT platforms and 5G connectivity in combination can deliver higher quality video at reliable speeds. Through the possibilities unlocked by live streaming in 4K and 360-degree videos, learners will be able to consume educational content of their choice at an enhanced level of immersiveness and engagement in multiple formats and modes.

Augmented Reality (AR) can help medical interns fully immerse themselves in training and practice, via virtualization, of a complex surgical procedure without putting any lives at risk or incurring huge expenses in the real world. NASA teaches budding astronauts how to take a walk on Mars employing visuals generated through AR. The Metaverse too will enable close to real-life experiences, a safe way to simulate learning experiences until a desired outcome has been achieved. It provides another dimension to educational storytelling and gamification to make learning more fun and engaging. For example, Arizona State University and Dreamscape Immersive, a VR entertainment and technology company have collaborated to create virtual zoology labs for an explorative learning approach inspired by the metaverse.

Gamifying education

As learners of every age are becoming more digitally savvy, gamification ensures engagement in a highly personal and interactive manner. Kindergarten can become more enjoyable with interactive games catering to young learners. Like Pearson’s interactive education app, which is brimming with images, videos, and interactive games at varying levels, difficulties, and types, to offer fully immersive learning, individualized experiences for children – each gets their own avatar and personalized learning journey. At the same time, teachers, and parents can track the child’s progress easily.

Traditional learning methods can be gamified and infused with elements of fun and healthy competition through interactive quizzes, dynamic leader boards, reward systems, badges to acknowledge and motivate learners. For example, Tinycards has gamified the flash card learning technique and made it more enjoyable. As the learner advances through the cards, their progress is tracked and earns them brownie points for every milestone achieved.

We are rapidly entering a future where education will find its place in a hybrid environment – the offline and online formats will coexist and support each other by bringing the best of their respective worlds. Rather than being seen as a makeshift alternative to physical/classroom learning, EdTech can potentially become the enabler of a robust and resilient system of education, acting as a multiplier to the current in-campus models. With the ability to extend the reach of education across geographies, reduce the burden on teachers, and include those sections who earlier did not have access to learning, the convergence of education and EdTech will see a new era emerge.

AR/VR Artificial Intelligence Mobile Tech Talk

How the next gen of education can be enabled by emerging technologies

Sep 28,2022 / By Vishnu Maji / No Comments

In 2021, online learning platform Coursera reported 20 million new learners in the year, equal to the total growth of the three years prior. The COVID-19 pandemic triggered an exponential jump in the already upward trajectory of online learning. Work from home, virtual classrooms, and time to pursue learning new skills saw the US recording the highest growth in online learning with more than 17 million registered learners followed by India, Mexico, Brazil, and China.

Amid the devastation caused by the pandemic, governments, teachers, students and corporates by benefited by accelerating digitalization efforts. For sure, today’s generation of digitally native learners lapped up this transition by educators. And although initially resistant, teachers discovered digital tools to be welcome assistants while managing schedules, keeping parents included, and doling out and marking assignments. The forced adoption of digital technologies, accompanied by wider access to smartphones, made online learning accessible and affordable to larger masses globally. The shift to remote working also saw more professionals sign up on Learning Management Solutions to upskill and keep pace with the evolving demands of the workplace, learning about emerging technologies, wellness and personal growth, and management behaviors.

With the global EdTech and Smart Classroom market size expected to reach US$ 259070 million by 2028, the future outlook for eLearning platforms and EdTech is certainly bright.

A lot depends, however, on how much EdTech companies and educators invest in the right set of technologies that fulfil the expectations of educators and learners. Whether in educational institutions or corporate learning, solutions must help the teaching community reduce the burden of administration and deliver affordable, quality education to their audience, which is increasingly relying on this format for their learning and training needs. Touted to be the mainstay of education in the future, smart classrooms will rely on a wide range of teaching tools and technologies to assist the learning experience end to end. Companies on their part, already heavily invest their learning budgets in online resources for their workforces.

With the education sector finally on board with digitalization, technology offers a delightful range of possibilities for EdTech to transform learning experiences.

The top five use cases of emerging technologies that will redefine education and the learning journey include:

1. Adaptive teaching that is human-centric

Learning is becoming more student centric with a growing preference for personalized experiences. While research suggests that modern-day learners prefer reading the more affordable and convenient digital form of their textbooks to the print version, Bay View Analytics research found that 43% of college faculty believe students retained knowledge better when learning from printed matter. Research also suggests that modern learners retain knowledge better when they are taught using multiple modalities and delivery methods.

The world of education has changed irrevocably, creating disparities in the teacher-learner dynamic. The role of the teacher has transformed too, becoming more significant – teachers are not supposed to simply pass on information but also required to function as facilitators of the learning journey. They are therefore, expected to switch modes to suit the student’s learning style and capacity. They also have to continuously monitor and assess the learner’s journey so as to customize and make the experience delightful and meaningful for their audience. Educators who have traditionally seen themselves as the controlling authority of educational material, now have to adapt their teaching mindsets to suit modern preferences and expectations of easy, inclusive accessibility.

Technology enabling adaptive teaching and learning experiences holds the key as it can deliver personalized, updated content that is focused on the unique needs and abilities of each learner. And the best thing is that adaptive learning works across all levels of education. Surgent CPA Review, for example, is an AI-driven, adaptive learning exam prep course. Its proprietary algorithm evaluates performance on questions, student learning styles, exam date available study hours etc. to produce tailor-made study plans. Prodigy is an educational math game that’s becoming popular globally because it can customize content that allows for different learning styles to address specific areas that pose learning difficulties. Room to Read, developed by Robosoft, is a leading non-profit organization based in California provides an interactive & feature-rich digital platform to foster a reading habit among children. Test Coach is another comprehensive online learning platform developed by Robosoft for students. It brings the best of both offline and online learning to the students by providing a seamless digital experience.

2. AI as a teaching assistant

Teachers bear a significant burden of administration, lesson planning, assignment grading, learner assessments and recommendations, reports and metrics on performance at an individual and group level. Artificial Intelligence (AI) lends itself to automation of certain daily administrative tasks like grading, report generation thus freeing up time for teachers and trainers to focus on improving core aspects of their course content and teaching methods.

AI supported by machine learning is used for customized content delivery, learning assessment, plagiarism checks, virtual assistance, multiple language support, and computer vision. AI tools like ElevateU help colleges assess student performance and decide on the content and format best suited for each student. Georgia State uses Jill Watson, a human-like yet affordable AI assistant to respond to student queries round-the-clock. An elementary school in New Jersey uses an AI-based teaching assistant to help teachers figure out problematic areas of learning mathematics and fine-tune learning methods for each young learner.

3. Learning companions to suit each learner’s pace

Assistive technology is increasing in acceptance as educators are able to extend the learning experience to students who are unable to attend regular classroom sessions. For example, those with special needs require simpler, easy access to educational content and personalized monitoring because of certain developmental challenges. Accounts of assistive technology like the one on robots helping preschoolers with autism practice non-verbal communications skills, have been making waves on the internet in recent years. The biggest advantage offered by these robots is that they can engage each student with the kind of individual attention and assistance required to help ease their learning journey.

AI can also play a valuable role in enhancing learning outcomes by identifying patterns in erroneous answers, areas of improvement in course material, and enabling individualized feedback messages relevant to a specific learner, which wouldn’t have been possible otherwise. Experts believe that AI can help provide feedback in alternative formats such as a video/audio message that may go down better with the recipient learner and help break down their resistance to consider criticism in a positive light.

4. Gamification and visualization of real-life situations

Augmented Reality (AR) can replace paper-based learning material as all that the learner requires is a smartphone. With a smartphone in almost every hand, it is much easier to create an immersive learning experience, for example, of plant life through a walk in the park. Smart classrooms that are more interactive, immersive and collaborative have also become readily available.

As learners of every age are becoming more digitally savvy, AR brings alive the visualization and ensures engagement through gamification in a highly personal and interactive manner. For example, medical interns can safely and fully immerse themselves in training and practice via the how-to virtualization of a complex surgical procedure without putting any lives at risk or incurring huge expenses in the real world. NASA teaches budding astronauts how to take a walk on Mars employing visuals generated through AR. At the other end of the spectrum, kindergarten can become more enjoyable with interactive games catering to young learners.

The Metaverse too will enable close to real-life experiences, a safe way to simulate learning experiences until a desired outcome has been achieved. Important and practical tasks such as performing advanced medical surgeries, conducting astrophysics experiments, visualization of a rocket launch etc. It provides another dimension to educational storytelling and gamification to make learning more fun and engaging. For example, Arizona State University and Dreamscape Immersive, a VR entertainment and technology company have collaborated to create virtual zoology labs for an explorative learning approach inspired by the metaverse.

5. Seamless consumption of multi-format, multi-genre content at the learner’s convenience

With video becoming a popular means of consuming content, digital devices, and broadcast technologies finally have an opportunity to converge. OTT platforms and 5G connectivity in combination can deliver higher quality video at reliable speeds. Live streaming in 4K, 360-degree videos, highly interactive experiences – the opportunities to generate an immersive learning experience are almost limitless.

For this to come about, EdTech needs to befriend emerging technologies such as OTT/5G, AI, AR/VR, metaverse, data analytics to enable seamless, enhanced learning experience while bringing more learners into its fold. This way, EdTech companies will also be able to move quickly to capitalize on new revenue streams that technology opens up as education settles into its next-gen avatar.

Resources