RealityLab
RealityLab
RealityLab
TIME
2022 - 2023
TEAM
Mankin
Cherry
MY ROLE
UI / UX Design
Coding
Overview
Overview
Professional virtual backgrounds and AR effects usually require expensive gear. You need a green screen, tracking hardware, and a powerful PC. For most everyday content creators, that setup is just way too costly and complicated.
Professional virtual backgrounds and AR effects usually require expensive gear. You need a green screen, tracking hardware, and a powerful PC. For most everyday content creators, that setup is just way too costly and complicated.
We wanted to break down those barriers with a simple mobile app. The goal was straightforward: allow anyone to stream or create video with professional virtual effects using just their phone, without needing a physical green screen or chroma key.
We wanted to break down those barriers with a simple mobile app. The goal was straightforward: allow anyone to stream or create video with professional virtual effects using just their phone, without needing a physical green screen or chroma key.
By putting the processing power into software, we made high-quality virtual production accessible, lightweight, and affordable for everyone.
By putting the processing power into software, we made high-quality virtual production accessible, lightweight, and affordable for everyone.
Some of our design inspirations
The Core Challenge
The Core Challenge
Creating a realistic virtual background usually requires a physical green screen, heavy camera rigs, and high-end computers. Our biggest goal was to tear all that down and bring everything onto a single iPad.
Creating a realistic virtual background usually requires a physical green screen, heavy camera rigs, and high-end computers. Our biggest goal was to tear all that down and bring everything onto a single iPad.
The real technical nightmare was blending real-life footage with 3D animated scenes seamlessly. We used ARKit and SceneKit to handle real-time spatial tracking, but balancing heavy graphical rendering with smooth device performance was tough. On top of that, we had to squeeze complex 3D mechanics into a simple UI so anyone could start broadcasting or recording without a steep learning curve.
The real technical nightmare was blending real-life footage with 3D animated scenes seamlessly. We used ARKit and SceneKit to handle real-time spatial tracking, but balancing heavy graphical rendering with smooth device performance was tough. On top of that, we had to squeeze complex 3D mechanics into a simple UI so anyone could start broadcasting or recording without a steep learning curve.
It was a constant battle between hardware limits, realistic graphics, and user simplicity.
It was a constant battle between hardware limits, realistic graphics, and user simplicity.
Old
New
Old
New
Old
New
A compact pop-up overlay that replaces full-screen views, enabling users to preview component choices directly over their presentation slides for a faster, distraction-free workflow.
A compact pop-up overlay that replaces full-screen views, enabling users to preview component choices directly over their presentation slides for a faster, distraction-free workflow.
Designed as the central hub for our project management ecosystem. This homepage allows users to effortlessly add, share, and present their work with a single click. It delivers an intuitive experience for creators at any level to showcase their work and collaborate.
Designed as the central hub for our project management ecosystem. This homepage allows users to effortlessly add, share, and present their work with a single click. It delivers an intuitive experience for creators at any level to showcase their work and collaborate.
This feature gives users real-time control to adjust the position, scale, and color of 3D objects against a live video background. We integrated scene presets, component quick-selection, and intuitive bottom bar controls so users can effortlessly build, tweak, or undo their designs. It streamlines the entire 3D customization process into a frictionless experience.
This feature gives users real-time control to adjust the position, scale, and color of 3D objects against a live video background. We integrated scene presets, component quick-selection, and intuitive bottom bar controls so users can effortlessly build, tweak, or undo their designs. It streamlines the entire 3D customization process into a frictionless experience.
Users can quickly jumpstart their creations with
high-quality 3D presets, then access the full scene editor to make advanced adjustments after upgrading.
Users can quickly jumpstart their creations with
high-quality 3D presets, then access the full scene editor to make advanced adjustments after upgrading.
IPad
IPhone
IPad
IPhone
IPad
IPhone
This feature empowers users to customize their media player layout, stream dynamic PowerPoint backgrounds, and manage multi-angle iPhone inputs through a seamless, highly intuitive control bar.
This feature empowers users to customize their media player layout, stream dynamic PowerPoint backgrounds, and manage multi-angle iPhone inputs through a seamless, highly intuitive control bar.
A smart recorder featuring hands-free gesture triggers, auto-optimized resolution settings, and intuitive visual cues for seamless device pairing.
A smart recorder featuring hands-free gesture triggers, auto-optimized resolution settings, and intuitive visual cues for seamless device pairing.
A real-time video editing tool that uses color-coded clips and instant 3D model modifications to streamline post-production.
A real-time video editing tool that uses color-coded clips and instant 3D model modifications to streamline post-production.
A streamlined paywall and onboarding experience that uses popular scene previews.
A streamlined paywall and onboarding experience that uses popular scene previews.
The Result & Reflection
The Result & Reflection
Building this AR app was a intense learning curve. In the end, we successfully brought the core vision to life. We created a working iPad application that allows creators to stream and record with realistic 3D virtual backgrounds using ARKit and SceneKit, completely free from physical green screens.
Building this AR app was a intense learning curve. In the end, we successfully brought the core vision to life. We created a working iPad application that allows creators to stream and record with realistic 3D virtual backgrounds using ARKit and SceneKit, completely free from physical green screens.
But looking back, the journey was far from smooth. Juggling heavy FYP development alongside exams and other course deadlines was a constant struggle. We hit major layout bugs and performance bottlenecks that pushed our schedule to the limit.
But looking back, the journey was far from smooth. Juggling heavy FYP development alongside exams and other course deadlines was a constant struggle. We hit major layout bugs and performance bottlenecks that pushed our schedule to the limit.
Our biggest compromise was compatibility. Due to time constraints and hardware rendering limits, we had to lock the initial build to iPad only. While I wish we could have expanded it to iPhones and other platforms, this limitation taught me a crucial lesson about scope management in real-world product development.
Our biggest compromise was compatibility. Due to time constraints and hardware rendering limits, we had to lock the initial build to iPad only. While I wish we could have expanded it to iPhones and other platforms, this limitation taught me a crucial lesson about scope management in real-world product development.
Overall, this project gave me a deep, hands-on understanding of spatial AR technology. It proved to me that we can make high-end video production accessible to everyday people. It also opened my eyes to where AR is heading next, especially when combined with real-time AI tracking in the future.
Overall, this project gave me a deep, hands-on understanding of spatial AR technology. It proved to me that we can make high-end video production accessible to everyday people. It also opened my eyes to where AR is heading next, especially when combined with real-time AI tracking in the future.
A video introducing the app
© Mankin Portfolio | Created by Mankin
Made in Framer
© Mankin Portfolio | Created by Mankin
Made in Framer
© Mankin Portfolio | Created by Mankin
Made in Framer