Posted by Manfred Ernst, Software Engineer
Great VR experiences make you feel like you're really somewhere else. To create deeply immersive experiences, there are a lot of factors that need to come together: amazing graphics, spatialized audio, and the ability to move around and feel like the world is responding to you.
Last year at I/O, we announced Seurat as a powerful tool to help developers and creators bring high-fidelity graphics to standalone VR headsets with full positional tracking, like the Lenovo Mirage Solo with Daydream. Seurat is a scene simplification technology designed to process very complex 3D scenes into a representation that renders efficiently on mobile hardware. Here's how ILMxLAB was able to use Seurat to bring an incredibly detailed 'Rogue One: A Star Wars Story' scene to a standalone VR experience.
Today, we're open sourcing Seurat to the developer community. You can now use Seurat to bring visually stunning scenes to your own VR applications and have the flexibility to customize the tool for your own workflows.
Behind the scenes - How Seurat works
Seurat works by taking advantage of the fact that VR scenes are typically viewed from within a limited viewing region, and leverages this to optimize the geometry and textures in your scene. It takes RGBD images (color and depth) as input and generates a textured mesh, targeting a configurable number of triangles, texture size, and fill rate, to simplify scenes beyond what traditional methods can achieve.
To demonstrate what Seurat can do, here's a snippet from Blade Runner: Revelations, which launched today with the Lenovo Mirage Solo.
Blade Runner: Revolution by Alcon Interactive and Seismic Games
The Blade Runner universe is known for its stunning worlds, and in Revelations, you get to unravel a mystery around fugitive Replicants in the futuristic but gritty streets. To create the look and feel for Revelations, Seismic used Seurat to bring a scene of 46.6 million triangles down to only 307,000, improving performance by more than 100x with almost no loss in visual quality:
Original scene:
Seurat-processed scene:
If you're interested in learning more about Seurat or trying it out yourself, visit the Seurat GitHub page to access the documentation and source code. We're looking forward to seeing what you build!
Spatial audio adds to your sense of presence when you're in VR or AR, making it feel and sound, like you're surrounded by a virtual or augmented world. And regardless of the display hardware you're using, spatial audio makes it possible to hear sounds coming from all around you.
Resonance Audio, our spatial audio SDK launched last year, enables developers to create more realistic VR and AR experiences on mobile and desktop. We've seen a number of exciting experiences emerge across a variety of platforms using our SDK. Recent examples include apps like Pixar's Coco VR for Gear VR, Disney's Star WarsTM: Jedi Challenges AR app for Android and iOS, and Runaway's Flutter VR for Daydream, which all used Resonance Audio technology.
To accelerate adoption of immersive audio technology and strengthen the developer community around it, we’re opening Resonance Audio to a community-driven development model. By creating an open source spatial audio project optimized for mobile and desktop computing, any platform or software development tool provider can easily integrate with Resonance Audio. More cross-platform and tooling support means more distribution opportunities for content creators, without the worry of investing in costly porting projects.
As part of our open source project, we're providing a reference implementation of YouTube's Ambisonic-based spatial audio decoder, compatible with the same Ambisonics format (Ambix ACN/SN3D) used by others in the industry. Using our reference implementation, developers can easily render Ambisonic content in their VR media and other applications, while benefiting from Ambisonics open source, royalty-free model. The project also includes encoding, sound field manipulation and decoding techniques, as well as head related transfer functions (HRTFs) that we've used to achieve rich spatial audio that scales across a wide spectrum of device types and platforms. Lastly, we're making our entire library of highly optimized DSP classes and functions, open to all. This includes resamplers, convolvers, filters, delay lines and other DSP capabilities. Additionally, developers can now use Resonance Audio's brand new Spectral Reverb, an efficient, high quality, constant complexity reverb effect, in their own projects.
We've open sourced Resonance Audio as a standalone library and associated engine plugins, VST plugin, tutorials, and examples with the Apache 2.0 license. Most importantly, this means Resonance Audio is yours, so you're free to use Resonance Audio in your projects, no matter where you work. And if you see something you'd like to improve, submit a GitHub pull request to be reviewed by the Resonance Audio project committers. While the engine plugins for Unity, Unreal, FMOD, and Wwise will remain open source, going forward they will be maintained by project committers from our partners, Unity, Epic, Firelight Technologies, and Audiokinetic, respectively.
If you're interested in learning more about Resonance Audio, check out the documentation on our developer site. If you want to get more involved, visit our GitHub to access the source code, build the project, download the latest release, or even start contributing. We're looking forward to building the future of immersive audio with all of you.
With ARCore and Google Lens, we're working to make smartphone cameras smarter. ARCore enables developers to build apps that can understand your environment and place objects and information in it. Google Lens uses your camera to help make sense of what you see, whether that's automatically creating contact information from a business card before you lose it, or soon being able to identify the breed of a cute dog you saw in the park. At Mobile World Congress, we're launching ARCore 1.0 along with new support for developers, and we're releasing updates for Lens and rolling it out to more people.
ARCore, Google's augmented reality SDK for Android, is out of preview and launching as version 1.0. Developers can now publish AR apps to the Play Store, and it's a great time to start building. ARCore works on 100 million Android smartphones, and advanced AR capabilities are available on all of these devices. It works on 13 different models right now (Google's Pixel, Pixel XL, Pixel 2 and Pixel 2 XL; Samsung's Galaxy S8, S8+, Note8, S7 and S7 edge; LGE's V30 and V30+ (Android O only); ASUS's Zenfone AR; and OnePlus's OnePlus 5). And beyond those available today, we're partnering with many manufacturers to enable their upcoming devices this year, including Samsung, Huawei, LGE, Motorola, ASUS, Xiaomi, HMD/Nokia, ZTE, Sony Mobile, and Vivo.
Making ARCore work on more devices is only part of the equation. We're bringing developers additional improvements and support to make their AR development process faster and easier. ARCore 1.0 features improved environmental understanding that enables users to place virtual assets on textured surfaces like posters, furniture, toy boxes, books, cans and more. Android Studio Beta now supports ARCore in the Emulator, so you can quickly test your app in a virtual environment right from your desktop.
Everyone should get to experience augmented reality, so we're working to bring it to people everywhere, including China. We'll be supporting ARCore in China on partner devices sold there— starting with Huawei, Xiaomi and Samsung—to enable them to distribute AR apps through their app stores.
We've partnered with a few great developers to showcase how they're planning to use AR in their apps. Snapchat has created an immersive experience that invites you into a "portal"—in this case, FC Barcelona's legendary Camp Nou stadium. Visualize different room interiors inside your home with Sotheby's International Realty. See Porsche's Mission E Concept vehicle right in your driveway, and explore how it works. With OTTO AR, choose pieces from an exclusive set of furniture and place them, true to scale, in a room. Ghostbusters World, based on the film franchise, is coming soon. In China, place furniture and over 100,000 other pieces with Easyhome Homestyler, see items and place them in your home when you shop on JD.com, or play games from NetEase, Wargaming and Game Insight.
With Google Lens, your phone's camera can help you understand the world around you, and, we're expanding availability of the Google Lens preview. With Lens in Google Photos, when you take a picture, you can get more information about what's in your photo. In the coming weeks, Lens will be available to all Google Photos English-language users who have the latest version of the app on Android and iOS. Also over the coming weeks, English-language users on compatible flagship devices will get the camera-based Lens experience within the Google Assistant. We'll add support for more devices over time.
And while it's still a preview, we've continued to make improvements to Google Lens. Since launch, we've added text selection features, the ability to create contacts and events from a photo in one tap, and—in the coming weeks—improved support for recognizing common animals and plants, like different dog breeds and flowers.
Smarter cameras will enable our smartphones to do more. With ARCore 1.0, developers can start building delightful and helpful AR experiences for them right now. And Lens, powered by AI and computer vision, makes it easier to search and take action on what you see. As these technologies continue to grow, we'll see more ways that they can help people have fun and get more done on their phones.
As developers, we all know that having the right assets is crucial to the success of a 3D application, especially with AR and VR apps. Since we launched Poly a few weeks ago, many developers have been downloading and using Poly models in their apps and games. To make this process easier and more powerful, today we launched the Poly API, which allows applications to dynamically search and download 3D assets at both edit and run time.
The API is REST-based, so it's inherently cross-platform. To help you make the API calls and convert the results into objects that you can display in your app, we provide several toolkits and samples for some common game engines and platforms. Even if your engine or platform isn't included in this list, remember that the API is based on HTTP, which means you can call it from virtually any device that's connected to the Internet.
Here are some of the things the API allows you to do:
If you are using Unity, we offer Poly Toolkit for Unity, a plugin that includes all the necessary functionality to automatically wrap the API calls and download and convert assets, exposing it through a simple C# API. For example, you can fetch and import an asset into your scene at runtime with a single line of code:
PolyApi.GetAsset(ASSET_ID, result => { PolyApi.Import(result.Value, PolyImportOptions.Default()); });
Poly Toolkit optionally also handles authentication for you, so that you can list the signed in user's own private assets, or the assets that the user has liked on the Poly website.
In addition, Poly Toolkit for Unity also comes with an editor window, where you can search for and import assets from Poly into your Unity scene directly from the editor.
If you are using Unreal, we also offer Poly Toolkit for Unreal, which wraps the API and performs automatic download and conversion of OBJs and Blocks models from Poly. It allows you to query for assets and filter results, download assets and import assets as ready-to-use Unreal actors that you can use in your game.
Not using a game engine? No problem! If you are developing for Android, check out our Android sample code, which includes a basic sample with no external dependencies, and also a sample that shows how to use the Poly API in conjunction with ARCore. The samples include:
If you are an iOS developer, we have two samples for you as well: one using SceneKit and one using ARKit, showing how to build an iOS app that downloads and imports models from Poly. This includes all the logic necessary to open an HTTP connection, make the API requests, parse the results, build the 3D objects from the data and place them on the scene.
For web developers, we also offer a complete WebGL sample using Three.js, showing how to get and display a particular asset, or perform searches. There is also a sample showing how to import and display Tilt Brush sketches.
No matter what engine or platform you are using, we hope that the Poly API will help bring high quality assets to your app and help you increase engagement with your users! You can find more information about the Poly API and our toolkits and samples on our developers site.
As humans, we rely on sound to guide us through our environment, help us communicate with others and connect us with what's happening around us. Whether walking along a busy city street or attending a packed music concert, we're able to hear hundreds of sounds coming from different directions. So when it comes to AR, VR, games and even 360 video, you need rich sound to create an engaging immersive experience that makes you feel like you're really there. Today, we're releasing a new spatial audio software development kit (SDK) called Resonance Audio. It's based on technology from Google's VR Audio SDK, and it works at scale across mobile and desktop platforms.
Bringing rich, dynamic audio environments into your VR, AR, gaming, or video experiences without affecting performance can be challenging. There are often few CPU resources allocated for audio, especially on mobile, which can limit the number of simultaneous high-fidelity 3D sound sources for complex environments. The SDK uses highly optimized digital signal processing algorithms based on higher order Ambisonics to spatialize hundreds of simultaneous 3D sound sources, without compromising audio quality, even on mobile. We're also introducing a new feature in Unity for precomputing highly realistic reverb effects that accurately match the acoustic properties of the environment, reducing CPU usage significantly during playback.
We know how important it is that audio solutions integrate seamlessly with your preferred audio middleware and sound design tools. With Resonance Audio, we've released cross-platform SDKs for the most popular game engines, audio engines, and digital audio workstations (DAW) to streamline workflows, so you can focus on creating more immersive audio. The SDKs run on Android, iOS, Windows, MacOS and Linux platforms and provide integrations for Unity, Unreal Engine, FMOD, Wwise and DAWs. We also provide native APIs for C/C++, Java, Objective-C and the web. This multi-platform support enables developers to implement sound designs once, and easily deploy their project with consistent sounding results across the top mobile and desktop platforms. Sound designers can save time by using our new DAW plugin for accurately monitoring spatial audio that's destined for YouTube videos or apps developed with Resonance Audio SDKs. Web developers get the open source Resonance Audio Web SDK that works in the top web browsers by using the Web Audio API.
DAW plugin for sound designers to monitor audio destined for YouTube 360 videos or apps developed with the SDK
By providing powerful tools for accurately modeling complex sound environments, Resonance Audio goes beyond basic 3D spatialization. The SDK enables developers to control the direction acoustic waves propagate from sound sources. For example, when standing behind a guitar player, it can sound quieter than when standing in front. And when facing the direction of the guitar, it can sound louder than when your back is turned.
Another SDK feature is automatically rendering near-field effects when sound sources get close to a listener's head, providing an accurate perception of distance, even when sources are close to the ear. The SDK also enables sound source spread, by specifying the width of the source, allowing sound to be simulated from a tiny point in space up to a wall of sound. We've also released an Ambisonic recording tool to spatially capture your sound design directly within Unity, save it to a file, and use it anywhere Ambisonic soundfield playback is supported, from game engines to YouTube videos.
If you're interested in creating rich, immersive soundscapes using cutting-edge spatial audio technology, check out the Resonance Audio documentation on our developer site, let us know what you think through GitHub, and show us what you build with #ResonanceAudio on social media; we'll be resharing our favorites.
For a virtual scene to be truly immersive, stunning visuals need to be accompanied by true spatial audio to create a realistic and believable experience. Spatial audio tools allow developers to include sounds that can come from any direction, and that are associated in 3D space with audio sources, thus completely enveloping the user in 360-degree sound.
Spatial audio helps draw the user into a scene and creates the illusion of entering an entirely new world. To make this possible, the Chrome Media team has created Songbird, an open source, spatial audio encoding engine that works in any web browser by using the Web Audio API.
The Songbird library takes in any number of mono audio streams and allows developers to programmatically place them in 3D space around the user. Songbird allows you to create immersive soundscapes, realistically reproducing reflection and reverb for the space you describe. Sounds bounce off walls and reflect off materials just as they would in real-life, capturing truly 360-degree sound. Songbird creates an ambisonic soundfield that can then be rendered in real-time for use in your application. We've partnered with the Omnitone project, which we blogged about last year, to add higher-order ambisonic support to Omnitone's binaural renderer to produce far more accurate sounding audio than ever before.
Songbird encapsulates Omnitone and with it, developers can now add interactive, full-sphere audio to any web based application. Songbird can scale to any order ambisonics, thereby creating a more realistic sound and higher performance than what is achievable through standard Web Audio API.
The implementation of Songbird is based on the Google spatial media specification. It expects mono input and outputs ambisonic (multichannel) ACN channel layout with SN3D normalization. Detailed documentation may be found here.
As the web emerges as an important VR platform for delivering content, spatial audio will play a vital role in users' embrace of this new medium. Songbird and Omnitone are key tools in enabling spatial audio on the web platform and establishing it as a preeminent platform for compelling VR experiences. Combining these audio experiences with 3D JavaScript libraries like three.js gives a glimpse into the future on the web.
This project was made possible through close collaboration with Google's Daydream and Web Audio teams. This collaboration allowed us to deliver similar audio capabilities to the web as are available to developers creating Daydream applications.
We look forward to seeing what people do with Songbird now that it's open source. Check out the code on GitHub and let us know what you think. Also available are a number of demos on creating full spherical audio with Songbird.
At Area 120, Google's internal workshop for experimental ideas, we're working on early-stage projects and quickly iterate to test concepts. We heard from developers that they're looking at how to make money to fund their VR applications, so we started experimenting with what a native, mobile VR ad format might look like.
Developers and users have told us they want to avoid disruptive, hard-to-implement ad experiences in VR. So our first idea for a potential format presents a cube to users, with the option to engage with it and then see a video ad. By tapping on the cube or gazing at it for a few seconds, the cube opens a video player where the user can watch, and then easily close, the video. Here's how it works:
Our work focuses on a few key principles - VR ad formats should be easy for developers to implement, native to VR, flexible enough to customize, and useful and non-intrusive for users. Our Area 120 team has seen some encouraging results with a few test partners, and would love to work with the developer community as this work evolves - across Cardboard (on Android and iOS), Daydream and Samsung Gear VR.
If you're a VR developer (or want to be one) and are interested in testing this format with us, please fill out this form to apply for our early access program. We have an early-stage SDK available and you can get up and running easily. We're excited to continue experimenting with this format and hope you'll join us for the ride!
Posted by Nathan Martz, Product Manager, Google VR
With Google Cardboard and Daydream, our Google VR team is working to bring virtual reality to everyone. In addition to making VR more accessible by using the smartphone in your pocket, we recently launched the Google VR SDK out of beta, with native integration for Unity and UE4, to help make it easier for more developers to join the fold.
To further support and encourage new developers to build VR experiences, we’ve partnered with Udacity to create the VR Developer Nanodegree. Students will learn how to create 3D environments, define behaviors, and make VR experiences comfortable, immersive, and performant.
Even with more than 50 million installs of Google Cardboard apps on Google Play, these are still the early days of VR. Students who complete the VR Developer Nanodegree learn by doing, and will graduate having completed a portfolio of VR experiences.
Learn more and sign up to receive VR Developer Nanodegree program updates at https://www.udacity.com/vr
At Google I/O, we announced Daydream—Google's platform for high quality, mobile virtual reality—and released early developer resources to get the community started with building for Daydream. Since then, the team has been hard at work, listening to feedback and evolving these resources into a suite of powerful developer tools.
Today, we are proud to announce that the Google VR SDK 1.0 with support for Daydream has graduated out of beta, and is now available on the Daydream developer site. Our updated SDK simplifies common VR development tasks so you can focus on building immersive, interactive mobile VR applications for Daydream-ready phones and headsets, and supports integrated asynchronous reprojection, high fidelity spatialized audio, and interactions using the Daydream controller.
To make it even easier to start developing with the Google VR SDK 1.0, we’ve partnered with Unity and Unreal so you can use the game engines and tools you’re already familiar with. We’ve also updated the site with full documentation, reference sample apps, and tutorials.
Native Unity integration
This release marks the debut of native Daydream integration in Unity, which enables Daydream developers to take full advantage of all of Unity’s optimizations in VR rendering. It also adds support for features like head tracking, deep linking, and easy Android manifest configuration. Many Daydream launch apps are already working with the newest integration features, and you can now download the new Unity binary here and the Daydream plugin here.
Native UE4 integration
We’ve made significant improvements to our UE4 native integration that will help developers build better production-quality Daydream apps. The latest version introduces Daydream controller support in the editor, a neck model, new rendering optimizations, and much more. UE4 developers can download the source here.
Get started today
While the first Daydream-ready phones and headset are coming this fall, you can start developing high-quality Daydream apps right now with the Google VR SDK 1.0 and the DIY developer kit.
We’re also opening applications to our Daydream Access Program (DAP) so we can work closely with even more developers building great content for Daydream. Submit your Daydream app proposal to apply to be part of our DAP.
When you create content for the Daydream platform, you know your apps will work seamlessly across every Daydream-ready phone and headset. Daydream is just getting started, and we’re looking forward to working together to help you build new immersive, interactive VR experiences. Stay tuned for more information about Daydream-ready phones and the Daydream headset and controller coming soon.
Posted by Robbie Tilton, UX Designer, Google VR
At Daydream Labs, we have experimented with social interactions in VR. Just like in real reality, people naturally want to share and connect with others in VR. As developers and designers, we are excited to build social experiences that are fun and easy to use—but it’s just as important to make it safe and comfortable for all involved. Over the last year, we’ve learned a few ways to nudge people towards positive social experiences.
What can happen without clear social norms
People are curious and will test the limits of your VR experience. For example, when some people join a multiplayer app or game, they might wonder if they can reach their hand through another player’s head or stand inside another avatar’s body. Even with good intentions, this can make other people feel unsafe or uncomfortable.
For example, in a shopping experiment we built for the HTC Vive, two people could enter a virtual storefront and try on different hats, sunglasses, and accessories. There was no limit to how or where they could place a virtual accessory, so some people stuck hats on friends anywhere they would stick—like in front of their eyes. This had the unfortunate effect of blocking their vision. If they couldn’t remove the hat in front of their eyes with their controllers, they had no other recourse than to take off their headset and end their VR experience.
Protecting user safety
Everyone should feel safe and comfortable in VR. If we can anticipate the actions of others, then we may be able to discourage negative social behavior before it starts. For example, by designing personal space around each user, you can prevent other people from invading that personal space.
We built an experiment around playing poker where we tried new ways to discourage trolling. If someone left their seat at the poker table, their environment desaturated to black and white and their avatar would disappear from the other player’s view. A glowing blue personal space bubble would guide the person back to their seat. We found it’s enough to prevent a player from approaching their opponents to steal chips or invade personal space.
Reward positive behavior
If you want people to interact in positive ways—like high-fiving 🙌 —try giving them an incentive. In one experiment, we detected when two different avatars “touched” each other’s hands at high speed. This triggered a loud slapping sound and a fireworks animation. It sounds simple, but people loved it. Meanwhile, if you tried to do something more aggressive, like punching an avatar’s body, nothing would happen. You can guess which behavior people naturally preferred.
Rob Jagnow, Software Engineer, Google VR
Whether you're playing a game or watching a video, VR lets you step inside a new world and become the hero of a story. But what if you want to tell a story of your own?
Producing immersive 3D animation can be difficult and expensive. It requires complex software to set keyframes with splined interpolation or costly motion capture setups to track how live actors move through a scene. Professional animators spend considerable effort to create sequences that look expressive and natural.
At Daydream Labs, we've been experimenting with ways to reduce technical complexity and even add a greater sense of play when animating in VR. In one experiment we built, people could bring characters to life by picking up toys, moving them through space and time, and then replay the scene.
As we saw people play with the animation experiment we built, we noticed a few things:
The need for complex metaphors goes away in VR: What can be complicated in 2D can be made intuitive in 3D. Instead of animating with graph editors or icons representing location, people could simply reach out, grab a virtual toy, and carry it through the scene. These simple animations had a handmade charm that conveyed a surprising degree of emotion.
The learning curve drops to zero: People were already familiar with how to interact with real toys, so they jumped right in and got started telling their stories. They didn't need a lengthy tutorial, and they were able to modify their animations and even add new characters without any additional help.
People react to virtual environments the same way they react to real ones: When people entered a playful VR environment, they understood it was safe space to play with the toys around them. They felt comfortable performing and speaking in funny voices. They took more risks knowing the virtual environment was designed for play.
To create more intricate animations, we also built another experiment that let people independently animate the joints of a single character. It let you record your character’s movement as you separately animated the feet, hands, and head — just like you would with a puppet.
VR allows us to rethink software and make certain use cases more natural and intuitive. While this kind of animation system won’t replace professional tools, it can allow anyone to tell their own stories. There are many examples of using VR for storytelling, especially with video and animation, and we’re excited to see new perspectives as more creators share their stories in VR.
Posted by Rob Jagnow, Software Engineer, Google VR
At Daydream Labs, we pair engineers with designers to rapidly prototype virtual reality concepts, and we’ve already started to share our learnings with the VR community. This week, we focus on social. In many of our experiments, we’ve found that being in VR with others amplifies and improves experiences in VR, as long as you take a few things into account. Here’s what we’ve learned so far:
Simplicity can be powerful: Avatars (or the virtual representations of people in VR) can be simplified to just a floating head with googly eyes and still convey a surprising degree of emotion, intent, and number of social cues. Eyes give people a location to look to and speak towards, but they also increase face-to-face communication by making even basic avatars feel more human. When we combine this with hands and a spatially-located voice, it comes together to create a sense of shared presence.
Connecting the real and the virtual: Even when someone is alone in VR, you can make them feel connected. For example, you can continue to carry a conversation even if you’re not in VR with them. Your voice can serve as a subtle reminder that they’re spanning two spaces—the real and the virtual. This asymmetric experience can be a fun way to help ground party games where one player is in VR but other players aren’t, like with charades or Pictionary.
But when someone else joins that virtual world with them, we’ve seen time and time again that the real world melts away. For most multiplayer activities, this is ideal because it makes the experience incredibly engaging.
Join the party: When you first start a VR experience with others, it can be tough to know where to begin. After all, it’s easier to join a party than to start one! Create shared goals for multi-player experiences. When you give people something to play with together, it can help them break the ice, allow them to make friends, and have more fun in VR.
You think you know somebody: Lastly, people who know each other offline immediately notice stature or differences in a person’s height in VR. We can re-calibrate environments to play with height and scale values to build a VR world where everyone appears to be the same height. Or we can adjust display settings to make each person feel like they’re the tallest person in the room. Height is such a powerful social cue in the real world and we can tune these settings in VR to nudge people into having more friendly, prosocial interactions.
If you’d like to learn more about Daydream Labs and what we’ve learned so far, check out our recent Lessons Learned from VR Prototyping talk at Google I/O.
Posted by Andrey Doronichev, Group Product Manager, Google VR
In Daydream Labs, the Google VR team explores virtual reality’s possibilities and shares what we learn with the world. While it’s still early days, the VR community has already come a long way in understanding what works well in VR across hardware, software, video, and much more. But, part of what makes developing for VR so exciting is that there’s still so much more to discover.
Apps are a big focus for Daydream Labs. In the past year, we’ve built more than 60 app experiments that test different use cases and interaction designs. To learn fast, we build two new app prototypes each week. Not all of our experiments are successful, but we learn something new with each one.
For example, in one week we built a virtual drum kit that used HTC Vive controllers as drumsticks. The following week, when we were debating how to make typing in VR more natural and playful, we thought — “what if we made a keyboard out of tiny drums?”
We were initially skeptical that drumsticks could be more efficient than direct hand interaction, but the result surprised us. Not only was typing with drumsticks faster than with a laser pointer, it was really fun! We even built a game that lets you track your words per minute (mine was 50 wpm!).
Daydream Labs is just getting started. This post is the first in an ongoing series sharing what we’ve learned through our experiments so stay tuned for more! You can also see more of what we’ve learned about VR interactions, immersion, and social design by watching our Google I/O talks on the live stream.
Posted by Nathan Martz, Product Manager, Daydream
Two years ago at Google I/O, we introduced Google Cardboard, a simple and fun way to experience virtual reality on your smartphone. Since then, we've grown the Google VR family with Expeditions and Jump, and this week at Google I/O, we announced Daydream, a platform for high quality mobile virtual reality.
We announced Jump, cameras and software to make producing beautiful VR video simple, at I/O last year. Jump cameras are now in the hands of media partners such as Paramount Pictures, The New York Times, and Discovery Communications. Virtual reality production companies including WEVR, Vrse, The Secret Location, Surreal, Specular Theory, Panograma, and RYOT also have cameras in hand. We can't wait to see the wide variety of immersive videos these creators will share with a growing VR audience.
To enable cameras in a range of shapes and sizes and price points. Today, the Jump ecosystem expands with two partnerships to build Jump cameras. First, we're working with Yi Technology on a rig based around their new 4K Action Cam, coming later this year.
With Jump, we've also seen incredible interest from filmmakers. Of course when you're creating the best content you want the absolute highest quality, cinema-grade camera available. To help create this, we're collaborating with IMAX to develop a very high-end cinema-grade Jump camera.
More than one million students from over 11 countries have taken an Expedition since we introduced the Google Expeditions Pioneer Program last May. The program lets students take virtual reality trips to over 200 places including Buckingham Palace, underwater in the Great Barrier Reef—and in seventh grader Lance Teeselink’s case—Dubai’s Burj Khalifa, the tallest building in the world.
And soon, students will have even more places to visit, virtually, thanks to new partnerships with the Associated Press and Getty Images. These partners will provide the Expeditions program with high-resolution VR imagery for current events to help students better understand what’s happening around the world.
Daydream is our new platform for high quality mobile virtual reality, coming this fall. Over time, Daydream will encompass VR devices in many shapes and sizes, and Daydream will enable high quality VR on Android smartphones.
We are working with a number of smartphone manufacturers to create a specification for Daydream-ready phones. These smartphones enable VR experiences with high-performance sensors for smooth, accurate head tracking, fast response displays to minimize blur, and powerful mobile processors. Daydream-ready phones take advantage of VR mode in Android N, a set of powerful optimizations for virtual reality built right into Android.
With Daydream, we've also created a reference design for a comfortable headset and an intuitive controller. And, yes we're building one too. The headset and controller work in tandem to provide rich, immersive experiences. Take a look at how the controller lets you interact in VR:
The most important part of virtual reality is what you experience. Some of the world's best content creators and game studios are bringing their content to Daydream. You will also have your favorite Google apps including Play Movies, Street View, Google Photos, and YouTube.
You can start building for Daydream today. The Google VR SDK now includes a C++ NDK. And if you develop with Unreal or Unity, Daydream will be natively supported by both engines. Visit the Daydream developer site where you can get access the tools. Plus, with Android N Developer Preview 3 you can use the Nexus 6P as a Daydream developer kit.
This is just the beginning for Daydream. We’ll be sharing much more on this blog over the coming months. We’re excited to build the next chapter of VR with you.
Posted by Jeff Nusz, Data Arts Team, Pixel Painter
Two weeks ago, we introduced Tilt Brush, a new app that enables artists to use virtual reality to paint the 3D space around them. Part virtual reality, part physical reality, it can be difficult to describe how it feels without trying it firsthand. Today, we bring you a little closer to the experience of painting with Tilt Brush using the powers of the web in a new Chrome Experiment titled Virtual Art Sessions.
Virtual Art Sessions lets you observe six world-renowned artists as they develop blank canvases into beautiful works of art using Tilt Brush. Each session can be explored from start to finish from any angle, including the artist’s perspective – all viewable right from the browser.
Participating artists include illustrator Christoph Niemann, fashion illustrator Katie Rodgers, sculptor Andrea Blasich, installation artist Seung Yul Oh, automotive concept designer Harald Belker, and street artist duo Sheryo & Yok. The artists’ unique approaches to this new medium become apparent when seeing them work inside their Tilt Brush creations. Watch this behind-the-scenes video to hear what the artists had to say about their experience:
Virtual Art Sessions makes use of Google Chrome’s V8 Javascript engine for high-performance processing power to render large volumes of data in real time. This includes point cloud data of the artist’s physical form, 3D geometry data of the artwork, and position data of the VR controllers. It also relies on Chrome’s support of WebM video and WebGL to produce the 360° representations of the artists and artwork – the artist portrayals alone require the browser to draw over 200,000 points at 30 times a second. For a deeper look, read the technical case study or browse the project code that is available open source from the site’s tech page.
We hope this experiment provides a window into the world of painting in virtual reality using Tilt Brush. We are excited by this new medium and hope the experience leaves you feeling the same. Visit g.co/VirtualArtSessions to start exploring.
