Posted by Michele Turner, Director of Product and Smart Home Ecosystem for Google Nest
To create a helpful home experience, we have focused on foundational features necessary to make it easier for people to manage their smart devices. But as people spend more and more time at home during these challenging times, it’s important that we invest in additional ways to work with developers to build a more useful connected home.
Today, at the "Hey Google" Smart Home Virtual Summit, we gave updates on our latest smart home initiatives, talked more in-depth about the new smart home controls in Android 11, and previewed some platform tools that we're investing in to make devices easier to set up and work with Google Assistant.
As many of us continue to stay home, smart devices are being used a lot more. With the biggest growth coming from entertainment devices, we’re increasing our support in this area with our Smart Home API.
Last year, we launched Google Assistant support for Smart Home for Entertainment Device (SHED) device types and traits, including TVs, remotes, set-top boxes, speakers, soundbars, and even game consoles from top brands like Xbox, Roku, Dish, and LG. And now, we are making these APIs public for any Smart TV, set-top box or game developers to use. SHED gives users the ability to control their favorite entertainment devices from any Assistant-enabled smart display, smart speaker or mobile device.
With the release of Android 11, coming out later this year, we are introducing a dedicated space for Smart Home controls that users can find quickly, and access any time. We’ve redesigned the power menu to make devices linked to Google Assistant just a button-press away.
Users with the Home App can choose all, or just their favorite controls to be in the space. For partners, you get this for free - there’s no new development work required. We’ll have sliders which will allow you to adjust specific settings, like the temperature of your thermostat in the morning, or how far to open the blinds. You can also customize what devices are visible from the control space and whether these devices can be accessible in your lockscreen.
With Android 11, we want to give users a quick and easy way to check or adjust the devices in their home. And as we continue to add new surfaces for device control, it becomes more critical to ensure we have accurate state. In the coming months, we’ll be introducing tools to measure your reliability and latency to help improve and debug state reporting. Once you hit key targets for reliability and latency, we will shift from a default of querying your state to using report state to render stateful controls. This will reduce query volume on your servers and improve the user experience with accurate device state across multiple surfaces.
In addition to state accuracy, the best user experience comes with strong reliability and low latency. To help achieve both, we launched local execution with the Local Home SDK back in April. As part of the Smart Home platform, local fulfillment can extend your Smart Home Action and routes commands to devices through the local network, benefitting users with reduced latency and higher reliability by removing an additional cloud hop.
To ease the development process, the Local Home platform supports both Chrome and Node.js runtime environments, as well as building and testing of apps on local development machines or personal servers. Once you've deployed your local fulfillment app, users will benefit immediately without having to upgrade hardware or manually update firmware. Nanoleaf and Yeelight have already enabled local execution for their devices. It’s available to all developers through the Actions on Google Console.
Implementing a high quality integration is important - it reduces churn and delights users. Yet, it’s still challenging to get users to discover these features, and we’re doing a couple of things on our end to increase the funnel of users linked to your action. We are excited to launch OAuth-based App Flip on the developer console today. With AppFlip, we streamline the standard account linking flow by flipping users from the Google Home App to the Partner app to gather consent without requiring the users to re-enter their credentials.
To increase awareness of your Action, you will soon be able to initiate the account linking flow within your app. There will also be more opportunities to increase awareness through feature promotion and in-app notification using your app, and we will have more details on discovery and linking opportunities later this year.
We know that visibility into the behavior of your smart home integrations is critical, from debugging in early development to detailed analytics in production. To enhance developer productivity, we've integrated with the powerful monitoring and troubleshooting tools available in Google Cloud Platform to provide detailed event logs and usage metrics for smart home projects.
We’ve also recently launched new tools to help developers improve the reliability of their integrations and aid in debugging and resolving issues quickly. You can view aggregate metrics directly in the developer console, or build logs-based metrics to find trends and gain deeper insights into common problems. Google Cloud Platform also enables developers to build custom alerts to quickly identify production issues.
You can also find a new Smart Home Analytics Dashboard accessible from the developer console and pre-populated with charts for common metrics such as Daily Active Users and Request Breakdown — giving you an overall picture of your integration's behavior. This dashboard is powered by new usage and performance metrics in Google Cloud Monitoring, giving you the power to set alerts and be notified if your integration has an issue. Get started today by going to the “Analytics” tab in the Actions console or the Google Cloud console to check out these new logs, metrics, and alerting capabilities for your projects!
Last year, we announced that we’re moving from the Works with Nest program to Works with Google Assistant and build on a foundation of privacy and data security to ensure users have confidence in how Google and our partners are protecting the consumer’s home data.
As part of that effort, we created the Device Access program to provide a way for partners to integrate directly with Nest devices. To support the Device Access program, we will soon launch the Device Access Console, a self-serve console that guides commercial developers through the different project phases: development, certification and pilot testing, and finally production.
For a commercial developer the console allows them to manage your various projects and integrations. It also provides development guides and trait documentation for all supported Nest devices. Individuals who want to create their own automations with their Nest devices will be able to do so with this console, but only for the homes they are a member of.
One of the most popular features with Nest users is the ability to automatically trigger routines based on whether users are Home or Away. Later this year, similar functionality will be available with Google Assistant through occupancy detection.
Sleep is also a critical part in maintaining our overall well-being as we stay more at home. Last year we launched the Gentle Sleep & Wake feature with Philips Hue, which slowly brightens or dims the lights at a specific time or can be tied to your morning alarms. Just say, “Turn on Gentle Wake up” to your bedroom speaker to ‘set it and forget it.’ The Light Effects trait is now public so all developers can integrate their native Sleep or Wake experiences - in fact LIFX has recently launched! We encourage you to build and integrate your own unique experiences. We’ll have a larger launch moment later this year when we launch emulated Sleep and Wake effects so that it’ll work out of the box for any smart light!
Another way partners will be able to innovate on our platform and provide more helpful experiences to users is by extending personal routines with custom routines designed by partners, available in the coming months. Developers will be able to create and suggest routines, not just for their devices, but that can work with other devices in a customer’s home. You’ll be able to create solutions for your customers that are based on your core business and bring value to your customers - whether it’s wellness, cleaning, or entertainment. Users will be able to browse and opt-in to approved routines and choose to have Nest and other devices react and participate in that routine.
Our Smart Home efforts have grown significantly over the past several years. We now have integrations with thousands of partners covering all the major connected product categories and devices, and will continue our ambitious goal to build deeper in-home integrations. Be sure to review our docs/samples/videos to learn about all the cool new stuff, and connect with us on our dev communities.
Posted by Toni Klopfenstein, Developer Advocate
Easy account linking helps create more helpful user experiences with Google Assistant and your products and services. Today, we are launching OAuth-based App Flip, a new feature to help you build a better mobile account linking experience. App Flip allows your users to seamlessly link their accounts to Google without having to re-enter their credentials if they have already signed in to your app on their device. This streamlined authorization flow helps minimize users dropping out of the account linking process, and makes it easier for users to integrate your smart devices into their smart homes. This feature is now available for all Smart Home Actions and is available in beta for other Conversational Actions.
When App Flip is implemented within your application, the Google Assistant or Google Home app automatically flips to your app when users initiate the account linking process. Once users consent to link their accounts, your app then requests an authorization code from your server, and Google handles the remainder of the account linking flow.
Account Linking Flows
By flipping directly to your local app, users can skip logging in again and can simply choose to link their accounts.
You can implement App Flip by enabling your app to accept a deep link that allows Google to connect. We also have test tools available for both Android and iOS to help you verify your app integration with App Flip.
For more details on how to use App Flip with your integration, check out the docs, or the sample Android and iOS apps. For Conversational Actions, please sign up for the Beta program.
We want to hear from you, so continue sharing your feedback with us, and engage with other Action developers in the /r/GoogleAssistantDev community. Follow @ActionsOnGoogle on Twitter for more of our team's updates, and tweet using #AoGDevs to share what you’re working on. We can’t wait to see what you build!
UPDATE: Apr 23, 2020: gcloud min version has been updated Apr 22, 2020: error injection planned for Apr 28 is CANCELLED, JSON-RPC and Global HTTP batch endpoint will perform normally. Next error injection window will be on May 26 as scheduled.
Apr 23, 2020: gcloud min version has been updated
Apr 22, 2020: error injection planned for Apr 28 is CANCELLED, JSON-RPC and Global HTTP batch endpoint will perform normally. Next error injection window will be on May 26 as scheduled.
We have invested heavily in our API and service infrastructure to improve performance and security and to add features developers need to build world-class APIs. As we make changes we must address features that are no longer compatible with the latest architecture and business requirements.
The JSON-RPC protocol (http://www.jsonrpc.org/specification) and Global HTTP Batch (example) are two such features. Our support for these features was based on an architecture using a single shared proxy to receive requests for all APIs. As we move towards a more distributed, high performance architecture where requests go directly to the appropriate API server we can no longer support these global endpoints.
Back in April, we released the first set of Smart Home Entertainment Device (SHED) types, including TV, set-top box, and remote, as well as the traits AppSelector, InputSelector, MediaState, TransportControl, and Volume. We are excited to announce the release of new Smart Home Entertainment Device (SHED) types and traits. These new device types and traits compliment the original set we released earlier this year, and help build out a more complete solution for smart home media and gaming devices. By implementing these types and traits on your entertainment devices, you can enable users to fully access device and media controls from any Assistant surface.
SHED Types and Traits
To expand the SHED options, we've released the following new device types for Smart Home:
We've also released the following new trait:
To ensure a consistent, high-quality experience for all end users, each of these device types require your service to report activityState and playbackState to Google using the ReportState API. This requirement improves the portability between media devices and helps the Assistant better understand user intents for these devices. By implementing the complete set of recommended device traits, you can further improve the quality of your smart home Action and improve device targeting for media playback command fulfilment.
activityState
playbackState
For more information on how to implement these new device features, check out the docs and samples. You can also join us at our "Hey Google" Smart Home Virtual Summit to learn more about these new features.
We want to hear from you, so continue sharing your feedback with us through the issue tracker, and engage with other smart home developers in the /r/GoogleAssistantDev community. Follow @ActionsOnGoogle on Twitter for more of our team's updates, and tweet using #AoGDevs to share what you’re working on. We can’t wait to see what you build!
Posted by Jose Ugia and Checkout.com
We sat down with Riaz Bordie, the CTO of Checkout.com, a leading international provider of online payment solutions, to get his advice to merchants and the developer community on how to think about future-proofing payments in the uncertain world we live in today.
Jose Ugia: What advice do you have for merchants and developers as it relates to payments in these difficult times?
Riaz: Merchants are seeing a polarizing impact of COVID-19 on their businesses. For those who have an online presence, you’re either seeing a lull in traffic or a spike.
If you’re a merchant who’s seeing traffic dwindle, it’s more important than ever to make sure every transaction counts. If you used to see 50 transactions a day and now you see 10, you want to make sure all 10 deliver. Work with your Payment Service Provider (PSP) to make sure your approval ratios are as optimal as possible -- a legitimate customer who gets declined incorrectly may not return to purchase as they have in the past. If your PSP supports alternative payment methods like Google Pay that decrease friction at checkout and local payment methods if you’re selling internationally, that’s ideal. Keep an eye on your PSP’s stacks and uptimes to make sure you’re not missing out on sales due to outages or technical issues.
If you’re a merchant seeing a spike in traffic, that’s great news! But it’s important to note that a sudden traffic increase without proper operational and infrastructure planning can lead to fraud spikes, decreases in approval ratios, and downtime. With higher sales velocity, risk related issues will multiply. You’ll see more attempted fraud as fraudsters take advantage of unsuspecting consumers, higher payment declines resulting from outdated issuer risk modeling and excessive chargeback levels, subscription cancellations, buyer’s remorse, among others. How are your payments infrastructure and operations equipped to handle all of this?
Make sure your infrastructure is capable of scaling up. If you don’t have autoscaling, you’ll need a team and processes in place to scale infrastructure for traffic spikes, and keep in mind this may get harder with people working remotely. Work your PSP and other providers to optimize your payments, risk models and chargeback handling during this challenging time.
For both types of merchants, it’s important to pay closer attention to performance of your payments system. This includes both ensuring that processes are working in an optimal way - especially given remote working situations and also ensuring that you are seeing efficiencies at scale.
Jose Ugia: How did you think about building a payments infrastructure that was scalable and future-proof at Checkout.com?
We knew in the beginning we wanted a unified API, which through a single integration gives a merchant access to any market via a range of payment methods and other facilities. We’ve worked hard to get acquiring licenses in as many markets as possible so we can bring acquiring in-house, which in turn gives us greater visibility on the entire payment flow. We have also invested in a gateway that can be consistently deployed in local geographies so that whether the merchant is in Dubai or Singapore, they are getting the most optimal traffic flow.
Any engineer knows that tech breaks. Those who win have a better plan for dealing with breakage efficiently, to consistently maintain high levels of service. We spend a lot of time and resources on making sure our stack is resilient and we have the right operational processes in place to both proactively monitor for potential issues and respond correctly when they come up.
Jose Ugia: Speaking of where things are headed, where do you see the future of payments going from a payment service provider perspective?
A few key trends I see:
Risk & Fraud Detection. AI/ML is improving every aspect of tech. Fraudsters will get smarter but so will fraud prevention - it’s a cat and mouse game. In payments, sophisticated risk engines offering ML-based transaction scoring and highly customizable rules builders, among other features, will get better at detecting fraud without compromising sales.
Global acceptance will continue to be complex but paramount. Offering a variety of payment methods is table stakes these days. More and more, we’ll see that local payment methods aren’t the alternative but instead the primary way consumers pay. For example, you need to have Giropay if you’re selling in Germany and Alipay if you’re selling in China if you want a high conversion rate. Ensure that you and your local entities have an optimized setup with your acquirer (ideally domestic where possible) focused on achieving the lowest costs and highest approval rates.
Embedded infrastructure. Merchants - especially enterprise players - will want increased visibility and more control on optimizing their payment systems. We offer this level of insight and flexibility to our merchants today via our APIs around risk, reconciliation, disputes, etc. But we’re headed toward a world where dedicated infrastructure will become part of the package and allow for complete data separation and zero contention.
Jose Ugia: How do you think these changes of payments infrastructure will impact consumers downstream?
Convenience is king among consumers. I believe that COVID-19 will accelerate the move toward a contactless payments society, with consumers relying more on digital wallets and opportunities to pay through their devices. I personally no longer take my wallet out with me when I leave the house. A couple of years ago that felt like a conscious decision - now it’s just part of everyday life to rely solely on my smartphone to pay.
In some regions like MENA, which has typically been a cash-on-delivery society, we’re seeing more merchants close off cash and impose digital payments, opening up more adoption of upfront e-commerce payments. As mandated payment methods begin to change consumer behavior (studies say it takes 2 months to change a habit), new ways of paying will be here to stay, even beyond COVID-19.
When creating scalable applications, consistent and reliable monitoring of resources is a valuable tool for any developer. Today we are releasing enhanced analytics and logging for Smart Home Actions. This feature enables you to more quickly identify and respond to errors or quality issues that may arise.
Request Latency Dashboard
You can now access the smart home dashboard with pre-populated metrics charts for your Actions on the Analytics tab in the Actions Console, or through Cloud Monitoring. These metrics help you quantify the health and usage of your Action, and gain insight into how users engage with your Action. You can view:
Successful Requests Dashboard
Cloud Logging provides detailed logs based on the events observed in Cloud Monitoring.
We've added additional features to the error logs to help you quickly debug why intents fail, which particular device commands malfunction, or if your local fulfilment falls back to cloud fulfilment.
New details added to the event logs include:
You can additionally export these logs through Cloud Pub/Sub, and build log-based metrics and alerts for your development teams to gain insights into common issues.
For more guidance on accessing your Smart Home Action analytics and logs, check out the developer guide or watch the video.
We want to hear from you! Continue sharing your feedback with us through the issue tracker, and engage with other smart home developers in the /r/GoogleAssistantDev community. Follow @ActionsOnGoogle on Twitter for more of our team's updates, and tweet using #AoGDevs to share what you’re working on. We can’t wait to see what you build!
Posted by Toni Klopfenstein, Developer Relations
Over the past year, we've been focused on building new tools and features to support our smart home developer community. Though we weren't able to engage with you in person at Google I/O, we are pleased to announce the "Hey Google" Smart Home Virtual Summit on July 8th - an opportunity for us to come together and dive into the exciting new and upcoming features for smart home developers and users.
Join us in the keynote where Michele Turner, the Product Management director of the Smart Home Ecosystem, will share our recent smart home product initiatives and how developers can benefit from these capabilities. She will also introduce new tools that make it easier for you to develop with Google Assistant. We will also be hosting a partner panel, where you can hear from industry leaders on how they navigate the impact of COVID-19 and their thoughts on the state of the industry.
Registration is FREE! Head on over to the Summit website to register and check out the schedule. Events will be held during EMEA, APAC, and AMER friendly times. We hope to see you and your colleagues there!
Posted by Rajat Paharia, Product Lead, AR Platform
Since the launch of ARCore, our developer platform for building augmented reality (AR) experiences, we've been focused on providing APIs that help developers seamlessly blend the digital and physical worlds.
At the end of last year, we announced a preview of the ARCore Depth API, which uses our depth-from-motion algorithms to generate a depth map with a single RGB camera. Since then, we’ve been working with select collaborators to explore how depth can be used across a range of use cases to enhance AR realism.
Today, we're taking a major step forward and announcing the Depth API is available in ARCore 1.18 for Android and Unity, including AR Foundation, across hundreds of millions of compatible Android devices.
Generate a depth map without specialized hardware to unlock capabilities like occlusion
As we highlighted last year, a key capability of the Depth API is occlusion: the ability for digital objects to accurately appear behind real world objects. This makes objects feel as if they’re actually in your space, creating a more realistic AR experience.
Illumix, the game studio behind Five Nights at Freddy’s AR: Special Delivery, uses occlusion to deepen the realism of the experience by allowing certain characters to hide behind objects for more startling jump scares.
Play Five Nights at Freddy’s AR: Special Delivery
While occlusion is an important capability, the ARCore Depth API unlocks more ways to increase realism and enables new interaction types. The ARCore Depth Lab spurred more ideas on how depth can be used, including realistics physics, surface interactions, environmental traversal, and more. Developers can now build on these ideas through the open sourced GitHub project.
Experiment with ARCore Depth Lab on the Google Play Store
The designers and engineers at Snap Inc. integrated several of these ideas into a set of Snapchat Lenses including the Dancing Hotdog and a new Android exclusive Undersea World Lens.
See how depth can add a layer of realism to your Snapchat experience
Snapchat Lens Creators can now download an ARCore Depth API template to create depth-based experiences for compatible Android devices. Sam Hare, Research Engineering Manager at Snap Inc, expressed his excitement, “We’re beginning to understand what kinds of depth capabilities are exciting for developers to build with. This single integration point streamlines and simplifies the development process and enables Lens Studio developers to easily take advantage of advanced depth capabilities.”
Another app that combines occlusion with other depth capabilities is Lines of Play, an Android experiment from the Google Creative Lab. Lines of Play lets users create domino art in AR, and uses depth information to showcase both occlusion and collisions. Design elaborate domino creations, topple them over and watch them collide with the furniture and walls in your room.
Watch as domino pieces topple into each other and onto your walls with Lines of Play
In addition to gaming and self-expression, depth can also be used to unlock new utility use cases. For example, the TeamViewer Pilot app, a remote assistance solution that enables AR annotations on video calls, uses depth to better understand the environment so experts around the world can more precisely apply real time 3D AR annotations for remote support and maintenance.
3D annotations help experts accurately highlight details in the TeamViewer Pilot app
Later this year, you will be able to try more depth-enabled AR experiences such as SKATRIX by Reality Crisis and SPLASHAAR by ForwARdgames, that use surface interactions and environmental traversal as they make rich use of the environment around you.
Check out surface interactions and environmental traversal in SKATRIX, and SPLASHAAR
While depth sensors, such as time-of-flight (ToF) sensors, are not required for the Depth API to work, having them will further improve the quality of experiences. Dr. Soo Wan Kim, Camera Technical Product Manager at Samsung commented on the future that the Depth API and ToF unlocks saying, “Depth will enrich user's AR experience in many perspectives. It will reduce scanning time, and can detect planes fast, even low textured planes. These will bring seamless experiences to users who will be able to use AR apps more easily and frequently.” In the coming months, Samsung will update their Quick Measure app to use the ARCore Depth API on the Galaxy Note10+ and Galaxy S20 Ultra.
Accurately measure with Quick Measure
To learn more and get started with the ARCore Depth API, get the SDK and visit the ARCore developer website.
Posted by Thye Yeow Bok, Developer Relations Program Manager
Apply now for the Google for Startups Accelerator: Southeast Asia
In the last few months, COVID-19 has ushered in an era of profound changes to the way we live and work, causing businesses to rethink strategies and product roadmaps. At the forefront of this change are startups, stepping up to solve for new and unforeseen challenges as they always have done — with agility, innovative technology, and resilience.
The Southeast Asia startup ecosystem has always been a hotbed for creativity and innovation. This part of the world has a rich history of homegrown entrepreneurs armed with solutions, oftentimes growing their local ideas to global companies. Startups are primed to develop solutions for the unique challenges we face today and we are committed to supporting them in that effort.
Today, I’m thrilled to announce that applications are open for Google for Startups Accelerator: Southeast Asia. This is a three month online accelerator program for high potential, early stage tech startups across the Southeast Asia region (Indonesia, Singapore, Malaysia, Thailand, Vietnam and the Philippines) and Pakistan. And this year, we’re looking particularly for startups who are solving for the challenges we face today: whether that’s startups looking at new healthcare, education, finance or logistics solutions in light of social distancing restrictions; using AI , ML or data analysis in meaningful ways; or using technology to make the world more inclusive for the elderly or people with disabilities.
Previously known as the Google Launchpad Accelerator, this program continues our longstanding commitment to help startups solve specific, technical challenges with Google support and resources. As part of the Google for Startups Accelerator, selected founders outline the top challenges their startup is facing, and are paired with relevant experts from Google and the industry to solve those challenges. Participating startups receive deep mentorship on both technical and business challenges as well as connections to relevant teams from across Google and its network of industry partners. In addition to mentorship and technical project support, the accelerator also includes deep dives and workshops focused on product design, customer acquisition, and leadership development for founders.
Applications are now open through July 19th 2020 for startups across Southeast Asia (Indonesia, Singapore, Malaysia, Thailand, Vietnam and the Philippines) and also in Pakistan.
We know that if startups succeed, our communities and economies do, too. We look forward to working with the next generation of founders and innovators who will help shape our economic recovery, and build a stronger long-term future in Southeast Asia and beyond.
Posted by Priyanka Vergadia, Developer Advocate
Google Cloud is a cloud computing platform that can be used to build and deploy applications. It allows you to take advantage of the flexibility of development while scaling the infrastructure as needed.
I'm often asked by developers to provide a list of Google Cloud architectures that help to get started on the cloud journey. Last month, I decided to start a mini-series on Twitter called “#13DaysOfGCP" where I shared the most common use cases on Google Cloud. I have compiled the list of all 13 architectures in this post. Some of the topics covered are hybrid cloud, mobile app backends, microservices, serverless, CICD and more. If you were not able to catch it, or if you missed a few days, here we bring to you the summary!
Series kickoff #13DaysOfGCP
Day 1
Day 2
Day 3
Day 4
Day 5
Day 6
Day 7
Day 8
Day 9
Day 10
Day 11
Day 12
Day 13
Wrap up!
We hope you enjoy this list of the most common reference architectures. Please let us know your thoughts in the comments below!
