A guest post by the Engineering team at SignAll | Twitter handle | MediaPipe team
Please note that the information, uses, and applications expressed in the below post are solely those of our guest author, SignAll.
SignAll SDK: Sign language interface using MediaPipe is now available for developers
When Google published the first versions of its on-device hand tracking technology in MediaPipe, the work could serve as a basis for developers to build sign language recognition solutions into their own apps. Later updates to this hand tracking solution have further improved its accuracy where other technologies have fallen short (Figure 1).
Figure 1. Illustrating MediaPipe’s improvement over time: hand skeleton tracking output of an older version (2020.02.10) and the latest version (2020.12.16). This handshape is used in sign language frequently, but often missed because of the lack of representation in training datasets.Watch the full video
SignAll is a startup working on sign language translation technology. Its mission is to make sign language interpretation universally available, both through communication between Deaf and hearing parties, and between Deaf individuals and computers. The SignAll’s products, which are used nationwide in the US for both communications and education, employ a complex multi-camera setup and gloves with colored markers. While sign languages’ complexity goes far beyond handshapes (facial features, body, grammar, etc.), it is true that the accurate tracking of the hands has been a huge obstacle in the first layer of processing – computer vision. MediaPipe unlocked the possibility to offer SignAll’s solutions not only glove free, but also by using a single camera. SignAll has just announced the availability of the first SDK of its kind, so developers can now enable sign language input in their apps.
The company recently published an interactive educational app in the App Store, that lets the user practice signing with immediate feedback, this app also serves as a demonstration of the possibilities with the SDK.
SignAll with MediaPipe Hands
Our system uses several layers for sign recognition, and each one uses more and more abstract data. The low-level layer extracts crucial hand, body, and face data from 2D and 3D cameras. In our first implementation, this layer detects the colors of the gloves and creates 3D hand data. Replacing this with MediaPipe Hands (supplemented by the MediaPipe Pose and MediaPipe Face Mesh) has been a game changer for using our system without gloves or special lighting.
Figure 2. Demo of our SignAll SDK developed using MediaPipe
As mentioned earlier, we use multiple cameras with depth sensors which are calibrated in the real world. This allows for a more accurate 3D world space than that of a local camera or tensor spaces, but requires the use of hand landmark detection for each camera. The cameras are placed distinctly to each other in position and orientation so the hands are visible more frequently, as one hand might cover the other from one camera but not necessarily from the others.
Figure 3. Corrected 3D hand shape based on three cameras. Due to the unique orientation, the detection from the front camera is wrong, but the side camera can correct the result.
The next step is to filter and smooth the data to replicate the precise measurements offered by our colored glove markers. Although SignAll’s markers are different from the landmarks given by MediaPipe, we used our hand model to generate colored markers from landmarks. Therefore, the new mocap data is fully compatible with the previous one.
Although we are focused mainly on the hands, we also integrated MediaPipe Pose and MediaPipe Face Mesh. The pose landmarks provide accurate hand position information, even when touching or close to each other.
While the two versions of mocap are compatible, the nature of the artifacts is different: direct measure of each marker and simulated markers from a globally detected hand. Due to this discrepancy, we had to refine the parameters on higher levels. On the other hand, we could still use our huge sign database for the gloveless configuration. By replacing the low-level data and refining our higher-level data, we could test our system without gloves. Going gloveless can be a huge step for using our sign recognition technology easily worldwide.
Figure 4. Demonstration of compatible mocap from different low-level trackings. The right side is without gloves; the left side is with gloves. This compatibility enables the usage of SignAll’s meticulously labeled dataset of 300,000+ sign language videos to be used for the training of recognition models based on different low-level data. Watch the full video
The SignAll system using MediaPipe framework
After integrating MediaPipe Hands into our system, we also wanted to take advantage of the customization and scaling opportunities provided by the MediaPipe framework on multiple platforms. This allowed us to not only prototype our research state methods in Python, but also deliver our end-user solutions for Windows, iOS, Android, and even the Web. Thanks to the similarities between our module graph system and the calculator graph of MediaPipe, our existing processing units can be reused in this new framework with minor modifications. With that said, the extended platform set also comes with other challenges, like using only a single 2D camera in most cases instead of a calibrated multi-camera system.
The models, algorithms and techniques we have used were mostly developed to work on our mocap data interpreted in the 3D global world. The data extracted from a single-camera setup, of course, cannot be as detailed. That is the reason we had to make some adjustments to our implementations, fine-tune the algorithms and add some extra logic (e.g., dynamically adapting to the changes of space resulted by the hand-held camera use-case). Luckily, the MediaPipe framework enables us to implement the core processing units in C++, so we can still benefit from the runtime-optimized core solutions we previously developed.
Some higher-level models trained on 3D data also needed to be re-trained in order to perform better on the data originating from a single 2D source. The MediaPipe landmarks are defined by 3D coordinates, which makes it possible to reuse the existing training methods and concepts. On the other hand, the 2D information is more directly extracted and therefore more stable than the third coordinate, which was taken into consideration while designing the training modifications.
Luckily, it is not necessary to make an entirely new data recording for this purpose. We can still use our huge video database annotated in great detail. The preprocessed mocap data we can extract from our recordings and interpreted in the 3D world can be used to simulate hand, skeleton, or face landmark detections in any virtual camera view.
Among the data on virtual camera views, we also use traditional 2D recordings in sufficient proportion to cover the unique noise characteristics of the landmark detections. Thanks to having most of these types of data in advance, we can focus on the most exciting part - trying the latest techniques and training new models.
Summary
The advancement made possible by MediaPipe enabled SignAll to change its model. In addition to offering all-in-one products for sign language education and translation, SignAll is now starting to offer an SDK for developers. The capabilities of this SDK depend on the type of the camera or cameras being used and the available computational capacity. The possible functions enabled by using the SDK vary from launching video calls by signing the contact’s name (watch a demo here), adding addresses into navigation by signing (as a counterpart to speech input), or ordering food on a fast-food restaurant’s kiosk or drive-thru. With its mission to make sign language an alternative everywhere that voice can be used, SignAll is excited to see more and more apps implementing this feature.
We are eager to try future updates of MediaPipe, which could bring us closer to our ultimate goal of having our solutions available for everyone on any device. The most awaited update is the ability to build custom MediaPipe graphs and add our own calculators for web-based solutions aided by the WebAssembly technology, so websites will be able to use a new level of accessibility features for Deaf visitors.
Posted by Erica Hanson, Global Program Manager, Google Developer Student Clubs
Hey, student developers! If you’re passionate about programming and are ready to use your technology skills to help your community, then you should become a Google Developer Student Clubs Lead!
Application forms for the upcoming 2021-2022 academic year are NOW OPEN. Get started at goo.gle/gdsc-leads.
Want to know more? Learn more about the program below.
Google Developer Student Clubs are university based community groups for students interested in Google developer technologies. With clubs hosted in 106 countries around the world, students from undergraduate and graduate programs with an interest in leading a community are welcome. Together, students learn the latest in Android App Development, Google Cloud Platform, Flutter, and so much more.
By joining a GDSC, students grow their knowledge in a peer-to-peer learning environment and put theory to practice by building solutions for local businesses and their community.
As a Google Developer Student Club Lead you will have the chance to…
Google Developer Student Clubs are now in 106 countries with 1250+ groups. Find a club near you or learn how to start your own, here.
We encourage students to submit their forms as soon as possible. You can learn more about your region’s application deadline, here. Make sure to learn more about our program criteria.
From working to solve the United Nations Sustainable Development Goals to helping local communities make informed voting decisions, Google Developer Student Club leads are learning valuable coding skills while making a true difference. As a lead from a Club in Kuala Lumpur, Malaysia put it,
“The secret to our club’s success was that we were able to cultivate a heart of service and a culture of open mentorship.”
We can’t wait to see what our next group of Google Developer Student Club leads will accomplish this year. Join the fun and get started, here.
*Google Developer Student Clubs are student-led independent organizations, and their presence does not indicate a relationship between Google and the students' universities.
Posted by Wesley Chun, Developer Advocate, Google Cloud
Since its initial launch in 2008 as the first product from Google Cloud, Google App Engine, our fully-managed serverless app-hosting platform, has been used by many developers worldwide. Since then, the product team has continued to innovate on the platform: introducing new services, extending quotas, supporting new languages, and adding a Flexible environment to support more runtimes, including the ability to serve containerized applications.
With many original App Engine services maturing to become their own standalone Cloud products along with users' desire for a more open cloud, the next generation App Engine launched in 2018 without those bundled proprietary services, but coupled with desired language support such as Python 3 and PHP 7 as well as introducing Node.js 8. As a result, users have more options, and their apps are more portable.
With the sunset of Python 2, Java 8, PHP 5, and Go 1.11, by their respective communities, Google Cloud has assured users by expressing continued long-term support of these legacy runtimes, including maintaining the Python 2 runtime. So while there is no requirement for users to migrate, developers themselves are expressing interest in updating their applications to the latest language releases.
Google Cloud has created a set of migration guides for users modernizing from Python 2 to 3, Java 8 to 11, PHP 5 to 7, and Go 1.11 to 1.12+ as well as a summary of what is available in both first and second generation runtimes. However, moving from bundled to unbundled services may not be intuitive to developers, so today we're introducing additional resources to help users in this endeavor: App Engine "migration modules" with hands-on "codelab" tutorials and code examples, starting with Python.
Each module represents a single modernization technique. Some are strongly recommended, others less so, and, at the other end of the spectrum, some are quite optional. We will guide you as far as which ones are more important. Similarly, there's no real order of modules to look at since it depends on which bundled services your apps use. Yes, some modules must be completed before others, but again, you'll be guided as far as "what's next."
More specifically, modules focus on the code changes that need to be implemented, not changes in new programming language releases as those are not within the domain of Google products. The purpose of these modules is to help reduce the friction developers may encounter when adapting their apps for the next-generation platform.
Central to the migration modules are the codelabs: free, online, self-paced, hands-on tutorials. The purpose of Google codelabs is to teach developers one new skill while giving them hands-on experience, and there are codelabs just for Google Cloud users. The migration codelabs are no exception, teaching developers one specific migration technique.
Developers following the tutorials will make the appropriate updates on a sample app, giving them the "muscle memory" needed to do the same (or similar) with their applications. Each codelab begins with an initial baseline app ("START"), leads users through the necessary steps, then concludes with an ending code repo ("FINISH") they can compare against their completed effort. Here are some of the initial modules being announced today:
webapp2
ndb
taskqueue
What should you expect from the migration codelabs? Let's preview a pair, starting with the web framework: below is the main driver for a simple webapp2-based "guestbook" app registering website visits as Datastore entities:
class MainHandler(webapp2.RequestHandler): 'main application (GET) handler' def get(self): store_visit(self.request.remote_addr, self.request.user_agent) visits = fetch_visits(LIMIT) tmpl = os.path.join(os.path.dirname(__file__), 'index.html') self.response.out.write(template.render(tmpl, {'visits': visits}))
A "visit" consists of a request's IP address and user agent. After visit registration, the app queries for the latest LIMIT visits to display to the end-user via the app's HTML template. The tutorial leads developers a migration to Flask, a web framework with broader support in the Python community. An Flask equivalent app will use decorated functions rather than webapp2's object model:
LIMIT
@app.route('/') def root(): 'main application (GET) handler' store_visit(request.remote_addr, request.user_agent) visits = fetch_visits(LIMIT) return render_template('index.html', visits=visits)
The framework codelab walks users through this and other required code changes in its sample app. Since Flask is more broadly used, this makes your apps more portable.
The second example pertains to Datastore access. Whether you're using App Engine's ndb or the Cloud NDB client libraries, the code to query the Datastore for the most recent limit visits may look like this:
limit
def fetch_visits(limit): 'get most recent visits' query = Visit.query() visits = query.order(-Visit.timestamp).fetch(limit) return (v.to_dict() for v in visits)
If you decide to switch to the Cloud Datastore client library, that code would be converted to:
def fetch_visits(limit): 'get most recent visits' query = DS_CLIENT.query(kind='Visit') query.order = ['-timestamp'] return query.fetch(limit=limit)
The query styles are similar but different. While the sample apps are just that, samples, giving you this kind of hands-on experience is useful when planning your own application upgrades. The goal of the migration modules is to help you separate moving to the next-generation service and making programming language updates so as to avoid doing both sets of changes simultaneously.
As mentioned above, some migrations are more optional than others. For example, moving away from the App Engine bundled ndb library to Cloud NDB is strongly recommended, but because Cloud NDB is available for both Python 2 and 3, it's not necessary for users to migrate further to Cloud Datastore nor Cloud Firestore unless they have specific reasons to do so. Moving to unbundled services is the primary step to giving users more flexibility, choices, and ultimately, makes their apps more portable.
For those who are interested in modernizing their apps, a complete table describing each module and links to corresponding codelabs and expected START and FINISH code samples can be found in the migration module repository. We are also working on video content based on these migration modules as well as producing similar content for Java, so stay tuned.
In addition to the migration modules, our team has also setup a separate repo to support community-sourced migration samples. We hope you find all these resources helpful in your quest to modernize your App Engine apps!
Recently young developers in Saudi Arabia from Google Developer Student Clubs, a program of university based community groups for students interested in Google technologies, came together to help local small businesses. As more companies across the globe rely on online sales, these students from Prince Sattam Bin Abdulaziz University noticed that many of their favorite local stores did not have a presence on the web.
So to help these local shops compete, these up-and-coming developers went into the community and began running workshops to teach local store owners the basics of building a website. Inspired by Google’s fundamentals of digital marketing course, these learning sessions focused on giving small business owners basic front-end skills, while introducing them to easy to use coding tools.
Image of Chrome Devtools
The first goal of these student-run workshops was to teach local store owners the basics of building web interfaces. In particular, they focused on websites that made it easy for customers to make purchases. To do this, the students first taught store owners the basics of HTML, CSS, and JS code. Then, they showed them how to deploy Chrome DevTools, a collection of web developer tools built directly into the Google Chrome browser that allows programmers to inspect and edit HTML, CSS, and JS code to optimize user experience.
Next, the students challenged participants to put their knowledge to use by creating demos of their businesses' new websites. The young developers again used Chrome DevTools to highlight the best practices for testing the demo sites on different devices and screen sizes.
Image of demo created and maintained in workshop.
With the basics of HTML, CSS, JS code, and Chrome DevTools covered, the students also wanted to give the store owners tools to help maintain their new websites. To do this, they introduced the small businesses to three toolkits:
With these basic coding skills, access to intuitive toolkits, and completed website demos, the local businesses owners now had everything they needed to launch their sites to the public - all thanks to a few dedicated students.
All over the world, students are coming together to learn programming and make a difference in their community as members of local Google Developer Student Clubs. Learn more on how to get involved in projects like this one, here.
Posted by Biswajeet Mallik, Program Manager, Google Developers India.
Image from Cloud Community Days India
Earlier this year, ten Google Developer Groups in India came together to host Google Cloud Community Days India, a two day event helping developers study for their upcoming Cloud Certification exams. To address the rising demand for professional certifications, the virtual event hosted over 63,000 developers, covered four main exam areas, and welcomed nine speakers. This was the second edition to the event series which started in 2019 in India.
By providing expert learning materials and mentorship, the event uniquely prepared developers for the Associate Cloud Engineer, Professional Data Engineer, Professional Cloud Machine Learning Engineer, and Professional Cloud Architect exams. Learn more below.
The Cloud Community Days event focused on helping developers study for four milestone certifications, tailored to engineers at four different stages of their career. The goal: help Google Developer Group members obtain the right credentials to improve their job prospects.
The event broke participants into breakout sessions based on which exam they were preparing to take. Since the certifications targeted professionals of all skill levels, study groups ranged from early career associates to late career executives. The learning groups were organized around the following certifications:
This learning session was created to help early career developers complete the first stepping stone exam. In particular, learning materials and speakers were curated to guide participants who had no prior experience, or very little, working on the Google Cloud Platform.
Workshops were mainly dedicated to assisting programmers who were familiar with building different applications but wished to show employers that they could deploy them on Google Cloud Platform.
Watch more from: Day 1, here. And day 2, here.
The next group brought together were data practitioners with special interests in data visualization and decision making. Workshops and learning activities helped these developers hone their large scale data and data driven decision making abilities.
Improving these skills are essential for passing the Professional Data Engineers certification and growing a programmer’s early career.
For these sessions, the Google Developer Group Cloud community paired experienced programmers with a significant interest in ML to form their study groups. The main driver in these learning activities was to help seasoned developers gain a deeper understanding of how to utilize Google Cloud ML services.
With significant emphasis being placed on machine learning in the ecosystem right now, Google Developer Group community leaders felt this certification could help developers make the leap into new leadership roles.
Lastly, this event paired experienced Cloud executives and professionals working in leading capacities for their organizations. For these sessions, speakers and activities had a specific scope: help high level professions be at the forefront of Google Cloud Platforms innovative capabilities.
Specifically, the Professional Cloud Architect Certification was created to help senior software engineers better design, scale and develop highly secure and robust applications.
Day 1, here. And day 2, here.
Overall, the community put together these resources to help developers feel more confident in their abilities, obtain tangible credentials, and in turn increase access to better job opportunities. As two participants recalled the event,
“This was the first time I attended the Google Developer Group event! It is an awesome package for learning in one place. All the fun activities were engaging and the panelist discussion was also very insightful. I feel proud to be a part of this grand GDG event.”
With Google Developer Groups, find a space to learn alongside a group of curious developers, all coming together to advance their careers from withinside a caring community of peers.
Want to know more about what Cloud Community days were like? Then watch their live recording below.
Ready to find a community event near you? Then get started at gdg.community.dev
Posted by Charles Maxson, Developer Advocate
Google Workspace Add-ons offer developers a simplified, structured, and safe way of integrating your solutions right within the Google Workspace user experience, allowing you to bring the logic and data of your application right within the reach of billions of Google Workspace users. So whether your goal is to help users avoid switching context from their inbox to your application, or to easily bring in data from your solution to Google Sheets, developing your own Google Workspace Add-ons makes a lot of sense to keep users productive, engaged and focused.
While the concept of Add-ons for Google Workspace isn’t new per se, building add-ons for Google Workspace has come a long way since they were first introduced some years back. Originally designed to allow solution developers to extend our collaboration apps: Google Docs, Sheets, Forms and Slides, it’s now possible to create a single add-on project for Google Workspace that spans the entire suite, including Gmail, Drive and Calendar.
The original design created for our collaboration apps also required you to use HTML, CSS and Google Apps Script to ‘hand roll’ elements like the user interface and events, requiring a bit more do-it-yourself effort (aka code) for developers, resulting in more inconsistency across the add-on market. That has evolved as Google Workspace Add-ons adopted Card-based interfaces more recently, allowing developers to simplify and standardize add-on building by leveraging just their knowledge of Google Apps Script.
Today we are pleased to announce that building Google Workspace Add-ons has evolved once again, this time to offer developers an alternative to using Apps Script for building add-ons with the general availability of Alternate Runtimes for Google Workspace Add-ons. Announced via an early access program mid last year, the release of Alternate Runtimes is a major breakthrough for Google Workspace developers who want to use their own development stack: hosting, tools, languages, packages, processes, etc.
While Alternate Runtimes enables the same functionality that Apps Script does for building add-ons, the flexibility and the freedom to choose your dev environment plus the opportunity to decouple from Apps Script will likely yield greater developer productivity and performance gains for future projects. This commonly requested feature by Google Workspace solution developers has finally become a reality.
Technically, there’s a little more effort in using the Alternate Runtimes method, as Apps Script does mask much of the complexity from the developer, but it's essentially swapping in JSON for Apps Script in rendering the Cards service-based interfaces needed to drive Google Workspace Add-ons. Learn more about getting started with Alternate Runtimes here or try the five minute Quickstart for Alternate Runtimes to see it in action.
Also note, whether you are just getting started or you are an experienced add-on builder, we have recently released the GWAO Card Builder tool that allows you to visually design the user interfaces for your Google Workspace Add-ons projects. It is a must-have for add-on developers using either Apps Script or Alternate Runtimes, enabling you to prototype and design Card UIs super fast without hassle and errors of hand coding JSON or Apps Script on your own.
Google Workspace Card Builder Design Tool
Included with this launch of Alternate Runtimes for general availability is also the debut of the Google Workspace Add-ons Cloud API, which allows you to completely forgo using Apps Script for managing Google Workspace Add-on deployments using Alternate Runtimes. Unlike using Alternate Runtimes during the beta program where you still needed to create an Apps Script project to stub out your project endpoints via the manifest file, the Google Workspace Add-ons Cloud API allows you to create and manage your add-on deployment lifecycle with a series of command line instructions.
With the Google Workspace Add-ons Cloud API you can create a deployment, install or delete a deployment, get a list of deployments, manage permissions and more. These are straightforward to use from a CLI like gcloud, which will help simplify developing and deploying Google Workspace Add-ons built via Alternate Runtimes. For documentation on how to use the new Add-ons Cloud API, refer back to the Quickstart: Create an add-on in a different coding language example.
While Alternate Runtimes for Google Workspace Add-ons is officially generally available as of today, a number of Google Cloud partner teams have already been working with the technology via our early adopter program. One of those Google Cloud partners, Zzapps based out of the Netherlands, has already been creating solutions using Alternate Runtimes in their work building Add-ons for customers.
We asked Riël Notermans, owner of Zzapps (and Google Developer Expert), whose teams have been developing on Google Workspace for over a decade, to share his team’s key takeaways on Alternate Runtimes. He offered not only his insights, but added a few screenshots of one of their recent projects to illustrate as well. In Riël’s own words: “Now that we can use Alternate Runtimes for Add-ons, it changes how we approach projects from start to finish. Prototyping with GSAO makes it possible for us to quickly draft an add-on’s functionality, creating trust and clearness about what we will deliver. Alternate Runtimes makes it possible to tap into our existing applications with almost no effort. We only need to create a JSON response to push a card to interact with add-ons. Our developers are able to work in their own environment, keeping our own tools and development flow. Here’s an example below using a Node.js Express server project that we used to set email signatures, adding a few routes for the card but using our existing logic. The add-on is used to control the functionality.”
Routing Add-on requests to existing logic
“Being able to update your deployment for local development for live testing, without having to create new versions constantly, drastically improves the development experience.”
Introduces advantage of instant testing of add-ons
“Because the Add-on runtimes has built-in authorization and tokens, it is really easy to safely interact with the users data without building complex backend authentication.”
Maximizing use of existing UI with Add-ons
“In the end, we still offer our users solutions for a great experience with a Google Workspace Add-on, while our developers get to use the tools and processes that make them more productive, capable and accomplished”
Creating Add-ons with Alternate Runtimes allows flexible, fast UI design
For More Information
If you want to learn more about using Alternate Runtimes for building Google Workspace Add-ons, here are some essential links for Google Workspace Add-on resources to get you started:
Posted by Ting Huang, Software Engineer
Some time ago, we announced that the Google Contacts API was being deprecated in favor of the People API, and it is scheduled for sunset on June 15, 2021. To aid in the process of migrating from Contacts API, we are pleased to announce that we have added two sets of new endpoints for working with contacts via the People API.
First, we now have new write endpoints that allow developers to create, delete, and update multiple contacts at once. In addition, we also have new read endpoints that allow developers to search a user’s contacts using a prefix query. Both will greatly improve working with the People API, so let’s take a quick look at how you can leverage these new endpoints today.
Applications need to be authorized to access the API, so to get started you will need to create a project on the Google Developers Console with the People API enabled to get access to the service. If you are new to the Google APIs, you can follow the steps here to begin accessing People API.
Once you’re authorized, you can simply create new contacts like this (using the Google APIs Client Library for Java):
Person person = new Person(); person.setNames(ImmutableList.of(new Name().setGivenName("John").setFamilyName("Doe"))); ContactToCreate contactToCreate = new ContactToCreate(); contactToCreate.setContactPerson(person); BatchCreateContactsRequest request = new BatchCreateContactsRequest(); request.setContacts(ImmutableList.of(contactToCreate)).setReadMask("names"); BatchCreateContactsResponse response = peopleService.people().batchCreateContacts(request).execute();
The scope your app needs to authorize with is https://www.googleapis.com/auth/contacts. Full documentation on the people.batchCreateContacts method is available here.
people.batchCreateContacts
Similarly, you can update existing contacts like this:
String resourceName = "people/c12345"; // existing contact resource name Person contactToUpdate = peopleService .people() .get(resourceName) .setPersonFields("names,emailAddresses") .execute(); contactToUpdate.setNames( ImmutableList.of(new Name().setGivenName("John").setFamilyName("Doe"))); BatchUpdateContactsRequest request = new BatchUpdateContactsRequest(); ImmutableMap<String, Person> map = ImmutableMap.of(contactToUpdate.getResourceName(), contactToUpdate); request.setContacts(map).setUpdateMask("names") .setReadMask("names,emailAddresses"); BatchUpdateContactsResponse response = peopleService.people().batchUpdateContacts(request).execute();
Full documentation on the people.batchUpdateContacts method is available here.
Working with Search Endpoints
You can search through the authenticated user’s contacts like this:
SearchResponse response = peopleService.people().searchContacts() .setQuery("query") .setReadMask("names,emailAddresses") .execute();
The scope your app needs to authorize with is https://www.googleapis.com/auth/contacts or https://www.googleapis.com/auth/contacts.readonly. Full documentation on the people.searchContacts method is available here.
You can also search through the authenticated user’s “other contacts” like this:
SearchResponse response = peopleService.otherContacts().search() .setQuery("query") .setReadMask("names,emailAddresses") .execute();
The scope your app needs to authorize with is https://www.googleapis.com/auth/contacts.other.readonly. Full documentation on the otherContacts.search method is available here.
Next Steps
We hope that these newly added features inspire you to create the next generation of cool web and mobile apps that delight your users and those in their circles of influence. To learn more about the People API, check out the official documentation here.
Posted by Jose Ugia and Steve Klebe
Steve Klebe forms partnerships that drive adoption of Google Pay. He's spent the last 9 years working for the Google Payments Business Development team, and possesses more than 40 years of experience with products and services related to payment processing, data security, and authentication.
Recently, Steve sat down for an interview with Jose Ugia, a Developer Relations Engineer on the Google Pay team.
Read the interview transcript for a deep overview of online payments.
Jose Ugia: Let’s get started with the basics. What is the typical sequence of events in processing an online credit-card payment?
Steve Klebe: This can happen in a few different ways, but let’s talk about the typical series of events:
The approval triggers a response chain. The network responds to the payment processor, the payment processor responds to the merchant, and the merchant responds to the consumer with something like, “Your payment has been accepted!”
This sequence of events happens in approximately 2 seconds, during which the transaction passes through multiple different systems in order to deliver a response to the consumer.
Jose Ugia: Most developers and businesses don’t think about these steps. When you think about chargebacks and fraud, this information is especially useful.
The next question is related to a concept that goes by many names in the industry. It's what we call a PSP or payment service provider, but others refer to it as a payment processor, payment provider, or payment gateway. What is this concept and why are there so many different terms for it?
Steve Klebe: Things evolve and sometimes different entities in the ecosystem create their own terms to differentiate themselves. It’s a big challenge in the payments industry; there are many terms for the same concepts.
The term PSP has an official meaning in the ecosystem, and it can represent companies that take on different roles in the payment sequence, which I outlined in the first question. However, we kept things simple for our merchant and developer partners. PSP defines the initial link between the merchant and the network, regardless of their roles. The role of the PSP is to make sure the merchant is legitimate and categorize the merchant as a retail store, restaurant, or something else.
The PSP is the entity through which the money flows, from the card issuer through the networks to the PSP. They provide consolidated reporting to the merchant and—most people don’t realize this—they also often hold the financial responsibility. If the merchant is fraudulent or goes out of business and there are lingering transactions, the PSP assumes financial responsibility for the merchants.
Jose: So, if I’m planning to accept payments online, do I need a PSP?
Steve Klebe: Yes, you absolutely need to have a PSP, but it doesn't matter to you as a merchant if the PSP is an official processor or a licensed agent of a processor.
Jose: Are there specific considerations that I have to account for as a merchant or developer when I choose a PSP to process credit-card payments?
Steve Klebe: Sometimes it’s tied to the shopping cart of your e-commerce platform, most of which embed one or more PSPs into their systems. Sometimes, the decision has been made for you. Other times, you have flexibility to choose whatever you want. Different PSPs have different expertise in different types of payments. For example, if you’re a merchant who focuses on a subscription model, there are certain PSPs who handle these types of payments better than others. If you’re going to sell globally, you need to pick a PSP with the maximum ability to support alternative payment methods from other countries. If you’re a restaurant and you need to do in-store and online payment processing, not all PSPs are equal in their ability to support different types of channels.
So, do some research, talk to peers in your industry to find out who they use and whether they’re satisfied, and make an intelligent choice. It can have fairly significant consequences if you need to do online ordering, but you picked a PSP who is competent at in-store purchases and doesn’t take e-commerce seriously.
Jose: Are you suggesting that I might need to integrate multiple PSPs to cover different scenarios?
Steve: Yes. Using multiple PSPs is not unusual. If you need to cover different scenarios, such as subscription payments, in-person payments, or online payments then this can be very common. If you need to change your PSP, it can affect you later. Your PSP choice becomes intertwined with your back-office operations and fulfillment. It’s not just an API; it becomes integrated into all aspects of the business supply chain, including customer servicing, revenue recognition, etc. and switching isn't easy.
Jose: I’ve seen some PSPs offering something called “hosted checkout”. How does that differ from a regular integration in my website or application?
Steve Klebe: There are typically two approaches: you integrate your PSP's API and you as the merchant typically control the checkout process directly with the consumer. In the case of Google Pay, you can add the Google Pay button to your checkout pages. That's typically used by medium-to-large merchants, while smaller merchants tend to gravitate towards this concept called a hosted order page, which has some limitations because the checkout occurs on a page that the PSP hosts and different PSPs have different hosted-order-page capabilities.
If you’re an API merchant, for your non-Google Pay transactions you have a responsibility to protect the card information of your customers. With a hosted order page, all the sensitive information is being hosted on a page from the PSP. The penalties for having card information stolen from your servers are very severe, so hosted order pages are popular, flexible, and customizable.
In Europe, hosted checkouts are popular because commerce is complicated with more than 20 countries, different currencies, and payment methods. A US merchant could survive with a much simpler array of payment options if the merchant plans to only sell within US borders.
We work with most major PSPs globally and have them implement Google Pay as a default option for hosted checkouts. Usually, this is enabled by default but the PSP gives the merchant a choice to opt out.
Jose: What are e-wallets, digital wallets, and other payment facilitators, and how do they differ from a PSP.
Steve Klebe: There are a lot of acronyms, and they can start blending together and sounding the same to someone new to the space. The metaphor for a digital wallet was originally developed to represent that whatever is in your physical wallet would ultimately be in your digital wallet. While PSPs facilitate online transactions, digital wallets are a form of payment. There are many benefits to offering a digital wallet like Google Pay. One of the most obvious being the ability for customers to checkout quickly, without needing to re-enter credit card and billing information for every single transaction .
In the case of Google Pay, you can store loyalty cards, boarding passes, payment cards, and receipts in your digital wallet and use it to transact in physical stores, online websites and applications alike. The metaphor has played out, but there are a lot of differences within the broad category of alternative payment methods and digital wallets.
Those differences are evolving. Today, we have Google Pay, Apple Pay, PayPal, Samsung Pay, WeChatPay, Alipay and others. In some cases, the app or the account is only a container for credentials. In other cases, it's the account of record for your money. For example, in Asia, you see the popularity of Alipay and WeChat Pay, which are actually like bank accounts. In India, the Google Pay for India app connects directly to the consumer’s bank account, and initiates the movement of money to the merchant’s bank account.
Jose: What is a tokenized card and how does it affect online transactions?
Steve Klebe: The word tokenization is a loaded word in our industry and it creates a bunch of confusion. Tokenization and encryption (which are sometimes confused) came about because of the growing popularity of cards, and the growing use and misuse of cards by people with good and bad intentions.
The concept of exchanging a card number with a token is applied by various parties at different stages of an online transaction:
Tokenization, at the network level, came about after the industry established a standard for protecting card data that’s now referred to as PCI, which is an industry consortium funded by the major card brands that established a single standard for security.
Similarly, to assist merchants with complying with PCI, most PSPs came up with a proprietary scheme to take the card number from the merchant and give the merchant a token or reference number. The PSP, within its secure environment, would hold the card and the merchant wouldn’t need to handle it anymore. This became a dominant approach after PCI took effect.
In addition, there are two types of tokens that are used at the network level:
Device-based tokens or DPAN
When you want to use an existing card on your phone as a payment method, the call gets made to the associated network, which then calls the bank that issued the card. A call then comes back to authenticate the consumer and the most common step is the consumer is asked to enter a one time passcode they received through text. After the bank confirms your identity, it sends a signal to the network and approves your card for digital payments. The network then takes the account number, converts it to a token, and returns it to your wallet provider who securely stores it on the phone.
E-commerce tokens
This is a brand new concept where a product like Google Pay, which helps to securely store millions of cards in its cloud, delivers them to the network for conversion to a token. The network validates the status of the card with the issuing bank, turns them into e-commerce tokens, and returns the tokens to Google. Now, when you shop on any device, Google can use one of these e-commerce tokens because the network and issuer authenticated them. Even if the underlying card changes completely or the expiration date gets updated, this all happens behind the scenes. This is not only convenient for customers, but it also helps protect their card and transaction information by keeping the actual credit card number unexposed and including a dynamic element that is different for every transaction.
Jose: What is the future of payments going to bring? What are you most excited about?
Steve Klebe: I would say, due to the changes our world is going through, we are rethinking how payments are changing. It’s hard to know what the ultimate impact will be, but it's been about mobile optimization during the last couple years. Every merchant and PSP realizes that they have to enhance their digital offerings, but it’s not going to be any one individual thing. I think it’s the entire holistic experience, whether it’s web, mobile, or in-store. All of a sudden, every merchant realizes that they need to be prepared to do payments contactlessly. Even if the consumer is standing in front of you, you have to be prepared to handle the payment without contact.
There is a clear divide between card present and card not present, and those areas are now blending together. The card industry doesn’t care whether the person is in front of you. If a payment is made digitally, there are alternative rules that apply to the merchant. Merchants need to be extremely cognizant of these rules and they need to do everything they can to optimize how they accept payments.
An exception would be where you can start shopping with a merchant on your desktop and complete transactions elsewhere while your goods remain in your shopping cart. Their systems have to be capable of multiplatform payments and that requires a fresh look at who your PSPs are because not all PSPs provide such capabilities.
Device-bound tokens are very 1990ish. The whole world is moving to the cloud. A device bound token needs to be reprovisioned every time I get a new phone, which is typically every 1-2 years, and that has to change. We live in a cloud-based world and people expect to authenticate themselves and start doing business, and payments have to work this way, too.
Jose: Thank you for the chat, Steve. It sounds like payments are changing a lot, adapting to the evolution of technology and we’re excited to see where these changes take us.
--
Interested in learning more about Google Pay APIs or have questions? Follow us @GooglePayDevs and let us know in the comments or tweet using #AskGooglePayDev! For any other Google Pay-related requests and questions, or to start your Google Pay integration, visit Google Pay Business Console.
Posted by Toni Klopfenstein, Developer Advocate
As more developers onboard to the Smart Home Actions platform, we have gathered feedback about the certification process for launching an Action. Today, we are pleased to announce we have updated our Actions policy to enable developers to more quickly develop their Actions, and to help streamline the certification and launch process for developers. These updates will also help to provide a consistent, cohesive experience for smart device users.
Ensuring each device type meets quality benchmark metrics provides end users with reliable and timely responses from their smart devices.With these policy updates, minimum latency and reliability metrics have been added to each device type guide. To ensure consistent device control and timely updates to Home Graph, all cloud controlled smart devices need to maintain a persistent connection through a hub or the device itself, and cannot rely on mobile devices and tablets.
Along with these quality benchmarks, we have also updated our guides with required and recommended traits for each device. By implementing these within an Action, developers can ensure their end users can trigger devices in a consistent manner and access the full range of device capabilities. To assist you in ensuring your Action is compliant with the updated policy, the Test Suite testing tool will now more clearly flag any device type or trait issues.
Smart home users care deeply about the safety and security of the devices integrated into their homes, so we have also updated our requirements for secondary user verification. This verification step must be implemented for any Action that can set a device in an unprotected state, such as unlocking a door, regardless of whether you are building a Conversational Action or Smart Home Action. Once configured with a secondary verification method, developers can provide users a way to opt out of this flow. For any developer wishing to include an opt-out selection to their customers, we have provided a warning message template to ensure users understand the security implications for turning this feature off.
For devices that may pose heightened safety risks, such as cooking appliances, we require UL certificates or similar certification forms to be provided along with the Test Suite results before an Action can be released to production.
These policy updates also will affect the use of the Works With Hey Google badge. The badge will only be available for use on marketing materials for new Smart Home Direct Actions that have successfully integrated any device types referenced.
Any Conversational Actions currently using the badge will not be approved for use for any new marketing assets, including packaging/product refreshes. Any digital assets using the badge will need to be updated to remove the badge by the end of 2021.
With the roll-out today, there will be a 1 month grace period for developers to update new integrations to match the new policy requirements. For Actions currently deployed to production, compliance will be evaluated when the Action is recertified. Once integrations have been certified and launched to production, Actions will need to be recertified annually, or any time new devices or device functionality is added to the Action. Notifications for recertification will be shared with the developer account associated with your Action in the console.
This policy grace-period ends April 12, 2021.
Please review the updated policy, as well as our updated docs for launching your Smart Home Action. You can also check out our policy video for more information.
We want to hear from you, so continue sharing your feedback with us through the issue tracker, and engage with other smart home developers in the /r/GoogleAssistantDev community. Follow @ActionsOnGoogle on Twitter for more of our team's updates, and tweet using #AoGDevs to share what you’re working on. We can’t wait to see what you build!
Posted by Leslie Garcia-Amaya, Global Product Partnerships Lead, Google Assistant Ashwin Karuhatty, Head of Global Product Partnerships, Google Assistant
The interest and adoption of voice technology reached an important inflection point last year with the pandemic, as we immediately saw Google Assistant play a bigger role in helping people manage more of their time at home, from juggling family activities to controlling their smart home devices.
To help brands and developers stay ahead of these trends and identify potential opportunities to create impactful voice experiences for their users, we spun up a series of virtual events to stay engaged with the community when many in-person industry events were cancelled. For example, we introduced VOICE Talks last April in partnership with Modev as a monthly series of digital events that connected Google business, engineering and product leaders directly with the voice-tech ecosystem and developer community. VOICE Talks also provided a platform to companies, like Sony, Bamboo Learning, American Express, Verizon, Headspace, Vizio, iRobot, Nike, Dunkin, to share best practices on how they integrated voice technology into their products. You can watch past episodes here.
The ecosystem support and participation has been incredible with over 110,000 subscribers for VOICE Talks, over 40,000 hours of content consumed and active ongoing viewership on YouTube. In addition, we saw a huge demand for country/region-specific content in India, and started the VOICE Talks India series, which has also been received very well.
Thanks to all the positive feedback from the community, we’re looking to double down on those efforts this year. In addition to hosting more VOICE Talks events, we’re expanding our collaboration with industry-recognized influencers through podcasts, livestreams and more to continue growing the community, such as:
Additionally, we’re excited to announce that Google Assistant is the first corporate sponsor of Women In Voice, a global non-profit with a mission to amplify women and diverse people in the voice technology field that has grown to 20 chapters in 15 countries since they launched in 2018. This sponsorship builds on the momentum Women In Voice established with Google Assistant at CES 2020, where they collaborated on a “Women In Tech & Allies” event. Tune in to womeninvoice.org to stay up to date on upcoming events and collaborations between Google Assistant and Women In Voice.
There’s now more ways to hear from us, share your feedback and learn about the latest trends in the space.
