Posted by Charles Maxson, Developer Advocate
Developing your own custom Google Chat bot is a great way for users and teams to interact with your solutions and services both directly and within context as they collaborate in Chat. More specifically, Chat bots can be used in group conversations to streamline workflows, assist with activities in the context of discussions, and provide information and notifications in real time. Chat bots can also be used in direct messages, offering a new way to optimize workflows and personal productivity, such as managing project tasks or reporting time activity. Because use cases for bots are varied, you can consistently reach a growing audience of Chat users over time, directly where they work and uh-hum, chat.
Once you’ve identified your specific use case for your custom Chat bot, how you design the bot itself is super important. Bots that are intuitive and easy to use see better adoption and develop a more loyal following. Those that are not as fluid or approachable, or come across as confusing and complicated to use, will likely miss the mark of becoming an essential “sticky” tool even if your back end is compelling. To help you build an engaging, must-have Google Chat bot, we recently added a one-two feature punch to the Chat bot framework that allows you to build a much richer bot experience than ever before!
The first new(er) feature that you can leverage to enhance the usability of your Chat bots are slash commands. Released a few months back, slash commands simplify the way users interact with your Chat bot, offering them a visual leading way to discover and execute your bot’s primary features. Unlike bots created prior to slash commands, where users had to learn what features a bot offered and then invoke the bot and type the command correctly to execute them, slash commands make Chat bot usage faster and help users get the most out of your bot.
Users can now simply type “/” in the message line to reveal a list of all the functions offered by the bots available to the room or direct message, and select the one to their liking to execute it. Slash commands can be invoked standalone (e.g. /help) or include user added text as parameters (e.g. /new_task review project doc ) that the developer can handle when invoked. To help make bot command discovery even simpler, the slash commands list filters matching commands once the user starts typing beyond the / (e.g. “/h” shows all commands beginning with H). This is super helpful as more and more bots are added to a room, and as more bots with slash commands are introduced by developers. Also included directly in the Slash Command UI is a description of what each command does (up to 50 characters), easing the guesswork out of learning.
As a developer, slash commands are straightforward to implement, and daresay essential in offering a better bot experience. In fact, if you have an existing Google Chat bot you’ve built and deployed, it’s likely more than worthwhile to revise your bot to include slash commands in an updated release.
To add slash commands to any Chat bot, you will need to register your commands in the Hangouts Chat API configuration page. (e.g. https://console.cloud.google.com/apis/api/chat.googleapis.com/hangouts-chat?project=<?yourprojectname?>) There is a section for slash commands that allows you to provide the /name and the description the user will see, along with the important Command Id unique identifier (a number between 1-1000) that you will later need to handle these events in your code.
When a user invokes your bot via a Slash Command, there is a slashCommand field attached to the message sent to the bot that indicates the call was initiated from a Slash Command. Remember users can still @mention your bot to call it directly by name without a / command and this helps you distinguish the difference. The message also includes the corresponding commandId for the invoked command based on what you set up in the bot configuration page, allowing you to identify the user’s requested command to execute. Finally, the message also offers additional annotations about the event and includes any argumentText supplied by the user already parsed from the command text itself.
slashCommand
commandId
argumentText
{ ... "message": { "slashCommand": { "commandId": 4 }, "annotations": [ { "length": 6, "slashCommand": { "type": "INVOKE", "commandId": 4, "bot": { "type": "BOT", "displayName": "Slashbot" }, "commandName": "/debug" }, "type": "SLASH_COMMAND" } ], ... "argumentText": " show code", "text": "/debug show code", ... }
Here is a simple example used to determine if a Slash Command was invoked by the user, and if so, runs the requested command identified by its Command Id.
function onMessage(event) { if (event.message.slashCommand) { switch (event.message.slashCommand.commandId) { case 1: // Command Id 1 return { 'text': 'You called commandId 1' } case 2: // Command Id 2 return { 'text': 'You called commandId 2' } case 3: // Help return { 'text': 'You asked for help' } } } }
The second part of the one-two punch of new Google Chat bots features are dialogs. This is a brand new capability being introduced to the Chat bot framework that allows developers to build user interfaces to capture inputs and parameters in a structured, reliable way. This is a tremendous step forward for bot usability because it will simplify and streamline the process of users interacting with bot commands. Now with dialogs, users can be led visually to supply inputs via prompts, versus having to rely on wrapping bot commands with natural language inputs -- and hoping they correctly executed syntax the bot could decipher.
For developers, you can design UIs that are targeted to work precisely with the inputs you need users to supply your commands, without having to parse out arguments and logically infer the intent of users. In the end, dialogs will greatly expand the type of solution patterns and use cases that Chat bots can handle, as well as making the experience truly richer and more rewarding for users and developers alike.
Technically, Chat bot dialogs leverage the aforementioned slash commands combined with the existing Google Workspace Add-on Card framework to support the creation and handling of dialogs. To get started, you create a Slash Command that will invoke your dialog by designating it’s Slash command triggers a dialog setting to true in the Slash Command configuration process, as seen below:
Once you have configured a Slash Command to trigger a dialog, it will send an onMessage event when it’s invoked as it would before, but now it includes new details that indicate it is representing a dialog request. To handle this event you can use the example above with non-dialog Slash Command, using the commandId you can use a switch to determine what the user requested.
switch
Designing the actual elements that the dialog renders is where you draw from the Google Workspace Add-on Card-based framework. If you’ve built a new generation of Google Workspace Add-on, this part will be familiar where you construct widgets, add headers and sections, create events, etc. In fact, you can even reuse or share some of your Add-on UIs within your Chat bots, but do note there currently is a lighter subset of elements available for bots. The benefits of using Cards allows you to build modern, consistently-styled user interfaces for your bots that doesn’t require that you get bogged down in low level details like managing tags or CSS. You can learn more about working with Cards starting here. To make building your Cards-based interfaces for Add-ons and Chat bots even easier, we have also just introduced the GWAO Card Builder tool, which employs a drag-n-drop visual designer to boost your development efforts.
Once you’ve assembled your Card’s widgets, to make it render as a dialog when invoked you must specify that its a DIALOG type within the action_response as seen stubbed out here below:
{ "action_response": { "type": "DIALOG", "dialog_action": { "dialog": { "body": { "sections": [ { "widgets": [ { "textInput": { "label": "Email", "type": "SINGLE_LINE", "name": "fieldEmail", "hintText": "Add others using a comma separator", ...
Now with a working dialog, all there is left to do is handle user events once it's displayed. Again this is similar to how you would handle events working with Cards within Add-ons. Your bot will receive an event that is type CARD_CLICKED with a DialogEventType set to SUBMIT_DIALOG. The actionMethodName value will let you know what element the user clicked to process the request, e.g. ‘assign’ as depicted below. The response includes the formInputs details which are the user provided inputs returned from the dialog, which you can process as your solution needs to.
actionMethodName
formInputs
{ dialogEventType: 'SUBMIT_DIALOG', type: 'CARD_CLICKED', action: { actionMethodName: 'assign' }, ... common: { hostApp: 'CHAT', formInputs: { 'whotochoose-dropdown': [Object], whotochoose: [Object], email: [Object] }, invokedFunction: 'assign' }, isDialogEvent: true }
Once your bot is finished processing its task, it can respond back to the user in one of two ways. The first is with a simple acknowledgement (aka OK) response letting them know their action was handled correctly and close out the dialog.
{ "action_response": { "type": "DIALOG", "dialog_action": { "action_status": "OK", ...
The other option is to respond with another dialog, allowing you to follow-up with a new or revised dialog useful for complex or conditional input scenarios. This is accomplished as it was originally when you called a dialog using a dialog card within an ActionResponse to get started.
{ "action_response": { "type": "DIALOG", "dialog_action": { "dialog": { ...
To get started building Google Workspace Chat bots, or to add slash commands and dialogs to your existing Chat bots, please explore the following resources:
Posted by Miguel Guevara, Product Manager, Privacy and Data Protection Office.
As developers, it’s our responsibility to help keep our users safe online and protect their data. This starts with building products that are secure by default, private by design, and put users in control. Everything we make at Google is underpinned by these principles, and we’re proud to be an industry leader in developing, deploying, and scaling new privacy-preserving technologies that make it possible to learn valuable insights and create helpful experiences while protecting our users’ privacy.
That’s why today, we are excited to announce that we’re open-sourcing a first-of-its-kind, general-purpose transpiler for Fully Homomorphic Encryption (FHE), which will enable developers to compute on encrypted data without being able to access any personally identifiable information.
A deeper look at the technology
With FHE, encrypted data can travel across the Internet to a server, where it can be processed without being decrypted. Google’s transpiler will enable developers to write code for any type of basic computation such as simple string processing or math, and run it on encrypted data. The transpiler will transform that code into a version that can run on encrypted data. This then allows developers to create new programming applications that don’t need unencrypted data. FHE can also be used to train machine learning models on sensitive data in a private manner.
For example, imagine you’re building an application for people with diabetes. This app might collect sensitive information from its users, and you need a way to keep this data private and protected while also sharing it with medical experts to learn valuable insights that could lead to important medical advancements. With Google’s transpiler for FHE, you can encrypt the data you collect and share it with medical experts who, in turn, can analyze the data without decrypting it - providing helpful information to the medical community, all while ensuring that no one can access the data’s underlying information.
In the next 10 years, FHE could even help researchers find associations between specific gene mutations by analyzing genetic information across thousands of encrypted samples and testing different hypotheses to identify the genes most strongly associated with the diseases they’re studying.
Making more products private by design
Our principle to make our products private by design drives us to build ground-breaking computing technologies that enable personalized experiences while protecting your private information. Privacy-preserving technologies are on the cutting-edge of Google’s innovations, and they have already shown great potential to help shape a more private internet.
In 2016, Google researchers invented Federated Learning, a technique that helps preserve privacy by keeping as much personal information on your device as possible. And in 2019, Google made its differential privacy library freely available to any organization or developer, an advanced anonymization technology that enables developers to learn from their data privately. No one has scaled the use of Differential Privacy more than we have.
We’ve been thrilled to see these technologies put to use across the globe; in France, for example, a startup called Arkhn has been able to accelerate scientific discovery using differential privacy to share data across hospitals.
We still have a ways to go before most computations happen with FHE -- but much as it took some time for HTTPS to take off and be widely adopted, today’s announcement is an important step towards bringing users helpful products that preserve their privacy and keep their data safe.
At Google, we know that open-sourcing our technologies with the developer community for feedback and use helps make them better. We will continue to invest and lead the privacy-preserving technology field by publishing new work, and open-sourcing it for everyone to use at scale - and we're excited to continue this practice by sharing this latest advancement with developers everywhere. We can't wait to see what you’ll build, and we look forward to collaborating on the journey towards a safer Internet.
#IamaGDE series presents: Google Maps
The Google Developers Experts program is a global network of highly experienced technology experts, influencers, and thought leaders who actively support developers, companies, and tech communities by speaking at events and publishing content.
Meet Josue Gutierrez — Maps, Web, Identity and Angular Google Developer Expert.
Josue currently works at the German company Boehringer Ingelheim and lives near Frankfurt. Before moving to Germany, Josue was working as a software engineer in Mexico, and before that, he spent almost a year in San Francisco as a senior front-end developer at Sutter Health.
Josue Gutierrez
Josue studied computer science and engineering as an undergraduate and learned algorithms and programming. His first language was C++, and he learned C and Python, but was drawn to web technologies.
“When I saw a web browser for the first time, it stuck with me,” he says, “It was changing in real time as you’re developing. That feeling is really cool. That’s why I went into frontend development.”
Josue has worked on multiple ecommerce projects focused on improving customers’ trade experience. He sees his role as creating something from scratch to help people improve lives.
“These opportunities we have as developers are great — to travel, work for many verticals, and learn many businesses,” he says. “In my previous job, I developed tech-oriented trade tools for research companies, to manipulate strings or formulas. I was on the team involved in writing these kinds of tools, so it was more about the trade experience for doctors.”
Getting involved in the developer community
Josue’s first trip outside Mexico, to San Francisco, exposed him to the many developer communities in the area, and he appreciated the supportive communities of people trying to learn together. Several of the people he met suggested he start his own meetup in Mexico City, to get more involved in Google technologies, so he launched an Angular community there. As he hunted for speakers to come to his Angular meetup, Josue found himself giving talks, too.
Then, the GDG Mexico leader invited Josue to give talks on Google for startups.
“That helped me get involved in the ecosystem,” Josue says. “I met a lot of people, and now many of them are good friends. It’s really exciting because you get connected with people with the same interests as you, and you all learn together.”
“I’m really happy to be part of the Google Maps ecosystem,” Josue says. “It’s super connected, with kind people, and now I know more colleagues in my area, who work for different companies and have different challenges. Seeing how they solve them is a good part of being connected to the product. I try to share my knowledge with other people and exchange points of view.”
Josue says 2020 provided interesting opportunities.
“This year was weird, but we also discovered more tools that are evolving with us, more functionalities in Hangouts and Meetup,” Josue says. “It’s interesting how people are curious to get connected. If I speak from Germany, I get comments from countries like Bolivia and Argentina. We are disconnected but increasing the number of people we engage with.”
He notes that the one missing piece is the face-to-face, spontaneous interactions of in-person workshops, but that there are still positives to video workshops.
“I think as communities, we are always trying to get information to our members, and having videos is also cool for posterity,” he says.
He is starting a Maps developer community in Germany.
“I have colleagues interested in trying to get a community here with a solid foundation,” he says. “We hope we can engage people to get connected in the same place, if all goes well.”
Favorite Maps features and current projects
As a frontend developer, Josue regards Google Maps Platform as an indispensable tool for brands, ecommerce companies, and even trucking companies.
“Once you start learning how to plant coordinates inside a map, how to convert information and utilize it inside a map, it’s easy to implement,” he says.
In 2021, Josue is working on some experiments with Maps, trying to make more real-time actualization, using currently available tools.
“Many of the projects I’ve been working on aren’t connected with ecommerce,” he says. “Many customers want to see products inside a map, like trucking products. I’ve been working in directories, where you can see the places related to categories — like food in Mexico. You can use Google Maps functionalities and extend the diversification of maps and map whatever you want.”
“Submission ID is really cool,” he adds. “You can do it reading the documentation, a key part of the product, with examples, references, and a live demo in the browser.”
Future plans
Josue says his goal going forward is to be as successful as he can at his current role.
“Also, sharing is super important,” he says. “My company encourages developer communities. It’s important to work in a place that matches your interests.”
Follow Josue on Twitter at @eusoj |Check out Josue’s projects on GitHub.
For more information on Google Maps Platform, visit our website or learn more about our GDE program.
Posted by Max Saltonstall
A little over 10 years ago, we launched the IT Residency Program (ITRP) at Google with a twofold mission: to provide exceptional tech support for Googlers and to empower the next generation of IT pioneers.
ITRP’s founding principle is learning and development. In addition to formal on-the-job IT support training, the program takes its residents through a focused career development path, including a hands-on rotation in a specific Google function in their chosen specialty. For their part, the residents, who come from a wide spectrum of often non-traditional backgrounds, bring with them a passion for learning. ITRP converts that passion into real-world experience and equips them for a lifelong career in tech.
Today, hundreds of Googlers are ITRP alums, working in disciplines ranging from site reliability engineering to security and privacy to program management and all points in between. Looking back on the program’s 10-plus years, we wanted to share some of their stories, their experiences and their triumphs. Look for more installments in this series in the weeks to come.
While studying anthropology in college, Kate Grant had a part time help desk job, helping solve people's computer problems (and sometimes the computer's people problems). And it was through that job that she realized she really liked the satisfaction she got from helping people so directly. At school, Kate could help students, faculty and staff — both technical folks and non-technical, and it was great to save the day for them, teach them, and constantly learn about new technologies.
Joining the Pride March in NYC as part of Google's presence
As graduation loomed close, Kate needed something to do afterwards, and she stumbled upon the ITRP posting. It looked like a great way to continue doing what she loved, and in a much bigger company, with more to learn. She applied, thinking "there's no way in hell" she would get this job.
Turns out she was wrong!
Starting at Google in August 2012, Kate worked in Mountain View, CA helping Googlers of all types with a broad range of tech challenges. She got to spend some time with the Search team at Google working on technical documentation, helping engineers at Google as well as web designers outside Google better understand Google search features. This experience gave Kate something cool she could show her family too, a very tangible "I made this" moment, which can be hard to come by in IT operations work.
During her time in the program, Kate also had the chance to spend some time working in the New York office, which felt good because it brought her closer to New Jersey, where she grew up. It was the first step towards coming back to the Northeast, a welcome return to a comfortable place. Eventually she'd move to New York to work in Google's NYC office full time, helping with IT operations and later managing junior IT help desk folks as the team expanded.
Neon welcome sign in the lobby of the Google NYC office
Working across teams and projects and help desks in ITRP helped Kate develop all kinds of skills, from improving proficiency with Linux, Windows, MacOS and Chrome to also developing better judgment and analysis skills when solving novel problems that walked into the support desk. And as she began helping to train new employees in orientation, covering IT, Security and Technology topics, she got practice improving her public speaking; by the end of it she was speaking to over 100 people at a time on Mondays in Mountain View as new employees learned the essentials of their work at Google.
While she had been considering a tech writing career path before, Kate realized that the work in ITRP helping people day-to-day let her wear many hats. She was writing documentation, that was one key component. But she also wrote code, managed programs and projects, mentored junior members of the team, and analyzed data. The breadth of the job matched her growth goals, and Kate ended up continuing her career in Techstop, transitioning into a role as a Corporate Operations Engineer in 2014.
The work remained in the front-line support team and gave Kate tremendous exposure to the wide array of people in the Google headquarters. She focused more on projects to improve the onboarding and daily operations of the support teams, helping to keep systems and services healthy. This involved scheduling, mentoring, training and helping the newer members of the team, mostly later cohorts of IT Residents.
Kate's experience training new employees, and mentoring new IT Residents, made her a great fit for a new opportunity on the onboarding team, where she worked to make the new employee experience more consistent and reliable between offices across Google globally. This began a traversal of different parts of Google's IT org, where she learned about how the company, and its many teams, operate. Kate's work focused on helping to create better, smoother, more automated processes for the teams in IT, and help to scale the successes they had already achieved.
Driving improvements to Google’s onboarding infrastructure was fun and satisfying, but Kate missed working with junior techs day to day, and mentoring them on their own career growth. Luckily in 2016 a new manager position opened up in NYC, and Kate jumped at it. She started leading a team of newer support technicians, including IT Residents. Now she could focus on making a really great support experience for Googlers, and give back to the program she had enjoyed by helping the folks on her team grow.
In 2019 Kate had the opportunity to move to Austin, TX to build out a new ITRP Hub location from scratch, and continues to manage IT Residences and Corporate Operations Engineers there today.
Coming full circle now Kate shared some great advice with us for those thinking about ITRP:
"Don't screen yourself out of the opportunity. Folks come into the program from all walks of life. You don't have to have a super fancy computer science degree. If you’re excited about technology and helping people, ITRP could be a great next step for you."
She reinforced that you get out of the program what you put into it: “If you take the time to be curious, to ask questions, to investigate, you will learn so much. It's really an endless amount of material you could absorb, and nobody can get through it all. But when you put in the work, it pays you back”.
And as people go through the ITR Program, they end up in all sorts of places. "There's no cookie cutter outcome… It doesn’t mean you’re doing something wrong if your path doesn’t look like someone else’s. The right way is what's right for you and your goals."
Posted by Toni Klopfenstein, Developer Advocate
Alerts for important device events, such as a delivery person arriving or the back door failing to lock, create a more beneficial and reassuring experience for your smart home device users.
As we announced at I/O, you can now add proactive notifications and follow-up responses to your Smart Home Action to alert users to events in a timely, relevant and helpful fashion and better engage with your end users.
Notifications can either alert a user to an event that has occurred without them proactively issuing a request through the Assistant, or as a follow-up to verify that the user's request has been fulfilled. Each device event that triggers one of these notifications has a unique event id, which helps the Assistant route it to the appropriate Home Graph users and Google Home Smart Speakers or Nest Smart Displays, depending on the notification type and priority. Notifications and follow-up responses can also provide users with additional information, such as error and exception codes, or timestamps for the event.
You can enable notifications on your existing devices once users opt-in to receive alerts by updating the device definition and requesting a SYNC intent. You can then send device notifications along with any applicable device state changes using the Home Graph API.
We are adding support for traits where asynchronous requirements are a core use case.The following device traits now support follow-up responses to user queries:
Additionally, we are launching proactive notification alerts for the following traits:
For more information, check out the developer guides and samples, or check out the Notifications video.
We want to hear from you, so continue sharing your feedback with us through the issue tracker, and engage with other smart home developers in the /r/GoogleAssistantDev community. Follow @ActionsOnGoogle on Twitter for more of our team's updates, and tweet using #AoGDevs to share what you’re working on. We can’t wait to see what you build!
Posted by HyeJung Lee and MJ You, Google ML Ecosystem Community Managers. Reviewed by Soonson Kwon, Developer Relations Program Manager.
Google Developers Experts is a community of passionate developers who love to share their knowledge with others. Many of them specialize in Machine Learning (ML). Despite many unexpected changes over the last months and reduced opportunities for various in person activities during the ongoing pandemic, their enthusiasm did not stop.
Here are some highlights of the ML GDE’s hard work during the Q1 2021 which contributed to the global ML ecosystem.
ML GDE YouTube channel
With the initiative and lead of US-based GDE Margaret Maynard-Reid, we launched the ML GDEs YouTube channel. It is a great way for GDEs to reach global audiences, collaborate as a community, create unique content and promote each other's work. It will contain all kinds of ML related topics: talks on technical topics, tutorials, interviews with another (ML) GDE, a Googler or anyone in the ML community etc. Many videos have already been uploaded, including: ML GDE’s intro from all over the world, tips for TensorFlow & GCP Certification and how to use Google Cloud Platform etc. Subscribe to the channel now!!
TensorFlow Everywhere
17 ML GDEs presented at TensorFlow Everywhere (a global community-led event series for TensorFlow and Machine Learning enthusiasts and developers around the world) hosted by local TensorFlow user groups. You can watch the recorded sessions in the TensorFlow Everywhere playlist on the ML GDE Youtube channel. Most of the sessions cover new features in Tensorflow.
International Women’s Day
Many ML GDEs participated in activities to celebrate International Women’s Day (March 8th). GDE Ruqiya Bin Safi (based in Saudi Arabia) cooperated with WTM Saudi Arabia to organize “Socialthon” - social development hackathons and gave a talk “Successful Experiences in Social Development", which reached 77K viervers live and hit 10K replays. India-based GDE Charmi Chokshi participated in GirlScript's International Women's Day event and gave a talk: “Women In Tech and How we can help the underrepresented in the challenging world”. If you’re looking for more inspiring materials, check out the “Women in AI” playlist on our ML GDE YouTube channel!
Mentoring
ML GDEs are also very active in mentoring community developers, students in the Google Developer Student Clubs and startups in the Google for Startups Accelerator program. Among many, GDE Arnaldo Gualberto (Brazil) conducted mentorship sessions for startups in the Google Fast Track program, discussing how to solve challanges using Machine Learning/Deep Learning with TensorFlow.
TensorFlow
Meanwhile in Europe, GDEs Alexia Audevart (based in France) and Luca Massaron (based in Italy) released “Machine Learning using TensorFlow Cookbook”. It provides simple and effective ideas to successfully use TensorFlow 2.x in computer vision, NLP and tabular data projects. Additionally, Luca published the second edition of the Machine Learning For Dummies book, first published in 2015. Her latest edition is enhanced with product updates and the principal is a larger share of pages devoted to discussion of Deep Learning and TensorFlow / Keras usage.
On top of her women-in-tech related activities, Ruqiya Bin Safi is also running a “Welcome to Deep Learning Course and Orientation” monthly workshop throughout 2021. The course aims to help participants gain foundational knowledge of deep learning algorithms and get practical experience in building neural networks in TensorFlow.
Nepal-based GDE Kshitiz Rimal gave a talk “TensorFlow Project Showcase: Cash Recognition for Visually Impaired" on his project which uses TensorFlow, Google Cloud AutoML and edge computing technologies to create a solution for the visually impaired community in Nepal.
On the other side of the world, in Canada, GDE Tanmay Bakshi presented a talk “Machine Learning-powered Pipelines to Augment Human Specialists” during TensorFlow Everywhere NA. It covered the world of NLP through Deep Learning, how it's historically been done, the Transformer revolution, and how using the TensorFlow & Keras to implement use cases ranging from small-scale name generation to large-scale Amazon review quality ranking.
Google Cloud Platform
We have been equally busy on the GCP side as well. In the US, GDE Srivatsan Srinivasan created a series of videos called “Artificial Intelligence on Google Cloud Platform”, with one of the episodes, "Google Cloud Products and Professional Machine Learning Engineer Certification Deep Dive", getting over 3,000 views.
Korean GDE Chansung Park contributed to TensorFlow User Group Korea with his “Machine Learning Pipeline (CI/CD for ML Products in GCP)” analysis, focused on about machine learning pipeline in Google Cloud Platform.
Last but not least, GDE Gad Benram based in Israel wrote an article on “Seven Tips for Forecasting Cloud Costs”, where he explains how to build and deploy ML models for time series forecasting with Google Cloud Run. It is linked with his solution of building a cloud-spend control system that helps users more-easily analyze their cloud costs.
If you want to know more about the Google Experts community and all their global open-source ML contributions, visit the GDE Directory and connect with GDEs on Twitter and LinkedIn. You can also meet them virtually on the ML GDE’s YouTube Channel!
Posted by Patricia Correa, Director, Global Developer Marketing
At Google Play we’re committed to helping all developers thrive, whether these are large multinational companies or small startups and indie game studios. They are all critical to providing the services and experiences that people around the world look for on their Android devices. The indie game developer community, in particular, constantly pushes the boundaries with their creativity and passion, and bring unique and diverse content to players everywhere.
To continue supporting indies, today we’re opening submissions for two of our annual developer programs - the Indie Games Accelerator and the Indie Games Festival. These programs are designed to help small games studios grow on Google Play, no matter what stage they are in:
This year the programs come with some changes, including more eligible markets and fully digital event experiences. Learn more below and apply by July 1st.
Accelerator: Get education and mentorship to supercharge your growth
If you’re an indie developer, early in your journey - either close to launching a new game or recently launched a title, this is the program for you. We’ll provide education and mentorship that will help you build, launch and grow successfully.
This year we have nearly doubled the eligible markets, with developers from over 70 countries being eligible to apply for the 2021 program.
Selected participants will be invited to take part in a 12-week online acceleration program. During this time you’ll get exclusive access to a community of Google and industry experts, as well as a network of other passionate developers from around the world looking to supercharge their growth.
Festival: win promotions that put your game in the spotlight
If you're an indie game developer who has recently launched a high quality game, this is your chance to have your game discovered by industry experts and players worldwide.
This year we will, again, host three competitions for developers from Japan, South Korea, and selected European countries.
Prizes include featuring on Google Play store, promotional campaigns worth 100,000 EUR, and more.
How useful did you find this blog post?
★ ★ ★ ★ ★
Posted by Posted by Erica Hanson, Senior Program Manager, Google Developer Student Clubs
Tamta Kapanadze wishes that she had learned sooner about careers in technology. By the time that the Georgian citizen learned about them, she was already a university student.
As Kapanadze continued her studies and her interest in technology grew, she wanted to spread the word about the growing field to high-school students in Georgia, a country where the industry is still small.
To do this, Kapanadze called in the support of Google Developer Student Clubs (GDSCs), community groups for college and university students interested in Google's developer technology. After Kapanadze graduated from university, she continued her work by organizing a chapter of Google Developer Groups (GDGs) for Kutaisi.
Google Developer Groups are the largest community network of professional developers in the world. The program consists of local chapters that provide inclusive environments open to everybody interested in tech. The chapters let members learn new skills, and meet other developers with similar interests through online and in-person events.
However, even after all that, Kapanadze still wanted to do more. She brought together friends and colleagues to partner with Mariam, GDSC Georgia American University Lead; Iliko, GDSC Georgia American University core team member; Giorgi, GDSC Tbilisi State University Lead; and Bakar, GDSC San Diego State University Lead. With this team assembled, they planned Tech Camp, a virtual technological learning experience that teaches high schoolers about tech fields and how to start careers in web development, game development, artificial intelligence, machine learning, and more.
Tamta Kapanadze.
While it's difficult enough to plan and execute a new event, Kapanadze and her partners didn't let the additional challenges of the last year stop their plans to launch Tech Camp. They wanted to publicize the event by mid-January, so they made a to-do list and set deadlines for themselves. After a few weeks of intense planning, they:
Kapanadze and her partners accepted applications for Tech Camp from Jan. 20 to Feb. 10 and announced their speakers to the public to keep the buzz about the event going. They originally hoped to receive 30 applications, but instead received 500 across their events. In particular, 300 students attended the speaker sessions and 50 students attended the coding sessions, where they would teach them about algorithms and the basics of C++.
Finally, the first day of Tech Camp arrived on Feb. 15. They began each session with fun icebreakers to help everybody feel comfortable, including themselves. Here's a timeline of what each day covered:
A screenshot of the virtual session about game development.
Everybody defines success differently, but for Kapanadze and her team it meant impacting at least one person. By this measure, Tech Camp succeeded because many of those who attended decided to pursue careers in tech. As for Kapanadze, she can’t wait to see what the future holds for Georgia's high schoolers and the country's growing tech industry.
To watch recordings from Tech Camp, please visit the playlist on YouTube.
The recordings available from Tech Camp.
For more information, find a Google Developers community group near you.
Posted by the Google Developers team
If you attended Google I/O this year, you probably stopped by the Google I/O photo booth for a selfie with our Google Developer mascots: Flutter’s Dash, Android Jetpack, Chrome’s Dino, and Firebase’s Sparky. If you didn’t, it’s not too late to jump in, take a selfie, and share it on social media! We loved seeing all of the pictures you posted and your favorite props! Want to learn more about building a camera plugin, layouts, and gestures used in a photo booth for Flutter on the web?
It took a combination of Google developer products to make the photo booth successful. The Flutter and Firebase teams joined forces to build a best in class example of Flutter on the web that used Firebase for hosting, auth, performance monitoring, and social sharing. Take a closer look at how the photo booth was built here and then grab the open source code on Github!
Flutter team members having fun in the photo booth!
Guest post by the Engineering teams at Mirru and Tweag
What is the Mirru App?
Mirru is a free and open source Android app under development with which one can control robotic prosthetic hands via hand tracking. With our app, a user can instantly mirror grips from their sound hand onto a robotic one, which could be 3D-printed and self-assembled at low cost. With Mirru, we want to provide a cheap, intuitive and open end-to-end alternative to existing, costly, cumbersome and proprietary technology.
Figure 1: A demonstration of using MediaPipe hand tracking to move a robotic hand’s fingers with the Mirru app.
The Mirru team is a collaboration between Violeta López and Vladimir Hermand, two independent designers and technologists currently based in Paris. To kickstart the project, the team took part in Tweag’s Open Source Fellowship program which provided funding, mentorship and data engineering expertise from one of their engineers, Dorran Howell. The fellowship helped get Mirru launched from the ground-up.
Our goal for the 3-month fellowship was to develop an initial version of the Android app that can control any bluetooth open source hand using computer vision techniques, and make the app available for free on the Google Play store so anyone can print their own hand, assemble it, and download the app. With the help of MediaPipe, we were able to quickly prototype our app without having to build our own machine learning model, as we didn’t have the resources or training data to do so.
Why use hand tracking?
Using your phone and a front-facing camera with hand tracking opens up a new, affordable, accessible, and versatile way to control prosthetics.
Let's say I'm a left hand amputee who owns a robotic prosthesis. Every day, I need my prosthetic hand to actuate a lot of different grip patterns. For example, I need to use a pinch or tripod grip to pick up small objects, or a fist grip to pick up objects like a fruit or a cup. I change and execute these grip patterns via myoelectric muscle sensors that allow me to, for example, open and close a grip by flexing and unflexing my upper limb muscles. These myoelectric muscle sensors are the main interface between my body and the prosthesis.
However, living with them is not as easy as it seems. Controlling the myoelectric sensors can take a lot of time to get used to, and many never do. It can also be quite expensive to get these sensors fitted by a prosthetist, especially for people in developing countries or anyone without health insurance. Finally, the number of grips on many devices currently on the market is limited to less than ten, and only few models come with ways to create custom grips, which are often cumbersome.
Mirru provides an alternative interface. Using just their phone, a tool many have access to, a user can digitally mirror their sound hand in real-time and communicate with their prosthesis in an intuitive way. This removes the expensive need to be fitted by a prosthetist and enables the user to quickly program an unlimited amount of grips. For now, Mirru stays away from electromyography altogether as reliable muscle sensors are expensive. The programmed grips therefore need to be triggered via the android phone, which is why this first version of our app is more suited for activities like sweeping, holding a book while reading it, or holding a cup or shopping bag. In the future we hope to combine myoelectric sensors with hand tracking to get the benefits of both.
Programming a grip with the Mirru app looks like the following: Let’s say that I want to grab an object with my robotic hand. I bring my prosthesis near the object and I then form the desired grip with my sound hand in front of my android phone and Mirru mirrors it in real-time to the prosthesis. I then lock my prosthesis into this new grip and free up my vsound hand. Finally I might save this grip for later use and add it to my library of grips.
Figure 2: A user tester using hand-tracking on their phone to program their prosthesis’s grip to pick up a measuring tape and measure with the other hand.
The Brunel Hand and the Mirru Arduino Sketch
In order to accomplish our goal of allowing as many people as possible to print, assemble, and control their own hand, we designed the Mirru android app to be compatible with any robotic hand that is controlled by a bluetooth-enabled Arduino board and servo motors.
For our project, we printed and assembled an open source robotic hand called the Brunel Hand made by Open Bionics. First, we 3D printed the Brunel Hand’s 3D printable files that are made available under the CC Attribution-Sharealike 4.0 International License. We then bought the necessary servos, springs, and screws to assemble the hand. In combination with printing and buying the servos, the hand costs around €500 to purchase and assemble.
The Brunel Hand comes with myoelectric-based firmware and a PCB board developed by Open Bionics, but since the hand is in essence just 4 servo motors, any microcontroller could be used. We ended up using an Adafruit ESP32 feather board for its bluetooth capabilities and created an Arduino sketch that can be downloaded, customized, and uploaded for anyone who is printing and assembling their own hand. They could then download the Mirru app to use as the control-interface for their printed hand.
There are many computer vision solutions available for hand tracking that could be used for this project, but we needed a fast, open source solution that didn’t require us to train our own model, and that could be used reliably on a portable device such as a phone.
MediaPipe provides great out of the box support for hand tracking, and since we didn’t have the training data or resources available to create a model from scratch, it was perfect for our team. We were able to build the Android example apps easily and were excited to find that the performance was promising. Even better, no tweaking on the ready-made hand tracking model or the graphs was necessary, as the hand landmark model provided all the necessary outputs for our prototype.
When testing the prosthesis on real users, we were happy to hear that many of them were impressed with how fast the app was able to translate their movements, and that nothing else exists on the market that allows you to make custom grips as fast and on-the-fly.
Figure 3: A user tester demonstrates how quickly the MediaPipe hand-tracking can translate her moving fingers to the movement of her prosthesis’s fingers.
To achieve the goals of the Mirru app, we need to use hand tracking to independently control each finger of the Brunel Hand in real-time. In the Brunel Hand, the index, middle, and ring fingers are actuated using servos that move at an angle from 0 to 180 degrees; 0 means the finger is fully upright and 180 means the finger is fully flexed down. As we lacked adequate training data to create a model from scratch that could calculate these servo angles for us, we opted to use a heuristic to relate the default hand tracking landmark outputs to the inputs required by our hardware for an initial version of our prototype.
Figure 4: In the lab testing the translation of the outputs to inputs with the app and the prototype.
We were initially unsure whether the estimated depth (Z) coordinate in the 3D landmarks would be accurate enough for the translation of inputs or if we would be limited to working in 2D. As an initial step, we recorded an example dataset and spun up a visualization of the points in a Jupyter Notebook with Plotly. We were immediately impressed by the quality and accuracy of the coordinates, considering that the technology only uses a single camera without any depth sensors. As noted in the MediaPipe documentation, The Z coordinates have a slightly different scale than the X/Y coordinates, but this didn’t seem to pose a significant challenge for our prototype.
Figure 5: A data visualization of the hand made up of 21 3D hand landmarks provided by MediaPipe.
Given the accuracy of the 3D landmarks, we opted to use a calculation in 3D for relating landmark outputs to the inputs required by the prosthesis. In our approach, we calculate the acute angles of the fingers in relation to the palm by calculating the angle between the finger direction and the normal of the plane defined by the palm. An angle of 0° corresponds to maximum closure of the finger, and an angle of 180° indicates a fully extended finger. We were able to calculate the finger direction by calculating the vector from the landmark at the base of the fingers to the landmark on the tip of the fingers.
Figure 6: Diagram showing the 3D landmarks and which ones we used to calculate the finger direction vector, the palm normal, and the angle that both form.
We calculate the palm normal by selecting three points in the plane of the palm. Using Landmark 0 as the reference point, we calculate the vectors for side 1 and side 2, and compute the cross product of those vectors to give us the palm normal. Finally, we compute the angle of the finger direction and the palm normal. This returns an angle in radians that we use to calculate degrees.
We had to do some extra processing to match the degrees of freedom for the thumb on our prosthetic hand. The thumb moves in more complex ways than the rest of the fingers. In order to get our app to work with the thumb, we did similar calculations for thumb direction and the palm normal, but we used different landmarks.
Once we do the calculation of the servo angles on the android phone, we send those values via bluetooth to the Arduino board, and the Arduino board moves the servos to the proper position. Due to some noise in the model outputs, we add a smoothing step to the pipeline, which is important so that the movements of the robotic fingers aren’t too jittery for precise grips.
Figure 7: A user tester makes a pinch grip on her prosthesis with the Mirru app.
The Mirru app and Mirru Arduino Sketch are designed to allow anyone to control an open source prosthesis with their sound hand and an Android phone. This is a novel and frugal alternative to muscle sensors, and MediaPipe has proven that it is the right tool for the essential hand tracking component of the full application. The Mirru team was able to get started quickly with MediaPipe’s out of the box solutions without having to gather any training data or having to design a model from scratch. The speed of the real-time translation from hand tracking points to the robotic hand has especially excited all of our users in our testing sessions and opens up many possibilities for the future of prostheses.
We see exciting potential for combining the MediaPipe hand tracking features with existing myoelectric prostheses which could open powerful and advanced ways to create and save custom prosthesis grips in real-time. Also, with the help of MediaPipe, we have been able to provide an open source alternative to proprietary prostheses without the need for myoelectric sensors or a visit to a prosthetist, at a cost that is much lower than what is already on the market, and whose source code can be customized and built-upon by other developers. Our team is excited to see what other ideas the open source community might come up with, and to see what hand tracking can bring to users and manufacturers of prostheses.
As for the current state of the Mirru application, we have yet to implement the possibility of recording and saving moving gestures that are longer sequences compared to the static grip positions. For example, imagine being able to record the movement of the fingers to play a bass line on a piano, like a loopable animated gif. There is a realm of possibilities for prostheses that is waiting to be explored, and we’re really happy that MediaPipe gives us access to it.
We are looking for contributors. If you have ideas or comments about this application, please reach out to mirruapp@gmail.com, or visit our GitHub.
This blog post is curated by Igor Kibalchich, ML Research Product Manager at Google AI.
