Posted by Miguel Guevara, Product Manager, Privacy and Data Protection Office.
As developers, it’s our responsibility to help keep our users safe online and protect their data. This starts with building products that are secure by default, private by design, and put users in control. Everything we make at Google is underpinned by these principles, and we’re proud to be an industry leader in developing, deploying, and scaling new privacy-preserving technologies that make it possible to learn valuable insights and create helpful experiences while protecting our users’ privacy.
That’s why today, we are excited to announce that we’re open-sourcing a first-of-its-kind, general-purpose transpiler for Fully Homomorphic Encryption (FHE), which will enable developers to compute on encrypted data without being able to access any personally identifiable information.
A deeper look at the technology
With FHE, encrypted data can travel across the Internet to a server, where it can be processed without being decrypted. Google’s transpiler will enable developers to write code for any type of basic computation such as simple string processing or math, and run it on encrypted data. The transpiler will transform that code into a version that can run on encrypted data. This then allows developers to create new programming applications that don’t need unencrypted data. FHE can also be used to train machine learning models on sensitive data in a private manner.
For example, imagine you’re building an application for people with diabetes. This app might collect sensitive information from its users, and you need a way to keep this data private and protected while also sharing it with medical experts to learn valuable insights that could lead to important medical advancements. With Google’s transpiler for FHE, you can encrypt the data you collect and share it with medical experts who, in turn, can analyze the data without decrypting it - providing helpful information to the medical community, all while ensuring that no one can access the data’s underlying information.
In the next 10 years, FHE could even help researchers find associations between specific gene mutations by analyzing genetic information across thousands of encrypted samples and testing different hypotheses to identify the genes most strongly associated with the diseases they’re studying.
Making more products private by design
Our principle to make our products private by design drives us to build ground-breaking computing technologies that enable personalized experiences while protecting your private information. Privacy-preserving technologies are on the cutting-edge of Google’s innovations, and they have already shown great potential to help shape a more private internet.
In 2016, Google researchers invented Federated Learning, a technique that helps preserve privacy by keeping as much personal information on your device as possible. And in 2019, Google made its differential privacy library freely available to any organization or developer, an advanced anonymization technology that enables developers to learn from their data privately. No one has scaled the use of Differential Privacy more than we have.
We’ve been thrilled to see these technologies put to use across the globe; in France, for example, a startup called Arkhn has been able to accelerate scientific discovery using differential privacy to share data across hospitals.
We still have a ways to go before most computations happen with FHE -- but much as it took some time for HTTPS to take off and be widely adopted, today’s announcement is an important step towards bringing users helpful products that preserve their privacy and keep their data safe.
At Google, we know that open-sourcing our technologies with the developer community for feedback and use helps make them better. We will continue to invest and lead the privacy-preserving technology field by publishing new work, and open-sourcing it for everyone to use at scale - and we're excited to continue this practice by sharing this latest advancement with developers everywhere. We can't wait to see what you’ll build, and we look forward to collaborating on the journey towards a safer Internet.
#IamaGDE series presents: Google Maps
The Google Developers Experts program is a global network of highly experienced technology experts, influencers, and thought leaders who actively support developers, companies, and tech communities by speaking at events and publishing content.
Meet Josue Gutierrez — Maps, Web, Identity and Angular Google Developer Expert.
Josue currently works at the German company Boehringer Ingelheim and lives near Frankfurt. Before moving to Germany, Josue was working as a software engineer in Mexico, and before that, he spent almost a year in San Francisco as a senior front-end developer at Sutter Health.
Josue Gutierrez
Josue studied computer science and engineering as an undergraduate and learned algorithms and programming. His first language was C++, and he learned C and Python, but was drawn to web technologies.
“When I saw a web browser for the first time, it stuck with me,” he says, “It was changing in real time as you’re developing. That feeling is really cool. That’s why I went into frontend development.”
Josue has worked on multiple ecommerce projects focused on improving customers’ trade experience. He sees his role as creating something from scratch to help people improve lives.
“These opportunities we have as developers are great — to travel, work for many verticals, and learn many businesses,” he says. “In my previous job, I developed tech-oriented trade tools for research companies, to manipulate strings or formulas. I was on the team involved in writing these kinds of tools, so it was more about the trade experience for doctors.”
Getting involved in the developer community
Josue’s first trip outside Mexico, to San Francisco, exposed him to the many developer communities in the area, and he appreciated the supportive communities of people trying to learn together. Several of the people he met suggested he start his own meetup in Mexico City, to get more involved in Google technologies, so he launched an Angular community there. As he hunted for speakers to come to his Angular meetup, Josue found himself giving talks, too.
Then, the GDG Mexico leader invited Josue to give talks on Google for startups.
“That helped me get involved in the ecosystem,” Josue says. “I met a lot of people, and now many of them are good friends. It’s really exciting because you get connected with people with the same interests as you, and you all learn together.”
“I’m really happy to be part of the Google Maps ecosystem,” Josue says. “It’s super connected, with kind people, and now I know more colleagues in my area, who work for different companies and have different challenges. Seeing how they solve them is a good part of being connected to the product. I try to share my knowledge with other people and exchange points of view.”
Josue says 2020 provided interesting opportunities.
“This year was weird, but we also discovered more tools that are evolving with us, more functionalities in Hangouts and Meetup,” Josue says. “It’s interesting how people are curious to get connected. If I speak from Germany, I get comments from countries like Bolivia and Argentina. We are disconnected but increasing the number of people we engage with.”
He notes that the one missing piece is the face-to-face, spontaneous interactions of in-person workshops, but that there are still positives to video workshops.
“I think as communities, we are always trying to get information to our members, and having videos is also cool for posterity,” he says.
He is starting a Maps developer community in Germany.
“I have colleagues interested in trying to get a community here with a solid foundation,” he says. “We hope we can engage people to get connected in the same place, if all goes well.”
Favorite Maps features and current projects
As a frontend developer, Josue regards Google Maps Platform as an indispensable tool for brands, ecommerce companies, and even trucking companies.
“Once you start learning how to plant coordinates inside a map, how to convert information and utilize it inside a map, it’s easy to implement,” he says.
In 2021, Josue is working on some experiments with Maps, trying to make more real-time actualization, using currently available tools.
“Many of the projects I’ve been working on aren’t connected with ecommerce,” he says. “Many customers want to see products inside a map, like trucking products. I’ve been working in directories, where you can see the places related to categories — like food in Mexico. You can use Google Maps functionalities and extend the diversification of maps and map whatever you want.”
“Submission ID is really cool,” he adds. “You can do it reading the documentation, a key part of the product, with examples, references, and a live demo in the browser.”
Future plans
Josue says his goal going forward is to be as successful as he can at his current role.
“Also, sharing is super important,” he says. “My company encourages developer communities. It’s important to work in a place that matches your interests.”
Follow Josue on Twitter at @eusoj |Check out Josue’s projects on GitHub.
For more information on Google Maps Platform, visit our website or learn more about our GDE program.
Posted by Max Saltonstall
A little over 10 years ago, we launched the IT Residency Program (ITRP) at Google with a twofold mission: to provide exceptional tech support for Googlers and to empower the next generation of IT pioneers.
ITRP’s founding principle is learning and development. In addition to formal on-the-job IT support training, the program takes its residents through a focused career development path, including a hands-on rotation in a specific Google function in their chosen specialty. For their part, the residents, who come from a wide spectrum of often non-traditional backgrounds, bring with them a passion for learning. ITRP converts that passion into real-world experience and equips them for a lifelong career in tech.
Today, hundreds of Googlers are ITRP alums, working in disciplines ranging from site reliability engineering to security and privacy to program management and all points in between. Looking back on the program’s 10-plus years, we wanted to share some of their stories, their experiences and their triumphs. Look for more installments in this series in the weeks to come.
While studying anthropology in college, Kate Grant had a part time help desk job, helping solve people's computer problems (and sometimes the computer's people problems). And it was through that job that she realized she really liked the satisfaction she got from helping people so directly. At school, Kate could help students, faculty and staff — both technical folks and non-technical, and it was great to save the day for them, teach them, and constantly learn about new technologies.
Joining the Pride March in NYC as part of Google's presence
As graduation loomed close, Kate needed something to do afterwards, and she stumbled upon the ITRP posting. It looked like a great way to continue doing what she loved, and in a much bigger company, with more to learn. She applied, thinking "there's no way in hell" she would get this job.
Turns out she was wrong!
Starting at Google in August 2012, Kate worked in Mountain View, CA helping Googlers of all types with a broad range of tech challenges. She got to spend some time with the Search team at Google working on technical documentation, helping engineers at Google as well as web designers outside Google better understand Google search features. This experience gave Kate something cool she could show her family too, a very tangible "I made this" moment, which can be hard to come by in IT operations work.
During her time in the program, Kate also had the chance to spend some time working in the New York office, which felt good because it brought her closer to New Jersey, where she grew up. It was the first step towards coming back to the Northeast, a welcome return to a comfortable place. Eventually she'd move to New York to work in Google's NYC office full time, helping with IT operations and later managing junior IT help desk folks as the team expanded.
Neon welcome sign in the lobby of the Google NYC office
Working across teams and projects and help desks in ITRP helped Kate develop all kinds of skills, from improving proficiency with Linux, Windows, MacOS and Chrome to also developing better judgment and analysis skills when solving novel problems that walked into the support desk. And as she began helping to train new employees in orientation, covering IT, Security and Technology topics, she got practice improving her public speaking; by the end of it she was speaking to over 100 people at a time on Mondays in Mountain View as new employees learned the essentials of their work at Google.
While she had been considering a tech writing career path before, Kate realized that the work in ITRP helping people day-to-day let her wear many hats. She was writing documentation, that was one key component. But she also wrote code, managed programs and projects, mentored junior members of the team, and analyzed data. The breadth of the job matched her growth goals, and Kate ended up continuing her career in Techstop, transitioning into a role as a Corporate Operations Engineer in 2014.
The work remained in the front-line support team and gave Kate tremendous exposure to the wide array of people in the Google headquarters. She focused more on projects to improve the onboarding and daily operations of the support teams, helping to keep systems and services healthy. This involved scheduling, mentoring, training and helping the newer members of the team, mostly later cohorts of IT Residents.
Kate's experience training new employees, and mentoring new IT Residents, made her a great fit for a new opportunity on the onboarding team, where she worked to make the new employee experience more consistent and reliable between offices across Google globally. This began a traversal of different parts of Google's IT org, where she learned about how the company, and its many teams, operate. Kate's work focused on helping to create better, smoother, more automated processes for the teams in IT, and help to scale the successes they had already achieved.
Driving improvements to Google’s onboarding infrastructure was fun and satisfying, but Kate missed working with junior techs day to day, and mentoring them on their own career growth. Luckily in 2016 a new manager position opened up in NYC, and Kate jumped at it. She started leading a team of newer support technicians, including IT Residents. Now she could focus on making a really great support experience for Googlers, and give back to the program she had enjoyed by helping the folks on her team grow.
In 2019 Kate had the opportunity to move to Austin, TX to build out a new ITRP Hub location from scratch, and continues to manage IT Residences and Corporate Operations Engineers there today.
Coming full circle now Kate shared some great advice with us for those thinking about ITRP:
"Don't screen yourself out of the opportunity. Folks come into the program from all walks of life. You don't have to have a super fancy computer science degree. If you’re excited about technology and helping people, ITRP could be a great next step for you."
She reinforced that you get out of the program what you put into it: “If you take the time to be curious, to ask questions, to investigate, you will learn so much. It's really an endless amount of material you could absorb, and nobody can get through it all. But when you put in the work, it pays you back”.
And as people go through the ITR Program, they end up in all sorts of places. "There's no cookie cutter outcome… It doesn’t mean you’re doing something wrong if your path doesn’t look like someone else’s. The right way is what's right for you and your goals."
Posted by Toni Klopfenstein, Developer Advocate
Alerts for important device events, such as a delivery person arriving or the back door failing to lock, create a more beneficial and reassuring experience for your smart home device users.
As we announced at I/O, you can now add proactive notifications and follow-up responses to your Smart Home Action to alert users to events in a timely, relevant and helpful fashion and better engage with your end users.
Notifications can either alert a user to an event that has occurred without them proactively issuing a request through the Assistant, or as a follow-up to verify that the user's request has been fulfilled. Each device event that triggers one of these notifications has a unique event id, which helps the Assistant route it to the appropriate Home Graph users and Google Home Smart Speakers or Nest Smart Displays, depending on the notification type and priority. Notifications and follow-up responses can also provide users with additional information, such as error and exception codes, or timestamps for the event.
You can enable notifications on your existing devices once users opt-in to receive alerts by updating the device definition and requesting a SYNC intent. You can then send device notifications along with any applicable device state changes using the Home Graph API.
We are adding support for traits where asynchronous requirements are a core use case.The following device traits now support follow-up responses to user queries:
Additionally, we are launching proactive notification alerts for the following traits:
For more information, check out the developer guides and samples, or check out the Notifications video.
We want to hear from you, so continue sharing your feedback with us through the issue tracker, and engage with other smart home developers in the /r/GoogleAssistantDev community. Follow @ActionsOnGoogle on Twitter for more of our team's updates, and tweet using #AoGDevs to share what you’re working on. We can’t wait to see what you build!
Posted by HyeJung Lee and MJ You, Google ML Ecosystem Community Managers. Reviewed by Soonson Kwon, Developer Relations Program Manager.
Google Developers Experts is a community of passionate developers who love to share their knowledge with others. Many of them specialize in Machine Learning (ML). Despite many unexpected changes over the last months and reduced opportunities for various in person activities during the ongoing pandemic, their enthusiasm did not stop.
Here are some highlights of the ML GDE’s hard work during the Q1 2021 which contributed to the global ML ecosystem.
ML GDE YouTube channel
With the initiative and lead of US-based GDE Margaret Maynard-Reid, we launched the ML GDEs YouTube channel. It is a great way for GDEs to reach global audiences, collaborate as a community, create unique content and promote each other's work. It will contain all kinds of ML related topics: talks on technical topics, tutorials, interviews with another (ML) GDE, a Googler or anyone in the ML community etc. Many videos have already been uploaded, including: ML GDE’s intro from all over the world, tips for TensorFlow & GCP Certification and how to use Google Cloud Platform etc. Subscribe to the channel now!!
TensorFlow Everywhere
17 ML GDEs presented at TensorFlow Everywhere (a global community-led event series for TensorFlow and Machine Learning enthusiasts and developers around the world) hosted by local TensorFlow user groups. You can watch the recorded sessions in the TensorFlow Everywhere playlist on the ML GDE Youtube channel. Most of the sessions cover new features in Tensorflow.
International Women’s Day
Many ML GDEs participated in activities to celebrate International Women’s Day (March 8th). GDE Ruqiya Bin Safi (based in Saudi Arabia) cooperated with WTM Saudi Arabia to organize “Socialthon” - social development hackathons and gave a talk “Successful Experiences in Social Development", which reached 77K viervers live and hit 10K replays. India-based GDE Charmi Chokshi participated in GirlScript's International Women's Day event and gave a talk: “Women In Tech and How we can help the underrepresented in the challenging world”. If you’re looking for more inspiring materials, check out the “Women in AI” playlist on our ML GDE YouTube channel!
Mentoring
ML GDEs are also very active in mentoring community developers, students in the Google Developer Student Clubs and startups in the Google for Startups Accelerator program. Among many, GDE Arnaldo Gualberto (Brazil) conducted mentorship sessions for startups in the Google Fast Track program, discussing how to solve challanges using Machine Learning/Deep Learning with TensorFlow.
TensorFlow
Meanwhile in Europe, GDEs Alexia Audevart (based in France) and Luca Massaron (based in Italy) released “Machine Learning using TensorFlow Cookbook”. It provides simple and effective ideas to successfully use TensorFlow 2.x in computer vision, NLP and tabular data projects. Additionally, Luca published the second edition of the Machine Learning For Dummies book, first published in 2015. Her latest edition is enhanced with product updates and the principal is a larger share of pages devoted to discussion of Deep Learning and TensorFlow / Keras usage.
On top of her women-in-tech related activities, Ruqiya Bin Safi is also running a “Welcome to Deep Learning Course and Orientation” monthly workshop throughout 2021. The course aims to help participants gain foundational knowledge of deep learning algorithms and get practical experience in building neural networks in TensorFlow.
Nepal-based GDE Kshitiz Rimal gave a talk “TensorFlow Project Showcase: Cash Recognition for Visually Impaired" on his project which uses TensorFlow, Google Cloud AutoML and edge computing technologies to create a solution for the visually impaired community in Nepal.
On the other side of the world, in Canada, GDE Tanmay Bakshi presented a talk “Machine Learning-powered Pipelines to Augment Human Specialists” during TensorFlow Everywhere NA. It covered the world of NLP through Deep Learning, how it's historically been done, the Transformer revolution, and how using the TensorFlow & Keras to implement use cases ranging from small-scale name generation to large-scale Amazon review quality ranking.
Google Cloud Platform
We have been equally busy on the GCP side as well. In the US, GDE Srivatsan Srinivasan created a series of videos called “Artificial Intelligence on Google Cloud Platform”, with one of the episodes, "Google Cloud Products and Professional Machine Learning Engineer Certification Deep Dive", getting over 3,000 views.
Korean GDE Chansung Park contributed to TensorFlow User Group Korea with his “Machine Learning Pipeline (CI/CD for ML Products in GCP)” analysis, focused on about machine learning pipeline in Google Cloud Platform.
Last but not least, GDE Gad Benram based in Israel wrote an article on “Seven Tips for Forecasting Cloud Costs”, where he explains how to build and deploy ML models for time series forecasting with Google Cloud Run. It is linked with his solution of building a cloud-spend control system that helps users more-easily analyze their cloud costs.
If you want to know more about the Google Experts community and all their global open-source ML contributions, visit the GDE Directory and connect with GDEs on Twitter and LinkedIn. You can also meet them virtually on the ML GDE’s YouTube Channel!
Posted by Patricia Correa, Director, Global Developer Marketing
At Google Play we’re committed to helping all developers thrive, whether these are large multinational companies or small startups and indie game studios. They are all critical to providing the services and experiences that people around the world look for on their Android devices. The indie game developer community, in particular, constantly pushes the boundaries with their creativity and passion, and bring unique and diverse content to players everywhere.
To continue supporting indies, today we’re opening submissions for two of our annual developer programs - the Indie Games Accelerator and the Indie Games Festival. These programs are designed to help small games studios grow on Google Play, no matter what stage they are in:
This year the programs come with some changes, including more eligible markets and fully digital event experiences. Learn more below and apply by July 1st.
Accelerator: Get education and mentorship to supercharge your growth
If you’re an indie developer, early in your journey - either close to launching a new game or recently launched a title, this is the program for you. We’ll provide education and mentorship that will help you build, launch and grow successfully.
This year we have nearly doubled the eligible markets, with developers from over 70 countries being eligible to apply for the 2021 program.
Selected participants will be invited to take part in a 12-week online acceleration program. During this time you’ll get exclusive access to a community of Google and industry experts, as well as a network of other passionate developers from around the world looking to supercharge their growth.
Festival: win promotions that put your game in the spotlight
If you're an indie game developer who has recently launched a high quality game, this is your chance to have your game discovered by industry experts and players worldwide.
This year we will, again, host three competitions for developers from Japan, South Korea, and selected European countries.
Prizes include featuring on Google Play store, promotional campaigns worth 100,000 EUR, and more.
How useful did you find this blog post?
★ ★ ★ ★ ★
Posted by Posted by Erica Hanson, Senior Program Manager, Google Developer Student Clubs
Tamta Kapanadze wishes that she had learned sooner about careers in technology. By the time that the Georgian citizen learned about them, she was already a university student.
As Kapanadze continued her studies and her interest in technology grew, she wanted to spread the word about the growing field to high-school students in Georgia, a country where the industry is still small.
To do this, Kapanadze called in the support of Google Developer Student Clubs (GDSCs), community groups for college and university students interested in Google's developer technology. After Kapanadze graduated from university, she continued her work by organizing a chapter of Google Developer Groups (GDGs) for Kutaisi.
Google Developer Groups are the largest community network of professional developers in the world. The program consists of local chapters that provide inclusive environments open to everybody interested in tech. The chapters let members learn new skills, and meet other developers with similar interests through online and in-person events.
However, even after all that, Kapanadze still wanted to do more. She brought together friends and colleagues to partner with Mariam, GDSC Georgia American University Lead; Iliko, GDSC Georgia American University core team member; Giorgi, GDSC Tbilisi State University Lead; and Bakar, GDSC San Diego State University Lead. With this team assembled, they planned Tech Camp, a virtual technological learning experience that teaches high schoolers about tech fields and how to start careers in web development, game development, artificial intelligence, machine learning, and more.
Tamta Kapanadze.
While it's difficult enough to plan and execute a new event, Kapanadze and her partners didn't let the additional challenges of the last year stop their plans to launch Tech Camp. They wanted to publicize the event by mid-January, so they made a to-do list and set deadlines for themselves. After a few weeks of intense planning, they:
Kapanadze and her partners accepted applications for Tech Camp from Jan. 20 to Feb. 10 and announced their speakers to the public to keep the buzz about the event going. They originally hoped to receive 30 applications, but instead received 500 across their events. In particular, 300 students attended the speaker sessions and 50 students attended the coding sessions, where they would teach them about algorithms and the basics of C++.
Finally, the first day of Tech Camp arrived on Feb. 15. They began each session with fun icebreakers to help everybody feel comfortable, including themselves. Here's a timeline of what each day covered:
A screenshot of the virtual session about game development.
Everybody defines success differently, but for Kapanadze and her team it meant impacting at least one person. By this measure, Tech Camp succeeded because many of those who attended decided to pursue careers in tech. As for Kapanadze, she can’t wait to see what the future holds for Georgia's high schoolers and the country's growing tech industry.
To watch recordings from Tech Camp, please visit the playlist on YouTube.
The recordings available from Tech Camp.
For more information, find a Google Developers community group near you.
Posted by the Google Developers team
If you attended Google I/O this year, you probably stopped by the Google I/O photo booth for a selfie with our Google Developer mascots: Flutter’s Dash, Android Jetpack, Chrome’s Dino, and Firebase’s Sparky. If you didn’t, it’s not too late to jump in, take a selfie, and share it on social media! We loved seeing all of the pictures you posted and your favorite props! Want to learn more about building a camera plugin, layouts, and gestures used in a photo booth for Flutter on the web?
It took a combination of Google developer products to make the photo booth successful. The Flutter and Firebase teams joined forces to build a best in class example of Flutter on the web that used Firebase for hosting, auth, performance monitoring, and social sharing. Take a closer look at how the photo booth was built here and then grab the open source code on Github!
Flutter team members having fun in the photo booth!
Guest post by the Engineering teams at Mirru and Tweag
What is the Mirru App?
Mirru is a free and open source Android app under development with which one can control robotic prosthetic hands via hand tracking. With our app, a user can instantly mirror grips from their sound hand onto a robotic one, which could be 3D-printed and self-assembled at low cost. With Mirru, we want to provide a cheap, intuitive and open end-to-end alternative to existing, costly, cumbersome and proprietary technology.
Figure 1: A demonstration of using MediaPipe hand tracking to move a robotic hand’s fingers with the Mirru app.
The Mirru team is a collaboration between Violeta López and Vladimir Hermand, two independent designers and technologists currently based in Paris. To kickstart the project, the team took part in Tweag’s Open Source Fellowship program which provided funding, mentorship and data engineering expertise from one of their engineers, Dorran Howell. The fellowship helped get Mirru launched from the ground-up.
Our goal for the 3-month fellowship was to develop an initial version of the Android app that can control any bluetooth open source hand using computer vision techniques, and make the app available for free on the Google Play store so anyone can print their own hand, assemble it, and download the app. With the help of MediaPipe, we were able to quickly prototype our app without having to build our own machine learning model, as we didn’t have the resources or training data to do so.
Why use hand tracking?
Using your phone and a front-facing camera with hand tracking opens up a new, affordable, accessible, and versatile way to control prosthetics.
Let's say I'm a left hand amputee who owns a robotic prosthesis. Every day, I need my prosthetic hand to actuate a lot of different grip patterns. For example, I need to use a pinch or tripod grip to pick up small objects, or a fist grip to pick up objects like a fruit or a cup. I change and execute these grip patterns via myoelectric muscle sensors that allow me to, for example, open and close a grip by flexing and unflexing my upper limb muscles. These myoelectric muscle sensors are the main interface between my body and the prosthesis.
However, living with them is not as easy as it seems. Controlling the myoelectric sensors can take a lot of time to get used to, and many never do. It can also be quite expensive to get these sensors fitted by a prosthetist, especially for people in developing countries or anyone without health insurance. Finally, the number of grips on many devices currently on the market is limited to less than ten, and only few models come with ways to create custom grips, which are often cumbersome.
Mirru provides an alternative interface. Using just their phone, a tool many have access to, a user can digitally mirror their sound hand in real-time and communicate with their prosthesis in an intuitive way. This removes the expensive need to be fitted by a prosthetist and enables the user to quickly program an unlimited amount of grips. For now, Mirru stays away from electromyography altogether as reliable muscle sensors are expensive. The programmed grips therefore need to be triggered via the android phone, which is why this first version of our app is more suited for activities like sweeping, holding a book while reading it, or holding a cup or shopping bag. In the future we hope to combine myoelectric sensors with hand tracking to get the benefits of both.
Programming a grip with the Mirru app looks like the following: Let’s say that I want to grab an object with my robotic hand. I bring my prosthesis near the object and I then form the desired grip with my sound hand in front of my android phone and Mirru mirrors it in real-time to the prosthesis. I then lock my prosthesis into this new grip and free up my vsound hand. Finally I might save this grip for later use and add it to my library of grips.
Figure 2: A user tester using hand-tracking on their phone to program their prosthesis’s grip to pick up a measuring tape and measure with the other hand.
The Brunel Hand and the Mirru Arduino Sketch
In order to accomplish our goal of allowing as many people as possible to print, assemble, and control their own hand, we designed the Mirru android app to be compatible with any robotic hand that is controlled by a bluetooth-enabled Arduino board and servo motors.
For our project, we printed and assembled an open source robotic hand called the Brunel Hand made by Open Bionics. First, we 3D printed the Brunel Hand’s 3D printable files that are made available under the CC Attribution-Sharealike 4.0 International License. We then bought the necessary servos, springs, and screws to assemble the hand. In combination with printing and buying the servos, the hand costs around €500 to purchase and assemble.
The Brunel Hand comes with myoelectric-based firmware and a PCB board developed by Open Bionics, but since the hand is in essence just 4 servo motors, any microcontroller could be used. We ended up using an Adafruit ESP32 feather board for its bluetooth capabilities and created an Arduino sketch that can be downloaded, customized, and uploaded for anyone who is printing and assembling their own hand. They could then download the Mirru app to use as the control-interface for their printed hand.
There are many computer vision solutions available for hand tracking that could be used for this project, but we needed a fast, open source solution that didn’t require us to train our own model, and that could be used reliably on a portable device such as a phone.
MediaPipe provides great out of the box support for hand tracking, and since we didn’t have the training data or resources available to create a model from scratch, it was perfect for our team. We were able to build the Android example apps easily and were excited to find that the performance was promising. Even better, no tweaking on the ready-made hand tracking model or the graphs was necessary, as the hand landmark model provided all the necessary outputs for our prototype.
When testing the prosthesis on real users, we were happy to hear that many of them were impressed with how fast the app was able to translate their movements, and that nothing else exists on the market that allows you to make custom grips as fast and on-the-fly.
Figure 3: A user tester demonstrates how quickly the MediaPipe hand-tracking can translate her moving fingers to the movement of her prosthesis’s fingers.
To achieve the goals of the Mirru app, we need to use hand tracking to independently control each finger of the Brunel Hand in real-time. In the Brunel Hand, the index, middle, and ring fingers are actuated using servos that move at an angle from 0 to 180 degrees; 0 means the finger is fully upright and 180 means the finger is fully flexed down. As we lacked adequate training data to create a model from scratch that could calculate these servo angles for us, we opted to use a heuristic to relate the default hand tracking landmark outputs to the inputs required by our hardware for an initial version of our prototype.
Figure 4: In the lab testing the translation of the outputs to inputs with the app and the prototype.
We were initially unsure whether the estimated depth (Z) coordinate in the 3D landmarks would be accurate enough for the translation of inputs or if we would be limited to working in 2D. As an initial step, we recorded an example dataset and spun up a visualization of the points in a Jupyter Notebook with Plotly. We were immediately impressed by the quality and accuracy of the coordinates, considering that the technology only uses a single camera without any depth sensors. As noted in the MediaPipe documentation, The Z coordinates have a slightly different scale than the X/Y coordinates, but this didn’t seem to pose a significant challenge for our prototype.
Figure 5: A data visualization of the hand made up of 21 3D hand landmarks provided by MediaPipe.
Given the accuracy of the 3D landmarks, we opted to use a calculation in 3D for relating landmark outputs to the inputs required by the prosthesis. In our approach, we calculate the acute angles of the fingers in relation to the palm by calculating the angle between the finger direction and the normal of the plane defined by the palm. An angle of 0° corresponds to maximum closure of the finger, and an angle of 180° indicates a fully extended finger. We were able to calculate the finger direction by calculating the vector from the landmark at the base of the fingers to the landmark on the tip of the fingers.
Figure 6: Diagram showing the 3D landmarks and which ones we used to calculate the finger direction vector, the palm normal, and the angle that both form.
We calculate the palm normal by selecting three points in the plane of the palm. Using Landmark 0 as the reference point, we calculate the vectors for side 1 and side 2, and compute the cross product of those vectors to give us the palm normal. Finally, we compute the angle of the finger direction and the palm normal. This returns an angle in radians that we use to calculate degrees.
We had to do some extra processing to match the degrees of freedom for the thumb on our prosthetic hand. The thumb moves in more complex ways than the rest of the fingers. In order to get our app to work with the thumb, we did similar calculations for thumb direction and the palm normal, but we used different landmarks.
Once we do the calculation of the servo angles on the android phone, we send those values via bluetooth to the Arduino board, and the Arduino board moves the servos to the proper position. Due to some noise in the model outputs, we add a smoothing step to the pipeline, which is important so that the movements of the robotic fingers aren’t too jittery for precise grips.
Figure 7: A user tester makes a pinch grip on her prosthesis with the Mirru app.
The Mirru app and Mirru Arduino Sketch are designed to allow anyone to control an open source prosthesis with their sound hand and an Android phone. This is a novel and frugal alternative to muscle sensors, and MediaPipe has proven that it is the right tool for the essential hand tracking component of the full application. The Mirru team was able to get started quickly with MediaPipe’s out of the box solutions without having to gather any training data or having to design a model from scratch. The speed of the real-time translation from hand tracking points to the robotic hand has especially excited all of our users in our testing sessions and opens up many possibilities for the future of prostheses.
We see exciting potential for combining the MediaPipe hand tracking features with existing myoelectric prostheses which could open powerful and advanced ways to create and save custom prosthesis grips in real-time. Also, with the help of MediaPipe, we have been able to provide an open source alternative to proprietary prostheses without the need for myoelectric sensors or a visit to a prosthetist, at a cost that is much lower than what is already on the market, and whose source code can be customized and built-upon by other developers. Our team is excited to see what other ideas the open source community might come up with, and to see what hand tracking can bring to users and manufacturers of prostheses.
As for the current state of the Mirru application, we have yet to implement the possibility of recording and saving moving gestures that are longer sequences compared to the static grip positions. For example, imagine being able to record the movement of the fingers to play a bass line on a piano, like a loopable animated gif. There is a realm of possibilities for prostheses that is waiting to be explored, and we’re really happy that MediaPipe gives us access to it.
We are looking for contributors. If you have ideas or comments about this application, please reach out to mirruapp@gmail.com, or visit our GitHub.
This blog post is curated by Igor Kibalchich, ML Research Product Manager at Google AI.
Posted by Brian Shen, Program Manager, Google Developers
Google Developer Groups are one of the largest community networks of developers in the world. Every group has an organizer that helps curate events based on the interests of their local developer community.
As we continue to explore how different Google Developer Groups build their communities, we interviewed Hebe He, an organizer of Google Developer Group Guangzhou in China. Learn more about how she is building the developer scene in China, thinking up new events for her community, and more below.
Hebe He, an organizer of Google Developer Group Guangzhou in China.
Tell us about yourself.
I am Hebe from China and I'm a native of Guangzhou. I'm the organizer of GDG Guangzhou, as well as an ambassador for Women Techmakers (WTM). I work at one of China's new electric-vehicle brands, where I'm responsible for the intelligent business operation of the Internet of Vehicles. I'm relatively outgoing and active, so I really like to deal with different people, whether it's at work or in other activities.
How did you learn about Google Developer Groups?
In 2014, I participated in GDG Guangzhou DevFest for the first time by coincidence and met the founder of GDG Guangzhou. Afterward, I joined the founder's company and volunteered at many GDG programs. In 2017, I officially became an organizer after the existing organizers recognized my ability and desire to contribute more to the GDG Guangzhou community.
Tell us more about Guangzhou and the developer community there.
Our community members are talented, passionate, and amazing. I see all kinds of possibilities in them. They're always excited for every event we hold, keep a fanatical attitude toward Google's technological innovation, and are particularly interested in Android, Kotlin, and Flutter.
What are events like in your community?
We highly value feedback from event participants, who are interested in a wide range of topics. For this reason, we generally use 15% of every event to cover non-technical topics, such as entrepreneurship, business management, and careers. For more comprehensive activities, such as DevFest, we increase the amount of non-technical content to roughly 30%.
What is your Google Developer Group focused on right now?
We devote most of our energy to improving the quality of activities. We try to add more elements to the event to strengthen the interaction of participants in hopes of improving the feedback mechanism and gaining more valuable suggestions for future event optimization. We also try to improve the quality of guests and themes, and pay more attention to event details, such as event announcements, registration, and check-in.
What’s your favorite community memory from a Google Developer Group event?
The memory that touches me the most is the construction of WTM Guangzhou. From the first event with only 80 developers to the audience of more than 500 people in recent years, it represents the recognition of, and support for, our events. There are many people who come to participate every year; some are actively encouraging their friends to participate and others are even urging us to hold events. They feel honored to be invited to our events and their enthusiasm endured during the pandemic.
What's next for you and your Google Developer Group?
There's still lots of room to grow in our community. We hope that we can continue to develop a Google Developer Group that reflects the best of Guangzhou. We also hope to find better ways to accumulate the experience shared by speakers and the value of community users.
If you want to grow your career and coding knowledge with people like Hebe He, join a Google Developer Group near you.
