Please note that the information, uses, and applications expressed in the below post are solely those of our guest author, Alfred Camera.
In this article, we’d like to give you a short overview of Alfred Camera and our experience of using MediaPipe to transform our moving object feature, and how MediaPipe has helped to get things easier to achieve our goals.
Fig.1 Alfred Camera Logo
Alfred Camera is a smart home app for both Android and iOS devices, with over 15 million downloads worldwide. By downloading the app, users are able to turn their spare phones into security cameras and monitors directly, which allows them to watch their homes, shops, pets anytime. The mission of Alfred Camera is to provide affordable home security so that everyone can find peace of mind in this busy world.
The Alfred Camera team is composed of professionals in various fields, including an engineering team with several machine learning and computer vision experts. Our aim is to integrate AI technology into devices that are accessible to everyone.
Alfred Camera currently has a feature called Moving Object Detection, which continuously uses the device’s camera to monitor a target scene. Once it identifies a moving object in the area, the app will begin recording the video and send notifications to the device owner. The machine learning models for detection are hand-crafted and trained by our team using TensorFlow, and run on TensorFlow Lite with good performance even on mid-tier devices. This is important because the app is leveraging old phones and we'd like the feature to reach as many users as possible.
We had started building our AI features at Alfred Camera since 2017. In order to have a solid foundation to support our AI feature requirements for the coming years, we decided to rebuild our real-time video analysis pipeline. At the beginning of the project, the goals were to create a new pipeline which should be 1) modular enough so we could swap core algorithms easily with minimal changes in other parts of the pipeline, 2) having GPU acceleration designed in place, 3) cross-platform as much as possible so there’s no need to create/maintain separate implementations for different platforms. Based on the goals, we had surveyed several open source projects that had the potential but we ended up using none of them as they either fell short on the features or were not providing the readiness/stabilities that we were looking for.
We started a small team to prototype on those goals first for the Android platform. What came later were some tough challenges way above what we originally anticipated. We ran into several major design changes as some key design basics were overlooked. We needed to implement some utilities to do things that sounded trivial but required significant effort to make it right and fast. Dealing with asynchronous processing also led us into a bunch of timing issues, which took the team quite some effort to address. Not to mention debugging on real devices was extremely inefficient and painful.
Things didn't just stop here. Our product is also on iOS and we had to tackle these challenges once again. Moreover, discrepancies in the behavior between the platform-specific implementations introduced additional issues that we needed to resolve.
Even though we finally managed to get the implementations to the confidence level we wanted, that was not a very pleasant experience and we have never stopped thinking if there is a better option.
Google open sourced MediaPipe project in June 2019 and it immediately caught our attention. We were surprised by how it is perfectly aligned with the previous goals we set, and has functionalities that could not have been developed with the amount of engineering resources we had as a small company.
We immediately decided to start an evaluation project by building a new product feature directly using MediaPipe to see if it could live up to all the promises.
To start the evaluation, we decided to migrate our existing moving object feature to see what exactly MediaPipe can do.
Our current Moving Object Detection pipeline consists of the following main components:
Our current implementation has taken GPU into consideration as much as we can. A series of shaders are created to perform the tasks above and the pipeline is designed to avoid moving pixels between CPU/GPU frequently to eliminate the potential performance hits.
The pipeline involves multiple ML models that are conditionally executed, mixed CPU/GPU processing, etc. All the challenges here make it a perfect showcase for how MediaPipe could help develop a complicated pipeline.
MediaPipe provides a lot of code samples for any developer to bootstrap with. We took the Object Detection on Android sample that comes with the project to start with because of the similarity with the back-end part of our pipeline. It did take us sometimes to fully understand the design concepts of MediaPipe and all the tools associated. But with the complete documentation and the great responsiveness from the MediaPipe team, we got up to speed soon to do most of the things we wanted.
That being said, there were a few challenges we needed to overcome on the road to full migration. Our original pipeline of Moving Object Detection takes the input frame asynchronously, but MediaPipe has timestamp bound limitations such that we cannot just show the result in an allochronic way. Meanwhile, we need to gather data through JNI in a specific data format. We came up with a workaround that conquered all the issues under the circumstances, which will be mentioned later.
After wrapping our models and the processing logics into calculators and wired them up, we have successfully transformed our existing implementation and created our first MediaPipe Moving Object Detection pipeline like the figure below, running on Android devices:
Fig.2 Moving Object Detection Graph
We do not block the video frame in the main calculation loop, and set the detection result as an input stream to show the annotation on the screen. The whole graph is designed as a multi-functioned process, the left chunk is the debug annotation and video frame output module, and the rest of the calculation occurs in the rest of the graph, e.g., low light detection, motion triggered detection, cropping of the area of interest and the detection process. In this way, the graph process will naturally separate into real-time display and asynchronous calculation.
As a result, we are able to complete a full processing for detection in under 40ms on a device with Snapdragon 660 chipset. MediaPipe’s tight integration with TensorFlow Lite provides us the flexibility to get even more performance gain by leveraging whatever acceleration techniques available (GPU or DSP) on the device.
The following figure shows the current implementation working in action:
Fig.3 Moving Object Detection running in Alfred Camera
After getting things to run on Android, Desktop GPU (OpenGL-ES) emulation was our next target to evaluate. We are already using OpenGL-ES shaders for some computer vision operations in our pipeline. Having the capability to develop the algorithm on desktop, seeing it work in action before deployment onto mobile platforms is a huge benefit to us. The feature was not ready at the time when the project was first released, but MediaPipe team had soon added Desktop GPU emulation support for Linux in follow-up releases to make this possible. We have used the capability to detect and fix some issues in the graphs we created even before we put things on the mobile devices. Although it currently only works on Linux, it is still a big leap forward for us.
Testing the algorithms and making sure they behave as expected is also a challenge for a camera application. MediaPipe helps us simplify this by using pre-recorded MP4 files as input so we could verify the behavior simply by replaying the files. There is also built-in profiling support that makes it easy for us to locate potential performance bottlenecks.
The result of the evaluation and the feedback from our engineering team were very positive and promising:
With just a few weeks working with MediaPipe, it has shown strong capabilities to fundamentally transform how we develop our products. Without MediaPipe we could have spent months creating the same features without the same level of performance.
Alfred Camera is designed to bring home security with AI to everyone, and MediaPipe has significantly made achieving that goal easier for our team. From Moving Object Detection to future AI-powered features, we are focusing on transforming a basic security camera use case into a smart housekeeper that can help provide even more context that our users care about. With the support of MediaPipe, we have been able to accelerate our development process and bring the features to the market at an unprecedented speed. Our team is really excited about how MediaPipe could help us progress and discover new possibilities, and is looking forward to the enhancements that are yet to come to the project.
We have invested heavily in our API and service infrastructure to improve performance and security and to add features developers need to build world-class APIs. As we make changes we must address features that are no longer compatible with the latest architecture and business requirements.
The JSON-RPC protocol (http://www.jsonrpc.org/specification) and Global HTTP Batch (example) are two such features. Our support for these features was based on an architecture using a single shared proxy to receive requests for all APIs. As we move towards a more distributed, high performance architecture where requests go directly to the appropriate API server we can no longer support these global endpoints.
Last month, Game Developers Conference (GDC) organizers made the difficult decision to postpone the conference. We understand this decision, as we have to prioritize the health and safety of our community. GDC is one of our most anticipated times of the year to connect with the gaming industry. Though we won’t be bringing the news in-person this year, we’re hosting the Google for Games Developer Summit, a free, digital-only experience where developers can watch the announcements and session content that was planned for GDC.
Google for Games Developer Summit
The Developer Summit kicks off on March 23rd at 9:00AM PT with our broadcasted keynote. Immediately following, we’ll be releasing a full lineup of developer sessions with over 10 hours of content to help take your games to the next level.
Here are some types of sessions to expect:
Sign up to stay informed at g.co/gamedevsummit.
Support for the game developer community
We recognize every developer is impacted differently by this situation. We’re coordinating with the GDC Relief Fund to sponsor and assist developers who’ve invested in this moment to further grow their games.
We also understand many developers were looking forward to sharing their content with peers. To help with this, developers can use YouTube to stream events from small to large using tools like Live Streaming and Premieres.
We can’t wait to share what we have in store for gaming. Discover the solutions our teams have been building to support the success of this community for years to come.
Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud
Students graduating from STEM majors at universities with development experience using industry APIs (application programming interfaces) have real-world practice that can prove valuable in terms of career readiness.
To that end, the Google Cloud team is creating a "Google Cloud for Student Developers" YouTube video series crafted specifically for the student developer audience.
While viewable by developers with any experience with Google Cloud, this series focuses on developing skills that will help student developers in their future careers. Google Cloud includes a pair of well-known product groups, Google Cloud Platform (GCP) as well as G Suite. While most equate GCP for developers and G Suite for users, many don't know that behind each G Suite application like Gmail, Google Drive, Calendar, Docs, Sheets, and Slides, are developer APIs.
The Google Cloud higher education team is happy to announce the first of a 5-episode mini-series to kickoff the video collection that shows student developers how they can code G Suite, starting with this first one introducing the G Suite developer landscape. Viewers will hear about the HTTP-based RESTful APIs as well as Google Apps Script, a serverless higher-level development environment that allows for automation, extension of G Suite app functionality, as well as integration of your apps with Gmail, Drive, Calendar, Docs, Sheets, Slides, and many more G Suite, Google, and even external services.
Succeeding episodes dig deeper into the RESTful APIs as well as Apps Script, with the final pair of videos showing students full-fledged apps they can build with G Suite developer tools. To learn more about integrating with G Suite, see its top-level documentation site and overview page as well as the set of all G Suite developer videos. Also stay tuned for new episodes in the series that focus on GCP developer tools. We look forward to seeing what you can build with G Suite, but also with GCP as well… or both at the same time!
Posted by Google Developer Studio
Today is International Women’s Day and we’re kicking off the celebration with a profile series featuring 20 tech trailblazers who have made significant contributions to the developer community. Many of the women we spoke to work directly with some of our educational outreach and inclusivity programs like Google Developer Groups and Women Techmakers, while others are Google Developers Experts or Googlers who do amazing work around the globe. One thing they all have in common is a dedication to making the developer community more approachable and inclusive for generations of women to come.
Read the interviews below to learn more about these amazing individuals whose passion and drive contribute to a better workplace and world.
We’re proud to celebrate #IWD2020 with them.
Editor’s Note: Guest authors Diego Moreno and Sophia Deng (@sophdeng) are from Gigster, a firm that builds dynamic teams made of top global talent who create industry-changing custom software.
Google Cloud provides multiple services for gathering and managing data. Google Forms paired with Google Sheets are quite popular as they require no engineering resources while being incredibly powerful, providing storage of up to 5 million rows of data and built-in analytics for small team projects.
At the other end of the spectrum, to support a high volume of users or data, Google Cloud provides advanced serverless platforms like Google App Engine (web app-hosting) and Google Cloud Functions (function/service-hosting) that can use Google Cloud Firestore for fast and scalable data storage. These are perfect for professional engineering teams that need autoscaling to respond to any level of user traffic and data input. Such apps can also be packaged into a container and deployed serverlessly on Google Cloud Run.
However, it's quite possible your needs are right in-between. Today, we're happy to present the Gigster story and their innovative use of Google Apps Script—a highly-accessible service conventionally relegated to simple macro and add-on development, but which Gigster used to its advantage, building robust systems to transform their internal operations. Apps Script is also serverless, meaning Gigster didn't have to manage any servers for their application nor did they need to find a place to host its source code.
Gigster enables distributed teams of software engineers, product managers and designers to build software applications for enterprise clients. Over the past five years, Gigster has delivered thousands of projects, all with distributed software teams. Our group, the Gigster Staffing Operations Team, is responsible for assembling these teams from Gigster’s network of over 1,000 freelancers.
Two years ago, our team began building custom software to automate the multi-stage and highly manual team staffing process. Building internal software has allowed the same-size Staffing Operations Team (3 members!) to enjoy a 60x reduction in time spent staffing each role.
The Apps Script ecosystem has emerged as the most critical component in our toolkit for building this internal software, due to its versatility and ease of deployment. We want to share how one piece of the staffing process has evolved to become more powerful over time thanks to Apps Script. Ultimately, we hope that sharing this journey enables all types of teams to build their own tools and unlock new possibilities.
Staffing is an operationally intensive procedure. Just finding willing and able candidates requires numerous steps:
To add complexity, many of these steps require working with different third-party applications. For awhile, we performed every step manually, tracking every piece of data generated in one central Sheet (the “Staffing Broadcast Google Sheet”). At a certain point, this back-and-forth work to log data from numerous applications became unsustainable. Although we leveraged Google Sheets features like Data Validation rules and filters, the Staffing Broadcast Sheet could not alleviate the high degree of manual processes that were required of the team.
The centralized Staffing Broadcast Google Sheet provided organization, but required a high degree of manual entry for tracking candidate decisions.
The key transformation was integrating Sheets data with third-party APIs via Apps Script. This enabled us to cut out the most time-consuming operations. We no longer had to flip between applications to message candidates, wait for their replies, and then manually track responses.
To interact with these APIs, we built a user interface directly into the Staffing Broadcast Google Sheet. By introducing an information module, as well as drop-down lists and buttons, we were able to define a small set of manual actions versus the much wider list of tasks the tool would perform automatically across multiple applications.
By integrating Google Apps Script with third-party APIs and creating a user interface, we evolved the Staffing Broadcast Tool to centralize and automate almost every step of the staffing process.
doPost() is the key function in our staffing tool that facilitates third-party services triggering our Apps Script automations. Below is a snippet of how we listened to candidates' responses from a third-party messaging application. In this case, queueing the third-party message in a Google Sheet so it can be processed with improved error-handling.
doPost()
/** * Receive POST requests and record to queue. */ doPost(e) { var payload = e.postData.contents; SpreadsheetApp.openById(SPREADSHEET_ID) .getSheetByName("Unprocessed") .appendRow([payload]); return ContentService.createTextOutput(""); // Return 200 }
Almost all manual work associated with finding candidates was automated through the combination of integrations with third-party APIs and having a user interface to perform a small, defined set of actions. Our team’s day-to-day became shockingly simple: select candidates to receive messages within the Staffing Broadcast Tool, then click the “Send Broadcast” button. That’s it. The tool handled everything else.
Leveraging Sheets as our foundation, we fundamentally transformed our spreadsheet into a custom software application. The spreadsheet went from a partially automated datastore to a tool that provided an end-to-end automated solution, requiring only the click of a few buttons to execute.
While satisfied, we understood that having our application live in Google Sheets had its limitations, namely, it was difficult for multiple team members to simultaneously use the tool. Using doGet(), the sibling to doPost(), we began building an HTML frontend to the Staffing Broadcast Tool. In addition to resolving difficulties related to multiple users being in a spreadsheet, it also allowed us to build an easier-to-use and more responsive tool by leveraging Bootstrap & jQuery.
doGet()
Having multiple users in a single Google Sheet can create conflicts, but Apps Script allowed us to build a responsive web app leveraging common libraries like Bootstrap & jQuery that eliminated those problems while providing an improved user experience.
When other teams at Gigster got wind of what we built, it was easy to grant access to others beyond the Staffing Operations Team. Since Apps Script is part of the G Suite developer ecosystem, we relied on Google’s security policies to help deploy our tools to larger audiences.
While this can be done through Google’s conventional sharing tools, it can also be done with built-in Apps Script functions like Session.getActiveUser() that allow us to restrict access to specific Google users. In our case, those within our organization plus a few select users.
To this day, we continue to use this third version of the Staffing Broadcast Tool in our daily operations as it supports 100% of all client projects at Gigster.
By fundamentally transforming the Staffing Broadcast Tool with Apps Script, Gigster’s Staffing Operations Team increased its efficiency while supporting the growth of our company. Inspired by these business benefits, we applied this application-building approach using Apps Script for multiple tools, including candidate searching, new user onboarding, and countless automations.
Our team’s psychological shift about how we view what we are capable of, especially as a non-engineering team, has been the most valuable part of this journey. By leveraging an already familiar ecosystem to build our own software, we have freed team members to become more self-sufficient and valuable to our customers.
To get started on your Apps Script journey, we recommend you check out the Apps Script Fundamentals playlist and the official documentation. And if you're a freelancer looking to build software applications for clients, we’re always looking for talented software engineers, product managers or designers to join Gigster’s Talent Network.
Thank you to Sandrine Bitton, the third member of the Staffing Operations Team, for all her help in the development of the Staffing Broadcast Tool.
Posted by the Google for Games Team
Last Friday, GDC 2020 organizers made the difficult decision to postpone the conference. We understand this decision, as we have to prioritize the health and safety of our community.
Every year, we look forward to the Game Developers Conference and surrounding events because it gives our teams a chance to connect with game developers, partners, and friends in the industry.
Although we won’t be connecting in-person this year, we’re still excited to share the latest announcements from Google with everyone through our digital experience. We'll be sharing plans for our digital experience in the coming days.
Thank you to all who keep this community thriving and check back soon at g.co/gdc2020 for more details.
In 2019, powerful conversations on how to solve climate change took place all over the world. In that spirit, the DevFest community recently joined the discussion by looking at how tech can change how we approach the environment. For our new readers, DevFests are community-led developer events, hosted by Google Developer Groups, that take place all around the world and are focused on building community around Google’s technologies.
From DevFests in the rainforests of Brazil to the Mediterranean Sea, our community has come together to take part in a high-powered exchange of ideas, focused on how to solve for the environment. Out of these DevFests have come 3 new solutions, aimed at using tech to create eco-friendly economies, safer seas, and care for crops. Check them out below!
Blockchain for Biodiversity comes out of the “Jungle’s DevFest”, an event hosted by GDG Manaus that took place in the Brazilian Rainforest with over 1,000 participants. Imagined by Barbara Schorchit’s startup GeneCoin, the idea focuses on using blockchain based solutions to track how “biodiversity friendly” different products and companies are. Her goal is to help create a more eco-friendly economy by providing consumers with information to hold companies accountable for their environmental footprint. This tool for tracking environmentally conscious products will allow consumers to better understand what materials goods are made from, what energy sources were used to produce them, and how their creation will impact climate change. Welcome to a whole new take on purchasing power.
The ECHO Marine Station comes from a partnership between the Italian Coast Guard and DevFest Mediterranean hosted by GDG Nebrodi, Palermo, and Gela. The marine vehicle was created with the hope of becoming the “space station of the seas” and is rechargeable via solar panels, equipped with low-consumption electric motors, and does not create waves or noise - so as to not disturb marine life. The DevFest team took the lead on developing the software and hardware for the station with the intention of creating a tool that can analyze and monitor any marine environment.
At the DevFest, two innovative ideas were proposed on how to best purpose the marine station. The first was to program it to collect radioactive ions released by nuclear power plants, with the goal of analyzing and eventually disposing the waste. The second was to program the station to carry out expeditions to collect water samples, analyze pollution levels, and report dangerous conditions back to the coast guard.
With DevFest and the Italian Coast Guard working together, the ECHO Marine Station is ready to change how we save our seas.
Doctor’s Eyes for Plants is an idea that came out of DevFest Vietnam’s “Green Up Your Code event”, hosted by GDG MienTrung, Vietnam. You can watch highlights from their event here. Created by local students, the program uses a public dataset with Tensorflow to train a region-convolutional neural network to recognize 6 diseases on 3 different plant species, all in 48 hours. The team was recently able to apply the technology to rice plants, which could revolutionize the world’s capacity and efficiency for growing the crop. The rewards of this technology could be unprecedented. The creation of a healthier eco-system paired with the chance to feed more people? Now that’s high impact.
Inspired by these stories? Find a Google Developer Groups community near you, at developers.google.com/community/gdg/
Today we're announcing some improvements to the review process for smart home Actions, aimed at making the experience more transparent in the Actions console and reducing the time it takes for your Actions to get approved.
The smart home review process is slightly different from other Actions on Google projects because it has two separate phases: a policy review and a certification review. All Actions undergo a policy review, which verifies that your Action follows the policy guidelines for Actions on Google. Smart home Actions go through certification review for additional quality assurance validation. Certification reviewers verify the content you provide through the certification request form, including your test suite for smart home results.
Glass Enterprise Edition 2 has helped people working in logistics, manufacturing, field services and a variety of other industries do their jobs more efficiently through hands-free access to the information and tools they need to do their job. Enterprises who have deployed Glass with experiences built by our network of solution providers, have seen faster production times, improved quality, and reduced costs.
Since Glass Enterprise Edition 2 launched last May, we’ve seen strong demand from developers and businesses who are interested in building new, helpful enterprise solutions for Glass. In order to make it easier for them to start working with Glass, they can now purchase devices directly from one of our hardware resellers, such as CDW, Mobile Advance or SHI.
Glass Enterprise Edition 2 is built on Android, so it’s easy for developers to work with, and for businesses to integrate the services and APIs (application programming interfaces) they already use. We’ve also shared new open source applications and code samples, including sample layouts and UI components that may be helpful examples for those just getting started developing for Glass.
We’re excited to see what kinds of new experiences and solutions developers will make for Glass to shape the future of work.
