Last year we introduced the developer preview of the Local Home SDK, a suite of local technologies to enhance your smart home integration with Google Assistant by adding local fulfillment. Since then, we've been hard at work incorporating your feedback and getting the experience ready for production. Starting today, we're exiting developer preview and allowing you to submit local fulfillment apps along with your smart home Action through the Actions console using Local Home SDK v1.0.
Adding local fulfillment for your smart home Action.
As part of the Smart Home platform, local fulfillment extends your smart home Action and routes commands to devices through the local network, benefitting users with reduced latency and higher reliability. If a local path cannot be successfully established, commands fall back to your cloud fulfillment.
The Local Home SDK v1.0 supports discovery of local devices over Wi-Fi using the mDNS, UDP, or UPnP protocols. Once a local path is established, apps can send commands to devices using TCP, UDP, or HTTP. For more details on the API changes in SDK v1.0, check out the changelog.
New multi-scan configuration UI.
The Local Home SDK configuration page in the Actions console now accepts JavaScript bundles for your local fulfillment app. When you are ready to publish your app, upload your JavaScript files to the console and submit your Action. For more details on submitting your smart home Action for review, see the smart home launch guide.
Upload your local fulfillment app.
We've updated the test suite for smart home to support local fulfillment as well. Be sure to self-test your local fulfillment before submitting your updated smart home Action for review. You must provide updated test suite results with your certification request when you submit.
To learn more about enhancing your smart home Actions with local fulfillment, check out the Introduction to Local Home SDK and the developer guide. Build your first local fulfillment app with the codelab, and go deeper with the samples and API reference.
We want to hear from you, so continue sharing your feedback with us through the issue tracker, and engage with other smart home developers in the /r/GoogleAssistantDev community. Follow @ActionsOnGoogle on Twitter for more of our team's updates, and tweet using #AoGDevs to share what you’re working on. We can’t wait to see what you build!
Calling all student developers: If you’re someone who wants to lead, is passionate about technology, loves problem-solving, and is driven to give back to your community, then Developer Student Clubs has a home for you. Interest forms for the upcoming 2020-2021 academic year are now available. Ready to dive in? Get started at goo.gle/dsc-leads.
Want to know more? Check out these details below.
What are Developer Student Clubs?
Developer Student Clubs (DSC) are university based community groups for students interested in Google developer technologies. With programs that meet in person and online, students from all undergraduate and graduate programs with an interest in growing as a developer are welcome. By joining a DSC, students grow their knowledge in a peer-to-peer learning environment and build solutions for local businesses and their community.
Why should I join?
- Grow your skills as a developer with training content from Google.
- Think of your own project, then lead a team of your peers to scale it.
- Build prototypes and solutions for local problems.
- Participate in a global developer competition.
- Receive access to select Google events and conferences.
- Gain valuable experience
Is there a Developer Student Club near me?
Developer Student Clubs are now in 68+ countries with 860+ groups. Find a club near you or learn how to start your own, here.
When do I need to submit the interest form?
You may express interest through the form until May 15th, 11:59pm PST. Get started here.
Make sure to learn more about our program criteria.
Our DSC Leads are working on meaningful projects around the world. Watch this video of how one lead worked to protect her community from dangerous floods in Indonesia. Similarly, read this story of how another lead helped modernize healthcare in Uganda.
We’re looking forward to welcoming a new group of leads to Developer Student Clubs. Have a friend who you think is a good fit? Pass this article along. Wishing all developer students the best on the path towards building great products and community.
*Developer Student Clubs are student-led independent organizations, and their presence does not indicate a relationship between Google and the students' universities.
At Google Pay, we’re always looking for ways to make things simple, helpful, and accessible for everyone, whether that’s consumers or developers. Today, we’re introducing a new resource for developers that does just that — the Business Console for Google Pay. The Business Console is a new tool that streamlines the way you integrate Google Pay into your apps and websites.
Many of you have already added support for Google Pay. In the process, you asked questions like:
We created the Business Console for Google Pay in response to your feedback. With the new console, you’ll be able to integrate Google Pay into your apps and websites more seamlessly, discover resources, get support at different stages throughout your integration, and keep track of your progress along the way.
And this is only the beginning. As we add new features, the Business Console will be your go-to place to manage all your new and existing integrations with Google Pay, see how your integrations perform over time, and add support for other business- and developer-focused products.
The new Business Console lets you simplify your Google Pay integrations by guiding you during the submission for approval and helping you keep track of progress.
Getting started is easy. Just head to pay.google.com/business/console. If you’ve already integrated with Google Pay, log in with your account to see your existing integrations or create new ones. And if you haven’t integrated with Google Pay yet, simply create your business profile, build an integration, and submit it for approval directly from the console.
Some businesses, system integrators, and developers have already started using the Business Console as part of our early-access program. “The new Google Pay Business Console helped us understand the integration requirements, and the examples made it easy to implement the Google Pay API into our website,” Gymondo GmbH CTO Christopher Weiss said. The Business Console also helped Weiss get their integration approved quickly. “Shortly after,” Weiss said, “we started seeing purchases coming from our customers paying with Google Pay."
We hope the new console makes your integration process go just as smoothly, and we’d love to hear about your experience. You can share any feedback from the menu within the console. We’re looking forward to learning how we can make Google Pay even more helpful in the future.
Recently, we introduced the "Google Cloud for Student Developers" video series to encourage students majoring in STEM fields to gain development experience using industry APIs (application programming interfaces) for career readiness. That first episode provided an overview of the G Suite developer landscape while this episode dives deeper, introducing G Suite's HTTP-based RESTful APIs, starting with Google Drive.
The first code sample has a corresponding codelab (a self-paced, hands-on tutorial) where you can build a simple Python script that displays the first 100 files or folders in your Google Drive. The codelab helps student (and professional) developers...
Please note that the information, uses, and applications expressed in the below post are solely those of our guest author, Alfred Camera.
In this article, we’d like to give you a short overview of Alfred Camera and our experience of using MediaPipe to transform our moving object feature, and how MediaPipe has helped to get things easier to achieve our goals.
Fig.1 Alfred Camera Logo
Alfred Camera is a smart home app for both Android and iOS devices, with over 15 million downloads worldwide. By downloading the app, users are able to turn their spare phones into security cameras and monitors directly, which allows them to watch their homes, shops, pets anytime. The mission of Alfred Camera is to provide affordable home security so that everyone can find peace of mind in this busy world.
The Alfred Camera team is composed of professionals in various fields, including an engineering team with several machine learning and computer vision experts. Our aim is to integrate AI technology into devices that are accessible to everyone.
Alfred Camera currently has a feature called Moving Object Detection, which continuously uses the device’s camera to monitor a target scene. Once it identifies a moving object in the area, the app will begin recording the video and send notifications to the device owner. The machine learning models for detection are hand-crafted and trained by our team using TensorFlow, and run on TensorFlow Lite with good performance even on mid-tier devices. This is important because the app is leveraging old phones and we'd like the feature to reach as many users as possible.
We had started building our AI features at Alfred Camera since 2017. In order to have a solid foundation to support our AI feature requirements for the coming years, we decided to rebuild our real-time video analysis pipeline. At the beginning of the project, the goals were to create a new pipeline which should be 1) modular enough so we could swap core algorithms easily with minimal changes in other parts of the pipeline, 2) having GPU acceleration designed in place, 3) cross-platform as much as possible so there’s no need to create/maintain separate implementations for different platforms. Based on the goals, we had surveyed several open source projects that had the potential but we ended up using none of them as they either fell short on the features or were not providing the readiness/stabilities that we were looking for.
We started a small team to prototype on those goals first for the Android platform. What came later were some tough challenges way above what we originally anticipated. We ran into several major design changes as some key design basics were overlooked. We needed to implement some utilities to do things that sounded trivial but required significant effort to make it right and fast. Dealing with asynchronous processing also led us into a bunch of timing issues, which took the team quite some effort to address. Not to mention debugging on real devices was extremely inefficient and painful.
Things didn't just stop here. Our product is also on iOS and we had to tackle these challenges once again. Moreover, discrepancies in the behavior between the platform-specific implementations introduced additional issues that we needed to resolve.
Even though we finally managed to get the implementations to the confidence level we wanted, that was not a very pleasant experience and we have never stopped thinking if there is a better option.
Google open sourced MediaPipe project in June 2019 and it immediately caught our attention. We were surprised by how it is perfectly aligned with the previous goals we set, and has functionalities that could not have been developed with the amount of engineering resources we had as a small company.
We immediately decided to start an evaluation project by building a new product feature directly using MediaPipe to see if it could live up to all the promises.
To start the evaluation, we decided to migrate our existing moving object feature to see what exactly MediaPipe can do.
Our current Moving Object Detection pipeline consists of the following main components:
Our current implementation has taken GPU into consideration as much as we can. A series of shaders are created to perform the tasks above and the pipeline is designed to avoid moving pixels between CPU/GPU frequently to eliminate the potential performance hits.
The pipeline involves multiple ML models that are conditionally executed, mixed CPU/GPU processing, etc. All the challenges here make it a perfect showcase for how MediaPipe could help develop a complicated pipeline.
MediaPipe provides a lot of code samples for any developer to bootstrap with. We took the Object Detection on Android sample that comes with the project to start with because of the similarity with the back-end part of our pipeline. It did take us sometimes to fully understand the design concepts of MediaPipe and all the tools associated. But with the complete documentation and the great responsiveness from the MediaPipe team, we got up to speed soon to do most of the things we wanted.
That being said, there were a few challenges we needed to overcome on the road to full migration. Our original pipeline of Moving Object Detection takes the input frame asynchronously, but MediaPipe has timestamp bound limitations such that we cannot just show the result in an allochronic way. Meanwhile, we need to gather data through JNI in a specific data format. We came up with a workaround that conquered all the issues under the circumstances, which will be mentioned later.
After wrapping our models and the processing logics into calculators and wired them up, we have successfully transformed our existing implementation and created our first MediaPipe Moving Object Detection pipeline like the figure below, running on Android devices:
Fig.2 Moving Object Detection Graph
We do not block the video frame in the main calculation loop, and set the detection result as an input stream to show the annotation on the screen. The whole graph is designed as a multi-functioned process, the left chunk is the debug annotation and video frame output module, and the rest of the calculation occurs in the rest of the graph, e.g., low light detection, motion triggered detection, cropping of the area of interest and the detection process. In this way, the graph process will naturally separate into real-time display and asynchronous calculation.
As a result, we are able to complete a full processing for detection in under 40ms on a device with Snapdragon 660 chipset. MediaPipe’s tight integration with TensorFlow Lite provides us the flexibility to get even more performance gain by leveraging whatever acceleration techniques available (GPU or DSP) on the device.
The following figure shows the current implementation working in action:
Fig.3 Moving Object Detection running in Alfred Camera
After getting things to run on Android, Desktop GPU (OpenGL-ES) emulation was our next target to evaluate. We are already using OpenGL-ES shaders for some computer vision operations in our pipeline. Having the capability to develop the algorithm on desktop, seeing it work in action before deployment onto mobile platforms is a huge benefit to us. The feature was not ready at the time when the project was first released, but MediaPipe team had soon added Desktop GPU emulation support for Linux in follow-up releases to make this possible. We have used the capability to detect and fix some issues in the graphs we created even before we put things on the mobile devices. Although it currently only works on Linux, it is still a big leap forward for us.
Testing the algorithms and making sure they behave as expected is also a challenge for a camera application. MediaPipe helps us simplify this by using pre-recorded MP4 files as input so we could verify the behavior simply by replaying the files. There is also built-in profiling support that makes it easy for us to locate potential performance bottlenecks.
The result of the evaluation and the feedback from our engineering team were very positive and promising:
With just a few weeks working with MediaPipe, it has shown strong capabilities to fundamentally transform how we develop our products. Without MediaPipe we could have spent months creating the same features without the same level of performance.
Alfred Camera is designed to bring home security with AI to everyone, and MediaPipe has significantly made achieving that goal easier for our team. From Moving Object Detection to future AI-powered features, we are focusing on transforming a basic security camera use case into a smart housekeeper that can help provide even more context that our users care about. With the support of MediaPipe, we have been able to accelerate our development process and bring the features to the market at an unprecedented speed. Our team is really excited about how MediaPipe could help us progress and discover new possibilities, and is looking forward to the enhancements that are yet to come to the project.
We have invested heavily in our API and service infrastructure to improve performance and security and to add features developers need to build world-class APIs. As we make changes we must address features that are no longer compatible with the latest architecture and business requirements.
The JSON-RPC protocol (http://www.jsonrpc.org/specification) and Global HTTP Batch (example) are two such features. Our support for these features was based on an architecture using a single shared proxy to receive requests for all APIs. As we move towards a more distributed, high performance architecture where requests go directly to the appropriate API server we can no longer support these global endpoints.
Last month, Game Developers Conference (GDC) organizers made the difficult decision to postpone the conference. We understand this decision, as we have to prioritize the health and safety of our community. GDC is one of our most anticipated times of the year to connect with the gaming industry. Though we won’t be bringing the news in-person this year, we’re hosting the Google for Games Developer Summit, a free, digital-only experience where developers can watch the announcements and session content that was planned for GDC.
Google for Games Developer Summit
The Developer Summit kicks off on March 23rd at 9:00AM PT with our broadcasted keynote. Immediately following, we’ll be releasing a full lineup of developer sessions with over 10 hours of content to help take your games to the next level.
Here are some types of sessions to expect:
Sign up to stay informed at g.co/gamedevsummit.
Support for the game developer community
We recognize every developer is impacted differently by this situation. We’re coordinating with the GDC Relief Fund to sponsor and assist developers who’ve invested in this moment to further grow their games.
We also understand many developers were looking forward to sharing their content with peers. To help with this, developers can use YouTube to stream events from small to large using tools like Live Streaming and Premieres.
We can’t wait to share what we have in store for gaming. Discover the solutions our teams have been building to support the success of this community for years to come.
Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud
Students graduating from STEM majors at universities with development experience using industry APIs (application programming interfaces) have real-world practice that can prove valuable in terms of career readiness.
To that end, the Google Cloud team is creating a "Google Cloud for Student Developers" YouTube video series crafted specifically for the student developer audience.
While viewable by developers with any experience with Google Cloud, this series focuses on developing skills that will help student developers in their future careers. Google Cloud includes a pair of well-known product groups, Google Cloud Platform (GCP) as well as G Suite. While most equate GCP for developers and G Suite for users, many don't know that behind each G Suite application like Gmail, Google Drive, Calendar, Docs, Sheets, and Slides, are developer APIs.
The Google Cloud higher education team is happy to announce the first of a 5-episode mini-series to kickoff the video collection that shows student developers how they can code G Suite, starting with this first one introducing the G Suite developer landscape. Viewers will hear about the HTTP-based RESTful APIs as well as Google Apps Script, a serverless higher-level development environment that allows for automation, extension of G Suite app functionality, as well as integration of your apps with Gmail, Drive, Calendar, Docs, Sheets, Slides, and many more G Suite, Google, and even external services.
Succeeding episodes dig deeper into the RESTful APIs as well as Apps Script, with the final pair of videos showing students full-fledged apps they can build with G Suite developer tools. To learn more about integrating with G Suite, see its top-level documentation site and overview page as well as the set of all G Suite developer videos. Also stay tuned for new episodes in the series that focus on GCP developer tools. We look forward to seeing what you can build with G Suite, but also with GCP as well… or both at the same time!
Posted by Google Developer Studio
Today is International Women’s Day and we’re kicking off the celebration with a profile series featuring 20 tech trailblazers who have made significant contributions to the developer community. Many of the women we spoke to work directly with some of our educational outreach and inclusivity programs like Google Developer Groups and Women Techmakers, while others are Google Developers Experts or Googlers who do amazing work around the globe. One thing they all have in common is a dedication to making the developer community more approachable and inclusive for generations of women to come.
Read the interviews below to learn more about these amazing individuals whose passion and drive contribute to a better workplace and world.
We’re proud to celebrate #IWD2020 with them.
Editor’s Note: Guest authors Diego Moreno and Sophia Deng (@sophdeng) are from Gigster, a firm that builds dynamic teams made of top global talent who create industry-changing custom software.
Google Cloud provides multiple services for gathering and managing data. Google Forms paired with Google Sheets are quite popular as they require no engineering resources while being incredibly powerful, providing storage of up to 5 million rows of data and built-in analytics for small team projects.
At the other end of the spectrum, to support a high volume of users or data, Google Cloud provides advanced serverless platforms like Google App Engine (web app-hosting) and Google Cloud Functions (function/service-hosting) that can use Google Cloud Firestore for fast and scalable data storage. These are perfect for professional engineering teams that need autoscaling to respond to any level of user traffic and data input. Such apps can also be packaged into a container and deployed serverlessly on Google Cloud Run.
However, it's quite possible your needs are right in-between. Today, we're happy to present the Gigster story and their innovative use of Google Apps Script—a highly-accessible service conventionally relegated to simple macro and add-on development, but which Gigster used to its advantage, building robust systems to transform their internal operations. Apps Script is also serverless, meaning Gigster didn't have to manage any servers for their application nor did they need to find a place to host its source code.
Gigster enables distributed teams of software engineers, product managers and designers to build software applications for enterprise clients. Over the past five years, Gigster has delivered thousands of projects, all with distributed software teams. Our group, the Gigster Staffing Operations Team, is responsible for assembling these teams from Gigster’s network of over 1,000 freelancers.
Two years ago, our team began building custom software to automate the multi-stage and highly manual team staffing process. Building internal software has allowed the same-size Staffing Operations Team (3 members!) to enjoy a 60x reduction in time spent staffing each role.
The Apps Script ecosystem has emerged as the most critical component in our toolkit for building this internal software, due to its versatility and ease of deployment. We want to share how one piece of the staffing process has evolved to become more powerful over time thanks to Apps Script. Ultimately, we hope that sharing this journey enables all types of teams to build their own tools and unlock new possibilities.
Staffing is an operationally intensive procedure. Just finding willing and able candidates requires numerous steps:
To add complexity, many of these steps require working with different third-party applications. For awhile, we performed every step manually, tracking every piece of data generated in one central Sheet (the “Staffing Broadcast Google Sheet”). At a certain point, this back-and-forth work to log data from numerous applications became unsustainable. Although we leveraged Google Sheets features like Data Validation rules and filters, the Staffing Broadcast Sheet could not alleviate the high degree of manual processes that were required of the team.
The centralized Staffing Broadcast Google Sheet provided organization, but required a high degree of manual entry for tracking candidate decisions.
The key transformation was integrating Sheets data with third-party APIs via Apps Script. This enabled us to cut out the most time-consuming operations. We no longer had to flip between applications to message candidates, wait for their replies, and then manually track responses.
To interact with these APIs, we built a user interface directly into the Staffing Broadcast Google Sheet. By introducing an information module, as well as drop-down lists and buttons, we were able to define a small set of manual actions versus the much wider list of tasks the tool would perform automatically across multiple applications.
By integrating Google Apps Script with third-party APIs and creating a user interface, we evolved the Staffing Broadcast Tool to centralize and automate almost every step of the staffing process.
doPost() is the key function in our staffing tool that facilitates third-party services triggering our Apps Script automations. Below is a snippet of how we listened to candidates' responses from a third-party messaging application. In this case, queueing the third-party message in a Google Sheet so it can be processed with improved error-handling.
doPost()
/** * Receive POST requests and record to queue. */ doPost(e) { var payload = e.postData.contents; SpreadsheetApp.openById(SPREADSHEET_ID) .getSheetByName("Unprocessed") .appendRow([payload]); return ContentService.createTextOutput(""); // Return 200 }
Almost all manual work associated with finding candidates was automated through the combination of integrations with third-party APIs and having a user interface to perform a small, defined set of actions. Our team’s day-to-day became shockingly simple: select candidates to receive messages within the Staffing Broadcast Tool, then click the “Send Broadcast” button. That’s it. The tool handled everything else.
Leveraging Sheets as our foundation, we fundamentally transformed our spreadsheet into a custom software application. The spreadsheet went from a partially automated datastore to a tool that provided an end-to-end automated solution, requiring only the click of a few buttons to execute.
While satisfied, we understood that having our application live in Google Sheets had its limitations, namely, it was difficult for multiple team members to simultaneously use the tool. Using doGet(), the sibling to doPost(), we began building an HTML frontend to the Staffing Broadcast Tool. In addition to resolving difficulties related to multiple users being in a spreadsheet, it also allowed us to build an easier-to-use and more responsive tool by leveraging Bootstrap & jQuery.
doGet()
Having multiple users in a single Google Sheet can create conflicts, but Apps Script allowed us to build a responsive web app leveraging common libraries like Bootstrap & jQuery that eliminated those problems while providing an improved user experience.
When other teams at Gigster got wind of what we built, it was easy to grant access to others beyond the Staffing Operations Team. Since Apps Script is part of the G Suite developer ecosystem, we relied on Google’s security policies to help deploy our tools to larger audiences.
While this can be done through Google’s conventional sharing tools, it can also be done with built-in Apps Script functions like Session.getActiveUser() that allow us to restrict access to specific Google users. In our case, those within our organization plus a few select users.
To this day, we continue to use this third version of the Staffing Broadcast Tool in our daily operations as it supports 100% of all client projects at Gigster.
By fundamentally transforming the Staffing Broadcast Tool with Apps Script, Gigster’s Staffing Operations Team increased its efficiency while supporting the growth of our company. Inspired by these business benefits, we applied this application-building approach using Apps Script for multiple tools, including candidate searching, new user onboarding, and countless automations.
Our team’s psychological shift about how we view what we are capable of, especially as a non-engineering team, has been the most valuable part of this journey. By leveraging an already familiar ecosystem to build our own software, we have freed team members to become more self-sufficient and valuable to our customers.
To get started on your Apps Script journey, we recommend you check out the Apps Script Fundamentals playlist and the official documentation. And if you're a freelancer looking to build software applications for clients, we’re always looking for talented software engineers, product managers or designers to join Gigster’s Talent Network.
Thank you to Sandrine Bitton, the third member of the Staffing Operations Team, for all her help in the development of the Staffing Broadcast Tool.
