Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud
In the previous episode of the Serverless Migration Station video series, developers learned how to containerize their App Engine code for Cloud Run using Docker. While Docker has gained popularity over the past decade, not everyone has containers integrated into their daily development workflow, and some prefer "containerless" solutions but know that containers can be beneficial. Well today's video is just for you, showing how you can still get your apps onto Cloud Run, even If you don't have much experience with Docker, containers, nor Dockerfiles.
Dockerfile
App Engine isn't going away as Google has expressed long-term support for legacy runtimes on the platform, so those who prefer source-based deployments can stay where they are so this is an optional migration. Moving to Cloud Run is for those who want to explicitly move to containerization.
Migrating to Cloud Run with Cloud Buildpacks video
So how can apps be containerized without Docker? The answer is buildpacks, an open-source technology that makes it fast and easy for you to create secure, production-ready container images from source code, without a Dockerfile. Google Cloud Buildpacks adheres to the buildpacks open specification and allows users to create images that run on all GCP container platforms: Cloud Run (fully-managed), Anthos, and Google Kubernetes Engine (GKE). If you want to containerize your apps while staying focused on building your solutions and not how to create or maintain Dockerfiles, Cloud Buildpacks is for you.
In the last video, we showed developers how to containerize a Python 2 Cloud NDB app as well as a Python 3 Cloud Datastore app. We targeted those specific implementations because Python 2 users are more likely to be using App Engine's ndb or Cloud NDB to connect with their app's Datastore while Python 3 developers are most likely using Cloud Datastore. Cloud Buildpacks do not support Python 2, so today we're targeting a slightly different audience: Python 2 developers who have migrated from App Engine ndb to Cloud NDB and who have ported their apps to modern Python 3 but now want to containerize them for Cloud Run.
ndb
Developers familiar with App Engine know that a default HTTP server is provided by default and started automatically, however if special launch instructions are needed, users can add an entrypoint directive in their app.yaml files, as illustrated below. When those App Engine apps are containerized for Cloud Run, developers must bundle their own server and provide startup instructions, the purpose of the ENTRYPOINT directive in the Dockerfile, also shown below.
entrypoint
app.yaml
ENTRYPOINT
Starting your web server with App Engine (app.yaml) and Cloud Run with Docker (Dockerfile) or Buildpacks (Procfile)
Procfile
In this migration, there is no Dockerfile. While Cloud Buildpacks does the heavy-lifting, determining how to package your app into a container, it still needs to be told how to start your service. This is exactly what a Procfile is for, represented by the last file in the image above. As specified, your web server will be launched in the same way as in app.yaml and the Dockerfile above; these config files are deliberately juxtaposed to expose their similarities.
Other than this swapping of configuration files and the expected lack of a .dockerignore file, the Python 3 Cloud NDB app containerized for Cloud Run is nearly identical to the Python 3 Cloud NDB App Engine app we started with. Cloud Run's build-and-deploy command (gcloud run deploy) will use a Dockerfile if present but otherwise selects Cloud Buildpacks to build and deploy the container image. The user experience is the same, only without the time and challenges required to maintain and debug a Dockerfile.
.dockerignore
gcloud run deploy
If you're considering containerizing your App Engine apps without having to know much about containers or Docker, we recommend you try this migration on a sample app like ours before considering it for yours. A corresponding codelab leading you step-by-step through this exercise is provided in addition to the video which you can use for guidance.
All migration modules, their videos (when available), codelab tutorials, and source code, can be found in the migration repo. While our content initially focuses on Python users, we hope to one day also cover other legacy runtimes so stay tuned. Containerization may seem foreboding, but the goal is for Cloud Buildpacks and migration resources like this to aid you in your quest to modernize your serverless apps!
Posted by Chris Curtis, Startup Marketing Manager at Google Cloud
We’re excited to announce our first-ever Google Cloud Startup Summit will be taking place on September 9, 2021.
We hope you will join us as we bring together our startup community, including startup founders, CTOs, VCs and Google experts to provide behind the scenes insights and inspiring stories of innovation. To kick off the event, we’ll be bringing in X’s Captain of Moonshots, Astro Teller, for a keynote focused on innovation. We’ll also have exciting technical and business sessions,with Google leaders, industry experts, venture investors and startup leaders. You can see the full agenda here to get more details on the sessions.
We can’t wait to see you at the Google Cloud Startup Summit at 10am PT on September 9! Register to secure your spot today.
Serverless Migration Station is a video mini-series from Serverless Expeditions focused on helping developers modernize their applications running on a serverless compute platform from Google Cloud. Previous episodes demonstrated how to migrate away from the older, legacy App Engine (standard environment) services to newer Google Cloud standalone equivalents like Cloud Datastore. Today's product crossover episode differs slightly from that by migrating away from App Engine altogether, containerizing those apps for Cloud Run.
There's little question the industry has been moving towards containerization as an application deployment mechanism over the past decade. However, Docker and use of containers weren't available to early App Engine developers until its flexible environment became available years later. Fast forward to today where developers have many more options to choose from, from an increasingly open Google Cloud. Google has expressed long-term support for App Engine, and users do not need to containerize their apps, so this is an optional migration. It is primarily for those who have decided to add containerization to their application deployment strategy and want to explicitly migrate to Cloud Run.
If you're thinking about app containerization, the video covers some of the key reasons why you would consider it: you're not subject to traditional serverless restrictions like development language or use of binaries (flexibility); if your code, dependencies, and container build & deploy steps haven't changed, you can recreate the same image with confidence (reproducibility); your application can be deployed elsewhere or be rolled back to a previous working image if necessary (reusable); and you have plenty more options on where to host your app (portability).
Legacy App Engine services are available through a set of proprietary, bundled APIs. As you can surmise, those services are not available on Cloud Run. So if you want to containerize your app for Cloud Run, it must be "ready to go," meaning it has migrated to either Google Cloud standalone equivalents or other third-party alternatives. For example, in a recent episode, we demonstrated how to migrate from App Engine ndb to Cloud NDB for Datastore access.
While we've recently begun to produce videos for such migrations, developers can already access code samples and codelab tutorials leading them through a variety of migrations. In today's video, we have both Python 2 and 3 sample apps that have divested from legacy services, thus ready to containerize for Cloud Run. Python 2 App Engine apps accessing Datastore are most likely to be using Cloud NDB whereas it would be Cloud Datastore for Python 3 users, so this is the starting point for this migration.
Because we're "only" switching execution platforms, there are no changes at all to the application code itself. This entire migration is completely based on changing the apps' configurations from App Engine to Cloud Run. In particular, App Engine artifacts such as app.yaml, appengine_config.py, and the lib folder are not used in Cloud Run and will be removed. A Dockerfile will be implemented to build your container. Apps with more complex configurations in their app.yaml files will likely need an equivalent service.yaml file for Cloud Run — if so, you'll find this app.yaml to service.yaml conversion tool handy. Following best practices means there'll also be a .dockerignore file.
appengine_config.py
lib
service.yaml
App Engine and Cloud Functions are sourced-based where Google Cloud automatically provides a default HTTP server like gunicorn. Cloud Run is a bit more "DIY" because users have to provide a container image, meaning bundling our own server. In this case, we'll pick gunicorn explicitly, adding it to the top of the existing requirements.txt required packages file(s), as you can see in the screenshot below. Also illustrated is the Dockerfile where gunicorn is started to serve your app as the final step. The only differences for the Python 2 equivalent Dockerfile are: a) require the Cloud NDB package (google-cloud-ndb) instead of Cloud Datastore, and b) start with a Python 2 base image.
gunicorn
requirements.txt
google-cloud-ndb
The Python 3 requirements.txt and Dockerfile
To walk developers through migrations, we always "START" with a working app then make the necessary updates that culminate in a working "FINISH" app. For this migration, the Python 2 sample app STARTs with the Module 2a code and FINISHes with the Module 4a code. Similarly, the Python 3 app STARTs with the Module 3b code and FINISHes with the Module 4b code. This way, if something goes wrong during your migration, you can always rollback to START, or compare your solution with our FINISH. If you are considering this migration for your own applications, we recommend you try it on a sample app like ours before considering it for yours. A corresponding codelab leading you step-by-step through this exercise is provided in addition to the video which you can use for guidance.
All migration modules, their videos (when published), codelab tutorials, START and FINISH code, etc., can be found in the migration repo. We hope to also one day cover other legacy runtimes like Java 8 so stay tuned. We'll continue with our journey from App Engine to Cloud Run ahead in Module 5 but will do so without explicit knowledge of containers, Docker, or Dockerfiles. Modernizing your development workflow to using containers and best practices like crafting a CI/CD pipeline isn't always straightforward; we hope content like this helps you progress in that direction!
Serverless Migration Station is a mini-series from Serverless Expeditions focused on helping users on one of Google Cloud's serverless compute platforms modernize their applications. The video today demonstrates how to migrate a sample app from Cloud NDB (or App Engine ndb) to Cloud Datastore. While Cloud NDB suffices as a current solution for today's App Engine developers, this optional migration is for those who want to consolidate their app code to using a single client library to talk to Datastore.
Cloud Datastore started as Google App Engine's original database but matured to becoming its own standalone product in 2013. At that time, native client libraries were created for the new product so non-App Engine apps as well as App Engine second generation apps could access the service. Long-time developers have been using the original App Engine service APIs to access Datastore; for Python, this would be App Engine ndb. While the legacy ndb service is still available, its limitations and lack of availability in Python 3 are why we recommend users switch to standalone libraries like Cloud NDB in the preceding video in this series.
While Cloud NDB lets users break free from proprietary App Engine services and upgrade their applications to Python 3, it also gives non-App Engine apps access to Datastore. However, Cloud NDB's primary role is a transition tool for Python 2 App Engine developers. Non-App Engine developers and new Python 3 App Engine developers are directed to the Cloud Datastore native client library, not Cloud NDB.
As a result, those with a collection of Python 2 or Python 3 App Engine apps as well as non-App Engine apps may be using completely different libraries (ndb, Cloud NDB, Cloud Datastore) to connect to the same Datastore product. Following the best practices of code reuse, developers should consider consolidating to a single client library to access Datastore. Shared libraries provide stability and robustness with code that's constantly tested, debugged, and battle-proven. Module 2 showed users how to migrate from App Engine ndb to Cloud NDB, and today's Module 3 content focuses on migrating from Cloud NDB to Cloud Datastore. Users can also go straight from ndb directly to Cloud Datastore, skipping Cloud NDB entirely.
Cloud NDB follows an object model identical to App Engine ndb and is deliberately meant to be familiar to long-time Python App Engine developers while use of the Cloud Datastore client library is more like accessing a JSON document store. Their querying styles are also similar. You can compare and contrast them in the "diffs" screenshot below and in the video.
The "diffs" between the Cloud NDB and Cloud Datastore versions of the sample app
All that said, this migration is optional and only useful if you wish to consolidate to using a single client library. If your Python App Engine apps are stable with ndb or Cloud NDB, and you don't have any code using Cloud Datastore, there's no real reason to move unless Cloud Datastore has a compelling feature inaccessible from your current client library. If you are considering this migration and want to try it on a sample app before considering for yours, see the corresponding codelab and use the video for guidance.
It begins with the Module 2 code completed in the previous codelab/video; use your solution or ours as the "START". Both Python 2 (Module 2a folder) and Python 3 (Module 2b folder) versions are available. The goal is to arrive at the "FINISH" with an identical, working app but using a completely different Datastore client library. Our Python 2 FINISH can be found in the Module 3a folder while Python 3's FINISH is in the Module 3b folder. If something goes wrong during your migration, you can always rollback to START, or compare your solution with our FINISH. We will continue our Datastore discussion ahead in Module 6 as Cloud Firestore represents the next generation of the Datastore service.
All of these learning modules, corresponding videos (when published), codelab tutorials, START and FINISH code, etc., can be found in the migration repo. We hope to also one day cover other legacy runtimes like Java 8 and others, so stay tuned. Up next in Module 4, we'll take a different turn and showcase a product crossover, showing App Engine developers how to containerize their apps and migrate them to Cloud Run, our scalable container-hosting service in the cloud. If you can't wait for either Modules 4 or 6, try out their respective codelabs or access the code samples in the table at the repo above. Migrations aren't always easy, and we hope content like this helps you modernize your apps.
Today we're introducing the first video showing long-time App Engine developers how to migrate from the App Engine ndb client library that connects to Datastore. While the legacy App Engine ndb service is still available for Datastore access, new features and continuing innovation are going into Cloud Datastore, so we recommend Python 2 users switch to standalone product client libraries like Cloud NDB.
This video and its corresponding codelab show developers how to migrate the sample app introduced in a previous video and gives them hands-on experience performing the migration on a simple app before tackling their own applications. In the immediately preceding "migration module" video, we transitioned that app from App Engine's original webapp2 framework to Flask, a popular framework in the Python community. Today's Module 2 content picks up where that Module 1 leaves off, migrating Datastore access from App Engine ndb to Cloud NDB.
webapp2
Migrating to Cloud NDB opens the doors to other modernizations, such as moving to other standalone services that succeed the original App Engine legacy services, (finally) porting to Python 3, breaking up large apps into microservices for Cloud Functions, or containerizing App Engine apps for Cloud Run.
App Engine's Datastore matured to becoming its own standalone product in 2013, Cloud Datastore. Cloud NDB is the replacement client library designed for App Engine ndb users to preserve much of their existing code and user experience. Cloud NDB is available in both Python 2 and 3, meaning it can help expedite a Python 3 upgrade to the second generation App Engine platform. Furthermore, Cloud NDB gives non-App Engine apps access to Cloud Datastore.
As you can see from the screenshot below, one key difference between both libraries is that Cloud NDB provides a context manager, meaning you would use the Python with statement in a similar way as opening files but for Datastore access. However, aside from moving code inside with blocks, no other changes are required of the original App Engine ndb app code that accesses Datastore. Of course your "YMMV" (your mileage may vary) depending on the complexity of your code, but the goal of the team is to provide as seamless of a transition as possible as well as to preserve "ndb"-style access.
with
The "diffs" between the App Engine ndb and Cloud NDB versions of the sample app
To try this migration yourself, hit up the corresponding codelab and use the video for guidance. This Module 2 migration sample "STARTs" with the Module 1 code completed in the previous codelab (and video). Users can use their solution or grab ours in the Module 1 repo folder. The goal is to arrive at the end with an identical, working app that operates just like the Module 1 app but uses a completely different Datastore client library. You can find this "FINISH" code sample in the Module 2a folder. If something goes wrong during your migration, you can always rollback to START, or compare your solution with our FINISH. Bonus content migrating to Python 3 App Engine can also be found in the video and codelab, resulting in a second FINISH, the Module 2b folder.
All of these learning modules, corresponding videos (when published), codelab tutorials, START and FINISH code, etc., can be found in the migration repo. We hope to also one day cover other legacy runtimes like Java 8 and others, so stay tuned! Developers should also check out the official Cloud NDB migration guide which provides more migration details, including key differences between both client libraries.
Ahead in Module 3, we will continue the Cloud NDB discussion and present our first optional migration, helping users move from Cloud NDB to the native Cloud Datastore client library. If you can't wait, try out its codelab found in the table at the repo above. Migrations aren't always easy; we hope this content helps you modernize your apps and shows we're focused on helping existing users as much as new ones.
The Google Cloud team recently introduced a series of codelabs (free, self-paced, hands-on tutorials) and corresponding videos designed to help users on one of our serverless compute platforms modernize their apps, with an initial focus on our earliest users running their apps on Google App Engine. We kick off this content by showing users how to migrate from App Engine's webapp2 web framework to Flask, a popular framework in the Python community.
While users have always been able to use other frameworks with App Engine, webapp2 comes bundled with App Engine, making it the default choice for many developers. One new requirement in App Engine's next generation platform (which launched in 2018) is that web frameworks must do their own routing, which unfortunately, means that webapp2 is no longer supported, so here we are. The good news is that as a result, modern App Engine is more flexible, lets users to develop in a more idiomatic fashion, and makes their apps more portable.
For example, while webapp2 apps can run on App Engine, Flask apps can run on App Engine, your servers, your data centers, or even on other clouds! Furthermore, Flask has more users, more published resources, and is better supported. If Flask isn't right for you, you can select from other WSGI-compliant frameworks such as Django, Pyramid, and others.
In this "Module 1" episode of Serverless Migration Station (part of the Serverless Expeditions series) Google engineer Martin Omander and I explore this migration and walk developers through it step-by-step.
In the previous video, we introduced developers to the baseline Python 2 App Engine NDB webapp2 sample app that we're taking through each of the migrations. In the video above, users see that the majority of the changes are in the main application handler, MainHandler:
MainHandler
Upon (re)deploying the app, users should see no visible changes to the output from the original version:
Today's video picks up from where we left off: the Python 2 baseline app in its Module 0 repo folder. We call this the "START". By the time the migration has completed, the resulting source code, called "FINISH", can be found in the Module 1 repo folder. If you mess up partway through, you can rewind back to the START, or compare your solution with ours, FINISH. We also hope to one day provide a Python 3 version as well as cover other legacy runtimes like Java 8, PHP 5, and Go 1.11 and earlier, so stay tuned!
All of the migration learning modules, corresponding videos (when published), codelab tutorials, START and FINISH code, etc., can all be found in the migration repo. The next video (Module 2) will cover migrating from App Engine's ndb library for Datastore to Cloud NDB. We hope you find all these resources helpful in your quest to modernize your serverless apps!
Earlier this year, the Google Cloud team introduced a series of codelabs (free, online, self-paced, hands-on tutorials) designed for technical practitioners modernizing their serverless applications. Today, we're excited to announce companion videos, forming a set of "learning modules" made up of these videos and their corresponding codelab tutorials. Modernizing your applications allows you to access continuing product innovation and experience a more open Google Cloud. The initial content is designed with App Engine developers in mind, our earliest users, to help you take advantage of the latest features in Google Cloud. Here are some of the key migrations and why they benefit you:
taskqueue
The "Serverless Migration Station" videos are part of the long-running Serverless Expeditions series you may already be familiar with. In each video, Google engineer Martin Omander and I explore a variety of different modernization techniques. Viewers will be given an overview of the task at hand, a deeper-dive screencast takes a closer look at the code or configuration files, and most importantly, illustrates to developers the migration steps necessary to transform the same sample app across each migration.
The baseline sample app is a simple Python 2 App Engine NDB and webapp2 application. It registers every web page visit (saving visiting IP address and browser/client type) and displays the most recent queries. The entire application is shown below, featuring Visit as the data Kind, the store_visit() and fetch_visits() functions, and the main application handler, MainHandler.
Visit
store_visit()
fetch_visits()
import os import webapp2 from google.appengine.ext import ndb from google.appengine.ext.webapp import template class Visit(ndb.Model): 'Visit entity registers visitor IP address & timestamp' visitor = ndb.StringProperty() timestamp = ndb.DateTimeProperty(auto_now_add=True) def store_visit(remote_addr, user_agent): 'create new Visit entity in Datastore' Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put() def fetch_visits(limit): 'get most recent visits' return (v.to_dict() for v in Visit.query().order( -Visit.timestamp).fetch(limit)) class MainHandler(webapp2.RequestHandler): 'main application (GET) handler' def get(self): store_visit(self.request.remote_addr, self.request.user_agent) visits = fetch_visits(10) tmpl = os.path.join(os.path.dirname(__file__), 'index.html') self.response.out.write(template.render(tmpl, {'visits': visits})) app = webapp2.WSGIApplication([ ('/', MainHandler), ], debug=True)
Upon deploying this application to App Engine, users will get output similar to the following:
This application is the subject of today's launch video, and the main.py file above along with other application and configuration files can be found in the Module 0 repo folder.
main.py
Each migration learning module covers one modernization technique. A video outlines the migration while the codelab leads developers through it. Developers will always get a starting codebase ("START") and learn how to do a specific migration, resulting in a completed codebase ("FINISH"). Developers can hit the reset button (back to START) if something goes wrong or compare their solutions to ours (FINISH). The hands-on experience helps users build muscle-memory for when they're ready to do their own migrations.
All of the migration learning modules, corresponding Serverless Migration Station videos (when published), codelab tutorials, START and FINISH code, etc., can all be found in the migration repo. While there's an initial focus on Python 2 and App Engine, you'll also find content for Python 3 users as well as non-App Engine users. We're looking into similar content for other legacy languages as well so stay tuned. We hope you find all these resources helpful in your quest to modernize your serverless apps!
Posted by Jennifer Kohl, Global Program Manager, Google Developer Communities
The Google Developer Groups Spotlight series interviews inspiring leaders of community meetup groups around the world. Our goal is to learn more about what developers are working on, how they’ve grown their skills with the Google Developer Group community, and what tips they might have for us all.
We recently spoke with Ilias Papachristos, Google Developer Group Cloud Thessaloniki Lead in Greece. Check out our conversation with Ilias on Cloud architecture, reading official documentation, and suggested resources to help developers grow professionally.
Tell us a little about yourself?
I’m a family man, ex-army helicopter pilot, Kendo sensei, beta tester at Coursera, Lead of the Google Developer Group Cloud Thessaloniki community, Google Cloud Professional Architect, and a Cloud Board Moderator on the Google Developers Community Leads Platform (CLP).
I love outdoor activities, reading books, listening to music, and cooking for my family and friends!
Can you explain your work in Cloud technologies?
Over my career, I have used Compute Engine for an e-shop, AutoML Tables for an HR company, and have architected the migration of a company in Mumbai. Now I’m consulting for a company on two of their projects: one that uses Cloud Run and another that uses Kubernetes.
Both of them have Cloud SQL and the Kubernetes project will use the AI Platform. We might even end up using Dataflow with BigQuery for the streaming and Scheduler or Manager, but I’m still working out the details.
I love the chance to share knowledge with the developer community. Many days, I open my PC, read the official Google Cloud blog, and share interesting articles on the CLP Cloud Board and GDG Cloud Thessaloniki’s social media accounts. Then, I check Google Cloud’s Medium publication for extra articles. Read, comment, share, repeat!
How did the Google Developer Group community help your Cloud career?
My overall knowledge of Google Cloud has to do with my involvement with Google Developer Groups. It is not just one thing. It’s about everything! At the first European GDG Leads Summit, I met so many people who were sharing their knowledge and offering their help. For a newbie like me it was and still is something that I keep in my heart as a treasure
I’ve also received so many informative lessons on public speaking from Google Developer Group and Google Developer Student Club Leads. They always motivate me to continue talking about the things I love!
What has been the most inspiring part of being a part of your local Google Developer Group?
Collaboration with the rest of the DevFest Hellas Team! For this event, I was a part of a small group of 12 organizers, all of whom never had hosted a large meetup before. With the help of Google Developer Groups, we had so much fun while creating a successful DevFest learning program for 360 people.
What are some technical resources you have found the most helpful for your professional development?
Besides all of the amazing tricks and tips you can learn from the Google Cloud training team and courses on the official YouTube channel, I had the chance to hear a talk by Wietse Venema on Cloud Run. I also have learned so much about AI from Dale Markovitz’s videos on Applied AI. And of course, I can’t leave out Priyanka Vergadia’s posts, articles, and comic-videos!
Official documentation has also been a super important part of my career. Here are five links that I am using right now as an Architect:
How did you become a Google Developer Group Lead?
I am a member of the Digital Analytics community in Thessaloniki, Greece. Their organizer asked me to write articles to start motivating young people. I translated one of the blogs into English and published it on Medium. The Lead of GDG Thessaloniki read them and asked me to become a facilitator for a Cloud Study Jams (CSJ) workshop. I accepted and then traveled to Athens to train three people so that they could also become CSJ facilitators. At the end of the CSJ, I was asked if I wanted to lead a Google Developer Group chapter. I agreed. Maria Encinar and Katharina Lindenthal interviewed me, and I got it!
What would be one piece of advice you have for someone looking to learn more about a specific technology?
Learning has to be an amusing and fun process. And that’s how it’s done with Google Developer Groups all over the world. Join mine, here. It’s the best one. (Wink, wink.)
Want to start growing your career and coding knowledge with developers like Ilias? Then join a Google Developer Group near you, here.
Posted by Wesley Chun, Developer Advocate, Google Cloud
Since its initial launch in 2008 as the first product from Google Cloud, Google App Engine, our fully-managed serverless app-hosting platform, has been used by many developers worldwide. Since then, the product team has continued to innovate on the platform: introducing new services, extending quotas, supporting new languages, and adding a Flexible environment to support more runtimes, including the ability to serve containerized applications.
With many original App Engine services maturing to become their own standalone Cloud products along with users' desire for a more open cloud, the next generation App Engine launched in 2018 without those bundled proprietary services, but coupled with desired language support such as Python 3 and PHP 7 as well as introducing Node.js 8. As a result, users have more options, and their apps are more portable.
With the sunset of Python 2, Java 8, PHP 5, and Go 1.11, by their respective communities, Google Cloud has assured users by expressing continued long-term support of these legacy runtimes, including maintaining the Python 2 runtime. So while there is no requirement for users to migrate, developers themselves are expressing interest in updating their applications to the latest language releases.
Google Cloud has created a set of migration guides for users modernizing from Python 2 to 3, Java 8 to 11, PHP 5 to 7, and Go 1.11 to 1.12+ as well as a summary of what is available in both first and second generation runtimes. However, moving from bundled to unbundled services may not be intuitive to developers, so today we're introducing additional resources to help users in this endeavor: App Engine "migration modules" with hands-on "codelab" tutorials and code examples, starting with Python.
Each module represents a single modernization technique. Some are strongly recommended, others less so, and, at the other end of the spectrum, some are quite optional. We will guide you as far as which ones are more important. Similarly, there's no real order of modules to look at since it depends on which bundled services your apps use. Yes, some modules must be completed before others, but again, you'll be guided as far as "what's next."
More specifically, modules focus on the code changes that need to be implemented, not changes in new programming language releases as those are not within the domain of Google products. The purpose of these modules is to help reduce the friction developers may encounter when adapting their apps for the next-generation platform.
Central to the migration modules are the codelabs: free, online, self-paced, hands-on tutorials. The purpose of Google codelabs is to teach developers one new skill while giving them hands-on experience, and there are codelabs just for Google Cloud users. The migration codelabs are no exception, teaching developers one specific migration technique.
Developers following the tutorials will make the appropriate updates on a sample app, giving them the "muscle memory" needed to do the same (or similar) with their applications. Each codelab begins with an initial baseline app ("START"), leads users through the necessary steps, then concludes with an ending code repo ("FINISH") they can compare against their completed effort. Here are some of the initial modules being announced today:
What should you expect from the migration codelabs? Let's preview a pair, starting with the web framework: below is the main driver for a simple webapp2-based "guestbook" app registering website visits as Datastore entities:
class MainHandler(webapp2.RequestHandler): 'main application (GET) handler' def get(self): store_visit(self.request.remote_addr, self.request.user_agent) visits = fetch_visits(LIMIT) tmpl = os.path.join(os.path.dirname(__file__), 'index.html') self.response.out.write(template.render(tmpl, {'visits': visits}))
A "visit" consists of a request's IP address and user agent. After visit registration, the app queries for the latest LIMIT visits to display to the end-user via the app's HTML template. The tutorial leads developers a migration to Flask, a web framework with broader support in the Python community. An Flask equivalent app will use decorated functions rather than webapp2's object model:
LIMIT
@app.route('/') def root(): 'main application (GET) handler' store_visit(request.remote_addr, request.user_agent) visits = fetch_visits(LIMIT) return render_template('index.html', visits=visits)
The framework codelab walks users through this and other required code changes in its sample app. Since Flask is more broadly used, this makes your apps more portable.
The second example pertains to Datastore access. Whether you're using App Engine's ndb or the Cloud NDB client libraries, the code to query the Datastore for the most recent limit visits may look like this:
limit
def fetch_visits(limit): 'get most recent visits' query = Visit.query() visits = query.order(-Visit.timestamp).fetch(limit) return (v.to_dict() for v in visits)
If you decide to switch to the Cloud Datastore client library, that code would be converted to:
def fetch_visits(limit): 'get most recent visits' query = DS_CLIENT.query(kind='Visit') query.order = ['-timestamp'] return query.fetch(limit=limit)
The query styles are similar but different. While the sample apps are just that, samples, giving you this kind of hands-on experience is useful when planning your own application upgrades. The goal of the migration modules is to help you separate moving to the next-generation service and making programming language updates so as to avoid doing both sets of changes simultaneously.
As mentioned above, some migrations are more optional than others. For example, moving away from the App Engine bundled ndb library to Cloud NDB is strongly recommended, but because Cloud NDB is available for both Python 2 and 3, it's not necessary for users to migrate further to Cloud Datastore nor Cloud Firestore unless they have specific reasons to do so. Moving to unbundled services is the primary step to giving users more flexibility, choices, and ultimately, makes their apps more portable.
For those who are interested in modernizing their apps, a complete table describing each module and links to corresponding codelabs and expected START and FINISH code samples can be found in the migration module repository. We are also working on video content based on these migration modules as well as producing similar content for Java, so stay tuned.
In addition to the migration modules, our team has also setup a separate repo to support community-sourced migration samples. We hope you find all these resources helpful in your quest to modernize your App Engine apps!
Posted by Biswajeet Mallik, Program Manager, Google Developers India.
Image from Cloud Community Days India
Earlier this year, ten Google Developer Groups in India came together to host Google Cloud Community Days India, a two day event helping developers study for their upcoming Cloud Certification exams. To address the rising demand for professional certifications, the virtual event hosted over 63,000 developers, covered four main exam areas, and welcomed nine speakers. This was the second edition to the event series which started in 2019 in India.
By providing expert learning materials and mentorship, the event uniquely prepared developers for the Associate Cloud Engineer, Professional Data Engineer, Professional Cloud Machine Learning Engineer, and Professional Cloud Architect exams. Learn more below.
The Cloud Community Days event focused on helping developers study for four milestone certifications, tailored to engineers at four different stages of their career. The goal: help Google Developer Group members obtain the right credentials to improve their job prospects.
The event broke participants into breakout sessions based on which exam they were preparing to take. Since the certifications targeted professionals of all skill levels, study groups ranged from early career associates to late career executives. The learning groups were organized around the following certifications:
This learning session was created to help early career developers complete the first stepping stone exam. In particular, learning materials and speakers were curated to guide participants who had no prior experience, or very little, working on the Google Cloud Platform.
Workshops were mainly dedicated to assisting programmers who were familiar with building different applications but wished to show employers that they could deploy them on Google Cloud Platform.
Watch more from: Day 1, here. And day 2, here.
The next group brought together were data practitioners with special interests in data visualization and decision making. Workshops and learning activities helped these developers hone their large scale data and data driven decision making abilities.
Improving these skills are essential for passing the Professional Data Engineers certification and growing a programmer’s early career.
For these sessions, the Google Developer Group Cloud community paired experienced programmers with a significant interest in ML to form their study groups. The main driver in these learning activities was to help seasoned developers gain a deeper understanding of how to utilize Google Cloud ML services.
With significant emphasis being placed on machine learning in the ecosystem right now, Google Developer Group community leaders felt this certification could help developers make the leap into new leadership roles.
Lastly, this event paired experienced Cloud executives and professionals working in leading capacities for their organizations. For these sessions, speakers and activities had a specific scope: help high level professions be at the forefront of Google Cloud Platforms innovative capabilities.
Specifically, the Professional Cloud Architect Certification was created to help senior software engineers better design, scale and develop highly secure and robust applications.
Day 1, here. And day 2, here.
Overall, the community put together these resources to help developers feel more confident in their abilities, obtain tangible credentials, and in turn increase access to better job opportunities. As two participants recalled the event,
“This was the first time I attended the Google Developer Group event! It is an awesome package for learning in one place. All the fun activities were engaging and the panelist discussion was also very insightful. I feel proud to be a part of this grand GDG event.”
With Google Developer Groups, find a space to learn alongside a group of curious developers, all coming together to advance their careers from withinside a caring community of peers.
Want to know more about what Cloud Community days were like? Then watch their live recording below.
Ready to find a community event near you? Then get started at gdg.community.dev
Posted by Travis Webb
This blog post describes common pitfalls and antipatterns to consider when migrating your mainframe workloads. It also helps you to understand and avoid them. Migrating or modernizing your mainframe workloads is complex and challenging, even under ideal conditions. If you avoid the antipatterns discussed in this document, you increase the odds of a successful transformation.
This blog post is useful whether you're planning to migrate your mainframe workloads to Google Cloud, to on-premises virtual machines, or to another cloud provider. It demonstrates how to remedy certain mainframe migration antipatterns using technology offerings from Google. In principle, however, you could apply these remedies to many kinds of transformations with different target platforms and architectures.
This blog post describes three common antipatterns:
These approaches can work in some narrow circumstances when migrating mainframe workloads. Avoid them, however, because they have a high probability of failure. For each antipattern discussed, you are given an overview of the antipattern, the typical rationale used to justify it, and the business and technical reasons that lead to failure.
In a big bang rewrite, you or your team manually rewrite and re-architect the legacy mainframe code into a modern language using modern design patterns. For example, you might form a development team to build a new Java application that replicates the business logic from a collection of legacy COBOL programs. Senior engineers who are familiar with the system often teach junior engineers the rationale behind the business logic to preserve institutional knowledge. The result is a new codebase using new programming languages and new documentation on a new platform.
Of the three antipatterns discussed in this document, the big bang rewrite requires the largest investment of capital and time to achieve success. It is capital-intensive and time-intensive because most organizations can’t resist the temptation to re-engineer and to improve business logic.
Re-engineering your systems using modern technologies allows for future innovation. Your senior engineers are moving on—to management, competitors, or retirement—and you need to transfer institutional knowledge to incoming staff. You expect those incoming staffers to re-engineer the system using the latest programming best practices. These less experienced engineers can rewrite module by module, and take advantage of current development methodologies and tools. Because you have all the code, you have an exact specification for what the new software needs to do, and can test against it. Access to the original code lets you compress the decades of investment into your original mainframe software into a modern application. At the same time, you are transferring institutional knowledge from your senior engineers to your junior engineers. At the end of the process, you'll have a new system consisting of well-engineered software built against modern design patterns and best practices.
This case is compelling and can help to convince your IT decision-makers. Though the approach appears rational, there are hidden pitfalls and risks that your team doesn’t recognize at the outset. Risks like budget overruns, unanticipated complexity, and staff turnover can derail a significant rewrite before realizing the benefits. As a result, big bang rewrites rarely equal the best-case scenarios presented to stakeholders. Often, they fail.
Big bang rewrites often suffer from the second system effect. Early in the project, they fall behind in schedule and budget. While you quickly develop prototypes, getting them to function in the same way as the original code is a long-tail effort that most teams underestimate. This unanticipated setback leads to the first major decision point in your project: How do I overcome these challenges but still achieve the outcomes that I need to make the project successful?
The first option: Continue to diligently plod the long path and adhere exactly to the original functionality. However, matching the new system precisely to the original functionality always takes longer than expected. This is true because the original code provides little or no improvement in productivity over a conventional specification. That means a significant engineering investment to understand the original code and reproduce it.
The second option: Implement the business logic differently. However, changes in business logic necessarily require changes to the business processes and downstream systems on which the original business logic depends. For example, you could have a web application that depends on the idiosyncratic behavior of your mainframe applications. Rather than incorporate these idiosyncrasies into the new, rewritten application, it is tempting to simplify and improve this behavior. However, that adds scope to the project. The chain reaction of further changes that are required in downstream systems introduce additional risk and prolong the rewrite effort.
If your production mainframe system requires ongoing maintenance or updates during the rewrite, you can compound these problems. For example, you might have a rules engine that powers a billing system on your mainframe. To support a new product launch, you need to add a feature to the rules engine to accommodate a new customer billing type. You also need to implement this new type in the current system and replicate it in the new system—possibly after the billing component was rewritten and tested. This maintenance and update scenario can occur many times during a big bang rewrite, setting the project back at each step, and increasing the odds of failure.
Even for companies that have the tenacity to see through a multi-year transformation effort, the raw cost of a rewrite is often prohibitive. When compared to all other approaches, a big bang rewrite is the costliest way to modernize your mainframe software. Often it has the least convincing return on investment (ROI) when factoring in the risks, unanticipated costs, and delays.
A lift-and-shift migration is an established method of moving an application from one system to another with minimal changes and downtime. It's commonly used to migrate virtual machines running on commodity hardware to virtual machines in a public cloud. You can take a similar approach with your mainframe migration.
Mainframe platforms are based on proprietary hardware rather than x86-based commodity hardware. Therefore, you must emulate your mainframe environment on x86-based machines. Doing so is required to move your applications directly from the mainframe into the cloud, as you would with virtual machines. To run your applications in the emulated environment, you recompile them using a compiler provided by your emulation vendor.
Lift-and-shift migration is often seen as the quickest way to get from an on-premises environment to the cloud. You can apply this same thinking to mainframe workloads. Strategic IT decisions are often most palatable when facing a key transition, such as a hardware refresh. Mainframe hardware investments are capital-intensive. Financing the purchase often adds debt or lease liabilities to your company's balance sheet. By moving to the public cloud, mainframe workloads can scale both up and down to optimize resource use and operational cost. When compared to other migration or modernization options, you can make a strong business case that a lift-and-shift migration provides the quickest ROI and carries the lowest risk.
The business risks of a lift-and-shift migration appear small compared to other approaches, but the potential benefits are even smaller. The benefits of migrating off the mainframe platform to the cloud don’t materialize, because you remain locked into the same mainframe ecosystem, but now with an extra dependency on an emulation layer. That dependency can result in a new set of technical challenges. Challenges that are often unfamiliar to the teams maintaining the mainframe software. Unfamiliarity can lead to additional reliance on a new, single-vendor cloud ecosystem.
By not changing your mainframe software, you avoid solving many important problems: scarce and shrinking mainframe talent, a static ecosystem, a lack of agility, and an inability to innovate. You're now running your legacy workloads in the cloud, but remain locked out of cloud innovations due to your continued reliance on proprietary platforms.
In this antipattern, the cost benefits that you relied on to justify the investment don’t materialize. While you might spend less after combining your cloud infrastructure costs with your new, ongoing, emulation software license fees, your savings don’t justify the investment. The outcome is that you've taken all the risks inherent in any migration, but have realized few of the benefits, if any.
In an in-place modernization, you focus on improving the quality, maintainability, and testability of your software while keeping it on your mainframe computers. You might choose this antipattern because you see mainframes as part of your future and know that you must modernize your application software accordingly.
You can rewrite your application software to use modern languages that run on the mainframe, or you can re-architect it in place. For a partial cloud-like experience you can install orchestration technologies, like Kubernetes.
Mainframe software presents challenges related to maintainability, innovation, agility, and extensibility. By re-architecting and re-engineering this software to align with modern standards and design patterns, you can avoid many of the pitfalls that disrupt large replatforming efforts. Moving off the mainframe is the single largest risk. By avoiding that move, you can improve the odds that your project succeeds. Of all the mainframe modernization approaches you might consider, an in-place modernization appears to be the lowest risk. There's no migration component, so there's no risk of downtime.
There is an ecosystem of vendors offering tools to help with mainframe development using modern methodologies. Therefore, the risk of being left to support the software on your own is low. An in-place modernization often takes longer than a lift-and-shift migration or a code conversion. By modernizing slowly, however, you afford your teams the time they need to learn new development processes. When you re-engineer and re-architect the codebase, you can perform a more rational analysis to better understand whether the mainframe is the appropriate long-term platform.
An in-place modernization suffers from many of the same challenges as the big bang rewrite. Any approach involving manually updating your mainframe software can have budget and time constraints. These efforts also often suffer from the second-system effect. Performance and correctness issues inevitably arise because rewriting business logic in a new language requires extensive testing before it aligns with the previous functionality. When management learns more about the modest benefits gained by running updated software on the same mainframe platform, expect their willingness to see through such a drawn-out and costly transformation to wane.
The biggest issue with an in-place modernization is that the ideal outcome leaves you many of the same problems that you started with. The mainframe is more than a piece of hardware. Using mainframes encompasses a talent pool, a software platform, and a vendor ecosystem. The trend for each of these variables is moving in the wrong direction. Every year the talent pool shrinks, the software platform becomes more isolated, and the vendor ecosystem consolidates.
Google Cloud offers various options and resources for you to find the necessary help and support to best use Google Cloud services:
There are more resources to help you to migrate workloads to Google Cloud in the Google Cloud migration center.
For more information about these resources, see the finding help section of Migration to Google Cloud: Getting started.
File backup isn't the most exciting topic while analyzing images with AI/ML is more interesting, so combining them probably isn't a workflow you think about often. However, by augmenting the former with the latter, you can build a more useful solution than without. Google provides a diverse array of developer tools you can use to realize this ambition, and in fact, you can craft such a workflow with Google Cloud products alone. More compellingly, the basic principle of mixing-and-matching Google technologies can be applied to many other challenges faced by you, your organization, or your customers.
The sample app presented uses Google Drive and Sheets plus Cloud Storage and Vision to make it happen. The use-case: Google Workspace (formerly G Suite) users who work in industries like architecture or advertising, where multimedia files are constantly generated. Every client job results in yet another Drive subfolder and collection of asset files. Successive projects lead to even more files and folders. At some point, your Drive becomes a "hot mess," making users increasingly inefficient, requiring them to scroll endlessly to find what they're looking for.
A user and their Google Drive files
How can Google Cloud help? Like Drive, Cloud Storage provides file (and generic blob) storage in the cloud. (More on the differences between Drive & Cloud Storage can be found in this video.)
Cloud Storage provides several storage classes depending on how often you expect to access your archived files. The less often files are accessed, the "colder" the storage, and the lower the cost. As users progress from one project to another, they're not as likely to need older Drive folders and those make great candidates to backup to Cloud Storage.
First challenge: determine the security model. When working with Google Cloud APIs, you generally select OAuth client IDs to access data owned by users and service accounts for data owned by applications/projects. The former is typically used with Workspace APIs while the latter is the primary way to access Google Cloud APIs. Since we're using APIs from both product groups, we need to make a decision (for now and change later if desired).
Since the goal is a simple proof-of-concept, user auth suffices. OAuth client IDs are standard for Drive & Sheets API access, and the Vision API only needs API keys so the more-secure OAuth client ID is more than enough. The only IAM permissions to acquire are for the user running the script to get write access to the destination Cloud Storage bucket. Lastly, Workspace APIs don't have their own product client libraries (yet), so the lower-level Google APIs "platform" client libraries serve as a "lowest common denominator" to access all four REST APIs. Those who have written Cloud Storage or Vision code using the Cloud client libraries will see something different.
The prototype is a command-line script. In real life, it would likely be an application in the cloud, executing as a Cloud Function or a Cloud Task running as determined by Cloud Scheduler. In that case, it would use a service account with Workspace domain-wide delegation to act on behalf of an employee to backup their files. See this page in the documentation describing when you'd use this type of delegation and when not to.
Our simple prototype targets individual image files, but you can continue to evolve it to support multiple files, movies, folders, and ZIP archives if desired. Each function calls a different API, creating a "service pipeline" with which to process the images. The first pair of functions are drive_get_file() and gcs_blob_upload(). The former queries for the image on Drive, grabs pertinent metadata (filename, ID, MIMEtype, size), downloads the binary "blob" and returns all of that to the caller. The latter uploads the binary along with relevant metadata to Cloud Storage. The script was written in Python for brevity, but the client libraries support most popular languages. Below is the aforementioned function pseudocode:
drive_get_file()
gcs_blob_upload()
def drive_get_file(fname): rsp = DRIVE.files().list(q="name='%s'" % fname).execute().get['files'][0] fileId, fname, mtype = rsp['id'], rsp['name'], rsp['mimeType'] blob = DRIVE.files().get_blob(fileId).execute() return fname, mtype, rsp['modifiedTime'], blob def gcs_blob_upload(fname, folder, bucket, blob, mimetype): body = {'name': folder+'/'+fname, 'uploadType': 'multipart', 'contentType': mimetype} return GCS.objects().insert(bucket, body, blob).execute()
Next, vision_label_img() passes the binary to the Vision API and formats the results. Finally that information along with the file's archived Cloud Storage location are written as a single row of data in a Google Sheet via sheet_append_row().
vision_label_img()
sheet_append_row()
def vision_label_img(img): body = {'requests': [{'image': {'content': img}, 'features': [{'type': 'LABEL_DETECTION'}]}]} rsp = VISION.images().annotate(body=body).execute().get['responses'][0] return ', '.join('(%.2f%%) %s' % (label['score']*100., label['description']) for label in rsp['labelAnnotations']) def sheet_append_row(sheet_id, row): rsp = SHEETS.spreadsheets().values().append(spreadsheetId=sheet_id, range='Sheet1', body={'values': row}).execute() return rsp.get('updates').get('updatedCells')
Finally, a "main" program that drives the workflow is needed. It comes with a pair of utility functions, _k_ize() to turn file sizes into kilobytes and _linkify() to build a valid Cloud Storage hyperlink as a spreadsheet formula. These are featured here:
_k_ize()
_linkify()
def _k_ize(nbytes): # bytes to KBs (not KiBs) as str return '%6.2fK' % (nbytes/1000.) def _linkify(bucket, fname): # make GCS hyperlink to bucket/folder/file tmpl = '=HYPERLINK("storage.cloud.google.com/{0}/{1}/{2}", "{2}")' return tmpl.format(bucket, folder, fname) def main(fname, bucket, SHEET_ID, folder): fname, mtype, ftime, data = drive_get_img(fname) gcs_blob_upload(fname, folder, bucket, data, mtype) info = vision_label_img(data) sheet_append_row(SHEET_ID, [folder, _linkify(bucket, fname), mtype, ftime, _k_ize(data), info])
While this post may feature just pseudocode, a barebones working version can be accomplished with ~80 lines of actual Python. The rest of the code not shown are constants, error-handling, and other auxiliary support. The application gets kicked off with a call to main() passing in a filename, the Cloud Storage bucket to archive it to, a Drive file ID for the Sheet, and a "folder name," e.g., a directory or ZIP archive. Running it several times results in a spreadsheet that looks like this:
main()
Image archive report in Google Sheets
Developers can build this application step-by-step with our "codelab" - codelabs are free, online, self-paced tutorials - which can be found here. As you journey through this tutorial, its corresponding open source repo features separate folders for each step so you know what state your app should be in after every implemented function. (NOTE: Files are not deleted, so your users have to decide when to their cleanse Drive folders.) For backwards-compatibility, the script is implemented using older Python auth client libraries, but the repo has an "alt" folder featuring alternative versions of the final script that use service accounts, Google Cloud client libraries, and the newer Python auth client libraries.
Finally to save you some clicks, here are links to the API documentation pages for Google Drive, Cloud Storage, Cloud Vision, and Google Sheets. While this sample app deals with a constrained resource issue, we hope it inspires you to consider what's possible with Google developer tools so you can build your own solutions to improve users' lives every day!
