Migrating from App Engine pull tasks to Cloud Pub/Sub (Module 19)

DEC 13, 2022

Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

Introduction and background

The Serverless Migration Station series is aimed at helping developers modernize their apps running one of Google Cloud's serverless platforms. The preceding (Migration Module 18) video demonstrates how to add use of App Engine's Task Queue pull tasks service to a Python 2 App Engine sample app. Today's Module 19 video picks up from where that leaves off, migrating that pull task usage to Cloud Pub/Sub.

Moving away from proprietary App Engine services like Task Queue makes apps more portable, giving them enough flexibility to:

Run on 2nd generation App Engine runtimes
Shift across to other serverless platforms, like Cloud Functions or Cloud Run (with or without Docker)
Move to VM-based services like GKE or Compute Engine, or to other compute platforms

Understanding the migrations

Module 19 consists of implementing three different migrations on the Module 18 sample app:

Migrate from App Engine NDB to Cloud NDB
Migrate from App Engine Task Queue pull tasks to Cloud Pub/Sub
Migrate from Python 2 to Python (2 and) 3

The NDB to Cloud NDB migration is identical to the Module 2 migration content, so it's not covered in-depth in Module 19. The original app was designed to be Python 2 and 3 compatible, so there's no work there either. Module 19 boils down to three key updates:

Setup: Enable APIs and create Pub/Sub Topic & Subscription
How work is created: Publish Pub/Sub messages instead of adding pull tasks
How work is processed: Pull messages instead of leasing tasks

Aside from these physical changes, a key hurdle to overcome is understanding the differences in terminology between pull tasks and Pub/Sub. The following chart attempts to demystify this so developers can more easily grasp how they differ:

Table of terminology with related GAE Pull Tasks and Cloud Pub/Sub

Terminology differences between App Engine pull tasks and Cloud Pub/Sub

Reflecting the chart, these differences can be summarized like this:

With Pull Queues, work is created in pull queues while work is sent to Pub/Sub topics
Task Queue pull tasks are called messages in Pub/Sub
With Task Queues, workers access pull tasks; with Pub/Sub, subscribers receive messages
Leasing a pull task is the same as pulling a message from a Pub/Sub topic via a subscription
Deleting a task from a pull queue when you're done is analogous to successfully acknowledging a Pub/Sub message

The video walks developers through the terminology as well as the code changes described above. Below is pseudocode implementing the key changes to the main application (new or updated lines of code bolded):

Table showing changes in code Before (Module 18) on the left, and After (Module 19) on
the right

Migration from App Engine Task Queue pull tasks to Cloud Pub/Sub

Observe how most of the code, especially app operations and data models are left relatively unchanged. The only visible changes are switching from App Engine NDB and Task Queue to Cloud NDB and Pub/Sub. Complete versions of the app before and after making the changes can be found in the Module 18 and Module 19 repo folders, respectively. In addition to the video, be sure to check out the Module 19 codelab which leads you step-by-step through the migrations discussed.

Wrap-up

Module 19 features a migration of App Engine pull tasks to Cloud Pub/Sub, but developers should note that Pub/Sub itself is not based on pull tasks. It is a fully-featured asynchronous, scalable messaging service that has many more features than the pull functionality provided by Task Queue. For example, Pub/Sub has other features like streaming to BigQuery and push functionality. Pub/Sub push operates differently than Task Queue push tasks, hence why we recommend push tasks be migrated to Cloud Tasks instead (see Module 8). For more information on all of its features, see the Pub/Sub documentation. Because Cloud Tasks doesn't support pull functionality, we turn to Pub/Sub instead for pull task users.

While we recommend users move to the latest offerings from Google Cloud, neither of those migrations are required, and should you opt to do so, can do them on your own timeline. In Fall 2021, the App Engine team extended support of many of the bundled services to 2nd generation runtimes (that have a 1st generation runtime), meaning you don't have to migrate to standalone Cloud services before porting your app to Python 3. You can continue using Task Queue in Python 3 so long as you retrofit your code to access bundled services from next-generation runtimes.

If you're using other App Engine legacy services be sure to check out the other Migration Modules in this series. All Serverless Migration Station content (codelabs, videos, source code [when available]) can be accessed at its open source repo. While our content initially focuses on Python users, the Cloud team is working on covering other language runtimes, so stay tuned. For additional video content, check out our broader Serverless Expeditions series.

Firebase Mobile Web Tutorials How-To Guides

Learn to build and run AI powered apps at Firebase Demo Day ‘24

Nov. 19, 2024

Cloud Cloud Announcements Problem-Solving

Now in Developer Preview: Enhancing Chat with the Google Chat API

Oct. 8, 2024

Cloud AI Cloud Announcements Industry Trends

Evolving the Responsible Generative AI Toolkit with new tools for every LLM

Oct. 23, 2024

Gemini AI Cloud Case Studies Announcements

Enhancing AI Powered Developer Tools with Gemini API

Nov. 14, 2024