Migrating from App Engine Blobstore to Cloud Storage (Module 16)

AUG 15, 2022

Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

Introduction and background

The most recent Serverless Migration Station video demonstrated how to add use of the App Engine's Blobstore service to a sample Python 2 App Engine app, kicking off the first of a 2-part series on migrating away from Blobstore. In today's Module 16 video, we complete this journey, arriving at Cloud Storage. Moving away from proprietary App Engine services like Blobstore makes apps more portable, giving them enough flexibility to:

Update to more recent language releases, say from Python 2 to 3 or Java 8 to 17, and run on App Engine's 2nd generation service
Shift across to other serverless platforms, like Cloud Functions or Cloud Run (with or without Docker), or
Move to VM-based services like GKE or Compute Engine, or to other compute platforms

Showing App Engine users how to migrate to Cloud Storage

As described previously, a Blobstore for Python 2 dependency on webapp made the Module 15 content more straightforward to implement if it was still using webapp2. To completely modernize this app here in Module 16, the following migrations should be carried out:

Migrate from webapp2 (and webapp) to Flask
Migrate from App Engine NDB to Cloud NDB
Migrate from App Engine Blobstore to Cloud Storage
Migrate from Python 2 to Python (2 and) 3

Performing the migrations

Prior to modifying the application code, a variety of configuration updates need to be made. Updates applying only to Python 2 feature a "Py2" designation while those migrating to Python 3 will see "Py3" annotations.

Remove the built-in Jinja2 library from app.yaml—Jinja2 already comes with Flask, so remove use of the older built-in version which may possibly conflict with the contemporary Flask version you're using. (Py2)
Use of Cloud client libraries (such as those for Cloud NDB and Cloud Storage) require a pair of built-in libraries, grpcio and setuptools, so add those to app.yaml (Py2)
Remove everything in app.yaml except for a valid runtime (Py3)
Add Cloud NDB and Cloud Storage client libraries to requirements.txt (Py2 & Py3)
Create an appengine_config.py supporting both built-in (those in app.yaml) and non built-in (those in requirements.txt) libraries used (Py2)

The Module 15 app already migrated away from webapp2's (Django) templating system to Jinja2. This is useful when migrating to Flask because Jinja2 is Flask's default template system. Switching from App Engine NDB to Cloud NDB is fairly straightforward as the latter was designed to be mostly compatible with the original. The only change visible in this sample app is to move Datastore calls into Python with blocks.

The most significant changes occur when moving the upload and download handlers from webapp to Cloud Storage. The video and corresponding codelab go more in-depth into the necessary changes, but in summary, these are the updates required in the main application:

webapp2 is replaced by Flask. Instead of using the older built-in version of Jinja2, use the version that comes with Flask.
App Engine Blobstore and NDB are replaced by Cloud NDB and Cloud Storage, respectively.
The webapp Blobstore handler functionality is replaced by a combination of the io standard library module plus components from Flask and Werkzeug. Furthermore, the handler classes and methods are replaced by Flask functions.
The main handler class and corresponding GET and POST methods are all replaced by a single Flask function.

The results

With all the changes implemented, the original Module 15 app still operates identically in Module 16, starting with a form requesting a visit artifact followed by the most recents visits page:

The sample app's artifact prompt page

The sample app's most recent visits page.

The only difference is that four migrations have been completed where all of the "infrastructure" is now taken care of by non-App Engine legacy services. Furthermore, the Module 16 app could be either a Python 2 or 3 app. As far as the end-user is concerned, "nothing happened."

Migrating sample app from App Engine Blobstore to Cloud Storage

Wrap-up

Module 16 featured four different migrations, modernizing the Module 15 app from using App Engine legacy services like NDB and Blobstore to Cloud NDB and Cloud Storage, respectively. While we recommend users move to the latest offerings from Google Cloud, migrating from Blobstore to Cloud Storage isn't required, and should you opt to do so, can do it on your own timeline. In addition to today's video, be sure to check out the Module 16 codelab which leads you step-by-step through the migrations discussed.

In Fall 2021, the App Engine team extended support of many of the bundled services to 2nd generation runtimes (that have a 1st generation runtime), meaning you are no longer required to migrate to Cloud Storage when porting your app to Python 3. You can continue using Blobstore in your Python 3 app so long as you retrofit the code to access bundled services from next-generation runtimes.

If you're using other App Engine legacy services be sure to check out the other Migration Modules in this series. All Serverless Migration Station content (codelabs, videos, source code [when available]) can be accessed at its open source repo. While our content initially focuses on Python users, the Cloud team is working on covering other language runtimes, so stay tuned. For additional video content, check out our broader Serverless Expeditions series.