Google Developers Blog: Streak brings CRM to the inbox with Google Cloud Platform

Blog

Streak brings CRM to the inbox with Google Cloud Platform

Tuesday, October 9, 2012

By Aleem Mawani, Co-Founder of StreakCross-posted with the Google App Engine BlogThis guest post was written by Aleem Mawani, Co-Founder of Streak, a startup alum of Y Combinator, a Silicon Valley incubator. Streak is a CRM tool built into Gmail. In this post, Aleem shares his experience building and scaling their product using Google Cloud Platform.Everyone relies on email to get work done – yet most people use separate applications from their email to help them with various business processes. Streak fixes this problem by letting you do sales, hiring, fundraising, bug tracking, product development, deal flow, project management and almost any other business process right inside Gmail. In this post, I want to illustrate how we have used Google Cloud Platform to build Streak quickly, scalably and with the ability to deeply analyze our data.

We use several Google technologies on the backend of Streak:

App Engine to serve our app.

App Engine Datastore to persist user data.

Memcache to make operations fast.

BigQuery to analyze our logs and power dashboards.

App Engine Search API to let users sift through their data.

Prediction API to machine learn over user data.

Google Translate API to translate our app to over 40 languages.

Our core learning is that you should use the best tool for the job. No one technology will be able to solve all your data storage and access needs. Instead, for each type of functionality, you should use a different service. In our case, we aggressively mirror our data in all the services mentioned above. For example, although the source of truth for our user data is in the App Engine Datastore, we mirror that data in the App Engine Search API so that we can provide full text search, Gmail style, to our users. We also mirror that same data in BigQuery so that we can power internal dashboards.

System Architecture

App Engine - We use App Engine for Java primarily to serve our application to the browser and mobile clients in addition to serving our API. App Engine is the source of truth for all our data, so we aggressively cache using Memcache. We also use Objectify to simplify access to the Datastore, which I highly recommend.

Google Cloud Storage - We mirror all of our Datastore data as well as all our log data in Cloud Storage, which acts as a conduit to other Google cloud services. It lets us archive the data as well as push it to BigQuery and the Prediction API.

BigQuery - Pushing the data into BigQuery allows us to run non-realtime queries that can help generate useful business metrics and slice user data to better understand how our product is getting used. Not only can we run complex queries over our Datastore data but also over all of our log data. This is incredibly powerful for analyzing the request patterns to App Engine. We can answer questions like:

Which requests cost us the most money?

What is the average response time for every URL on our site over the last 3 days?

BigQuery helps us monitor error rates in our application. We process all of our log data with debug statements, as well as something called an “error type” for any request that fails. If it’s a known error, we'll log something sensible, and we log the exception type if we haven’t seen it before. This is beneficial because we built a dashboard that queries BigQuery for the most recent errors in the last hour grouped by error type. Whenever we do a release, we can monitor error rates in the application really easily.

A Streak dashboard powered by BigQuery showing current usage statistics

In order to move the data into Cloud Storage from the Datastore and LogService, we developed an open source library called Mache. It’s a drop-in library that can be configured to automatically push data into BigQuery via Cloud Storage. The data can come from the Datastore or from LogService and is very configurable - feel free to contribute and give us feedback on it!
Google Cloud Platform also makes our application better for our users. We take advantage of the App Engine Search API and again mirror our data there. Users can then query their Streak data using the familiar Gmail full text search syntax, for example, “before:yesterday name:Foo”. Since we also push our data to the Prediction API, we can help users throughout our app by making smart suggestions. In Streak, we train models based on which emails users have categorized into different projects. Then, when users get a new email, we can suggest the most likely box that the email belongs to.

One issue that arises is how to keep all these mirrored data sets in sync. It works differently for each service based on the architecture of the service. Here’s a simple breakdown:

Having these technologies easily available to us has been a huge help for Streak. It makes our products better and helps us understand our users. Streak’s user base grew 30% every week for 4 consecutive months after launch, and we couldn’t have scaled this easily without Google Cloud Platform. To read more details on why Cloud Platform makes sense for our business, check out our case study and our post on the Google Enterprise blog.Aleem Mawani is the co-founder of Streak.com, a CRM tool built into Gmail. Previously, Aleem worked on Google Drive and various ads products at Google. He has a degree from the University of Waterloo in Software engineering and an MBA from Harvard University.Posted by Scott Knaster, Editor

Google

Labels: app engine , bigquery , cloud , guest post , prediction api

Labels

#freeandopen
#GooglePlay #AndroidDevStory #PlayStore #DeveloperConsole #StoreListingExperiments
#io12
#io13
#io14
#io15
#io16
#io2012
#io2013
#io2014
+1
20% project
3d
about.com
accessibility
Administrative APIs
AdMob
Ads
adsense
advogato
africa
agpl
ajax
ajax apis
ajax search
ajax search books news apis
all for good
amarok
analytics
android
Android Studio
android wear
apache
api
apis
apis console
apis explorer
apis. charts
app engine
app indexing
app indexing api
App Invites
apple
apps
apps script
asia
atom publishing protocol
Auth
authentication
authsub
awards
axsjax
barcodes
beacon
beacons
bespin
best practices
bigquery
blogger
book search
books API
bootcamp
browser
building ajax apps
buzz
c++
caja
caldav
calendar
camino
campfire one
caption
cardboard
CardDAV
cast
chinese
chrome
chrome apps
chrome dev summit
chrome devtools
chrome experiment
chrome extensions
chrome os
chrome web store
chromecast
chromium
chronoscope
cifs
classes
classroom api
client libraries
closure tools
cloud
cloud datastore
cloud platform
cloud portability
cloud services
cloud sql
cloud storage
cms
CocoaPods
code for educators
code jam
code review
code-in
codeedu
coffee with a googler
collada
commerce
community
compression
compressorhead
computing heritage
conferences
contacts api
contest
contextual gadgets
conversations
couchdb
countdown to I/O 2012
courses
creative commons
cricket
crisis response
cryptography
css
css3
custom search
custom search api
danish linux forum
dart
datastore
design
devart
develop
developer
developer expert
developers
developers. meetup
devfest
devfest developer chrome maps social wave apps
discovery service
django
dns
docs
documentation
documents list api
dojo
doodles
dot net
doubleclick
dreamweaver
Drive
drupal
eclipse
eclipsecon
eddystone
education
email
EMEA
enterprise
event
events
evolution
execution api
extensions
faster web
featured
feeds
finance
Firebase
Firebase Analytics
Firebase Cloud Messaging
firebug
firefox
firevox
fitness
font api
fosdem
freebsd
freenet
Fridaygram
fusion tables
gadgets
games
gcc
GDA
gdata
gdd07
gdd08
gdd09
GDD11
GDE
gdg
gdl
gdl weekly
gears
geo
geolocation
geoserver
getpaid
ghop
git
github
gmail
Gmail APIs
gnome
gnome women's summer outreach program
Go
goo.gl
Google
Google APIs
google apps
google apps api
google apps for your domain
google apps marketplace
google buzz
google cast
google chart api
google checkout
google chrome
Google Cloud Messaging
google cloud storage
google code
google code project hosting
google code search
google code university
google compute engine
google data apis
google data protocol
google developer day
google developer days
Google Developers Academy
Google Developers Live
Google Developers site
Google Developers University Consortium
google docs
google doctype
Google Drive
google earth
google fit
Google Fonts
google friend connect
google gadgets
google gears
google grants
google health
Google I/O
Google Identity Platform
google io
google mashup editor
Google Noto fonts
google play services
Google Science Fair
Google sheets
Google Spreadsheets API
google storage
google summer of code
Google tech talk
google technoloy user groups
google tv
google visualization api
google wallet
Google Wave
google web elements
google web toolkit
google.org
google+
googlecast
googleio
googlenew
googlewebelements googleio
GPE
green linux
gsoc
gtags
gtug
guest post
guice
gulp
GWSOP
gwt
hackathon
hacking
hackthon
hangouts
haproxy
hg
hibernate
howto
hpux
html
html5
I/O Extended
I/O Live
ical
identity
ietf
ignite
igoogle
iguanas
iiw
image search
Imara
in-app payments
internationalization
internet explorer
internet of things
interviews
ios
iOS SDK
ipad
iphone
israel
jaiku
java
javascript
jetpack
joomla
joomladayus2007
joomladayusa
karaoke
KDE
KDE 4.0
kernel
kernel summit
khronos
kids
kml
korean
labs
latitude
lca
lessons
licenses
linux
linux foundation
linux summit
linux virtual server
linuxconf eu
LoCo
london
mac
MacFuse
machine learning
maps
maps apis
Marketplace
material design
MDL
meetup
mercurial
MIT CSAIL
mobile
mod_pagespeed
Moderator
MOOC
mozilla
mylar
myspace
MySQL
mythtv
named
narratives
native client
nearby
netbsd
non-profit
nonsense
nosql
notifications
nss
O3D
oauth
OAuth playground
objective-c
OCaml
ocr
ODF
oha
OOXML
open data
open source
open source blog
open web
openajax alliance
opengl
openid
opensocial
openssh
openssl
oreilly
orkut
oscon
oscon2007
osi
oss devs
ossjam
osx
pactester
page speed
payments
performance
phone
photos
picasa
picasa web
places API
play services
playground
plone
plone sprint
podcast
polymer
portugal
posix
PowerMeter API
prediction api
preview
programming
project hosting
Project Loon
Project Tango
proximity
pubsubhubbub
py3k
python
python sprint
rails
random hacks of kindness
reader
Remote Config
research
result snippets
rhino
salesforce
samba
sandbox
scalability
screencast
sdk
search
security
service worker
Sheets API
shindig
shopping
sidewiki
sign-in
silverstripe
sitemaps
sites api
sixapart
sketchup
Smart Lock for Passwords
soap search api
soc
social
social graph
solaris
souders
spa2007
spdy
speakers
speed
speed tracer
standards
startup
storage
Street View
student programs
students
stuff
subscribed links
subversion
summer of code
SVG
sxsw
syndication
tasks API
templates
testing
themes
tool
tools
topp
training
tranparency
transit
translate
tutorials
tv
ubiquitous computing
ubiquity
ubucon
ubuntu
Udacity
unit test
unix
video
videos
Vim
virtual keyboard
virtual reality
visualization
VR
Wearables
web animations api
web components
web designer
web exponents
web fonts
web performance
web platform docs
webfonts
webgl
webmaster
WebP
website optimizer
weekly roundup
WhiteHouse.gov
Who's at Google I/O
windows
windows programming
Winter of Code
wtm
xauth
yahoo
youtube
zlib
zurich
ZXing

Company-wide

Official Google Blog
Public Policy Blog
Student Blog

Products

Google for Work Blog
Chrome Blog
Official Android Blog

Developers

Ads Developer Blog
Android Developers Blog
Developers Blog

Google
Privacy
Terms