Google Developers Blog: cloud

Cloud Platform at Google I/O - new Big Data, Mobile and Monitoring products

Wednesday, June 25, 2014

By Greg DeMichillie, Google Cloud Platform teamGoogle I/OIntroducing Google Cloud DataflowFlumeMillWheelDebug, trace and monitor your application in production New features for mobile developmentSnapchatRising StarGoogle Cloud DatastoreGoogle App EngineGoogle Compute EngineDatastore APIAndroid StudioJava ServletJava EndpointsApp Engine backend with Google Cloud MessagingApp Engine plug-in for Gradle,Greg DeMichillie has spent his entire career working on developer platforms for web, mobile, and the cloud. He started as a software engineer before making the jump to Product Management. When not coding, he's an avid photographer and gadget geek.Posted by Louis Gray, GooglerApache, Nginx, MongoDB, MySQL, Tomcat, IIS, Redis, Elasticsearch and Cassandra are trademarks of their respective owners.

Google I/O 2014 I/O Livestreams I/O Bytes Videos +Google Developers

Google Design Android Wear Android TV Android Auto

Explore the conference with the Google I/O 2014 Android app:

Updates to Google BigQuery following Cloud Platform Live

Monday, March 31, 2014

By Felipe Hoffa, Cloud Platform teamCross-posted from the Google Cloud Platform BlogEditor's note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.announcedGoogle BigQuery1000x increase in streaming capacitywe announced the ability to stream datacheck out the live demo from the keynoteTABLE_DATE_RANGE()documentation and syntax for table wildcard functionsImproved SQL support and table viewsGoogle Cloud SDK)User-defined metadataJSON parsing functionsJSON_EXTRACTJSON_EXTRACT_SCALARJSONPathSELECT json_extract_scalar( "{'book': { 'category':'fiction', 'title':'Harry Potter'}}", "$.book.category"); Fast parallel exportsexporting data documentationMassive price reductionsprice reductionR has its own connectortutorialPython pandas toovideo we made with PearsonActiveRecord connectorlogs with fluentdFelipe Hoffa is part of the Cloud Platform Team. He'd love to see the world's data accessible for everyone in BigQuery.Posted by Louis Gray, Googler

Bringing together the best of PaaS and IaaS

Thursday, March 27, 2014

By Navneet Joneja, Cloud Platform TeamCross-posted from the Google Cloud Platform blogEditor’s note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.Managed Virtual MachinesDeployment Manager

Managed Virtual MachinesGoogle Cloud Platform LivManaged Virtual MachinesGoogle Compute EngineGoogle App Enginevm:truehereIntroducing Google Cloud Deployment Manager Google Cloud Deployment Manager

The attributes of any Compute Engine virtual machines (e.g. instance type, network settings, persistent disk, VM metadata).

Health checks and auto-scaling

Startup scripts that can be used to launch applications or other configuration management software (like Puppet, Chef or SaltStack)

Deployment ManagerUpdate on operating systemsSuSE and Red Hat Enterprise Linuxsign up for accessSimple, lower pricesmentioned on Tuesday

We’ve dropped prices by 32% across the board.

We’re introducing sustained-use discounts, which lower your effective price as your usage goes up. Discounts start when you use a VM for over 25% of the month and increase with usage. When you use a VM for an entire month, this amounts to an additional 30% discount.

only pay for the minutes that you use

dedicated memcacheDatastoreSNI SSLPageSpeed hearing from youhereNavneet Joneja, Senior Product ManagerPosted by Louis Gray, Googler

New developer experiences for Cloud Platform

Wednesday, March 26, 2014

By Cody Bratt, Google Cloud Platform teamCross-posted from the Google Cloud Platform blogEditor's note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.Google Cloud PlatformIsolating production problems fasterGoogle App Engine

You can see here where Brad introduced an error into production.

Screen Shot 2014-03-21 at 1.36.21 PM.png

Shortening your time to fix

Fix simple problems directly in the browser

Trigger your push-to-deploy setup instantly

An integrated ecosystem of tools you knowGit push-to-deployo does the Google Cloud Platform team

Push-to-deploy now supports Java builds

The release history showing build, test and deployment status

Google Cloud SDKrecently posted gcloudinit

Cody Bratt is a product manager on the Google Cloud Platform teamPosted by Louis Gray, Googler

Google Cloud Platform Live: Blending IaaS and PaaS, Moore’s Law for the cloud

Tuesday, March 25, 2014

By Urs Hölzle, Senior Vice PresidentEditors note: Tune in to Google Cloud Platform Live for more information about our announcements. And join us during our 27-city Google Cloud Platform Roadshow which kicks off in Paris on April 7. Google Cloud Platform LiveCloud PlatformIndustry-leading, simplified pricingMoore's Law

Compute Engine reduced by 32% across all sizes, regions, and classes.

App Engine pricing simplified, with significant reductions in database operations and front-end compute instances.

Cloud Storage is now priced at a consistent 2.6 cents per GB. That's roughly 68% less for most customers.

Google BigQuery on-demand prices reduced by 85%.

Sustained-Use discountsSustained-Use Discounts

Sustained-Use Discounts automatically reward users who run VMs for over 25% of any calendar monthMaking developers more productive in the cloud

Build, test, and release in the cloud, with minimal setup or changes to your workflow. Simply commit a change with git and we'll run a clean build and all unit tests.

Aggregated logs across all your instances, with filtering and search tools.

Detailed stack traces for bugs, with one-click access to the exact version of the code that caused the issue. You can even make small code changes right in the browser.

Introducing Managed Virtual Machines Managed VMsExpanded Compute Engine operating system supportWindows Server 2008 R2Red Hat Enterprise LinuxSUSE Linux Enterprise ServerReal-Time Big DataBigQuery Streamingreserved query capacityConclusionPosted by Louis Gray, Googler

GA Release for Google APIs Client Library for .NET

Monday, March 17, 2014

By Dan Ciruli, Google Cloud Platform TeamNuGetOAuth 2.0uploadsdownloadsbatchingfifty Google APIsGetting StartedhereDan Ciruli is a Product Manager in the Cloud Platform Team intent on making developers' lives easier.Posted by Louis Gray, Googler

Announcing the global Cloud Platform Developer Roadshow

Tuesday, March 11, 2014

By Greg DeMichillie, Google Cloud Platform teamCross-posted from the Google Cloud Platform blogCloud Platform Developer RoadshowGoogle Cloud Platform Liveregister herejoin usGreg DeMichillie has spent his entire career working on developer platforms for web, mobile, and the cloud. He started as a software engineer before making the jump to Product Management. When not coding, he's an avid photographer and gadget geek.Posted by Louis Gray, Googler

Get started at no cost with a faster, larger Cloud SQL database

Thursday, November 8, 2012

By Joe Faith, Product ManagerCross-posted with the Official Google Enterprise BlogGoogle App EngineGoogle Cloud SQL

More Storage: We’re increasing the available storage on Cloud SQL to 100GB – ten times more than what used to be available.

Faster Reads: We’re increasing the maximum size of instances to 16GB RAM, a 4 times increase in the amount of data you can cache.

Faster Writes: We’re adding functionality for optional asynchronous replication, which gives the write performance of a non-replicated database, but the availability of a replicated one.

EU datacenter availability: Now you can choose to store your data and run your Cloud SQL database instance in either our US or EU data centers.

Integration with Google Apps Script: We’re making it quick and easy for businesses using Google Apps to use Cloud SQL. Publish and share data with Google Sheets, add data to Google Sites pages or create simple Google Forms without worrying about hosting or configuring servers.

Introducing a new trial offer Sign upJoe Faith is a Product Manager on the Google Cloud Team. In a previous life he was a researcher in machine learning, bioinformatics, and information visualization, and was founder of charity fundraising site Fundraising Skills.Posted by Scott Knaster, Editor

Streak brings CRM to the inbox with Google Cloud Platform

Tuesday, October 9, 2012

By Aleem Mawani, Co-Founder of StreakCross-posted with the Google App Engine BlogThis guest post was written by Aleem Mawani, Co-Founder of Streak, a startup alum of Y Combinator, a Silicon Valley incubator. Streak is a CRM tool built into Gmail. In this post, Aleem shares his experience building and scaling their product using Google Cloud Platform.Everyone relies on email to get work done – yet most people use separate applications from their email to help them with various business processes. Streak fixes this problem by letting you do sales, hiring, fundraising, bug tracking, product development, deal flow, project management and almost any other business process right inside Gmail. In this post, I want to illustrate how we have used Google Cloud Platform to build Streak quickly, scalably and with the ability to deeply analyze our data.

We use several Google technologies on the backend of Streak:

App Engine to serve our app.

App Engine Datastore to persist user data.

Memcache to make operations fast.

BigQuery to analyze our logs and power dashboards.

App Engine Search API to let users sift through their data.

Prediction API to machine learn over user data.

Google Translate API to translate our app to over 40 languages.

Our core learning is that you should use the best tool for the job. No one technology will be able to solve all your data storage and access needs. Instead, for each type of functionality, you should use a different service. In our case, we aggressively mirror our data in all the services mentioned above. For example, although the source of truth for our user data is in the App Engine Datastore, we mirror that data in the App Engine Search API so that we can provide full text search, Gmail style, to our users. We also mirror that same data in BigQuery so that we can power internal dashboards.

System Architecture

App Engine - We use App Engine for Java primarily to serve our application to the browser and mobile clients in addition to serving our API. App Engine is the source of truth for all our data, so we aggressively cache using Memcache. We also use Objectify to simplify access to the Datastore, which I highly recommend.

Google Cloud Storage - We mirror all of our Datastore data as well as all our log data in Cloud Storage, which acts as a conduit to other Google cloud services. It lets us archive the data as well as push it to BigQuery and the Prediction API.

BigQuery - Pushing the data into BigQuery allows us to run non-realtime queries that can help generate useful business metrics and slice user data to better understand how our product is getting used. Not only can we run complex queries over our Datastore data but also over all of our log data. This is incredibly powerful for analyzing the request patterns to App Engine. We can answer questions like:

Which requests cost us the most money?

What is the average response time for every URL on our site over the last 3 days?

BigQuery helps us monitor error rates in our application. We process all of our log data with debug statements, as well as something called an “error type” for any request that fails. If it’s a known error, we'll log something sensible, and we log the exception type if we haven’t seen it before. This is beneficial because we built a dashboard that queries BigQuery for the most recent errors in the last hour grouped by error type. Whenever we do a release, we can monitor error rates in the application really easily.

A Streak dashboard powered by BigQuery showing current usage statistics

In order to move the data into Cloud Storage from the Datastore and LogService, we developed an open source library called Mache. It’s a drop-in library that can be configured to automatically push data into BigQuery via Cloud Storage. The data can come from the Datastore or from LogService and is very configurable - feel free to contribute and give us feedback on it!
Google Cloud Platform also makes our application better for our users. We take advantage of the App Engine Search API and again mirror our data there. Users can then query their Streak data using the familiar Gmail full text search syntax, for example, “before:yesterday name:Foo”. Since we also push our data to the Prediction API, we can help users throughout our app by making smart suggestions. In Streak, we train models based on which emails users have categorized into different projects. Then, when users get a new email, we can suggest the most likely box that the email belongs to.

One issue that arises is how to keep all these mirrored data sets in sync. It works differently for each service based on the architecture of the service. Here’s a simple breakdown:

Having these technologies easily available to us has been a huge help for Streak. It makes our products better and helps us understand our users. Streak’s user base grew 30% every week for 4 consecutive months after launch, and we couldn’t have scaled this easily without Google Cloud Platform. To read more details on why Cloud Platform makes sense for our business, check out our case study and our post on the Google Enterprise blog.Aleem Mawani is the co-founder of Streak.com, a CRM tool built into Gmail. Previously, Aleem worked on Google Drive and various ads products at Google. He has a degree from the University of Waterloo in Software engineering and an MBA from Harvard University.Posted by Scott Knaster, Editor

Now in BigQuery: batch queries and a connector for Excel

Wednesday, August 29, 2012

By Ryan Boyd, Developer Advocate for Cloud Data ServicesBatch priority queriessignificantly lower cost    Job job = new Job();    JobConfiguration config = new JobConfiguration();
   JobConfigurationQuery queryConfig = new JobConfigurationQuery();
   config.setQuery(queryConfig);
   job.setConfiguration(config);
   queryConfig.setQuery(querySql);
   queryConfig.setPriority("BATCH");
   com.google.api.services.bigquery.Bigquery.Jobs.Insert insert =      bigquery.jobs().insert(projectId, job); Developers GuideBigQuery Connector for ExcelGoogle Apps Script integrationtry it right nowGoogle+holding office hoursFriday at 10 AM PDTMicrosoft and Excel are registered trademarks of Microsoft CorporationRyan Boyd is a Developer Advocate, focused on cloud data services. He's been at Google for 6 years and previously helped build out the Google Apps ISV ecosystem. He published his first book "Getting Started with OAuth 2.0" with O'Reilly.Posted by Scott Knaster, Editor

Helping developers build more applications on Google BigQuery

Tuesday, July 24, 2012

Amanda

Ju-kay

By Ju-kay Kwek and Amanda Bradford, Google BigQuery Teamlaunchedbusiness problems

Import data from multiple sources into BigQueryInformaticaPervasive SoftwareTalendSQLstreamTribusPointBuild rich interactive dashboards on BigQueryQlikTechJaspersoftBime AnalyticsMetric InsightsPixelfishGoogle BigQueryJu-kay Kwek is the Product Management Lead for Google's Cloud Big Data initiative. In this role, he focuses on creating services that enable businesses and developers to harness Google's unparalleled data processing infrastructure and algorithms to tackle Big Data needs.Amanda Bradford works in Business Development at Google, driving strategic alliances and partnerships for Google and specializing in the Google Cloud Platform.Posted by Scott Knaster, Editor

Google Compute Engine: Computing without limits

Thursday, June 28, 2012

By Craig McLuckie, Product Manager, Google Compute EngineGoogle Compute Engine

Scale. At Google we tackle huge computing tasks all the time, like indexing the web, or handling billions of search queries a day. Using Google's data centers, Google Compute Engine reduces the time to scale up for tasks that require large amounts of computing power. You can launch enormous compute clusters - tens of thousands of cores or more.

Performance. Many of you have learned to live with erratic performance in the cloud. We have built our systems to offer strong and consistent performance even at massive scale. For example, we have sophisticated network connections that ensure consistency. Even in a shared cloud you don’t see interruptions; you can tune your app and rely on it not degrading.

Value. Computing in the cloud is getting even more appealing from a cost perspective. The economy of scale and efficiency of our data centers allows Google Compute Engine to give you 50% more compute for your money than with other leading cloud providers. You can see pricing details here.

Compute. Launch Linux VMs on-demand. 1, 2, 4 and 8 virtual core VMs are available with 3.75GB RAM per virtual core.

Storage. Store data on local disk, on our new persistent block device, or on our Internet-scale object store, Google Cloud Storage.

Network. Connect your VMs together using our high-performance network technology to form powerful compute clusters and manage connectivity to the Internet with configurable firewalls.

Tooling. Configure and control your VMs via a scriptable command line tool or web UI. Or you can create your own dynamic management system using our API.

RightScalePuppet LabsOpsCodeNumerateCliqrMapRheresign up hereCraig McLuckie is the Product Management Lead for Google Compute Engine. He spends his days working with an amazing engineering team to open Google’s infrastructure to the world.Posted by Scott Knaster, Editor

Pricing plan announced for Google Cloud SQL

Thursday, May 10, 2012

By Joe Faith, Product ManagerDaffodil

For developers who want to try out the service, or who have lightweight applications, we offer a flexible "per use" pricing scheme. For example, you can get started with a cloud hosted MySQL database for around a dollar per month. You pay for just what you use.

For developers with more traffic, there are package plans that are more economical and help you predict your costs in advance.

https://developers.google.com/cloud-sql/docs/billinghttps://developers.google.com/cloud-sql/Joe Faith is a Product Manager on the Google Cloud Team. In a previous life he was a researcher in machine learning, bioinformatics, and information visualization, and was founder of charity fundraising site Fundraising Skills. Posted by Scott Knaster, Editor

Using Google BigQuery to learn from GitHub data

Thursday, May 3, 2012

By Ilya Grigorik, Web Performance EngineerGitHub ArchiveGoogle BigQueryDremel

chart showing number of commits by language

moresign up for BigQuerygithubarchive.orgGitHub Data ChallengeIlya Grigorik is a Web Performance Engineer and Advocate at Google, an open-source evangelist, and an analytics geek. You can find him on GitHub under igrigorik, and blogging about web performance at igvita.com.Posted by Scott Knaster, Editor

Google BigQuery brings Big Data analytics to all businesses

Tuesday, May 1, 2012

By Ju-kay Kwek, Product Manager, BigQueryBigQueryannouncingClariticsCrystalloidsBigQueryUI or REST interfacesign uppricing plancontact a sales representativeuser forums+Google Enterprise pageJu-kay Kwek is the Product Management Lead for Google's Cloud Big Data initiative. In this role, he focuses on creating services that enable businesses and developers to harness Google's unparalleled data processing infrastructure and algorithms to tackle Big Data needs.Posted by Scott Knaster, Editor

Fridaygram: goodbye to 2011

Friday, December 23, 2011

By Scott Knaster, Google Code Blog Editornew APIs and client librariesHTML5 and Chromemaking the web fasterAndroidGoogle+Google Appscloud topicscommerce and in-app paymentsannouncements, details, and recapsguest posts during Google I/ODart previewSwiffy launchFridaygramsall the postsone more Easter egg

Google Prediction API: faster, easier to use, and more accurate

Thursday, December 22, 2011

By Marc Cohen, Developer Relations

Faster on-ramp: We’ve made it easier to get started by enabling you to create an empty model (by sending a trainedmodels.insert request with no storageDataLocation specified) and add training data using the trainedmodels.update method. This change allows you to submit your model contents without needing to stage the data in Google Cloud Storage.

Improved updates: The algorithms used to implement model updates (adding additional data to existing models) have been modified to work faster than ever.

More classification algorithms: We’ve increased the number of classification algorithms used to build predictive models, resulting in across-the-board improvements in accuracy.

Integration with Google Apps Script: Prediction services are now available as part of Google Apps Script, which means you can integrate prediction services with Google Docs, Google Maps, Gmail, and other great Google products.

Marc Cohen is a member of Google’s Developer Relations Team in Seattle. When not teaching Python programming and listening to indie rock music, he enjoys using the Google Prediction API to peer into the future.Posted by Scott Knaster, Editor

Labels

Archive