Going For Google Scale

“Google scale” can refer to a few different concepts, but it generally relates to the massive infrastructure and operational capabilities of Google. This includes its ability to handle vast amounts of data, process complex computations, and serve billions of users with various services. It can also refer to the process of scaling up Google Cloud Platform (GCP) resources to meet increasing demands.

Platform as a service ( PaaS ) and Backend as a Service ( BaaS ) have a lot to answer for in the mindset of many a developer, myself included, who thinks to themselves ‘but its got to scale’ and with that in mind, Google scale being the goal! This can be rather naive and an over simplistic viewpoint, as scaling to even lower levels is a multi faceted series of steps in a long journey, even for Google.

A principle of scale can be expressed as something that can be N+1 and be recreated 1..N times. A solution is relatively easy to produce for a single, bespoke use case. But a bit more effort is needed for this to be reproduced and re-used for multiple purposes.

Infrastructure as Code ( IaC ) helps us get to this end goal. It is not just a way of automating deployments. It is about making something available to more than a single instance. It can make that service recoverable, ephemeral even. IaC is, I believe, like the magic that Arthur C Clarke spoke of when he likened what the ancients may have perceived as our present day technology to be.

It is a diamond of thought expressed as code, often times imperative in nature initially, but can become declarative over time.

We can be discouraged from learning about and using IAC by an overriding desire to ‘ship fast’, when the only obvious way to efficiently do this is to drop our code into a PaaS or SaaS that does all the hard stuff for you. This is at a price that may start small but will assuredly ‘scale’ to something the provider wants it to. True, many projects never reach viral status but they will inevitably be used over time and we will always end up paying more for their success when running on someone else’s cloud services.

An anti pattern I have come across, when applied to IaC can be ‘don’t sweat the small stuff’. Service delivery and the grind of making a laptop based software work as a deliverable service in the cloud can seen as a chore. After all, the developer needs to be focusing on the next big idea, not how to do the boring stuff!

Having spent 10’s of years doing both application development and delivery I can attest that this is far from reality. Becoming involved in service delivery is going to influence application development even at the micro level and will influence how software is developed throughout its entire life-cycle. Missing this out is missing a (very big) trick.

Having knowledge and appreciation of how things get delivered in ‘cloud services’ needs to be center most in our thoughts when starting a project, else we will be bound to build in problems that someone else needs to solve further down the track.

Another misconception can be that of over engineering. This, I believe, is the biggest problem of all in today’s technology stacks that results in the creation of technical debt. The complexity of getting “all the things” into a project from the beginning can easily translate to mind bending, eye watering complexity in simple applications that consequently deliver overly complex, difficult to maintain solutions.

Some people, I have seen, will try to implement every pattern and buzz word idea into every software they deliver just to show that they can do this. Perhaps this is also to improve career and resume but not to deliver simplicity. Simple is not necessarily stupid but simple can be stupid simple.

Going to Google scale has to start somewhere and even Google have to run up a service in single instances to begin with then scale over time.

Scaling then becomes a journey and is a series of steps forward that ultimately may become something like Netflix in the same way that all journeys to Rome start with the first step.

But its just that, a step by step process, not a performance.