The Network is the Computer

Go and the Localization of Distributed Computing

Go makes it natural to take techniques we use to design distributed systems and reuse them to build non-distributed systems. Goroutines, channels, interfaces, maps, and the select statement are primitives that have conceptual cousins in nodes, queues, rpc, caches and scheduling. When we build programs in this fashion, we inherit a whole body of proven techniques for verification, measurement, and optimization ("scaling") that have been developed for the distributed world. This relationship is explored and demonstrated through the evolution of a real Go program.

The first part of the talk is a brief explanation of these concepts and how they are related to each other not just semantically but also in how they influence the structure of the systems that are built with them. Of particular importance will be the assertion that Go implicitly encourages this type of program composition both by the primitives it provides and the ones that it does not provide.

The second part of the talk will involve the creation, in a number of steps, of a simple data processing program. The initial iteration will be straight forward and perform poorly, and gradually through measurement (profiling and instrumentation) we will identify bottlenecks and improve performance using analogues of classic distributed system techniques:

splitting processing up into multiple stages (pipelining)
adaptively scaling goroutines in the pipeline (elastic scaling)
improving concurrency of a map-based cache (sharding)

Beginner and Intermediate Gophers should come away with a solid approach to designing and optimizing Go programs; expert Gophers might have concepts that they understand implicitly laid out explicitly.

Beginners often find it hard to assemble Go's powerful primitives; they're used to having more of the puzzle filled in for them by frameworks. Too often lip service is paid to the idea of composition, orthogonality, emergent behavior, and the agility of disposing with complex frameworks without simple but compelling examples. Go's widespread adoption in the cloud will make the philosophical core of the talk resonate with the community. I have been using Go for 3 years and building these types of systems with Go in production for over a year.