Seeking Simplicity

A blog about DevOps, clouds, architecture, and everything else...

Learning to Go, Part I: Interfaces

This article was originally published in the April 2014 issue of NFJS the Magazine.

This article begins an introductory series on the Go programming language. Go is a language optimized for large-scale software engineering and is rapidly becoming the language of choice for building cloud services. It does this in a very interesting way, optimizing for simplicity rather than complexity and taking a “less is exponentially more” approach.

Three ways in which this really stands out are:

  • A pragmatic approach to component design (through implicitly satisfied interfaces)

  • The need to forget much of what you’ve learned about object-oriented programming (and to prefer composition over inheritance)

  • A concurrency model that mere mortals can use to great effect (via goroutines and channels)

Overall, it’s a language with the productivity of a dynamically-typed, interpreted language with the safety and efficiency found in a statically-typed, compiled language.

This series will focus on learning Go from the perspective of Java developers. We’ll begin with design ideas informed by Go’s primary method of abstraction: interfaces.

What If I Told You Classes Were Unnecessary?

Object-oriented programming (OO) has dominated large-scale software engineering for quite some time now. While OO finds its roots in simulation programming, it moved into and began to dominate the mainstream of business programming with the advent of languages like C++ and Java. When we design programs in an OO language, we usually begin with an object model, defining a taxonomy of objects representing all of the entities, or nouns, in our business domain. Languages like C++ and Java support this via the class construct, with a class defining a template for creating a particular type of object. Classes normally can be extended, creating subclasses. This allows us to define a hierarchy of types that inherit characteristics and behavior from one another.

But what if I told you that classes were unnecessary? This may seem like heresy, but we already know it. JavaScript has the feel of an OO language, but it does not have classes at all. In fact, it utilizes what’s called prototype-based programming [PBP]. With prototype-based programming, we achieve reuse by cloning existing objects that serve as prototypes.

Those of us that have surfed the wave of functional programming have discovered that it’s indeed possible to structure large programs around a very basic set of types (sets, lists, maps, etc.), along with a large collection of behaviors, or functions, that can operate on those types. In these languages, we don’t really see anything resembling an object!

The Go programming language is somewhat unique in that it offers many OO-like constructs, but does not offer either classes or prototypes. But it is also not correct to refer to it as a functional language per se.

Favoring Composition Over Inheritance

While the Java programming language has always included a type system featuring inheritance, it has long been considered best practice to favor composition over inheritance. When Joshua Bloch, now Chief Java Architect at Google and then Distinguished Engineer at Sun Microsystems, wrote his seminal work Effective Java [EJ], he included as Item #16: “Favor composition over inheritance.”

When we use inheritance, we must be careful to implement subclasses such that they are in an is-a relationship with their parents. In other words, any context that expects an instance of the parent type should also work well with an instance of the subtype. We call this substitutabilty, which is expressed well by the Liskov Substitution Principle [LSP]. When we don’t follow this principle, we usually end up creating a lot of code that uses type checking to implement special behaviors. In doing so, we create fragile systems that don’t abide by the Open-Closed Principle [OCP].

When we instead utilize composition, we build classes that are in a has-a relationship with their components. If we’d like to encapsulate the relationship between a class and its components, we can apply delegation, forwarding method calls to the underlying component. This makes it appear as if our object is-a instance of another type in terms of behavior, without the problems associated with inheritance-based reuse.

The Go programming language also emphasizes composition over inheritance, in that it does not even provide inheritance as a language construct! Instead, it provides the ability to use composition at the type and behavior level. Let’s see how.

Structs FTW!

In Go we can define composite data types using structs. You may be familiar with structs if you’ve spent any time programming in C. Structs are simply a sequence of fields, each of which has a name and a type. If you take a look at A Point struct, you’ll see that we’ve defined a struct representing a point in Cartesian space.

A Point struct
type Point struct {
        X, Y float64
}

New Go types are defined using the type keyword. We give our type the name Point. It’s important to note that we’ve used uppercase here. Visibility in Go is determined by the case of an identifier. All code in Go is defined within a package. If you’d like your types and functions to be visible beyond the confines of your package, you need to export them by starting them with an uppercase letter. If you use a lowercase letter or underscore (_), they will be unexported, roughly equivalent to the Java concept of private.

You’ll quickly notice that we’ve defined two fields in our struct, X and Y, both of which take on the type float64, representing a 64-bit floating-point number.

Note
Unlike Java, Go places the type declarations after the field names (the same applies when declaring function arguments) - you’ll get used to this eventually!
The keen observer will also note the case of the field names. Just as case matters when dealing with type visibility beyond package boundaries, case also matters when dealing with field visibility beyond struct boundaries. Because both X and Y are uppercase, they are exported from the struct. Had they been x and y, they would be unexported, and therefore only visible within the struct. To find out how we’ll deal with that, hold on until the next section!

Update (2014-07-09): My understanding of this point was corrected on Reddit. Case only affects visibility with respect to packages, so x and y would also be accessible to any code within the package enclosing the struct. See this example.

So how do we create one of our Points? We have a couple of options. First, we can utilize the new function:

        p := *new(point.Point)
        p.X = 1
        p.Y = 2

We prefixed our call to new with an asterisk (*) in order to dereference the pointer returned by new. Yes, that’s right, Go has pointers. With that said, there’s no pointer arithmetic, so they’re not quite as scary as what we find in languages like C and C++. They almost represent a happy medium between C and Java.

Note
You also don’t have to worry about freeing memory! Go is a garbage collected language.

Built-in types in Go start zeroed, rather than nil. In the case of float64, they take on the value 0. It’s good practice for our types to start zeroed as well, and we get that for free here. Our newly initialized Point represents the Cartesian coordinate (0,0). The next two statements move our Point to (1,2).

We can eliminate the two step process by utilizing a composite literal:

        p = point.Point{X: 1, Y: 2}

A composite literal creates a new instance each time it is evaluated, and initializes each field with the given value. In addition, this particular Point starts out dereferenced. If we actually want a pointer to it, we need to prefix the literal with an ampersand (&).

Note
It is generally considered idiomatic Go to prefer the composite literal style to using new.

Enough about structs, on to methods!

Methods

Go methods are just functions that happen to have a receiver argument. You can find one in Method for translating Points.

Method for translating Points
func (p Point) Translate(xDist float64, yDist float64) Point {
        return Point{p.X + xDist, p.Y + yDist}
}

The Translate method takes two distances along the x and y-axes, and translates a point from its current position to a new position by returning a new Point literal that combines the current x and y coordinates with those distances. We call Translate in Calling Translate, and the output is found in Output of calling Translate.

Calling Translate
        q := p.Translate(5, 5)
        fmt.Printf("Translated %v to %v\n", p, q)
Output of calling Translate
$ go run point.go
Translated {1 2} to {6 7}

Because we’re not operating on a pointer to a Point, there’s no way for us to affect the existing instance. If we couple this with unexported fields, we’d have completely controlled access to a Point instance’s fields.

If we want to mutate a Point, we need our method to use a pointer as the receiver argument rather than a value. We do this in Method for translating a Point via a pointer.

Method for translating a Point via a pointer
func (p *Point) TranslatePointer(xDist float64, yDist float64) {
        p.X = p.X + xDist
        p.Y = p.Y + yDist
}

As you can see, rather than creating a new Point and returning it, this method returns no value and directly mutates the Point referred to by the pointer passed in as a receiver. Notice that Go conveniently dereferences the pointer automatically when using the dot (.) operator to access fields. We call this method on a pointer to a Point in Calling TranslatePointer (with output in Output of calling TranslatePointer), again using an ampersand (&) to tell Go we’d like a pointer to a Point, not a Point value.

Calling TranslatePointer
        qP := &point.Point{X: 1, Y: 2}
        qP.TranslatePointer(5, 5)
        fmt.Printf("Translated using pointer to %v\n", qP)
Output of calling TranslatePointer
$ go run point.go
Translated using pointer to &{6 7}

Now that we have the capability to create types with associated methods, let’s look at how we can compose new composite types from existing ones.

Type composition with structs

Let’s imagine that we’d like to extend the concept of Point and add the notion of color. If we were working in Java, we might do the following:

public class ColorPoint extends Point {
  private Color color;

  // rest omitted...
}

Since Go doesn’t have the concept of classes or inheritance, we need to work with structs to accomplish our goal. We can compose a type called ColorPoint by embedding the Point type and then adding the additional field representing color (Embedding Point into ColorPoint).

Embedding Point into ColorPoint
const (
        BLUE  = iota
        RED   = iota
        GREEN = iota
)

type ColorPoint struct {
        Point Point
        Color int
}

First note the use of the const keyword. We’re defining a set of integer constants to represent our colors. This is the closest approximation that Go has to an enumerated type (such as Java’s enum) and is usually more that sufficient. The predeclared identifier iota represents successive untyped integer constants. It is reset to 0 whenever const appears again in the source. This has the effect in our code of setting BLUE to 0, RED to 1, and GREEN to 2.

Next we define our ColorPoint type as a struct. We embed Point as an exported field also called Point, and we define an additional exported field called Color that is typed as an int. Now, in Creating and printing a ColorPoint, we’ll create an instance of ColorPoint, and then we’ll print out both the previously defined Point instance as well as the ColorPoint instance (results in Output of creating and printing a ColorPoint).

Creating and printing a ColorPoint
        r := point.ColorPoint{Point: point.Point{X: 1, Y: 4}, Color: point.BLUE}

        fmt.Printf("Point: %v\n", p)
        fmt.Printf("Color Point: %v\n", r)
Output of creating and printing a ColorPoint
$ go run point.go
Point: {1 2}
Color Point: {{1 4} 0}

We’ve now successfully created a composite type, but we’re missing something. Let’s press on.

Houston, we have a problem…

Here’s a summary of our problem:

  • We have Points.

  • We have ColorPoints.

  • ColorPoints are like Points, but they are definitely not Points since Go does not have inheritance.

  • That said, we’d like to be able to write methods that can interoperate between them. What do we do?

As an example, since both Point and ColorPoint have x and y coordinates, it might be interesting if we could compute the Euclidean distance [EUC] between them. As a reminder, the formula can be found in Euclidean distance formula.

distance formula
Figure 1. Euclidean distance formula

As expressed here, you can think of p1 and q1 as the x coordinates for the two points, and p2 and q2 as the y coordinates for the two points. So how do we implement a method that will allow us to compute this formula in such a way that will work with Points, ColorPoints, or both?

Go Interfaces

Fortunately Go contributes a very powerful version of interfaces to the abstraction conversation. Go interfaces describe groups of behaviors in the form of method signatures that implementors of the interface must implement. So far this doesn’t sound unlike Java interfaces. Let’s take a look at two Go interfaces in Positioner and Distancer interfaces.

Positioner and Distancer interfaces
type Positioner interface {
        Coordinates() Point
}

type Distancer interface {
        DistanceTo(p Positioner) float64
}

Notice that interfaces are themselves types. This is important, because it allows us to define function and method signatures that accept interfaces as arguments. Positioner’s Coordinates() method should provide us the position of any implementor in terms of a Point. Distancer’s DistanceTo() method will calculate the distance between any implementor and a Positioner.

Note
It is idiomatic in Go for interface names to be suffixed with “-er.”

However, unlike Java interfaces, where we might write:

public class Point implements Positioner, Distancer {
  // implementation omitted...
}

Go does not have an implements keyword. In fact, if you think about the way we’ve defined structs and methods so far, the only place in which we indicated any attachment between a struct and a method is in the method signature itself, in the form of the receiver parameter. So how does this work in Go?

Let’s talk about how interfaces are satsified.

In languages like Java, an interfaces is satisfied explicitly. Classes are tagged with implements Interface. The compiler then looks up that interface and identifies all of the method signatures defined by it. It then examines the class to ensure that all of those method signatures have a concrete implementation in either the class itself or one of its parents.

In Go, interfaces are satisfied implicitly. We do not tag structs in any way. For a type to implement an interface, it simply needs to implement all of the method signatures defined by that interface. When we use a type in the context of an interface (e.g. we pass a type into a function that expects an interface as one of its arguments), the compiler will check to see if that type satisfies the interface.

Let’s see how this falls out in practice.

First, let’s define a function that can calculate the distance between two Positioners (Calculates the distance between two Positioners).

Calculates the distance between two Positioners
func distanceBetween(a Positioner, b Positioner) float64 {
        p := a.Coordinates()
        q := b.Coordinates()
        sqOfXDist := math.Pow(p.X-q.X, 2)
        sqOfYDist := math.Pow(p.Y-q.Y, 2)
        return math.Sqrt(sqOfXDist + sqOfYDist)
}

Next, in Point satisfies Positioner and Distancer, we satisfy both interfaces for Point.

Point satisfies Positioner and Distancer
func (p Point) Coordinates() Point {
        return p
}

func (p Point) DistanceTo(pos Positioner) float64 {
        return distanceBetween(p, pos)
}

In ColorPoint satisfies Positioner and Distancer, we satisfy both interfaces for ColorPoint.

ColorPoint satisfies Positioner and Distancer
func (cp ColorPoint) Coordinates() Point {
        return cp.Point
}

func (cp ColorPoint) DistanceTo(pos Positioner) float64 {
        return distanceBetween(cp, pos)
}

This all results in our ability to interchange Points and ColorPoints in calls to DistanceTo() (Calculating the distance between Point and ColorPoint). The output of these calls is found in Output of calculating the distance between Point and ColorPoint.

Calculating the distance between Point and ColorPoint
        fmt.Printf("Dist b/w p and q = %v\n", p.DistanceTo(r))
        fmt.Printf("Dist b/w q and p = %v\n", r.DistanceTo(p))
Output of calculating the distance between Point and ColorPoint
$ go run point.go
Dist b/w p and q = 2
Dist b/w q and p = 2

At first glance this may not seem so impressive, nor may it seem to be that great of an advantage over what’s available to us in Java. The power, however, is hiding just under the surface. When we think about implementing an interface in Java, we usually are thinking in terms of taxonomy. Classes extend other classes, and as we’ve previously stated, that means that a child class ought to be substitutable for its parent [LSP]. When we implement an interface in Java, we are also usually thinking that the class is a version of that interface. We might have expressed our Cartesian coordinate taxonomy in Java as:

public interface Coordinate {
  double distanceTo(Coordinate c);
}

public class Point implements Coordinate {
  // implementation omitted
}

public class ColorPoint extends Point implements Coordinate {
  // implementation omitted
}

When we do this, we soon hear ourselves talking about Coordinates as things, not as groups of behaviors. Let’s contrast this with Go’s implicit satisfaction by adding an additional example. Perhaps our program’s purpose is to keep track of animals in a wildlife preserve. It’s quite natural that we’d have an Animal type, and that type would have some way of keeping track of the animal’s current location (Animal struct).

Animal struct
type Animal struct {
        Name string
        X, Y float64
}

In order to perform our desired distance calculations, we need Animal to satisfy our two interfaces (Animal satisfies Positioner and Distancer).

Animal satisfies Positioner and Distancer
func (a Animal) Coordinates() point.Point {
        return point.Point{X: a.X, Y: a.Y}
}

func (a Animal) DistanceTo(pos point.Positioner) float64 {
        thing := pos.Coordinates()
        sqOfXDist := math.Pow(a.X-thing.X, 2)
        sqOfYDist := math.Pow(a.Y-thing.Y, 2)
        return math.Sqrt(sqOfXDist + sqOfYDist)
}
Note
Because distanceBetween() was not exported from the point package, we cannot use it in the animal package. Sometimes you’ll run into this situation in Go, which prefers “dependency hygiene” over reuse.

And now, we can perform our desired calculations (Mixing Animals and Points using interfaces). We now know the distance between our penguin and our original point p, and we also know that given the proximity of the seal (Output of mixing Animals and Points using interfaces), our penguin needs to start running!

Mixing Animals and Points using interfaces
        penguin := animal.Animal{Name: "penguin", X: 1, Y: 1}
        seal := animal.Animal{Name: "seal", X: 1, Y: 4}

        fmt.Printf("Dist b/w penguin and seal = %v\n", penguin.DistanceTo(seal))
        fmt.Printf("Dist b/w penguin and point = %v\n", penguin.DistanceTo(p))
Output of mixing Animals and Points using interfaces
$ go run point.go
Dist b/w penguin and seal = 3
Dist b/w penguin and point = 1

Now for the test. Is it proper to think of an Animal as being a Distancer or Positioner in terms of taxonomy? Not really. In fact, that seems like a coupling of concerns. And if we were implementing this program in Java, a naive translation would probably cause us to do the following:

public class Animal implements Positioner, Distancer

So to summarize, Go interfaces allow us to use arbitrary types in contexts expecting a particular interface type, as long as the type in question implements all of the methods defined by that interface. Since interfaces are satisfied implicitly, we’re no longer pressured toward treating interfaces as part of a type taxonomy. Instead, we’re able to focus on them as groups of related behaviors.

This may seem like a concept popularized by dynamic languages called “duck typing” [DT]. In most cases, you call a method blindly, and allow the runtime dispatch system to determine if the object can respond to that method. While this is a similar concept, it is not a very good description of how Go works. Go actually employs “structural typing” [ST], which uses the compiler to determine interface satisfaction in a type safe manner.

Conclusion

I hope you’ve enjoyed this brief introduction to the Go programming language, as well as one of its most powerful features: interfaces. With interfaces we’re able to take a more pragmatic approach to component design, as we’re not forced to think in terms of taxonomies and deep type hierarchies. In the next installment we’ll explore Go’s approach to concurrency. Until next time my fellow gophers!

References

Microservices Are SOLID

Microservices are often described as small, loosely coupled applications that follow the UNIX philosophy of “doing one thing well.” They have also been related to the Single Responsibility Principle, the first of the five principles making up SOLID. A microservices-based architecture is typically constructed around a set of common patterns. This set of patterns is actually consistent with all of the SOLID principles when thought of at the architectural rather than the class/module level. In this article, we’ll gain an understanding of each of the SOLID principles and how they relate to microservices.

A SOLID Architecture?

I have spent a significant portion of the last three years speaking and writing about the SOLID principles of object-oriented design. I spent the first year teaching SOLID much as expressed by Robert C. “Uncle Bob” Martin in his foundational book, Agile Software Development: Principles. About the same time I was embarking on a reintroduction to and exploration of functional programming, spending a significant amount of time with Clojure. Retraining my mind for functional thinking while simultaneously teaching the SOLID principles resulted in a perfect thought storm in my mind, resulting in the following question:

Is there an overlap between functional programming and the SOLID principles?

The thought journey that followed led me to start giving the talk entitled “Functional SOLID” on August 25, 2012. That same month I began publishing a four-part series of articles by the same name. In both of these presentations of my ideas, I related a couple of foundational memes:

  • Functional programming, especially within the Clojure programming language, provides wonderful constructs for building programs consistent with the SOLID principles.

  • The SOLID principles actually transcend all of programming, regardless of the particular programming paradigm (structured, object-oriented, functional) employed.

It is this second meme that provides the impetus for this particular article. At the conclusion of both presentations, I refer to Rich Hickey’s seminal Strange Loop keynote, “Simple Made Easy”. In his presentation, Hickey decries our continual conflation of the ideas of simplicity and ease by tracing the origins of both words:

  • Simple, from the Latin simplex, meaning “one fold or braid” (opposed to complex, meaning “many folds or braids,”) is an objective concept. In software we can relate it to the degree of interleaving of concerns in software components.

  • Easy, from the Latin adjacens, means “to lie near.” While “hard” does not trace to a root meaning “to lie far,” we can still understand that “easy” is a relative concept. What lies near, or is easy to me, is not necessarily easy or near to you. In software we can relate it to the programming languages, paradigms, framweworks, technologies, etc. with which we are intimately familiar.

This analysis led me to restate the SOLID principles in terms of another Rich Hickey word, complectedness, or the degree to which software concerns are interleaved:

Single Responsibility Principle

Complecting responsibilities leads to rigid and/or fragile design.

Open-Closed Principle

Complecting concretions of an abstraction in such a way that new concretions adversely affect existing, working concretions is problemmatic.

Liskov Substitution Principle

Reuse via inheritance is dangerous. It often complects entities not in a true “is-a” relationship, which leads to non-substitutability.

Interface Segregation Principle

Don’t complect clients with uninteresting operations by complecting unrelated groups of operations in a single entity!

Dependency Inversion Principle

Transitive dependency leads to transitive complectedness!

As you can see, we’ve now stated the principles independently of any programming-paradigm specific language. It is this restatement that cemented in my mind the idea that SOLID actually transcends all of software engineering — software engineering principles that are universally true, regardless of the context in which they are applied. So, we can easily walk these principles up the abstraction ladder into the world of architecture.

Microservices Introduced

With businesses built around software now disrupting multiple industries that appeared to have stable leaders, the need has emerged for enterprises to create “software factories” built around the following principles:

  • Streaming customer feedback directly into rapid, iterative cycles of application development;

  • Horizontally scaling applications to meet user demand;

  • Compatibility with an enormous diversity of clients, with mobility (smartphones, tablets, etc.) taking the lead;

  • Continuous delivery of value, shrinking the cycle time from concept to cash.

Infrastructure has taken the lead in adapting to meet these needs with the move to the cloud, and Platform as a Service (PaaS) has raised the level of abstraction to a focus on an ecosystem of applications and services. However, most applications are still developed as if we’re living in the previous generation of both business and infrastructure: the monolithic application. Microservices — small, loosely coupled applications that follow the Unix philosophy of “doing one thing well” — represent the application development side of enabling rapid, iterative development, horizontal scale, polyglot clients, and continuous delivery. They also enable us to scale application development and eliminate long term commitments to a single technology stack.

I won’t belabor the introduction to microservices anymore, as a wealth of reading is already available all over the web. As a jumping off point, I invite the reader to dive into James Lewis’ and Martin Fowler’s excellent coverage of the topic.

SOLID Microservices

Let’s now get to the heart of the matter: how do the patterns associated with a microservices architecture overlap with the SOLID principles? Let’s walk through each, briefly relating them in their natural context, and then swinging them into our microservices discussion.

Single Responsibility Principle

The Single Responsibility Principle (SRP) is stated by Martin as “each software module should have one and only one reason to change” [SRP]. Of all of the SOLID principles, the SRP is the one I’ve most often seen cited in the context of microservices.

One common thread between how Martin relates SOLID and microservices is change. Change in software is inevitable and constant. Requirements are realized as responsibilities doled out to various software modules. Requirements change leads to changes in responsibilities. If we couple responsibilities in a single module, then change to one responsibility can affect another unrelated responsibility simply due to its location. In other words, change one thing, sometimes another unrelated thing breaks. Risk goes up; change velocity goes down.

A monolithic architecture, no matter how modular on the inside, couples responsibilities together. Change cycles are coupled, increasing the risk associated with frequent deployments. Effective continuous delivery is far more difficult, as the release management process reimposes the waterfall process on the agile development team. If we instead separate architectural responsibilities into different microservices, we can decouple those change cycles, thus decreasing the risks associated with frequent deployments. Continuous delivery becomes more easily attainable.

The most common technique I’ve seen applied to decomposing a monolith into microservices is the bounded context from Domain-Driven Design. We identify discrete business capabilities, each of which owns and governs its own discrete segment of the overall data model for an organization. A microservice implements each business capability, encapsulating its data segment behind an often RESTful API. Overlaps between the capabilities (e.g. a shipping service and ordering service will both likely have the notion of customer, likely governed by a customer service) are realized by mappings in higher-order microservices or by utilizing hypermedia.

Open-Closed Principle

The Open-Closed Principle (OCP), first coined by Bertrand Meyer [OOSC], states that “software entities should be open for extension, but closed for modification.” Again we relate this principle to change. We should be able to change what a module does as software requirements change, but we should be able to do so without modifying any existing, working code.

At face value this looks impossible. How can we change the behavior of a module without changing its code? The key is in how we define the facade of the module, thinking at the appropriate level of abstraction.

Let’s draw an example from Java’s standard library. What if my client code is provided an instance of java.util.HashMap, and I instead want sorted keys? I would need to not only provide an instance of java.util.TreeMap to my client, but I would also need to change all of the existing references. If I instead refer to the map abstraction as java.util.Map (a Java interface), then I can provide my client with the new Map type without changing any code. By utilizing the appropriate module facade, we can decouple an abstraction from the its larger set of derivative behaviors.

What is our microservice facade? The API of course! As long as a given microservice continues to fulfill the contract expressed by its API, it should be possible to swap in new behaviors without changing any existing client code. This becomes supremely important when we consider the term of our commitment to a particular technology stack. Monolithic architectures are not closed to this particular type of modification, and the risk of incorporating new technology into an existing monolith can be very high. Microservices drastically reduce the risk associated with experimenting, even in production, with new technology stacks, and increase our ability to use the right tool for the job.

Another important technique enabled by the open-closed nature of microservices is polyglot persistence. By encapsulating the data store technology used for a particular business capability behind its facade (e.g. a recommendations service is very amenable to graph databases), we can hide the presence of that data store behind a microservice API. This enables us to both experiment with and utilize various data stores in advantageous contexts without polluting the overall service ecosystem with the semantics of each store.

Liskov Substitution Principle

The Liskov Substitution Principle (LSP) was born the same year as Meyer’s OCP, written down by Barbara Liskov. The LSP is concerned with types and subtypes, focused on the idea that “subtypes must be substitutable for their base types.” In object-oriented terms, drawing again from the Java language, if a class extends from a parent class or implements a parent interface, we should be able to use that class in the context of any code expecting an instance of the parent. If at any time that code context exhibits aberrant behavior, we have violated the LSP with our class.

Extending the idea of object-oriented inheritance to logical architecture is a bit of a stretch, but let’s give it a try. We’ll start by again considering the microservice’s facade, or its API. From the client’s perspective, the API represents the “base type” for our microservice. So long as any microservice we swap in properly fulfills this API, we can say it’s consistent with the LSP.

It’s unlikely that we’ll often substitute different implementations of the same API at runtime, and it’s unclear to me what a child microservice might look like. However, consider the case of services that implement the same API, but that must implement different business rules or policies given the legal jurisdiction governing the data. Further, consider that regulatory compliance dicates that those services actually are deployed and run in the same geographic location governed by that legal jurisdiction. We could implement each instance of this API as a separate microservice and deploy each of them in the appropriate geography. From the client’s perspective, the substitution would be transparent (thus abiding by the LSP), and the “polymorphic” substitution could be performed by another higher-order microservice or global site-selection mechanism.

Interface Segregation Principle

The Interface Segregation Principle (ISP) is stated in Martin’s book as “clients should not be forced to depend on methods they do not use.” Martin introduces the concept of so-called “fat interfaces,” or interfaces whose method set is not cohesive. One can divide their method sets into multiple groups, each group serving a different set of interested clients. The primary reason for seeking to separate these groups into different modules is to prevent change driven by one set of clients from affecting other distinct groups of clients.

API’s implemented via monolithic architectures cannot abide by the ISP. Adding or improving capabilities to serve one group of clients must involve minimally a redeployment of all of the capabilities affecting all clients. More likely, a lengthy regression test phase will also be required, as we must ensure that these additional or improved capabilities have not damaged the system’s other capabilities.

Microservices, when designed well around bounded contexts, also abide by the ISP, as we enforce a hard boundary between interfaces by separating them into discrete, independently deployable units.

Dependency Inversion Principle

The Dependency Inversion Principle (DIP) tells us that “abstractions should not depend upon details. Details should depend upon abstractions.” Stated another way, “high-level modules should not depend on low-level modules.” Our abstractions, or higher-level modules, are what codify the important business knowledge inherent in a body of software, whereas our details, or lower-level modules, represent the mechanical recipes for carrying them out. One of the promises of the other principles is the ability to “swap out” the details beneath the abstractions when it becomes advantageous. However, when our higher-level modules have direct dependency on our lower-level modules, swapping out details often causes the abstraction itself to have to change. “Absurd” is Uncle Bob’s description of this situation.

The DIP typically deals with this scenario by defining service interfaces for each module. If a module requires services that are not relevant to its bounded context, rather than implementing them itself or directly delegating to a dependency, it instead declares a signature for that service within its service interface. This interface then becomes a secondary abstraction expressing all of the collaboration a module intends to do. Possible collaborators then cooperate with the module by implementing its service interface. In this way, they become dependent on the module, rather than the module becoming dependent on the collaborator!

In a microservices architecture, the DIP finds its realization in the API Gateway pattern. An API Gateway acts as a single point of entry into a microservices architecture for a given client. It plays a multi-faceted role in serving the diverse clients (i.e. disparate mobile device platforms) of the architecture by:

  • reducing the chattiness of the network by reducing the number of services consulted;

  • performing protocol translation (e.g. AMQP to HTTP) when a particular protocol is not well supported by the client;

  • aggregating service responses concurrently to reduce response latency;

  • transforming service responses to service the needs of specific devices, screen sizes, and use cases.

The important thing to note is that the “API” defined by the API Gateway is owned by the client in much the same way as the service interface is owned by a higher-level module. In this way, we invert the dependency between clients and the microservices themselves. Consult “Optimizing the Netflix API” for a fantastic example of this architectural pattern.

Conclusion

Experience has taught me that ease is often cheap but illusory, but that simplicity is a pearl of great price. Microservices are not easy, but they are simple. One of the reasons for their simplicity is what I see as their strong compatibility with the SOLID principles, not only of object-oriented design, but of all of software engineering. By resisting the temptation to interleave distinct business capabilities, we retain the ability to develop and deploy them in an agile manner. I hope you’ve found some value in this article, and I even hope you’ve found some things with which you disagree. Please sound off in the blogosphere, the Twitterverse, or wherever suits your fancy.

Cheers!

References

BOSH and Cloud API Compatibility

The gauntlet has again been dropped in the world of cloud interoperability. The dueling factions include those asserting that competitors to Amazon’s web services (principally OpenStack) must adopt AWS’s API’s in order to remain viable, and those that believe such “API cloning” will do nothing more than stunt innovation. If you were to ask me, I’d say that we’ve seen this play out before. Remember the “Clone Wars” that began in the late 1980’s and that persisted for the better part of two decades? A huge cast of competitors battled for the title of “best PC that’s not manufactured by IBM.” How did that play out? For a relatively short period of time, having the best PC “designed for Microsoft Windows,” along with the leanest supply chain (see Dell), paved a golden path to victory. And then Steve Jobs returns to Apple, and now better than 50% of the laptops running in the Starbucks in which I’m writing this blog have a shiny white fruit on their lids. As it turns out, “going your own way” can work out awfully well.

But that’s not the angle I want to take in this discussion. Let’s dig deeper into what the two sides have to say.

The battle was first renewed with Cloud Scaling CTO Randy Bias’ Open Letter to the OpenStack Community. Randy adopts the position that full-compatibility with the AWS API’s is necessary for OpenStack’s survival. The gist of his argument is that Amazon currently dominates public cloud, supporting this via a comparison between Amazon’s and Rackspace’s growth rates since 2009, and that they also “control the innovation curve” as they push “new features into production at a breathtaking pace.” Furthermore, he asserts that any hope for survival with respect to competing cloud platforms is limited to the hybrid cloud space, providing enterprises with the capability to seamlessly migrate workloads between the public cloud and private, on-premises clouds. Therefore, OpenStack must adopt API compatibility with AWS in order to become the enterprise choice for hybrid cloud.

A few days later, Rackspace’s “Startup Liaison Officer” Robert Scoble responded with his own Open Letter. Scoble makes some interesting counterpoints, most notably the argument that customers don’t adopt cloud platforms because of API compatibility with Amazon, but because of the promise of a “10x improvement” to their own business. In order to provide such improvements, cloud platform competitors must not shackle themselves to a “de facto standard” API, but rather must focus their limited resources on driving those 10x improvements in infrastructure capability.

So by now you must be wondering, whose side am I on? I’m on the side of innovation. But that doesn’t necessarily put me in either camp. I think the end goals of both parties are things that we want:

  • Freedom: the ability to migrate workloads between cloud infrastructure providers without needing to significantly alter the behavior of the workload itself.
  • Innovation: the ability to realize capabilities that don’t exist today that will solve emerging problems (particularly those related to the explosion of generated and archived data).

Spending development cycles on API compatibility will certainly slow anyone’s ability to innovate. And what is API compatibility anyway? I believe that much of the concern rests on the large investment many enterprises have (or believe they will need to create) in bespoke automation written to a particular vendor’s API. Having recently left a large-scale project that generated thousands of lines of such automation to drive consumption of a particular vendor’s infrastucture platform, and that was in the near term planning to migrate to another platform, I can tell you that this concern is very real. But simply stating that “your existing code will work when you target our API” does not compatibility make. As Amazon continues to deploy new features at their breathtaking pace, how will OpenStack and other platforms keep up?

For API compatibility to be real, a “technology compatibility kit” (TCK) is needed. For those in the Java world, TCK’s are near and dear. Java itself is not a particular implementation, but a standard API that invites competing implementations and innovation. But for any competing implementation to call itself “Java,” it must pass the tests contained within the TCK. An AWS TCK is really the only true way to ensure API compatibility. But I think it’s hard to argue that Amazon has any real business interest in creating and sharing one.

There is another way. Perhaps we should stop creating bespoke automation and rally around a common standard toolkit for managing large-scale cloud application deployments. This toolkit could provide mechanisms for configuration management, orchestration, health management, and rolling upgrades. It could further, as part of its architecture, build an adapter layer between its core components and the underlying infrastructure provider. Plugins could then be developed to provide the toolkit with the ability to manage all of the common infrastructure providers.

Enter BOSH and it’s Cloud Provider Interface (CPI) layer. BOSH was initially developed as the means of deploying and managing the Cloud Foundry PaaS platform, but it’s much more generally applicable. BOSH can today deploy any distributed system, unchanged, to any of several popular IaaS providers: VMware vSphere, VMware vCloud Director, Amazon Web Services, and OpenStack. Heresy you say! Not so. Just ask Colin Humphreys of CloudCredo, who recently described their wildly successful deployment of Cloud Foundry to a hybrid composition of vSphere and AWS-based clouds. He recently presented a technical deep dive in Pivotal’s offices in which he made the statement (paraphrasing) “I took the same Cloud Foundry bits that were running on AWS and deployed them unchanged to vSphere using BOSH.” As anyone can tell, this isn’t just theory, it’s production.

So how then does BOSH make this happen? A trip into the code for the BOSH CPI “interface” will show a list of core infrastructure capabilities that BOSH requires:

  • current_vm_id
  • create_stemcell
  • delete_stemcell
  • create_vm
  • delete_vm
  • has_vm?
  • reboot_vm
  • set_vm_metadata
  • configure_networks
  • create_disk
  • delete_disk
  • attach_disk
  • snapshot_disk
  • delete_snapshot
  • detach_disk
  • get_disks

All interactions between BOSH and the underlying infrastructure provider pass through these core methods. As long as a CPI exists that exposes these capabilities to BOSH, BOSH can deploy and manage the lifecycle of Cloud Foundry (or any other distributed system described by a BOSH release) on an infrastructure provider.

So how hard is it to provide the CPI’s for both AWS and OpenStack? If you choose simple metrics like number of classes (NOC) and lines of code (LOC), not that hard.

You can find the CPI implementations at these links:

First we’ll generate the metrics for AWS:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ find ./bosh_aws_cpi/lib -name "*.rb" -exec wc -l {} \;
       2 ./bosh_aws_cpi/lib/bosh_aws_cpi.rb
      68 ./bosh_aws_cpi/lib/cloud/aws/aki_picker.rb
      39 ./bosh_aws_cpi/lib/cloud/aws/availability_zone_selector.rb
     651 ./bosh_aws_cpi/lib/cloud/aws/cloud.rb
      22 ./bosh_aws_cpi/lib/cloud/aws/dynamic_network.rb
      30 ./bosh_aws_cpi/lib/cloud/aws/helpers.rb
     171 ./bosh_aws_cpi/lib/cloud/aws/instance_manager.rb
      25 ./bosh_aws_cpi/lib/cloud/aws/manual_network.rb
      37 ./bosh_aws_cpi/lib/cloud/aws/network.rb
      89 ./bosh_aws_cpi/lib/cloud/aws/network_configurator.rb
     189 ./bosh_aws_cpi/lib/cloud/aws/resource_wait.rb
      68 ./bosh_aws_cpi/lib/cloud/aws/stemcell.rb
     114 ./bosh_aws_cpi/lib/cloud/aws/stemcell_creator.rb
      30 ./bosh_aws_cpi/lib/cloud/aws/tag_manager.rb
       7 ./bosh_aws_cpi/lib/cloud/aws/version.rb
      44 ./bosh_aws_cpi/lib/cloud/aws/vip_network.rb
      43 ./bosh_aws_cpi/lib/cloud/aws.rb

We’ll also want the total LOC:

1
2
$ find ./bosh_aws_cpi/lib -name "*.rb" -exec wc -l {} \; | awk '{ sum += $1 } END { print sum }'
1629

And now let’s do the same for OpenStack:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ find ./bosh_openstack_cpi/lib -name "*.rb" -exec wc -l {} \;
       4 ./bosh_openstack_cpi/lib/bosh_openstack_cpi.rb
     867 ./bosh_openstack_cpi/lib/cloud/openstack/cloud.rb
      28 ./bosh_openstack_cpi/lib/cloud/openstack/dynamic_network.rb
     131 ./bosh_openstack_cpi/lib/cloud/openstack/helpers.rb
      34 ./bosh_openstack_cpi/lib/cloud/openstack/manual_network.rb
      37 ./bosh_openstack_cpi/lib/cloud/openstack/network.rb
     159 ./bosh_openstack_cpi/lib/cloud/openstack/network_configurator.rb
      16 ./bosh_openstack_cpi/lib/cloud/openstack/tag_manager.rb
       8 ./bosh_openstack_cpi/lib/cloud/openstack/version.rb
      50 ./bosh_openstack_cpi/lib/cloud/openstack/vip_network.rb
      39 ./bosh_openstack_cpi/lib/cloud/openstack.rb
$ find ./bosh_openstack_cpi/lib -name "*.rb" -exec wc -l {} \; | awk '{ sum += $1 } END { print sum }'
1373

So, summarizing:

CPI Number of Classes (NOC) Lines of Code (LOC)
Amazon AWS 17 1629
OpenStack 11 1373

Let’s make a couple of points about these metrics. First of all, the two CPI’s do not use a common foundation. The AWS CPI uses the AWS SDK for Ruby while the OpenStack CPI uses Fog. Fog could also have been used as the foundation for the AWS CPI, but the CPI authors presumably thought it better to stick with the toolkit provided by Amazon. This is a minor point, however, as both of these toolkits essentially provide simple wrappers around the infrastructure providers’ REST API’s. It’s doubtful that using a common API wrapper for both CPI’s would have substantially changed the metrics presented here.

Second, obviously NOC and LOC are rather naive metrics. It’s incredibly possible to write terse code that is opaque, buggy, and hard to maintain or enhance. In fact, according to Code Climate, both of the top-level implementation classes for these CPI’s have quite a lot of room for improvement:

With that said, it is rather amazing that one could encapuslate all of the infrastructure-specific implementation necessary to deploy and manage a distributed system as powerful as Cloud Foundry in less than twenty classes and 1700 lines of code.

So, to summarize where we’ve been, while there’s an impressive war of words out there regarding API compatibility with Amazon AWS, Cloud Foundry and BOSH don’t necessarily need to take sides. If OpenStack chooses to embrace the AWS API’s, the BOSH AWS CPI will be there waiting. However, if OpenStack chooses to follow in the footsteps of Apple and take the road less traveled, the OpenStack CPI is ready and waiting to evolve with it. Should Google Compute Engine or Microsoft’s Azure gain a foodhold on the innovation curve, they are presumably a relatively simple CPI away from joining the BOSH ecosystem. So if you really want “cloud freedom,” BOSH is leading the charge.

Blue-Green Deployments on Cloud Foundry

One of the great things about Cloud Foundry is that it is a great enabler. Tall words. But what do they mean? Essentially, Cloud Foundry (and any other well-designed PaaS) enables us to do things as developers and operators that would be extremely difficult in a traditional deployment environments. One particularly valuable area of enablement is our new found ability to practice Continous Delivery, meaning that we continuously prove our ability to deliver working software by continuously treating each code commit to a system as if it could be deployed to a production environment. We do this by shipping these commits through what’s called a “deployment pipeline,” which consists of a series of build-test-deploy cycles that prove out a commit’s suitability for production deployment. At the end of the pipeline we can either deploy automatically to our production environment (i.e. continuous deployment), or we can have a business decision to deploy that “deployable artifact” or not.

One particular practice associated with Continuous Delivery is Blue-Green Deployments. Martin Fowler describes these very well at the link provided, but I’ll summarize briefly here:

  • Cut-over to a new software version is tricky, and must be quick in order to minimize downtime events.
  • Blue-green deployments ensure the parallel existence of two, identical (as possible) production environments.
  • At any given point, only one (e.g. blue) services production traffic.
  • New deploys are made to the other (e.g. green) environment. Final smoke testing is performed here.
  • When green is determined ready, we begin routing traffic to it.
  • We then stop routing traffic to blue.

Of course, there are several concerns that must be dealt with at the application level in order for this to work (datastores should support concurrent usage by two app versions, long running requests may be killed, etc.). What we’ll focus on in this post is how Cloud Foundry supports the mechanics summarized above.

We’ll begin with a basic Spring application named ms-spr-demo. This app takes users to a simple web page announcing the ubiquitous “Hello World!” message. We’ll utilize the cf command-line interface to push the application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
$ cf push --path build/libs/cf-demo.war
Name> ms-spr-demo

Instances> 1

Memory Limit> 512M

Creating ms-spr-demo... OK

1: ms-spr-demo
2: none
Subdomain> ms-spr-demo

1: cfapps.io
2: mattstine.com
3: none
Domain> 1

Creating route ms-spr-demo.cfapps.io... OK
Binding ms-spr-demo.cfapps.io to ms-spr-demo... OK

Create services for application?> n

Save configuration?> y

Saving to manifest.yml... OK
Uploading ms-spr-demo... OK
Starting ms-spr-demo... OK
-----> Downloaded app package (9.5M)
Installing java.
Downloading JDK...
Copying openjdk-1.7.0_25.tar.gz from the buildpack cache ...
Unpacking JDK to .jdk
Downloading Tomcat: apache-tomcat-7.0.41.tar.gz
Copying apache-tomcat-7.0.41.tar.gz from the buildpack cache ...
Unpacking Tomcat to .tomcat
Copying mysql-connector-java-5.1.12.jar from the buildpack cache ...
Copying postgresql-9.0-801.jdbc4.jar from the buildpack cache ...
Copying auto-reconfiguration-0.6.8.jar from the buildpack cache ...
-----> Uploading droplet (48M)
-----> Uploaded droplet
Checking ms-spr-demo...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  1/1 instances: 1 running
OK

The end result of this cf push event is that an application is now serving requests at http://ms-spr-demo.cfapps.io. The following graphic shows the state of our system, with the CF Router sending traffic to our application:

Next, we make a slight change to our application. Rather that saying “Hello World!” we decide to make it say “Goodbye World!” We build a new war file for the application. Rather than letting cf prompt us this time, we’ll make use of the manifest.yml file that we saved from our previous push. However, we’ll rename the application and provide a new route. Take a look:

1
2
3
4
5
6
7
---
applications:
- name: ms-spr-demo-green
  memory: 512M
  instances: 1
  url: ms-spr-demo-green.cfapps.io
  path: build/libs/cf-demo.war

As you can see, we’re calling our new application version ms-spr-demo-green and we’re mapping it to the URL ms-spr-demo-green.cfapps.io. Let’s push the application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
Using manifest file manifest.yml

Creating ms-spr-demo-green... OK

1: ms-spr-demo-green
2: none
Subdomain> ms-spr-demo-green

1: cfapps.io
2: mattstine.com
3: none
Domain> 1

Creating route ms-spr-demo-green.cfapps.io... OK
Binding ms-spr-demo-green.cfapps.io to ms-spr-demo-green... OK
Uploading ms-spr-demo-green... OK
Starting ms-spr-demo-green... OK
-----> Downloaded app package (9.5M)
Installing java.
Downloading JDK...
Copying openjdk-1.7.0_25.tar.gz from the buildpack cache ...
Unpacking JDK to .jdk
Downloading Tomcat: apache-tomcat-7.0.41.tar.gz
Copying apache-tomcat-7.0.41.tar.gz from the buildpack cache ...
Unpacking Tomcat to .tomcat
Copying mysql-connector-java-5.1.12.jar from the buildpack cache ...
Copying postgresql-9.0-801.jdbc4.jar from the buildpack cache ...
Copying auto-reconfiguration-0.6.8.jar from the buildpack cache ...
-----> Uploading droplet (48M)
-----> Uploaded droplet
Checking ms-spr-demo-green...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  1/1 instances: 1 running
OK

We now have two instances of the application running, each of them using distinct routes:

Now it’s time for the magic to happen. We’ll map our original URL route (ms-spr-demo.cfapps.io) to our “green” instance. This is accomplished very simply by using cf:

1
2
cf map --app ms-spr-demo-green --host ms-spr-demo --domain cfapps.io
Binding ms-spr-demo.cfapps.io to ms-spr-demo-green... OK

The CF router immediately begins to load balance requests between each instance of the application, as shown here:

Now our router will send requests to ms-spr-demo.cfapps.io to both instances of the application, while ms-spr-demo-green.cfapps.io only services the “green” instance. Once we determine that all is well, it’s time to stop routing requests to the “blue” instance. This is just as simple using cf:

1
2
cf unmap --url ms-spr-demo.cfapps.io --app ms-spr-demo
Unbinding ms-spr-demo.cfapps.io from ms-spr-demo... OK

Our “blue” instance is now no longer receiving any web traffic:

We can now decomission our “blue” instance, or we can leave it around for a period of time in case we decide we need to roll back our changes. The important thing is that our customers suffered absolutely no downtime!

Clojure on Cloud Foundry

I was inspired by Brian McClain’s post on bringing Haskell to Cloud Foundry using Cloud Foundry v2 buildpacks, so I decided to go on a buildpack journey of my own. Since Clojure is the language I most enjoying “toying around with,” I thought I’d try to deploy a simple Clojure web application using the Heroku Clojure Buildpack.

To reiterate some of the coolness around buildpacks, they are what allows a PaaS like Cloud Foundry or Heroku to support various runtimes without first building that support into the core platform. If your favorite language or framework runtime isn’t available, there’s absolutely nothing stopping you from providing your own buildpack to support it. Stuff can get crazy – McClain has even hinted at bringing FORTRAN support to Cloud Foundry.

I decided for my experiment to build a VERY basic “Hello World” style application using Ring, which is “a Clojure web applications library inspired by Python’s WSGI and Ruby’s Rack.” The easiest way to get started building Ring applications is to utilize the popular Clojure build tool Leiningen.

First I started by creating a new project:

1
2
$ lein new hello-cf
$ cd hello-cf

The next task was to add the Ring dependencies to my project.clj file:

1
2
3
4
5
6
7
8
(defproject hello-cf "0.1.0-SNAPSHOT"
  :min-lein-version "2.0.0"
  :description "Hello Clojure on Cloud Foundry!"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.4.0"]
                 [ring/ring-core "1.1.8"]
                 [ring/ring-jetty-adapter "1.1.8"]])

Then it was time to create the Ring application itself, by editing src/hello_cf/core.clj:

1
2
3
4
5
6
7
8
9
10
(ns hello-cf.core
  (:use [ring.adapter.jetty :only [run-jetty]]))

(defn handler [request]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body "Hello Cloud Foundry from heroku-buildpack-clojure!"})

(defn -main [port]
  (run-jetty handler {:port (Integer. port)}))

Let’s break this down a bit. The handler function will handle any HTTP request that hits our application, and will return an “OK” response containing a pleasant message indicating that we’ve succeeded. That’s really about it. Our application is complete. We can test it out by running the following:

1
2
3
$ lein trampoline run -m hello-cf.core 8080
2013-05-29 22:42:52.576:INFO:oejs.Server:jetty-7.6.1.v20120215
2013-05-29 22:42:52.804:INFO:oejs.AbstractConnector:Started SelectChannelConnector@0.0.0.0:8080

Hitting http://localhost:8080 in the browser confirms that we’re well on our way. Now it’s time to trying pushing the application to Cloud Foundry. As Brian stated in his blog, one of the stellar aspects of Cloud Foundry buildpacks is that they are approximately the same as Heroku buildpacks. Practically, this means that one should be able to utilize a Heroku buildpack on Cloud Foundry with minimal or no modifications. Let’s put that theory to the test, shall we? Before we do, let’s create a Procfile quickly to let the buildpack know what we want to run:

1
web: lein with-profile production trampoline run -m hello-cf.core $PORT

And on with the push:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
cf push hello-cf --buildpack=git://github.com/heroku/heroku-buildpack-clojure.git
Using manifest file manifest.yml

Creating hello-cf... OK

1: hello-cf
2: none
Subdomain> hello-cf

1: mstine.cf-app.com
2: none
Domain> mstine.cf-app.com

Binding hello-cf.mstine.cf-app.com to hello-cf... OK
Uploading hello-cf... OK
Starting hello-cf... OK
-----> Downloaded app package (12K)
Initialized empty Git repository in /tmp/buildpacks/heroku-buildpack-clojure.git/.git/
Installing heroku-buildpack-clojure.git.
-----> Installing OpenJDK 1.6...done
-----> Installing Leiningen
       Downloading: leiningen-2.1.2-standalone.jar
       Writing: lein script
-----> Building with Leiningen
       Running: lein with-profile production compile :all
       Retrieving lein-standalone-repl/lein-standalone-repl/0.1.5/lein-standalone-repl-0.1.5.pom from clojars
       Retrieving lein-standalone-repl/lein-standalone-repl/0.1.5/lein-standalone-repl-0.1.5.jar from clojars
       Performing task 'compile' with profile(s): 'production'
       Retrieving org/clojure/clojure/1.4.0/clojure-1.4.0.pom from
       ...
       Compiling hello-cf.core
-----> Uploading staged droplet (66M)
-----> Uploaded droplet
Checking hello-cf...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
  0/1 instances: 1 starting
  0/1 instances: 1 down
  0/1 instances: 1 starting
  0/1 instances: 1 flapping
Application failed to start.

Drat. Let’s take a quick look at the logs to see what may be awry:

1
2
Reading logs/stderr.log... OK
/home/vcap/app/.lein/bin/lein: line 42: java: command not found

Ah-hah! Looks like the existing buildpack is making some assumptions about the structure of our application that no longer hold true on Cloud Foundry. So, I followed in Brian’s footsteps and forked away. One small commit looks like it ought to fix the problem. Let’s give it another try:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
cf push hello-cf --buildpack=git://github.com/mstine/heroku-buildpack-clojure.git
Using manifest file manifest.yml

Not applying manifest changes without --reset
See `cf diff` for more details.

Uploading hello-cf... OK
Changes:
  buildpack: 'git://github.com/heroku/heroku-buildpack-clojure.git' -> 'git://github.com/mstine/heroku-buildpack-clojure.git'
Updating hello-cf... OK
Stopping hello-cf... OK

Starting hello-cf... OK
-----> Downloaded app package (8.0K)
-----> Downloaded app buildpack cache (17M)
Initialized empty Git repository in /tmp/buildpacks/heroku-buildpack-clojure.git/.git/
Installing heroku-buildpack-clojure.git.
-----> Installing OpenJDK 1.6...done
-----> Using cached Leiningen 2.1.2
       Writing: lein script
-----> Building with Leiningen
       Running: lein with-profile production compile :all
       Performing task 'compile' with profile(s): 'production'
       Compiling hello-cf.core
-----> Uploading staged droplet (66M)
-----> Uploaded droplet
Checking hello-cf...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  1/1 instances: 1 running
OK

BOOM!

I quickly pointed my browser, and as yesterday’s tweet indicates, success:

Score another win for Cloud Foundry’s buildpack support. I’m now toying with the idea of doing something of a world tour of LISP on Cloud Foundry. My next candidate may be Scheme.

Into the Crucible

Wow…it seems I only post to this blog toward the end of May. Well, that all changes now. You see, as of June 3, 2013, this blog is going to become one of many aspects of my new “day job.” On Monday, I start my life as a Community Engineer with Cloud Foundry by Pivotal. What’s a Community Engineer? Quite honestly, I’m not completely sure of the answer to that question yet. But given the many conversations I’ve had over the past few weeks, it seemingly fits right in with the bridge-building roles I’ve played many times over the course of my career. In this case, I have one foot squarely planted in the world of Cloud Foundry engineering, and one foot squarely planted out in the world with you guys and gals - the community. My job is to help you understand how we are quite literally seeking to “build a new platform for a new era.”

Of course, this is a journey that for me started a few years ago. In my previous life as a front-line development manager, I helped lead an agile transformation within my team with “ruthless automation” playing a central role in everything that our team did. However, it seemed that the better we “did agile,” the more pain we felt when dealing with those outside of our circle of control. It was only years later, after reading Eliyahu Goldratt’s The Goal and coming to an understanding of his Theory of Constraints, that I realized what was happening. Our constraints had moved “out of the plant,” if you will, and landed in the world of operations. Even without this understanding, I developed a keen interest in this newly emerging topic called “DevOps” and began to explore the ideas emerging around agile operations and infrastructure as code. I started playing with tools like Puppet, Chef, and Vagrant, and taught sessions on all three of them at the Project Automation Experience in 2011.

You can read my last entry and find out that not much later I joined VMware as a Senior Consultant for its Cloud Application Platform. I was hired into that role based on my extensive background in enterprise Java and the Spring ecosystem, but it was nothing short of a staffing accident that I found myself thrust into a role on a virtualization platform provisioning team helping to build out a private self-service cloud! I was steadily getting carried further away from my role as an application architect, steadily becoming assimilated into that mysterious world of web operations that I knew so little about. These experiences, along with my continued reading and thinking about the worlds of DevOps, Lean, and Kanban, have quite literally changed the way I look at the world of software engineering (or as I prefer to think of it now, value delivery through product engineering that just so happens to involve software!). These experiences have formed around me a crucible, melting me that I might be poured into a new professional mold.

So now it’s time to plunge into the world of building the leading open platform as a service, and to help YOU experience the HUGE can of @cloudfoundry awesome that we at Pivotal are about to unleash on the world. Sound good to you? Join us!

The Relaunch

I have rebooted this blog many times over the last several years. If you’ve been a reader of my blog in the past, you will have noticed significant changes. If you’re new here, welcome!

This reboot has been in the works for several months now, even though I’ve probably spent far less than 24 active hours working on it. Life as an “itinerant” consultant and conference speaker is extremely busy compared to what I was doing on May 16, 2012 (the date of my last blog posting). At that time I was in the middle of a transition from 3+ years as a front-line manager of a software development team into a lead software architect role. Since that time I’ve changed jobs (careers?) twice:

  • June 2011 - February 2012: During this time I was titled as a “Technical Architect” at AutoZone’s corporate offices in Memphis, TN. My focus was the modernization of their retail store management system architecture and technical stack. While there I also did a fair amount of agile coaching.
  • February 2012 - Present: In January, VMware came calling, wanting to know if I would join the vFabric Cloud Application Platform PSO as a Senior Consultant. After a few weeks of interviews and careful consideration, I made the jump. I’m now in the midst of helping a Fortune 50 company build a brand new “private cloud” hosting platform for their next generation of application development.

During that time I also significantly increased my participation on the No Fluff Just Stuff tour. In other words, I’ve BEEN BUSY.

At any rate, I have for quite awhile now wanted to get back into the practice of writing. However, I’ve wanted to do so outside the constraints of a hosted platform like Blogger or WordPress.com. Those systems place far too many constraints on how your blog works, and they also happen to be the only place that your data is housed. Sure, I could write my posts in Vim and then copy/paste them into the platform, but that’s annoying. I wanted to manage my writing using a plain text editor, mark it up using something lightweight like Markdown, check my writing into version control using Git, generate my site using a tool like Jekyll, and then push my site to the cloud. What was the easiest way to achieve all of this goodness? Octopress seemed to fit the bill.

So, what you now see is the result of my labor in migrating from WordPress.com to Octopress, hosted at Heroku. One day (perhaps) I’ll write up the process that I used. It was a bit convoluted and involved gigantic piles of yak hair, so I’d rather not consider that piece right now.

What I will be focusing on quite a bit is the idea of simplicity. For more than a decade I have wandered through the barren wasteland that is accidental complexity in software. A few months ago I viewed Rich Hickey’s keynote session at StrangeLoop 2011 entitled “Simple Made Easy”. It is Rich’s definition of simplicity in software that has inspired the brand new title of this blog, wherein I will chronicle my continuing quest to seek out simplicity in software structure.

Alas, it is time to return to billable work. I shall endeavor to post again in far less than a year. So, as we say in the south, “Ya’ll come back now, you hear?”

Design Meeting Patterns/Antipatterns

For those of you that don’t know, I recently returned to the technical ranks as a Software Architect after a three-year stint in management. To make a long story short, I now love my job again. Perhaps I’ll write the long story in a future blog entry. On to the topic at hand. Today I led the first significant design discussion that I have led in quite a long time. A few minutes afterward, I was already reflecting on what had occurred and how. The very first thing that I noticed was my drastically different approach to the task. I felt that my facilitation skills were more mature and effective than I remember from previous similar engagements. I’m not sure if the time I spent in management (where I facilitated quite a few meetings, though much more business-focused ones) helped, but something certainly has.

I also noticed several things that I’ll call “pattern vs. antipattern” tugs of war. Quite often during the meeting I felt the group trying to move our discussion in one direction that I, for better or for worse, thought was ineffective. I would then redirect us on to what I felt was a better path. In true catalog form, here’s the list:

  • Understand the Problem vs. Jump to the Solution - only a couple of minutes into our discussion solutions were being flung about like drunken darts. This situation almost always leads to suboptimal or faulty solutions. As a facilitator, try to ensure that the problem has been clearly stated. If at all possible, write it down on a whiteboard. Make sure everyone agrees that the problem as stated is the problem the group is there to solve. Sure enough, after performing this exercise, we all had a very different and clear understanding of the problem than that with which we walked in the door.

  • Assume the Worst vs. Assume the Best - occasionally the exact details of a requirement are unclear, and not assuming something will totally derail the design discussion. You have a couple of choices. The first is to halt the discussion and get the details cleared up. This is clearly the best solution, as you’ll no longer have to assume anything. However, it can be the case that the person who can clear things up isn’t available. Or in some cases, the question you’ll be asking will require another meeting at another level of the organization. If you find yourself in that spot, and you can’t wait (we couldn’t!), then the best approach is to work from the worst possible case scenario. You’ll then be in the best position to handle whatever answer comes your way. However, our tendency is often to assume the best (“That will never happen!”). Fight that tendency. However, whatever you choose, follow up at your earliest opportunity.

  • Basing Decisions on the Current Situation vs. Basing Decisions on History - many times the group wanted to veer off into safer territory. In some cases, a possible solution departed significantly from the current design. While this is a valid concern (we do want consistency of design across the system where possible), it is certainly not a trump card. Occasionally the situation at hand will merit a significant departure from the current design. Another way history can rear its ugly head is the assertion that we’ve always solved similar problems like ‘x,’ so we should do so with this problem as well. Again, note the word “similar.” All problems are somewhat different and have their own eccentricities. So, rather than working from history, I pushed us back to a clean slate with the statement “Let’s stop thinking about the past and start from scratch. We may very well come up with the same solution you guys are proposing, but I’d rather do so through our own objective analysis and not instinct.” Guess what. We came up with a different solution that we all felt better about.

  • Shooting for the “Best” Solution vs. the “Easiest” Solution - now sometimes we can’t afford the best solution. I grant that. However, I’m trying to fight the tendency to immediately jump to the “easiest thing that could possibly work.” Often this pops up in the first P vs. AP - if we don’t clearly understand the problem, sometimes an easy solution jumps out that doesn’t deal with the underlying details we’ve yet to uncover. Also, sometimes the best solution is quite simple and elegant. It doesn’t necessarily have to be harder and more complex than the easiest solution. In fact, sometimes the “easiest” solution leads to the most accidental complexity in the long-run. So, shoot for the best solution you can come up with and only then, optimize for cost.

  • Present Possible Solutions Objectively vs. My Solution is the Best! - one would hope that we all start here, but we don’t. We tend to like our own solutions to problems and want them to “win.” Our ego can get in the way of even hearing an alternate solution presented by another team member. I point you to my colleague Ted Neward’s post for more on “egoless programming.” So, as a facilitator, you’ve got to make sure that all solutions are presented objectively. I often had to say things like “OK, let’s assume before we ever get started that this is a good solution to the problem and not hack away at it until its fully presented, and we all understand it.” In the end, this insistence led us to choose a solution that none of us (myself included) originally thought we’d pick.

  • Validating from Code vs. Validating from Memory - more often than not, questions about the existing design/code/behavior will come up. Rather than scratching your head and trying to remember what you wrote six months ago, pull up the code and find out. I can’t tell you the number of meetings I’ve attended where baseless assertions were made about existing code, only to require another meeting the next day to revisit the whole discussion once those assertions were proven wrong. Again, as a facilitator, I directed us to solve every problem for which all of the facts were available. We inserted placeholders in our solution where questions remained. Guess what we’re doing now? Well, I’m blogging about the meeting, but the rest of us are validating from code. Tomorrow will fill in the blanks!

Selenium Conference 2011

I completed an interest survey for a potential Selenium-focused conference several weeks ago, and I’m excited to let you know that the “powers that be” have decided that the conference is going to happen! I have already submitted my “Executable Specifications: Automating Your Requirements Document with Geb and Spock” talk as a potential session. Whether it makes the conference program or not, I plan on attending the event. Here are the details:

Selenium Conference 2011

Join members of the growing Selenium community for 3 jam-packed days of talks, workshops, lightning talks, and hack sessions. Hear speakers from around the world talk about the present and future of automated testing, share ideas with fellow Selenium developers, including Core Committers, and take part in shaping the future success of the Selenium project.

When: April 4-6, 2011 Where: Marines’ Memorial Club & Hotel, 609 Sutter St, San Francisco, CA 94102 USA Register now to be a speaker or sponsor. More details: http://www.seleniumconf.com/