Seeking Simplicity

A blog about DevOps, clouds, architecture, and everything else...

BOSH and Cloud API Compatibility

The gauntlet has again been dropped in the world of cloud interoperability. The dueling factions include those asserting that competitors to Amazon’s web services (principally OpenStack) must adopt AWS’s API’s in order to remain viable, and those that believe such “API cloning” will do nothing more than stunt innovation. If you were to ask me, I’d say that we’ve seen this play out before. Remember the “Clone Wars” that began in the late 1980’s and that persisted for the better part of two decades? A huge cast of competitors battled for the title of “best PC that’s not manufactured by IBM.” How did that play out? For a relatively short period of time, having the best PC “designed for Microsoft Windows,” along with the leanest supply chain (see Dell), paved a golden path to victory. And then Steve Jobs returns to Apple, and now better than 50% of the laptops running in the Starbucks in which I’m writing this blog have a shiny white fruit on their lids. As it turns out, “going your own way” can work out awfully well.

But that’s not the angle I want to take in this discussion. Let’s dig deeper into what the two sides have to say.

The battle was first renewed with Cloud Scaling CTO Randy Bias’ Open Letter to the OpenStack Community. Randy adopts the position that full-compatibility with the AWS API’s is necessary for OpenStack’s survival. The gist of his argument is that Amazon currently dominates public cloud, supporting this via a comparison between Amazon’s and Rackspace’s growth rates since 2009, and that they also “control the innovation curve” as they push “new features into production at a breathtaking pace.” Furthermore, he asserts that any hope for survival with respect to competing cloud platforms is limited to the hybrid cloud space, providing enterprises with the capability to seamlessly migrate workloads between the public cloud and private, on-premises clouds. Therefore, OpenStack must adopt API compatibility with AWS in order to become the enterprise choice for hybrid cloud.

A few days later, Rackspace’s “Startup Liaison Officer” Robert Scoble responded with his own Open Letter. Scoble makes some interesting counterpoints, most notably the argument that customers don’t adopt cloud platforms because of API compatibility with Amazon, but because of the promise of a “10x improvement” to their own business. In order to provide such improvements, cloud platform competitors must not shackle themselves to a “de facto standard” API, but rather must focus their limited resources on driving those 10x improvements in infrastructure capability.

So by now you must be wondering, whose side am I on? I’m on the side of innovation. But that doesn’t necessarily put me in either camp. I think the end goals of both parties are things that we want:

  • Freedom: the ability to migrate workloads between cloud infrastructure providers without needing to significantly alter the behavior of the workload itself.
  • Innovation: the ability to realize capabilities that don’t exist today that will solve emerging problems (particularly those related to the explosion of generated and archived data).

Spending development cycles on API compatibility will certainly slow anyone’s ability to innovate. And what is API compatibility anyway? I believe that much of the concern rests on the large investment many enterprises have (or believe they will need to create) in bespoke automation written to a particular vendor’s API. Having recently left a large-scale project that generated thousands of lines of such automation to drive consumption of a particular vendor’s infrastucture platform, and that was in the near term planning to migrate to another platform, I can tell you that this concern is very real. But simply stating that “your existing code will work when you target our API” does not compatibility make. As Amazon continues to deploy new features at their breathtaking pace, how will OpenStack and other platforms keep up?

For API compatibility to be real, a “technology compatibility kit” (TCK) is needed. For those in the Java world, TCK’s are near and dear. Java itself is not a particular implementation, but a standard API that invites competing implementations and innovation. But for any competing implementation to call itself “Java,” it must pass the tests contained within the TCK. An AWS TCK is really the only true way to ensure API compatibility. But I think it’s hard to argue that Amazon has any real business interest in creating and sharing one.

There is another way. Perhaps we should stop creating bespoke automation and rally around a common standard toolkit for managing large-scale cloud application deployments. This toolkit could provide mechanisms for configuration management, orchestration, health management, and rolling upgrades. It could further, as part of its architecture, build an adapter layer between its core components and the underlying infrastructure provider. Plugins could then be developed to provide the toolkit with the ability to manage all of the common infrastructure providers.

Enter BOSH and it’s Cloud Provider Interface (CPI) layer. BOSH was initially developed as the means of deploying and managing the Cloud Foundry PaaS platform, but it’s much more generally applicable. BOSH can today deploy any distributed system, unchanged, to any of several popular IaaS providers: VMware vSphere, VMware vCloud Director, Amazon Web Services, and OpenStack. Heresy you say! Not so. Just ask Colin Humphreys of CloudCredo, who recently described their wildly successful deployment of Cloud Foundry to a hybrid composition of vSphere and AWS-based clouds. He recently presented a technical deep dive in Pivotal’s offices in which he made the statement (paraphrasing) “I took the same Cloud Foundry bits that were running on AWS and deployed them unchanged to vSphere using BOSH.” As anyone can tell, this isn’t just theory, it’s production.

So how then does BOSH make this happen? A trip into the code for the BOSH CPI “interface” will show a list of core infrastructure capabilities that BOSH requires:

  • current_vm_id
  • create_stemcell
  • delete_stemcell
  • create_vm
  • delete_vm
  • has_vm?
  • reboot_vm
  • set_vm_metadata
  • configure_networks
  • create_disk
  • delete_disk
  • attach_disk
  • snapshot_disk
  • delete_snapshot
  • detach_disk
  • get_disks

All interactions between BOSH and the underlying infrastructure provider pass through these core methods. As long as a CPI exists that exposes these capabilities to BOSH, BOSH can deploy and manage the lifecycle of Cloud Foundry (or any other distributed system described by a BOSH release) on an infrastructure provider.

So how hard is it to provide the CPI’s for both AWS and OpenStack? If you choose simple metrics like number of classes (NOC) and lines of code (LOC), not that hard.

You can find the CPI implementations at these links:

First we’ll generate the metrics for AWS:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ find ./bosh_aws_cpi/lib -name "*.rb" -exec wc -l {} \;
       2 ./bosh_aws_cpi/lib/bosh_aws_cpi.rb
      68 ./bosh_aws_cpi/lib/cloud/aws/aki_picker.rb
      39 ./bosh_aws_cpi/lib/cloud/aws/availability_zone_selector.rb
     651 ./bosh_aws_cpi/lib/cloud/aws/cloud.rb
      22 ./bosh_aws_cpi/lib/cloud/aws/dynamic_network.rb
      30 ./bosh_aws_cpi/lib/cloud/aws/helpers.rb
     171 ./bosh_aws_cpi/lib/cloud/aws/instance_manager.rb
      25 ./bosh_aws_cpi/lib/cloud/aws/manual_network.rb
      37 ./bosh_aws_cpi/lib/cloud/aws/network.rb
      89 ./bosh_aws_cpi/lib/cloud/aws/network_configurator.rb
     189 ./bosh_aws_cpi/lib/cloud/aws/resource_wait.rb
      68 ./bosh_aws_cpi/lib/cloud/aws/stemcell.rb
     114 ./bosh_aws_cpi/lib/cloud/aws/stemcell_creator.rb
      30 ./bosh_aws_cpi/lib/cloud/aws/tag_manager.rb
       7 ./bosh_aws_cpi/lib/cloud/aws/version.rb
      44 ./bosh_aws_cpi/lib/cloud/aws/vip_network.rb
      43 ./bosh_aws_cpi/lib/cloud/aws.rb

We’ll also want the total LOC:

1
2
$ find ./bosh_aws_cpi/lib -name "*.rb" -exec wc -l {} \; | awk '{ sum += $1 } END { print sum }'
1629

And now let’s do the same for OpenStack:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ find ./bosh_openstack_cpi/lib -name "*.rb" -exec wc -l {} \;
       4 ./bosh_openstack_cpi/lib/bosh_openstack_cpi.rb
     867 ./bosh_openstack_cpi/lib/cloud/openstack/cloud.rb
      28 ./bosh_openstack_cpi/lib/cloud/openstack/dynamic_network.rb
     131 ./bosh_openstack_cpi/lib/cloud/openstack/helpers.rb
      34 ./bosh_openstack_cpi/lib/cloud/openstack/manual_network.rb
      37 ./bosh_openstack_cpi/lib/cloud/openstack/network.rb
     159 ./bosh_openstack_cpi/lib/cloud/openstack/network_configurator.rb
      16 ./bosh_openstack_cpi/lib/cloud/openstack/tag_manager.rb
       8 ./bosh_openstack_cpi/lib/cloud/openstack/version.rb
      50 ./bosh_openstack_cpi/lib/cloud/openstack/vip_network.rb
      39 ./bosh_openstack_cpi/lib/cloud/openstack.rb
$ find ./bosh_openstack_cpi/lib -name "*.rb" -exec wc -l {} \; | awk '{ sum += $1 } END { print sum }'
1373

So, summarizing:

CPI Number of Classes (NOC) Lines of Code (LOC)
Amazon AWS 17 1629
OpenStack 11 1373

Let’s make a couple of points about these metrics. First of all, the two CPI’s do not use a common foundation. The AWS CPI uses the AWS SDK for Ruby while the OpenStack CPI uses Fog. Fog could also have been used as the foundation for the AWS CPI, but the CPI authors presumably thought it better to stick with the toolkit provided by Amazon. This is a minor point, however, as both of these toolkits essentially provide simple wrappers around the infrastructure providers’ REST API’s. It’s doubtful that using a common API wrapper for both CPI’s would have substantially changed the metrics presented here.

Second, obviously NOC and LOC are rather naive metrics. It’s incredibly possible to write terse code that is opaque, buggy, and hard to maintain or enhance. In fact, according to Code Climate, both of the top-level implementation classes for these CPI’s have quite a lot of room for improvement:

With that said, it is rather amazing that one could encapuslate all of the infrastructure-specific implementation necessary to deploy and manage a distributed system as powerful as Cloud Foundry in less than twenty classes and 1700 lines of code.

So, to summarize where we’ve been, while there’s an impressive war of words out there regarding API compatibility with Amazon AWS, Cloud Foundry and BOSH don’t necessarily need to take sides. If OpenStack chooses to embrace the AWS API’s, the BOSH AWS CPI will be there waiting. However, if OpenStack chooses to follow in the footsteps of Apple and take the road less traveled, the OpenStack CPI is ready and waiting to evolve with it. Should Google Compute Engine or Microsoft’s Azure gain a foodhold on the innovation curve, they are presumably a relatively simple CPI away from joining the BOSH ecosystem. So if you really want “cloud freedom,” BOSH is leading the charge.

Blue-Green Deployments on Cloud Foundry

One of the great things about Cloud Foundry is that it is a great enabler. Tall words. But what do they mean? Essentially, Cloud Foundry (and any other well-designed PaaS) enables us to do things as developers and operators that would be extremely difficult in a traditional deployment environments. One particularly valuable area of enablement is our new found ability to practice Continous Delivery, meaning that we continuously prove our ability to deliver working software by continuously treating each code commit to a system as if it could be deployed to a production environment. We do this by shipping these commits through what’s called a “deployment pipeline,” which consists of a series of build-test-deploy cycles that prove out a commit’s suitability for production deployment. At the end of the pipeline we can either deploy automatically to our production environment (i.e. continuous deployment), or we can have a business decision to deploy that “deployable artifact” or not.

One particular practice associated with Continuous Delivery is Blue-Green Deployments. Martin Fowler describes these very well at the link provided, but I’ll summarize briefly here:

  • Cut-over to a new software version is tricky, and must be quick in order to minimize downtime events.
  • Blue-green deployments ensure the parallel existence of two, identical (as possible) production environments.
  • At any given point, only one (e.g. blue) services production traffic.
  • New deploys are made to the other (e.g. green) environment. Final smoke testing is performed here.
  • When green is determined ready, we begin routing traffic to it.
  • We then stop routing traffic to blue.

Of course, there are several concerns that must be dealt with at the application level in order for this to work (datastores should support concurrent usage by two app versions, long running requests may be killed, etc.). What we’ll focus on in this post is how Cloud Foundry supports the mechanics summarized above.

We’ll begin with a basic Spring application named ms-spr-demo. This app takes users to a simple web page announcing the ubiquitous “Hello World!” message. We’ll utilize the cf command-line interface to push the application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
$ cf push --path build/libs/cf-demo.war
Name> ms-spr-demo

Instances> 1

Memory Limit> 512M

Creating ms-spr-demo... OK

1: ms-spr-demo
2: none
Subdomain> ms-spr-demo

1: cfapps.io
2: mattstine.com
3: none
Domain> 1

Creating route ms-spr-demo.cfapps.io... OK
Binding ms-spr-demo.cfapps.io to ms-spr-demo... OK

Create services for application?> n

Save configuration?> y

Saving to manifest.yml... OK
Uploading ms-spr-demo... OK
Starting ms-spr-demo... OK
-----> Downloaded app package (9.5M)
Installing java.
Downloading JDK...
Copying openjdk-1.7.0_25.tar.gz from the buildpack cache ...
Unpacking JDK to .jdk
Downloading Tomcat: apache-tomcat-7.0.41.tar.gz
Copying apache-tomcat-7.0.41.tar.gz from the buildpack cache ...
Unpacking Tomcat to .tomcat
Copying mysql-connector-java-5.1.12.jar from the buildpack cache ...
Copying postgresql-9.0-801.jdbc4.jar from the buildpack cache ...
Copying auto-reconfiguration-0.6.8.jar from the buildpack cache ...
-----> Uploading droplet (48M)
-----> Uploaded droplet
Checking ms-spr-demo...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  1/1 instances: 1 running
OK

The end result of this cf push event is that an application is now serving requests at http://ms-spr-demo.cfapps.io. The following graphic shows the state of our system, with the CF Router sending traffic to our application:

Next, we make a slight change to our application. Rather that saying “Hello World!” we decide to make it say “Goodbye World!” We build a new war file for the application. Rather than letting cf prompt us this time, we’ll make use of the manifest.yml file that we saved from our previous push. However, we’ll rename the application and provide a new route. Take a look:

1
2
3
4
5
6
7
---
applications:
- name: ms-spr-demo-green
  memory: 512M
  instances: 1
  url: ms-spr-demo-green.cfapps.io
  path: build/libs/cf-demo.war

As you can see, we’re calling our new application version ms-spr-demo-green and we’re mapping it to the URL ms-spr-demo-green.cfapps.io. Let’s push the application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
Using manifest file manifest.yml

Creating ms-spr-demo-green... OK

1: ms-spr-demo-green
2: none
Subdomain> ms-spr-demo-green

1: cfapps.io
2: mattstine.com
3: none
Domain> 1

Creating route ms-spr-demo-green.cfapps.io... OK
Binding ms-spr-demo-green.cfapps.io to ms-spr-demo-green... OK
Uploading ms-spr-demo-green... OK
Starting ms-spr-demo-green... OK
-----> Downloaded app package (9.5M)
Installing java.
Downloading JDK...
Copying openjdk-1.7.0_25.tar.gz from the buildpack cache ...
Unpacking JDK to .jdk
Downloading Tomcat: apache-tomcat-7.0.41.tar.gz
Copying apache-tomcat-7.0.41.tar.gz from the buildpack cache ...
Unpacking Tomcat to .tomcat
Copying mysql-connector-java-5.1.12.jar from the buildpack cache ...
Copying postgresql-9.0-801.jdbc4.jar from the buildpack cache ...
Copying auto-reconfiguration-0.6.8.jar from the buildpack cache ...
-----> Uploading droplet (48M)
-----> Uploaded droplet
Checking ms-spr-demo-green...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  1/1 instances: 1 running
OK

We now have two instances of the application running, each of them using distinct routes:

Now it’s time for the magic to happen. We’ll map our original URL route (ms-spr-demo.cfapps.io) to our “green” instance. This is accomplished very simply by using cf:

1
2
cf map --app ms-spr-demo-green --host ms-spr-demo --domain cfapps.io
Binding ms-spr-demo.cfapps.io to ms-spr-demo-green... OK

The CF router immediately begins to load balance requests between each instance of the application, as shown here:

Now our router will send requests to ms-spr-demo.cfapps.io to both instances of the application, while ms-spr-demo-green.cfapps.io only services the “green” instance. Once we determine that all is well, it’s time to stop routing requests to the “blue” instance. This is just as simple using cf:

1
2
cf unmap --url ms-spr-demo.cfapps.io --app ms-spr-demo
Unbinding ms-spr-demo.cfapps.io from ms-spr-demo... OK

Our “blue” instance is now no longer receiving any web traffic:

We can now decomission our “blue” instance, or we can leave it around for a period of time in case we decide we need to roll back our changes. The important thing is that our customers suffered absolutely no downtime!

Clojure on Cloud Foundry

I was inspired by Brian McClain’s post on bringing Haskell to Cloud Foundry using Cloud Foundry v2 buildpacks, so I decided to go on a buildpack journey of my own. Since Clojure is the language I most enjoying “toying around with,” I thought I’d try to deploy a simple Clojure web application using the Heroku Clojure Buildpack.

To reiterate some of the coolness around buildpacks, they are what allows a PaaS like Cloud Foundry or Heroku to support various runtimes without first building that support into the core platform. If your favorite language or framework runtime isn’t available, there’s absolutely nothing stopping you from providing your own buildpack to support it. Stuff can get crazy – McClain has even hinted at bringing FORTRAN support to Cloud Foundry.

I decided for my experiment to build a VERY basic “Hello World” style application using Ring, which is “a Clojure web applications library inspired by Python’s WSGI and Ruby’s Rack.” The easiest way to get started building Ring applications is to utilize the popular Clojure build tool Leiningen.

First I started by creating a new project:

1
2
$ lein new hello-cf
$ cd hello-cf

The next task was to add the Ring dependencies to my project.clj file:

1
2
3
4
5
6
7
8
(defproject hello-cf "0.1.0-SNAPSHOT"
  :min-lein-version "2.0.0"
  :description "Hello Clojure on Cloud Foundry!"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.4.0"]
                 [ring/ring-core "1.1.8"]
                 [ring/ring-jetty-adapter "1.1.8"]])

Then it was time to create the Ring application itself, by editing src/hello_cf/core.clj:

1
2
3
4
5
6
7
8
9
10
(ns hello-cf.core
  (:use [ring.adapter.jetty :only [run-jetty]]))

(defn handler [request]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body "Hello Cloud Foundry from heroku-buildpack-clojure!"})

(defn -main [port]
  (run-jetty handler {:port (Integer. port)}))

Let’s break this down a bit. The handler function will handle any HTTP request that hits our application, and will return an “OK” response containing a pleasant message indicating that we’ve succeeded. That’s really about it. Our application is complete. We can test it out by running the following:

1
2
3
$ lein trampoline run -m hello-cf.core 8080
2013-05-29 22:42:52.576:INFO:oejs.Server:jetty-7.6.1.v20120215
2013-05-29 22:42:52.804:INFO:oejs.AbstractConnector:Started SelectChannelConnector@0.0.0.0:8080

Hitting http://localhost:8080 in the browser confirms that we’re well on our way. Now it’s time to trying pushing the application to Cloud Foundry. As Brian stated in his blog, one of the stellar aspects of Cloud Foundry buildpacks is that they are approximately the same as Heroku buildpacks. Practically, this means that one should be able to utilize a Heroku buildpack on Cloud Foundry with minimal or no modifications. Let’s put that theory to the test, shall we? Before we do, let’s create a Procfile quickly to let the buildpack know what we want to run:

1
web: lein with-profile production trampoline run -m hello-cf.core $PORT

And on with the push:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
cf push hello-cf --buildpack=git://github.com/heroku/heroku-buildpack-clojure.git
Using manifest file manifest.yml

Creating hello-cf... OK

1: hello-cf
2: none
Subdomain> hello-cf

1: mstine.cf-app.com
2: none
Domain> mstine.cf-app.com

Binding hello-cf.mstine.cf-app.com to hello-cf... OK
Uploading hello-cf... OK
Starting hello-cf... OK
-----> Downloaded app package (12K)
Initialized empty Git repository in /tmp/buildpacks/heroku-buildpack-clojure.git/.git/
Installing heroku-buildpack-clojure.git.
-----> Installing OpenJDK 1.6...done
-----> Installing Leiningen
       Downloading: leiningen-2.1.2-standalone.jar
       Writing: lein script
-----> Building with Leiningen
       Running: lein with-profile production compile :all
       Retrieving lein-standalone-repl/lein-standalone-repl/0.1.5/lein-standalone-repl-0.1.5.pom from clojars
       Retrieving lein-standalone-repl/lein-standalone-repl/0.1.5/lein-standalone-repl-0.1.5.jar from clojars
       Performing task 'compile' with profile(s): 'production'
       Retrieving org/clojure/clojure/1.4.0/clojure-1.4.0.pom from
       ...
       Compiling hello-cf.core
-----> Uploading staged droplet (66M)
-----> Uploaded droplet
Checking hello-cf...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
  0/1 instances: 1 starting
  0/1 instances: 1 down
  0/1 instances: 1 starting
  0/1 instances: 1 flapping
Application failed to start.

Drat. Let’s take a quick look at the logs to see what may be awry:

1
2
Reading logs/stderr.log... OK
/home/vcap/app/.lein/bin/lein: line 42: java: command not found

Ah-hah! Looks like the existing buildpack is making some assumptions about the structure of our application that no longer hold true on Cloud Foundry. So, I followed in Brian’s footsteps and forked away. One small commit looks like it ought to fix the problem. Let’s give it another try:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
cf push hello-cf --buildpack=git://github.com/mstine/heroku-buildpack-clojure.git
Using manifest file manifest.yml

Not applying manifest changes without --reset
See `cf diff` for more details.

Uploading hello-cf... OK
Changes:
  buildpack: 'git://github.com/heroku/heroku-buildpack-clojure.git' -> 'git://github.com/mstine/heroku-buildpack-clojure.git'
Updating hello-cf... OK
Stopping hello-cf... OK

Starting hello-cf... OK
-----> Downloaded app package (8.0K)
-----> Downloaded app buildpack cache (17M)
Initialized empty Git repository in /tmp/buildpacks/heroku-buildpack-clojure.git/.git/
Installing heroku-buildpack-clojure.git.
-----> Installing OpenJDK 1.6...done
-----> Using cached Leiningen 2.1.2
       Writing: lein script
-----> Building with Leiningen
       Running: lein with-profile production compile :all
       Performing task 'compile' with profile(s): 'production'
       Compiling hello-cf.core
-----> Uploading staged droplet (66M)
-----> Uploaded droplet
Checking hello-cf...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
Staging in progress...
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  0/1 instances: 1 starting
  1/1 instances: 1 running
OK

BOOM!

I quickly pointed my browser, and as yesterday’s tweet indicates, success:

Score another win for Cloud Foundry’s buildpack support. I’m now toying with the idea of doing something of a world tour of LISP on Cloud Foundry. My next candidate may be Scheme.

Into the Crucible

Wow…it seems I only post to this blog toward the end of May. Well, that all changes now. You see, as of June 3, 2013, this blog is going to become one of many aspects of my new “day job.” On Monday, I start my life as a Community Engineer with Cloud Foundry by Pivotal. What’s a Community Engineer? Quite honestly, I’m not completely sure of the answer to that question yet. But given the many conversations I’ve had over the past few weeks, it seemingly fits right in with the bridge-building roles I’ve played many times over the course of my career. In this case, I have one foot squarely planted in the world of Cloud Foundry engineering, and one foot squarely planted out in the world with you guys and gals - the community. My job is to help you understand how we are quite literally seeking to “build a new platform for a new era.”

Of course, this is a journey that for me started a few years ago. In my previous life as a front-line development manager, I helped lead an agile transformation within my team with “ruthless automation” playing a central role in everything that our team did. However, it seemed that the better we “did agile,” the more pain we felt when dealing with those outside of our circle of control. It was only years later, after reading Eliyahu Goldratt’s The Goal and coming to an understanding of his Theory of Constraints, that I realized what was happening. Our constraints had moved “out of the plant,” if you will, and landed in the world of operations. Even without this understanding, I developed a keen interest in this newly emerging topic called “DevOps” and began to explore the ideas emerging around agile operations and infrastructure as code. I started playing with tools like Puppet, Chef, and Vagrant, and taught sessions on all three of them at the Project Automation Experience in 2011.

You can read my last entry and find out that not much later I joined VMware as a Senior Consultant for its Cloud Application Platform. I was hired into that role based on my extensive background in enterprise Java and the Spring ecosystem, but it was nothing short of a staffing accident that I found myself thrust into a role on a virtualization platform provisioning team helping to build out a private self-service cloud! I was steadily getting carried further away from my role as an application architect, steadily becoming assimilated into that mysterious world of web operations that I knew so little about. These experiences, along with my continued reading and thinking about the worlds of DevOps, Lean, and Kanban, have quite literally changed the way I look at the world of software engineering (or as I prefer to think of it now, value delivery through product engineering that just so happens to involve software!). These experiences have formed around me a crucible, melting me that I might be poured into a new professional mold.

So now it’s time to plunge into the world of building the leading open platform as a service, and to help YOU experience the HUGE can of @cloudfoundry awesome that we at Pivotal are about to unleash on the world. Sound good to you? Join us!

The Relaunch

I have rebooted this blog many times over the last several years. If you’ve been a reader of my blog in the past, you will have noticed significant changes. If you’re new here, welcome!

This reboot has been in the works for several months now, even though I’ve probably spent far less than 24 active hours working on it. Life as an “itinerant” consultant and conference speaker is extremely busy compared to what I was doing on May 16, 2012 (the date of my last blog posting). At that time I was in the middle of a transition from 3+ years as a front-line manager of a software development team into a lead software architect role. Since that time I’ve changed jobs (careers?) twice:

  • June 2011 - February 2012: During this time I was titled as a “Technical Architect” at AutoZone’s corporate offices in Memphis, TN. My focus was the modernization of their retail store management system architecture and technical stack. While there I also did a fair amount of agile coaching.
  • February 2012 - Present: In January, VMware came calling, wanting to know if I would join the vFabric Cloud Application Platform PSO as a Senior Consultant. After a few weeks of interviews and careful consideration, I made the jump. I’m now in the midst of helping a Fortune 50 company build a brand new “private cloud” hosting platform for their next generation of application development.

During that time I also significantly increased my participation on the No Fluff Just Stuff tour. In other words, I’ve BEEN BUSY.

At any rate, I have for quite awhile now wanted to get back into the practice of writing. However, I’ve wanted to do so outside the constraints of a hosted platform like Blogger or WordPress.com. Those systems place far too many constraints on how your blog works, and they also happen to be the only place that your data is housed. Sure, I could write my posts in Vim and then copy/paste them into the platform, but that’s annoying. I wanted to manage my writing using a plain text editor, mark it up using something lightweight like Markdown, check my writing into version control using Git, generate my site using a tool like Jekyll, and then push my site to the cloud. What was the easiest way to achieve all of this goodness? Octopress seemed to fit the bill.

So, what you now see is the result of my labor in migrating from WordPress.com to Octopress, hosted at Heroku. One day (perhaps) I’ll write up the process that I used. It was a bit convoluted and involved gigantic piles of yak hair, so I’d rather not consider that piece right now.

What I will be focusing on quite a bit is the idea of simplicity. For more than a decade I have wandered through the barren wasteland that is accidental complexity in software. A few months ago I viewed Rich Hickey’s keynote session at StrangeLoop 2011 entitled “Simple Made Easy”. It is Rich’s definition of simplicity in software that has inspired the brand new title of this blog, wherein I will chronicle my continuing quest to seek out simplicity in software structure.

Alas, it is time to return to billable work. I shall endeavor to post again in far less than a year. So, as we say in the south, “Ya’ll come back now, you hear?”

Design Meeting Patterns/Antipatterns

For those of you that don’t know, I recently returned to the technical ranks as a Software Architect after a three-year stint in management. To make a long story short, I now love my job again. Perhaps I’ll write the long story in a future blog entry. On to the topic at hand. Today I led the first significant design discussion that I have led in quite a long time. A few minutes afterward, I was already reflecting on what had occurred and how. The very first thing that I noticed was my drastically different approach to the task. I felt that my facilitation skills were more mature and effective than I remember from previous similar engagements. I’m not sure if the time I spent in management (where I facilitated quite a few meetings, though much more business-focused ones) helped, but something certainly has.

I also noticed several things that I’ll call “pattern vs. antipattern” tugs of war. Quite often during the meeting I felt the group trying to move our discussion in one direction that I, for better or for worse, thought was ineffective. I would then redirect us on to what I felt was a better path. In true catalog form, here’s the list:

  • Understand the Problem vs. Jump to the Solution - only a couple of minutes into our discussion solutions were being flung about like drunken darts. This situation almost always leads to suboptimal or faulty solutions. As a facilitator, try to ensure that the problem has been clearly stated. If at all possible, write it down on a whiteboard. Make sure everyone agrees that the problem as stated is the problem the group is there to solve. Sure enough, after performing this exercise, we all had a very different and clear understanding of the problem than that with which we walked in the door.

  • Assume the Worst vs. Assume the Best - occasionally the exact details of a requirement are unclear, and not assuming something will totally derail the design discussion. You have a couple of choices. The first is to halt the discussion and get the details cleared up. This is clearly the best solution, as you’ll no longer have to assume anything. However, it can be the case that the person who can clear things up isn’t available. Or in some cases, the question you’ll be asking will require another meeting at another level of the organization. If you find yourself in that spot, and you can’t wait (we couldn’t!), then the best approach is to work from the worst possible case scenario. You’ll then be in the best position to handle whatever answer comes your way. However, our tendency is often to assume the best (“That will never happen!”). Fight that tendency. However, whatever you choose, follow up at your earliest opportunity.

  • Basing Decisions on the Current Situation vs. Basing Decisions on History - many times the group wanted to veer off into safer territory. In some cases, a possible solution departed significantly from the current design. While this is a valid concern (we do want consistency of design across the system where possible), it is certainly not a trump card. Occasionally the situation at hand will merit a significant departure from the current design. Another way history can rear its ugly head is the assertion that we’ve always solved similar problems like ‘x,’ so we should do so with this problem as well. Again, note the word “similar.” All problems are somewhat different and have their own eccentricities. So, rather than working from history, I pushed us back to a clean slate with the statement “Let’s stop thinking about the past and start from scratch. We may very well come up with the same solution you guys are proposing, but I’d rather do so through our own objective analysis and not instinct.” Guess what. We came up with a different solution that we all felt better about.

  • Shooting for the “Best” Solution vs. the “Easiest” Solution - now sometimes we can’t afford the best solution. I grant that. However, I’m trying to fight the tendency to immediately jump to the “easiest thing that could possibly work.” Often this pops up in the first P vs. AP - if we don’t clearly understand the problem, sometimes an easy solution jumps out that doesn’t deal with the underlying details we’ve yet to uncover. Also, sometimes the best solution is quite simple and elegant. It doesn’t necessarily have to be harder and more complex than the easiest solution. In fact, sometimes the “easiest” solution leads to the most accidental complexity in the long-run. So, shoot for the best solution you can come up with and only then, optimize for cost.

  • Present Possible Solutions Objectively vs. My Solution is the Best! - one would hope that we all start here, but we don’t. We tend to like our own solutions to problems and want them to “win.” Our ego can get in the way of even hearing an alternate solution presented by another team member. I point you to my colleague Ted Neward’s post for more on “egoless programming.” So, as a facilitator, you’ve got to make sure that all solutions are presented objectively. I often had to say things like “OK, let’s assume before we ever get started that this is a good solution to the problem and not hack away at it until its fully presented, and we all understand it.” In the end, this insistence led us to choose a solution that none of us (myself included) originally thought we’d pick.

  • Validating from Code vs. Validating from Memory - more often than not, questions about the existing design/code/behavior will come up. Rather than scratching your head and trying to remember what you wrote six months ago, pull up the code and find out. I can’t tell you the number of meetings I’ve attended where baseless assertions were made about existing code, only to require another meeting the next day to revisit the whole discussion once those assertions were proven wrong. Again, as a facilitator, I directed us to solve every problem for which all of the facts were available. We inserted placeholders in our solution where questions remained. Guess what we’re doing now? Well, I’m blogging about the meeting, but the rest of us are validating from code. Tomorrow will fill in the blanks!

Selenium Conference 2011

I completed an interest survey for a potential Selenium-focused conference several weeks ago, and I’m excited to let you know that the “powers that be” have decided that the conference is going to happen! I have already submitted my “Executable Specifications: Automating Your Requirements Document with Geb and Spock” talk as a potential session. Whether it makes the conference program or not, I plan on attending the event. Here are the details:

Selenium Conference 2011

Join members of the growing Selenium community for 3 jam-packed days of talks, workshops, lightning talks, and hack sessions. Hear speakers from around the world talk about the present and future of automated testing, share ideas with fellow Selenium developers, including Core Committers, and take part in shaping the future success of the Selenium project.

When: April 4-6, 2011 Where: Marines’ Memorial Club & Hotel, 609 Sutter St, San Francisco, CA 94102 USA Register now to be a speaker or sponsor. More details: http://www.seleniumconf.com/

2010 in Review: The Automated Analysis…

The stats helper monkeys at WordPress.com mulled over how this blog did in 2010, and here’s a high level summary of its overall blog health:

Healthy blog!

The Blog-Health-o-Meter™ reads Fresher than ever.

Crunchy numbers

            ![Featured image](http://s0.wp.com/i/annual-recap/abstract-stats-1.png)

A helper monkey made this abstract painting, inspired by your stats.

A Boeing 747-400 passenger jet can hold 416 passengers. This blog was viewed about 6,600 times in 2010. That’s about 16 full 747s.

In 2010, there were 15 new posts, growing the total archive of this blog to 115 posts. There were 24 pictures uploaded, taking up a total of 979kb. That’s about 2 pictures per month.

The busiest day of the year was June 4th with 139 views. The most popular post that day was Don’t build software that’s TOO smart!.

Where did they come from?

The top referring sites in 2010 were agile.dzone.com, java.dzone.com, twitter.com, Google Reader, and javablogs.com.

Some visitors came searching, mostly for matt stine, groovy http post, groovy post, groovy url post, and groovy url encode.

Attractions in 2010

These are the posts and pages that got the most views in 2010.

1

                [Don't build software that's TOO smart!](http://mattstine.com/2010/06/03/dont-build-software-thats-too-smart/) June 2010  

1 comment

2

                [Groovy: Post to a URL](http://mattstine.com/2009/04/25/groovy-post-to-a-url/) April 2009  

4 comments

3

                [Resume](http://mattstine.com/resume/) March 2009                                           

4

                [Deploying Grails with Groovy](http://mattstine.com/2009/03/29/deploying-grails-with-groovy/) March 2009                                            

5

                [LOTY Time Again: Scala or Clojure?!?!](http://mattstine.com/2009/04/02/loty-time-again-scala-or-clojure/) April 2009  

17 comments

Selenium 2.0 Refcard

Today DZone released my first Refcard: ”Selenium 2.0: Using the WebDriver API to Create Robust User Acceptance Tests.” I have been interested in writing a Refcard for a long time but have never pulled the trigger. For whatever reason, I decided to jump on the bandwagon last month. Writing this card was quite a challenge, as it’s difficult to boil down a big topic into only six pages while keeping it useful. Thus far the feedback from the community has been largely positive.

If you’re looking for your next automated testing tool for browser-based user acceptance tests, download the card and let me know what you think!

Agile Zone Roundup

I’ve been really quiet on this blog lately. There have been multiple reasons for that. I’ve been extremely busy preparing for conferences this Fall including SpringOne/2GX and The Rich Web Experience. I’ve also stayed busy writing for DZone. I have my first DZone Refcard coming out soon, which is focused on Web Driver/Selenium 2. I’ve also been writing my regular articles for Agile Zone. Here’s a roundup: