Multiversion Push To Deploy with Jenkins, BitBucket, and GAE

Round Two in my ongoing battle to get CI to work (Round 1 was a bit of a wash).  I had a pretty straight forward goal in mind this time: Push a branch to BitBucket, initiate a Jenkins build job, and push to a specific version of the app on AppEngine based on which branch was pushed (development, staging, master).

Basic Setup and Deployment

I was able to make some headway following Google's Push-to-Deploy article, but gcloud didn't give me the flexibility I needed, so I started looking into using appcfg.sh directly.  The documentation from Google was helpful in getting that set up. The part that proved a bit tricky was getting authentication for jenkins set up.  Ultimately, I just ran appcfg.sh from the terminal with sudo -u jenkins, which prompted me for an access code:

I used just sudo above because I didn't feel like deleting jenkin's credential file, so this is trying to create one for root.  Once you enter the code from the URL, it creates the credential file ~/.config/gcloud/application_default_credentials.json that stores the authorization, so you don't need to do this over and over.

At first I was calling Maven from a shell script, but I'm started to get a better feel for what Jenkins will make easier, and figured out how to use the "Maven top level target" build action.  So now my build steps look like this:

I could easily clean up the deployment script by removing all the comments, and come to think of it I really don't need to cd into the workspace directory, I could just set the target path to $WORKSPACE/target/conference-1.0.  But this works.

To get it to deploy to a specific version, I have three seperate jobs: Push-to-Deploy, Push-to-Stage, and Push-to-Dev.  There is very little that varies between the three jobs.  The builds operate on the master, staging, and development branches respectively, and are triggered when the BitBucket webhook notifies Jenkins of a change. The setup for "Push-to-Dev" is below:

In BitBucket, you can set a webhook under Setting.  There is also a way to integrate with Jenkins via a service but it seemed more complicated, the webhook was pretty straightforward:

Or so it seems... turns out the BitBucket Plugin for Jenkins does not yet support the webhook JSON payload format, so using the "Webhook" will cause it to bomb out (and not build).  The "Jenkins" service is one option but it's tied to one project, so I would need three of them.  I opted to just use a "POST" service pointing that the bitbucket plugin URL on my Jenkins server.

And that is pretty much all there is to it for the versioning.  Any time a change is pushed for the dev, staging, or master branch, Jenkins will build and deploy the appropriate version.  There is, however, one drawback to this:  since we aren't setting namespaces in the code, all the data is shared across all versions.  As is, this would be unacceptable for a production deployment because your live data and test data would be intermingled, so you wouldn't have a true sandbox for testing.  Luckily, implementing namespaces is not super difficult.  A couple of the API documents came in very useful:
Namespaces Java API - Java and Implementing Multitenancy Using Namespaces.

Namespaceing Data

To implement Namespaces basically involves creating a filter that sets the namespace at the beginning of each request.  I took the code from the sample java project on the Namespaces Java API page and tweaked it so that the "stage" and "dev" versions used the "test" namespace, and production uses the "prod" namespace.  This means stage and dev will share test data, and prod will have all the live data.

A parred down version of my filter class is below (cut out comments and extra code paths):

public class NamespaceFilter implements Filter {
 
  enum NamespaceStrategy {
    SERVER_NAME,
    GOOGLE_APPS_DOMAIN,
    EMPTY,
    VERSION
  }
  
  private NamespaceStrategy strategy = NamespaceStrategy.SERVER_NAME;
  private FilterConfig filterConfig;

  @Override
  public void init(FilterConfig config) throws ServletException {
    this.filterConfig = config;
    String namespaceStrategy = config.getInitParameter("namespace-strategy");
    if (namespaceStrategy != null) {
      try {
        strategy = NamespaceStrategy.valueOf(namespaceStrategy);
      } catch (IllegalArgumentException exception) {
        // Badly configured namespace-strategy
        filterConfig.getServletContext().log(
            "web.xml filter config \"namespace-strategy\" setting " +
            "to \"" + namespaceStrategy + "\" is invalid. Select " +
            "from one of : " + 
            Arrays.asList(NamespaceStrategy.values()).toString());
        throw new ServletException(exception);
      }
    }
  }
  
  @Override
  public void destroy() {
    this.filterConfig = null;
  }

  @Override
  public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
      throws IOException, ServletException {

      System.out.println("Current Namespace:" + NamespaceManager.get());
    if (NamespaceManager.get() == null) {
      switch (strategy) {
        case VERSION : {
            //this will give us the version in the form of <version>.<timestamp>
            //dev and stage will use "test", and prod will use "prod"
            String rawVersion =  SystemProperty.applicationVersion.get();
            System.out.println("Version: " + rawVersion);
            
            if(Pattern.matches("prod.*", rawVersion))
                NamespaceManager.set("prod");
            else
                NamespaceManager.set("test");
            break;
          }
      }
    }

    chain.doFilter(request, response) ;
  }
}

And this is the configuration added to web.xml:

    <!-- Configure the namespace filter. -->
  <filter>
    <filter-name>NamespaceFilter</filter-name>
    <filter-class>com.google.devrel.training.conference.filter.NamespaceFilter</filter-class>
    <init-param>
      <param-name>namespace-strategy</param-name>
      <!-- Change this to the desired strategy, see NamespaceFilter.NamespaceStrategy -->
      <param-value>VERSION</param-value>
    </init-param>
  </filter>
  <filter-mapping>
      <filter-name>NamespaceFilter</filter-name>
      <url-pattern>/*</url-pattern>
  </filter-mapping>

With that in place, creating datastore entities within the application would put them in the correct namespacing "bucket", which can be explored seperately from the datastore browser:

Next step is to get code coverage and static analysis using JaCoCo and SonarQube fully functional, but I'll leave that to a future post.

One last thing I did is install the Embeddable build status plugin so that I could include the build status in the README that shows up on the BitBucket repo page.  I see that more and more and I think it is pretty cool: