Continuous Delivery on OpenShift

layout: true
name: top-bar
.left[![HealthPartners](/images/hplogo.gif)]

---

class: center

# Continuous Delivery on OpenShift

<hr />

## Building a functional build/deploy system at HealthPartners

---

# Philosophies

1. High availability for production
2. Secure credentials for all environments
3. Good default behavior
4. Continuous delivery – on demand from the business perspective
5. Everything comes from source control
6. Keep things ephemeral, record externally

---

# Cluster Design

* Build tools run on bld1 cluster
* Applications have three environments (dev, uat, prod)
.center[![Clusters](/images/cluster-design.png)]

---

# Tools

* Jenkins
* Bitbucket and Gitlab (long story)
* Nexus and Artifactory (long story)
* Jira & HipChat
* Hashicorp Vault
* Fortify
* Sonarqube
* ServiceNow

---

# OpenShift Concepts

* `BuildConfig` object integrates with Jenkins using `openshift-sync` plugin
* Jenkins uses `openshift-login` plugin to delegate access
* Jenkins pod running in each OpenShift namespace
* Each team owns one or more namespaces

---

# Jenkins Concepts

* Shared libraries create functions that can be called from Jenkinsfile
* Groovy scripts stored in `vars/` directory
  * Must declare `public def call()` method
* Groovy classes stored in `src/` directory
* Resources stored in `resources/` directory
  * Loaded with `libraryResource` function
* More info at https://jenkins.io/doc/book/pipeline/shared-libraries/
---

# Let's Dive In

```groovy
library identifier: 'hp-pipeline-library@master', retriever: modernSCM(
    [$class: 'GitSCMSource',
    remote: 'ssh://git@git.healthpartners.com/paas/hp-pipeline-library.git',
    credentialsId: "${env.NAMESPACE}-bitbucketsecret"]
)

hpPipeline([
  appName: "my-service",
  deployPom: "my-service/pom.xml",
  team: "Cloud"
])
```

.center[![BlueOcean](/images/jenkinsBO.png)]

---

# Developer Viewpoint

* Create your git repository with Jenkinsfile and pipeline.yaml at the root
* `oc login https://openshift-bld1.com && oc project MYPROJECT`
* `oc create -f pipeline.yaml`
* Jenkins pod in the namespace runs the build

---

# The Pipeline

1. Prepare
2. Build
3. Test
4. Pre-deploy
5. Deploy
6. Verify
7. Repeat 4-6

---

# The Code

hpPipeline.groovy
```groovy
def call(pipelineConfig) {
	new PipelineWrapper(this).wrapPipeline(pipelineConfig) { config ->
        def target = [namespace: env.NAMESPACE,
          environment: OpenshiftEnvironments.getBuildEnvironment()]
		DefaultEvaluator.evaluateDefault(config,
            'buildStage')?.call(config, target, this)
		def envArray = DefaultEvaluator.evaluateDefault(config,
            'deploymentEnvironments').call(config)
		for (String environment : envArray) {
            target << [environment: environment]
			beginEnvironment(environment, config)
			deployToEnvironment(environment, config)
			postDeployStages(environment, config)
		}
	}
}
```

---

# Preparation

* Set some base variables

```groovy
def wrapPipeline(pipelineConfig, closure) {
    pipelineConfig.audit = [:]
    pipelineConfig.defaults = new DefaultEvaluator().initialize()
    if (!pipelineConfig.openshift) {
       pipelineConfig.openshift = [:]
    }
    if (!pipelineConfig.skipNotify) {
        pipelineConfig.skipNotify = false
    }
    pipelineConfig.pipelineJobId = UUID.randomUUID().toString()
    ...
```

---

# Preparation

* Tell people what is happening

```groovy
context.notifyHipChatRoom([
  team: pipelineConfig.team,
  message: "Jenkins build started: ${detail}",
  skipNotify: pipelineConfig.skipNotify
])
context.audit([
        message: "Jenkins build started",
        build: context.env.BUILD_DISPLAY_NAME,
        job: context.env.JOB_NAME
], pipelineConfig)
```

---

# Preparation

* Enforce specific properties

```groovy
if (!pipelineConfig.appName) {
    error "You must define an appName property"
}
if(!pipelineConfig.team) {
    error "You must define a team property"
}
```

---

# Preparation

* Set global environment variables for the job

```groovy
env.DOCKER_HOST = 'docker.healthpartners.com'
env.FORTIFY_CLD_CTRL_HOST = 'https://fortify.healthpartners.com/cloud-ctrl'
env.FORTIFY_SSC_HOST = 'https://fortify.healthpartners.com/ssc'
env.FORTIFY_TOKEN = 'my-fav-token'
env.MAVEN_IMAGE_HASH = 'sha256:abc123'
env.FORTIFY_IMAGE_HASH = 'sha256:def456'
env.NODE_IMAGE_HASH = 'sha256:ghi789'
env.JIRA_HOST = 'jira.healthpartners.com'
env.SONARQUBE_URL = "http://sonarqube.${env.NAMESPACE}.svc:9000".toString()
env.AUDIT_URL = 'https://auditing-service.healthpartners.com/'
```

---

# Tangent: Defaults and Overrides

* Goal: Be able to override anything
* Sensible defaults in a place you can find them
* Created a groovy specific structure Evaluator/Configurer
* Single place where defaults are stored and evaluated
* Each default contributor is specific to a category
  * e.g. `MavenDefaultContributors`
---

# Tangent: Defaults and Overrides

* Groovy truthiness makes life easy here

```groovy
def evaluateDefault(config, String key) {
    def overrides = config?.overrides?.get(key)
    //single argument function
    def metaClassArgs = overrides?.metaClass?.respondsTo(overrides,
        'call', [Object.class].toArray())
    if (overrides && metaClassArgs && !metaClassArgs.isEmpty()) {
        return overrides.call(config)
    } else if (overrides){
        //its not a closure, so it must be a value: return it
        return overrides
    } else {
        //attempt the default closure, then the default value
        return defaultClosures?.get(key)?.call(config) ?:
                defaultValues[key]
    }
}
```

---

# Tangent: Defaults and Overrides

```groovy
class MavenDefaultContributors implements DefaultContributor {
    @Override
    LinkedHashMap getClosures() {
        return [
                packaging: {it.packaging},
                compilePom: {it.pomPath ?: it.appDir ? it.appDir + "/pom.xml" : null},
                packageFlags: {it.packageFlags ?: ''},
                deployFlags: {it.deployFlags ?: ''},
                integrationTestFlags: {it.integrationTestFlags ?: ''},
				webTestFlags: {it.webTest?.mavenFlags ?: '' }
        ]
    }

@Override
    LinkedHashMap getValues() {
        return [
                packaging: 'jar'
        ]
    }
}
```

---

# Tangent: Defaults and Overrides

* This structure allows developers to override anything that is keyed by a contributor

```groovy

hpPipeline([
  ...
  overrides: [
    packageFlags: "-PmyFavoriteProfile -Dprop=value"
  ]
])
```

---

# Build Stage

* Goal: Get a validated docker image to deploy
* Steps:
  * Package artifact
  * Run unit/component tests
  * Run s2i build from base image
  * Publish image to external registry

---

# Testing

* Unit & Component tests run before pre-deployment begins
* Generally JUnit tests, building @SpringBootTest component tests to load parts of the application
* H2 Database testing

---

# Pre-Deployment

* Approve the environment
* Check that the cluster is up
* Change management (will go into detail later)

```groovy
def call(envKey, config) {

def envConfig = OpenshiftEnvironments.getEnv(envKey)

getEnvironmentApproval(envConfig, config)

checkEnvironmentAvailability(envConfig)

jiraValidator(config, envConfig)

changeStart(config, envConfig)
}
```

---

# Deployment

* Use the API to update the deploymentconfig and associated objects
* Right now, we are using Jenkins to generate YAML file manifest

```groovy
def objects = [
    'BuildConfig' : new BuildConfig().create(config, target),
    'ImageStream' : new ImageStream().create(config, target),
    'DeploymentConfig' : new DeploymentConfig().create(config, target),
    'Service' : new Service().create(config, target),
]
```

---

# Verification

* Post deployment separated into "Phases" and run in parallel
* SonarQube scan phase
* Fortify scan phase
* Separate scan from validating results
* Integration tests live with the source code
* Zephyr tests from JIRA
* Selenium tests through a Selenium grid
* Change management verification

---

# Verification

```groovy
if(!config.buildOnly) {
    addPhase(postDeployMap, new IntegrationTestPhase(this),
      environment, config, null)
}
if (config.zephyr) {
    if (config.zephyr.environments.contains(environment)) {
        addPhase(postDeployMap, new ZephyrTestPhase(this),
          environment, config, null)
    }
}
if (project.buildType == 'maven') {
    if (environment.contains('dev')) {
        addPhase(postDeployMap,
          new StaticCodeScanPhase(this), environment, config,
          [new SonarQubeReporting(this), new FortifyReporting(this)])
    }
    else if(environment.contains('uat'))
    {
        addPhase(postDeployMap,
          new StaticCodeValidationPhase(this),
          environment, config, null)
    }
}
```

---

# Change Management

* ServiceNow TableAPI
  * Much of the actions in the ServiceNow UI are available via API
* Standard change request process
  * Change request templates that can be instantiated to changes
  * Simplifies approval process for low priority changes
* Opt in:
  ```groovy
  hpPipeline([
    ...
    cmdb_item: "HealthPartners.com Services",
    autoCreateChange: true,
    ...
  ])
  ```

---

# Change Management

* Created a Jenkins plugin to carry out standard actions
* https://github.com/jenkinsci/service-now-plugin
  * Manage tables
  * Attach files
  * Update state
* Provides a simple entrypoint for making the necessary API calls

---

# Change Management

* Remember the philosophy: Keep things ephemeral, record externally
* ServiceNow is our main audit store
  * Test results
  * JIRA ticket number
  * Git commit info
  * Scan results
  * Deployment time
  * Deployment results

---

# Change Management: Results

* Can go from commit to production in 18 minutes
* Have completed over 10 deploys in a single day
* Multiple other companies using our plugin
* Validated our change management approach with our audit/compliance groups

---

# Visualizing the Data

* Created our own audit datastore and send data to Splunk
* Splunk: Short term event view
* Data store: Long term auditing view of things that happen in the pipeline
  * Approvals
  * Jobs
  * Deployments
  * Validations

---

# Visualizing the Data

.center[![Deployments](/images/deploymentLog.png)]

---

# Multi-Faceted Approach

* This is one of many of these libraries we have written:
  * Liquibase pipelines
  * ConfigMap and Secret management
  * Namespace management
  * Orchestrate Ansible Tower jobs for OpenShift infrastructure
  * Even manage our weblogic servers through these pipelines(!)

---

# Building for the future

* Istio for canary deployments and advanced deployment strategy
* More complete testing strategies
* Read data from metrics stores to know about deployment success
* Tracking historical data

---

# OpenSource

* Working to open-source hp-pipeline-library
* Open sourced our testing library
* Our team manages two Jenkins plugins (service-now and prometheus)
* Open sourced a library that translates Prometheus alerts to ServiceNow (Snogo)
* Looking for more ways to contribute to the community

---

# References

1. https://github.com/jenkinsci/service-now-plugin
2. https://github.com/HealthPartners/jenkins-shared-library-test-helper
3. https://github.com/HealthPartners/snogo
4. https://jenkins.io/doc/book/pipeline/shared-libraries/