{"data":{"site":{"siteMetadata":{"title":"IJHDev Blog","author":"Ian Hayward"}},"markdownRemark":{"id":"815cb5b9-4fba-51fd-99a3-a27d55dcc983","html":"

A guide to building a simple pipeline from PR to deploy with a scalable solution. (TurtleWare 2.0 pt3)

\n\n

This is part 3 of a 3 part blog post about making v2 of turtle wear watch face. You can check out part 1 here and you can check out part 2 here

\n

This post will also be covering everything from PR to releasing on a Kubernetes cluster. We will be skipping some of the basics of CI/CD pipelines and what they are, if you are not sure what that is check out this blog post on a mobile CI/CD here.

\n

Git, pull requests and pipeline starting point

\n

\"GitHub\"

\n

A theme with this guide will be getting free stuff wherever able, the cool thing about open source projects is there is a lot of options. For a good solid Git repo with a good pull request system and plugins to use all these free tools, you can’t go wrong with GitHub. Trunk based development for git is still the preferred method for having something easy to admin but fast to release. For setting that kind of environment up on GitHub as well as protecting your open source project, make sure you have the following settings:

\n

In options:

\n\n

In branches:

\n\n

Inside the branch protection rule:

\n\n

There are other things you can set up here for to depending on what you need such as Wikis and Social Preview images but you can play around there for whatever suits your project best.

\n

SonarCloud

\n

\"SonarCloud\"

\n

One important thing to add to your pipeline is static code analysis. This doesn’t replace the need for other engineers reviewing code, rather it acts as an extra reviewer who is obsessed with code rules. The tool we are going to use in this setup is SonarCloud as it’s free for open-source code and easily plugs into GitHub. You can sign up for a SonarCloud account by using your GitHub auth, this way you can point it to your project so it can analyze it in just a couple of steps. You can set the code analyser to fire every time you trigger a build and it will comment on the pull request. It will cover a range of rules from code duplication, vulnerabilities including OWASP Top 10 bugs, and Unit test coverage. Going back to GitHub as well, we can now set the “Require status checks to pass before merging” to require SonarCloud to pass before allowing merging.

\n

DockerHub

\n

\"DockerHub\"

\n

So while there are more feature-rich automated building tools available, if your plan is to containerise your program using Docker then there is a simple automated builder in Docker called DockerHub. Before you sign up for that it’s better if you set up a working docker file for your Go program.

\n

So for Turtle Wear the docker image is quite simple and looks like this:

\n
FROM golang:alpine AS builder\n\nRUN apk update && apk add --no-cache git ca-certificates \n&& update-ca-certificates\nRUN adduser -D -g '' appuser\n\nWORKDIR $GOPATH/src/turtle-wear-api\n\nCOPY . .\n\nRUN go mod download && go mod verify\nRUN GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go test -cover ./...\nRUN GOOS=linux GOARCH=amd64 CGO_ENABLED=0 \ngo build -o /go/bin/turtle-wear-api ./cmd/turtle-functions/main.go\n\nFROM scratch\nCOPY --from=builder /etc/ssl/certs /etc/ssl/certs\nCOPY --from=builder /etc/passwd /etc/passwd\nCOPY --from=builder /go/bin/turtle-wear-api /go/bin/turtle-wear-api\nUSER appuser\nENTRYPOINT [\"/go/bin/turtle-wear-api\"]
\n

Credit to Nathan Dane for showing me his implementation and working with me on modifying it for the project needs.

\n

With Golang you can get your unit tests running right inside the docker image as well. The line “RUN GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go test -cover ./…” is what runs all the unit tests in the app and will fail the build if one doesn’t pass. Also the line “RUN apk update && apk add —no-cache git ca-certificates && update-ca-certificates” was needed otherwise the container couldn’t make external requests.

\n

Once you have a docker file setup and working, Login to DockerHub and click “Create repository”. Add a name and a description, select public and then connect up your GitHub account and point it to your Golang build with the docker file. From here DockerHub will take the first build and if it’s all working put the image up ready from use. In the same way we did with SonarCloud we can go back to GitHub and set the “Require status checks to pass before merging” to require DockerHub to pass before allowing merging.

\n

Kubernetes in Google Cloud Platform

\n

\"kubernetes\"

\n

“Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery.” Is the business talk way of explaining what they are. In very layman’s terms, it’s a way to deploy containers that are self-sustaining, update automatically, and can grow and shrink as demand requires.

\n

There are a few different services to get Kubernetes up and running easily enough, AWS has EKS, Azure has AKS, IBM has… IBM Kubernetes, and Google have GKE. GKE is what we will be using for this guide as it’s the right balance between giving you control and doing things for you. Plus they give you $300 free credit to use in your first year so you can learn the system safely.

\n

To start with, create a new project on the Google Cloud Platform. To do this, first go to the sidebar, click “IAM & admin” then “Manage resources”. On this page click Create Project, then in the new project window that appears, enter a project name and other applicable details, then click create.

\n

With that setup, you can now navigate to the Kubernetes Engine in the same left sidebar and click on clusters. Click on create cluster and you should be brought to a new screen with a lot of options. First pay attention to the very left-hand side, here are some templates with descriptions to make choosing the right cluster easier. For the purpose of this guide, we will be using “Your first cluster” as it’s the cheapest and what we are looking for. If you are building something that has higher requirements then feel free to select a different template.

\n

\"templates\"

\n

After selecting your template you can move over to the next section to the right. Starting from the top we have Name which is simple enough. deally, remember to name it with “cluster” at the end as you will be naming a few things when setting this up and it’s good to keep a naming convention for clarity later on. Next is Location type, choose zonal unless you really need always-on coverage assurance. Zone or Region just allows you to select where your cluster is hosted. Everything else here is fine as standard for whatever template you’ve chosen, click create when you’re ready. Once that’s done you will be back on the Clusters screen, give your new cluster a few minutes to finish setting up before continuing on.

\n

\"cluster

\n
Not ready yet
\n

\"cluster

\n
Green tick is good to go
\n

Next, we need to deploy our container, so click Deploy at the top of the clusters screen. The screen shows Container first, we can take the image path from our DockerHub build, press done then continue. It then moves on to Configuration, here you can name the Application and Namespace, again name them “app” and “namespace” at the end to keep that clarity. At the bottom, it allows you to select your newly created cluster then click Deploy.

\n

\"deploying

\n
More loading times
\n

This will bring you inside the deployment details page, the next thing to do here is on a prompt near the top of the page which mentions exposing the service. If you click that you will go to another screen where you can map the port for inside the container and the outside world. I highly recommend keeping the port to 80 but using whatever port you have set up inside your code for the target port. Click Expose here and it will start opening up the port.

\n

\"Expose\nWith that all done you now have an External endpoint IP address. Use this to run tests on your code to make sure in the container is working correctly before moving on.

\n

DNS Setup

\n

\"DNS

\n

With the IP address from GKE all setup the pipeline is complete, but there is some quality of life things we can do before we call it a full success. Firstly having a raw IP address isn’t the nicest looking way to connect up an endpoint, you can very easily use a web domain you have to be your new endpoint name. If you go to your DNS records for that domain and do the following:

\n\n

If you give it 5-10 mins for the DNS to propagate then give the same tests you were running on the IP address to the name in your DNS record you will find that everything works just the same but looks a lot nicer now.

\n

Swagger docs

\n

\"Swagger\"

\n

Allowing anyone to understand how your endpoints work and allow them to test them out is a great way to help people on board and save you a lot of time explaining how it works. Swagger Hub offers a pretty good free hosted setup so long as again it’s open source. The site has an editor and a coding mode so you can set up the page, I would recommend looking at one of their demo pages to understand the layout properly. You can check out the one I made for Turtle Wear API here

\n

Wrapping up

\n

\"The

\n

There you go, you now have a pipeline so when you deploy new code to your Golang project it will check for code issues, run the unit tests, build into a container and then deploy to a scalable solution. This also wraps up the Turtle Wear trilogy of blog posts about my journey of doing a v2 of the app, hopefully it’s of some help to you!

\n","frontmatter":{"title":"Golang automated pipeline","date":"December 09, 2019","featuredImage":{"childImageSharp":{"sizes":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAIABQDASIAAhEBAxEB/8QAFgABAQEAAAAAAAAAAAAAAAAAAAMF/8QAFQEBAQAAAAAAAAAAAAAAAAAAAAH/2gAMAwEAAhADEAAAAduZZQH/xAAWEAEBAQAAAAAAAAAAAAAAAAABABD/2gAIAQEAAQUChc//xAAUEQEAAAAAAAAAAAAAAAAAAAAQ/9oACAEDAQE/AT//xAAUEQEAAAAAAAAAAAAAAAAAAAAQ/9oACAECAQE/AT//xAAWEAADAAAAAAAAAAAAAAAAAAAAECL/2gAIAQEABj8CKX//xAAZEAACAwEAAAAAAAAAAAAAAAABEQAQIVH/2gAIAQEAAT8hmlAOIuv/2gAMAwEAAgADAAAAEHAf/8QAFREBAQAAAAAAAAAAAAAAAAAAARD/2gAIAQMBAT8QZ//EABQRAQAAAAAAAAAAAAAAAAAAABD/2gAIAQIBAT8QP//EABoQAAMAAwEAAAAAAAAAAAAAAAABETFBgaH/2gAIAQEAAT8QdjkuqKelDmROqo//2Q==","aspectRatio":2.4518388791593697,"src":"/static/goline-62aadd8af557846018fcb4e9333040a7-6b3a1.jpg","srcSet":"/static/goline-62aadd8af557846018fcb4e9333040a7-182ff.jpg 213w,\n/static/goline-62aadd8af557846018fcb4e9333040a7-7f5c7.jpg 425w,\n/static/goline-62aadd8af557846018fcb4e9333040a7-6b3a1.jpg 850w,\n/static/goline-62aadd8af557846018fcb4e9333040a7-45cc1.jpg 1275w,\n/static/goline-62aadd8af557846018fcb4e9333040a7-055dc.jpg 1400w","sizes":"(max-width: 850px) 100vw, 850px"}}}}}},"pageContext":{"slug":"/open-source-pipeline/","previous":{"fields":{"slug":"/making-wear-os/"},"excerpt":"Making a wearOS watch face and avoiding some of the faff. (TurtleWare 2.0 pt2)\n","frontmatter":{"title":"Building a WearOS Watch face","date":"November 18, 2019","featuredImage":{"childImageSharp":{"sizes":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAIABQDASIAAhEBAxEB/8QAFgABAQEAAAAAAAAAAAAAAAAAAAEE/8QAFQEBAQAAAAAAAAAAAAAAAAAAAAL/2gAMAwEAAhADEAAAAcUKig//xAAYEAADAQEAAAAAAAAAAAAAAAAAARMREv/aAAgBAQABBQKaONJo/8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAwEBPwE//8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAgEBPwE//8QAFBABAAAAAAAAAAAAAAAAAAAAEP/aAAgBAQAGPwJ//8QAGRAAAgMBAAAAAAAAAAAAAAAAAREAECFB/9oACAEBAAE/IZAOEsq//9oADAMBAAIAAwAAABAAD//EABYRAAMAAAAAAAAAAAAAAAAAAAABEf/aAAgBAwEBPxCIiP/EABURAQEAAAAAAAAAAAAAAAAAAAAR/9oACAECAQE/EFf/xAAZEAACAwEAAAAAAAAAAAAAAAAAEQEhYVH/2gAIAQEAAT8Q0H8I23TQ/9k=","aspectRatio":2.4518388791593697,"src":"/static/trtlwearbig-040e897d594d2e7aa6b51466324174e4-6b3a1.jpg","srcSet":"/static/trtlwearbig-040e897d594d2e7aa6b51466324174e4-182ff.jpg 213w,\n/static/trtlwearbig-040e897d594d2e7aa6b51466324174e4-7f5c7.jpg 425w,\n/static/trtlwearbig-040e897d594d2e7aa6b51466324174e4-6b3a1.jpg 850w,\n/static/trtlwearbig-040e897d594d2e7aa6b51466324174e4-45cc1.jpg 1275w,\n/static/trtlwearbig-040e897d594d2e7aa6b51466324174e4-055dc.jpg 1400w","sizes":"(max-width: 850px) 100vw, 850px"}}}}},"next":{"fields":{"slug":"/golden-path-document/"},"excerpt":"A golden path document is the best way for new starters to come in strong and help those that come after them\n","frontmatter":{"title":"Setting up a Golden Path Document","date":"February 4, 2020","featuredImage":{"childImageSharp":{"sizes":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAIABQDASIAAhEBAxEB/8QAFwABAAMAAAAAAAAAAAAAAAAAAAMEBf/EABYBAQEBAAAAAAAAAAAAAAAAAAIDBP/aAAwDAQACEAMQAAABr6RicAFP/8QAGBABAQADAAAAAAAAAAAAAAAAAQIAAyL/2gAIAQEAAQUCLzV1FiV//8QAFxEBAQEBAAAAAAAAAAAAAAAAAQACEf/aAAgBAwEBPwHK3W//xAAWEQEBAQAAAAAAAAAAAAAAAAABAhD/2gAIAQIBAT8BoM//xAAYEAEAAwEAAAAAAAAAAAAAAAAAAQIRIf/aAAgBAQAGPwJtocf/xAAYEAEBAQEBAAAAAAAAAAAAAAABABExQf/aAAgBAQABPyEBgeg5ZpaX/9oADAMBAAIAAwAAABAMP//EABYRAQEBAAAAAAAAAAAAAAAAAAABYf/aAAgBAwEBPxC5s//EABYRAQEBAAAAAAAAAAAAAAAAAAABEf/aAAgBAgEBPxCZMf/EABoQAQADAAMAAAAAAAAAAAAAAAEAESExQXH/2gAIAQEAAT8Qxgr7O9BvhqM7CWW7P//Z","aspectRatio":2.4518388791593697,"src":"/static/goldenpathlarge-cee690aa1396307abec7f48cd3a95f08-6b3a1.jpg","srcSet":"/static/goldenpathlarge-cee690aa1396307abec7f48cd3a95f08-182ff.jpg 213w,\n/static/goldenpathlarge-cee690aa1396307abec7f48cd3a95f08-7f5c7.jpg 425w,\n/static/goldenpathlarge-cee690aa1396307abec7f48cd3a95f08-6b3a1.jpg 850w,\n/static/goldenpathlarge-cee690aa1396307abec7f48cd3a95f08-45cc1.jpg 1275w,\n/static/goldenpathlarge-cee690aa1396307abec7f48cd3a95f08-055dc.jpg 1400w","sizes":"(max-width: 850px) 100vw, 850px"}}}}}}}