An Introduction to Continuous Integration and Workflows

A well-defined SDLC practice in a typical organization generally has projects running with users and roles. These users design, develop, test and deploy the jobs as per the business need/requirement. But have you ever wondered - what happens to the code after that? What if multiple developers want to work on the same job? How would you store this code, and how does one ensure that other developers always pick the right version? Well, welcome to the world of "Continuous Integration." In this blog, I will be highlighting the process of Continuous Integration (CI), the importance of being continuous and how to achieve that using Talend CI Builder with Jenkins and Maven plugins.

Let's first understand a few basic terminologies:

Continuous Integration: CI is a development practice where members of a team integrate their work frequently, with each integration being verified by an automated build to detect errors as quickly as possible.

Continuous Testing: CT means that every time an integration is done, predefined test cases are run to ensure that the new code doesn't break the existing system.

Continuous Deployment: CD means that you deliver software in continuous incremental fashion and deploy frequently. Here, deploying could be to a test environment or pre-prod environments.

So, for an organization to be being continuous it should be CI, CT, and CD driven and it must be inclusive in the Software Development Lifecycle (SDLC). The diagram given below shows the phases from the SDLC life cycle and the areas of CI, CT and CD.

So, what are the benefits of being continuous? If implemented correctly and practiced regularly, being continuous helps in reducing integration problems there by allowing you to deliver jobs/code/software more rapidly.

Continous Integration Development Cycle.

Also, by integrating regularly, you can detect errors quickly, and locate them more easily. With the usage of the right tools, one can have fewer conflicts and easy conflict resolution while integrating the code. The most important point to note is that you end up with less of a chance of breaking what already exists, and even if it breaks, it's easier to solve/recover.

Continuous Workflows Overview

For continuous integration, there is a need to have a repository where in the code could be saved, retrieved and maintained. The repository must be good enough to provide the developers with a powerful version controlling system. Though there are a number of CI tools available, I suggest trying out Git. Git is one of the version control systems (VCS) for tracking code changes and coordinating work on the code among multiple people. It is primarily used for source code management in software development, but it can be used to keep track of changes in any set of files. It provides few common workflow models:

Centralized Workflow

This flow uses a central repository to serve as the single point-of-entry for all changes to the project. The default development branch is called master and all changes are committed into this branch. This workflow doesn't require any other branches besides master. A typical centralized workflow life cycle would be as follows:

• Developers start by cloning the central repositories in their own local copies of the project. They edit jobs and commit changes locally. Once the changes are tested, the developer "Push" their local master branch to the central repository.
• Managing Conflicts: the central repository represents the official project, so if the local job changes conflicts with upstream commits, Git will pause the process and give a chance to manually resolve the conflicts. This makes it easy for developers to manage the merges.

You might have noticed that centralized workflow is more like SVN with few Git features. This would be great for transitioning teams off SVN, however, it doesn't use the distributed nature of Git.

Feature Branch Workflow

The core idea behind the Feature Branch workflow is that all feature development should take place in a dedicated branch instead of the master Git makes no technical distinction between the master branch and feature branches. So, developers can edit, stage and commit like they did in the Centralized workflow. Here, a typical workflow would look like:

• Developers create a new branch every time they start work on a new feature.
• Feature branches should have descriptive names like issue-#1061, Jira-190. The idea is to give a clear, highly focused purpose to each branch.

Gitflow Workflow

This defines a strict branching model designed around the project release. This provides a robust framework for managing larger projects. This is similar to the Feature Branch workflow except it assigns very specific roles to different branches and defines how and when they should interact. It also uses individual branches for preparing, maintaining, and recording releases. Like the previous workflow, developers work locally and push branches to the central repo. The only difference is the branch structure of the project. You define Historical Branches, Feature Branches, Release Branches, and Maintenance Branches.

Forking Workflow

This is fundamentally different than the other workflows. Here, instead of using a single server-side central repository, it gives every developer a server-side repository. This means that each contributor has not one, but two Git repositories: a private local one and a public server-side one. Note that the Forking workflow is not supported by Talend. A typical workflow would look like:

• The Forking workflow beings with an official public repository stored on a server. But when a new developer wants to start working on the project, they do not directly clone the official repository; instead, they fork the official repository to create a copy of it on the server.
• This new copy serves as their personal public repository - no other developers are allowed to push to it, but they can pull changes from it.
• When they are ready to publish a local commit, they push the commit to their public repository - not the official one. Then they file a pull request with the main repository, which lets the project maintainer know that an update is ready to be integrated.
• The main advantage of this workflow is that contributions can be integrated without the need for everybody to push to the official repository.
• Developers push to their own server-side repositories and only the project maintainer can push to the official repository.

In the next part of this blog series, we will look at the daily routine of a developer in SDLC phases with centralized workflow model, and how Talend helps the organization in being continuous.