Python CI with Github Actions

When developing projects in python, it’s important to have a reliable CI workflow that can perform the following functions.

• Linting and Formatting
• Unit Tests with Code Coverage
• Security Scanning for potential vulnerabilities
• Building the source code
• Deploying the build to a non-production environment for integration tests

Before we dive into these steps, let’s briefly discuss our branching / release strategy. The most common strategies are GitFlow, Github flow, Trunk based, Gitlab flow, and variations of each. For the majority of my projects I prefer a trunk-based strategy with small, frequent updates to my main production branch.

A typical change in the production source code starts with pulling down main and creating a new branch. Let’s call the new branch feature-request-1. I’ll make my changes to the source code in this branch and when I am ready for the code to be merged into main, I will submit a pull request.

The pull-request to main is an event in Github actions that we will use to trigger a series of steps in our python workflow.

Step 1. Lint and Format our code using pylint and black. This will make our code easy to read in a standard format and also check for syntax errors. 

Step 2. Run our unit tests. As a rule of thumb, I strive for 80% of my code to have test coverage. It's not always feasible with large codebases, however greater coverage ensures the code is functioning as intended. 

Step 3. Security scanning using Github or other 3rd party tools to find any vulnerabilities in the code before we build.  It's a good idea to fail the CI process here if the number of vulnerabilities exceed a defined threshold. 

Step 4. Execute the python build process to build our source package.  I prefer using the Github Commit SHA as the version number.  This is achieved by updating the pyproject.toml file with dynamic versioning using setuptools-scm

Step 5. Deploy our source code package to our Development environment for further testing. Once deployed, pull request reviewers (other team members) can run integration tests to validate the integrity of the code before approving and merging into the main branch. 

We can chain these jobs sequentially or have them execute independently. It’s also a good practice to write the results of these jobs as a comment on the pull request so that other code reviewers or approvers can quickly see that these jobs have succeeded before reviewing the code itself. The pull request can be rejected if our jobs fail, saving time for those required to review.

Artifacts

Python packages can be stored as GitHub artifacts and used in subsequent jobs.

Python build packages are initially created with the Commit SHA as the version to quickly identify each package with a specific commit. Release versions will be built using tags created with semantic versioning.

Github Packages does not support PyPi repositories, therefore I use AWS CodeArtifact since most of my projects are deployed in AWS.

Release Process

Once the pull request is approved and the commit is merged into main, at this point it’s ready for a Release.

The source of the release is main and a new tag with semantic version format (v.0.0.1) is created.

Release notes are added and the repository is tagged, then bundled as a tarball (in this case). The python package is rebuilt using the v0.0.1 tag as the version and then deployed to the Development environment followed by Production.

For more information, including some sample workflows please see the link below.

Sample Reusable Workflows:

https://github.com/platformdevorg/shared-workflows