Improving Code Quality with Sonar, GitHub and Azure DevOps CI

In my previous post I walked through the set up of a dotnet core project in Sonar Cloud, showing how, with a few settings in your csproj file and a couple of dotnet tools, you can create a code coverage report and upload your project and report to a Sonar Cloud project for full analysis.

Useful as this is, discipline in code quality is rarely maintained unless you build these quality ‘gates’ into a CI pipeline, and this is the next step for me in my GitHub project.

First of all, here’s my existing CI pipeline:

# the build will trigger on any changes to the master branch
trigger:
  branches:
    include:
    - master
  paths:
    exclude:
    - README.md;azure-pipelines.yml;LICENSE;.gitignore

# stop 2 build trigger from the submission of the Pull Request so we only get one from the CI merge
pr: none

# the build will run on a Microsoft hosted agent, using the lastest Windows VM Image
pool:
  vmImage: 'windows-latest'

variables:
  configuration: 'Release'

steps:

- task: DotNetCoreCLI@2
  displayName: dotnet build
  inputs:
    command: build
    configuration: $(configuration)
    
- task: DotNetCoreCLI@2
  displayName: dotnet test
  inputs:
    command: test
    projects: '**/*[Tt]ests.csproj'
    configuration: $(configuration)

- task: DotNetCoreCLI@2
  displayName: dotnet pack
  inputs:
    command: pack
    configuration: $(configuration)
    packDirectory: '$(Build.ArtifactStagingDirectory)/packages'

- task: PublishBuildArtifacts@1
  displayName: publish artifacts
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'

This pipeline is currently designed to trigger on a merge into master (note PRs will not currently trigger a build) and enforce a successful test run on the project. Therefore my only quality gates are a successful build and successful test run. No quality check on unit test coverage or 3rd party code review.

What I want to end up with is a Build policy that requires the following to pass before allowing a Pull Request to be completed. Because a CI pipeline can completely successfully even if a Quality gate or other policy requirement fails, I’m going to leave the above CI pipeline as-is, i.e. it will only run on a successful merge to master (which can only happen after a successful PR has completed) and create a new CI pipeline that does not publish any artifacts so we have no risk of any artifacts from a failed PR getting deployed.

My first target is to get dotnet test to complete code coverage and create a report than can be viewed within the DevOps process. Looking at my most recent CI run I can see the following analysis:

You can see that we get a brief summary and a Tests tab which shows the following:

To create a test report I am going to back to Daniel Palme‘s excellent Report Generator tool and use the cobertura report format with Azure Pipeline formatting (so it fits in nicely with my Azure DevOps project). I’m not going to add in the Sonar analysis until my next step but will prep for Sonar by including the opencover report format. I also need to downgrade the dotnet sdk used to build the solution to 2.1 as Sonar does not suport analysis for projects built using the 3.x sdk. Note this does not impact on the version of your app as the dotnet core 2.1 sdk can build dotnet core 3.x projects.

Here’s my new CI pipeline:

# the build will ne triggered by a branch policy on the master branch (i.e. through a Pull Request)
trigger: none

# the build will run on a Microsoft hosted agent, using the lastest Windows VM Image
pool:
  vmImage: 'windows-latest'

variables:
  configuration: 'Release'
  dotnetSdkVersion: '3.1.102'

steps:
- task: UseDotNet@2
  displayName: 'Use dotnet Core SDk $(dotnetSdkVersion)'
  inputs:
    version: '$(dotnetSdkVersion)'

# sonar requirement - cannot analyse projects built using the dotnet core 3.x sdk so need to include the 2.1 sdk
- task: UseDotNet@2
  displayName: 'Dropping .net core sdk to 2.1.505 for SonarCloud'
  inputs:
    version: '2.1.505'

- task: DotNetCoreCLI@2
  displayName: 'Restore sln dependencies'
  inputs:
    command: 'restore'
    projects: 'AgileTea.Persistence.sln'

- task: DotNetCoreCLI@2
  displayName: 'Build the solution'
  inputs:
    command: 'build'
    arguments: '--no-restore'
    configuration: $(configuration)
    projects: 'AgileTea.Persistence.sln'   

- task: DotNetCoreCLI@2
  displayName: 'Install Report Generator tool'
  inputs:
    command: custom
    custom: tool
    arguments: 'install --global dotnet-reportgenerator-globaltool'
    
- task: DotNetCoreCLI@2
  displayName: 'Run unit tests - $(configuration)'
  inputs:
    command: test
    arguments: '--no-build /p:CollectCoverage=true /p:CoverletOutputFormat="cobertura%2copencover" /p:CoverletOutput=$(Build.SourcesDirectory)/TestResults/Coverage/'
    publishTestResults: true
    projects: 'AgileTea.Persistence.sln'
    configuration: $(configuration)

- script: |
    reportgenerator -reports:$(Build.SourcesDirectory)/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/TestResults/Reports/ -reporttypes:HtmlInline_AzurePipelines
  displayName: 'Create code coverage report'

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage report'
  inputs:
    codeCoverageTool: 'cobertura'
    summaryFileLocation: '$(Build.SourcesDirectory/**/coverage.cobertura.xml)'

To get this started we have to now add it into our DevOps project. Remember this is a new pipeline, we’re not editing the existing one so I need to create a new pipeline from this file within DevOps:

First connect to my code – which is on GitHub

Selecting my repository I then need to configure the pipeline using my new ci pipeline yml file:

As I have not completed a PR for this branch change yet (although I have committed and pushed), I need to select the new branch from the dropdown in order to see my new Ci pipeline in the dropdown:

This will load your yaml into the editor and it’s a good place to review your code in case anything incorrect pops out at you. If not, you can always give it a test run and that will verify the file. I did this and it immediately found a syntax error for me (I always get caught out by incorrect indents)!

Multiple Test Projects

It’s no uncommon for devs to split out their test projects into the various sections of your solution. You might not want a 1:1 mapping but with multi-project solutions it can be useful to group tests into separate projects. There is downside to this approach when using dotnet test and coverlet to create your coverage report in your CI pipeline. When you run dotnet test on your solution with the coverlet options added, you will create a code coverage report per unit test project. Your local dev report generator tool will happily suck up these multiple files automatically and smash them together for your report (picks up a MultiReport Parser):

This MultiReportParser is not available in the Azure DevOps pipeline through the Report Generator Global tool to be used in the pipeline. Instead, we have three options:

1. Smash all the unit test projects into one thereby simplifying everything and generating a single, complete, code coverage file.
2. Use the /p:MergeWith=”…” coverlet option and run a dotnet test per project to get coverlet to produce a single , complete, code coverage file.
3. Use a script, i.e. Powershell, to do this for us.

I’m not a big fan of adding a load of Power Shell script calls into a CI pipeline and it hides a lot of functionality. I think if you either bite the bullet and add your tests into a single unit test project or use the coverlet mergeWith option. I’ve only got 2 unit test projects so it’s a good excuse to try out the mergeWith option.

# the build will be triggered by a branch policy on the master branch (i.e. through a Pull Request)
trigger: none

# the build will run on a Microsoft hosted agent, using the lastest Windows VM Image
pool:
  vmImage: 'windows-latest'

variables:
  configuration: 'Release'
  dotnetSdkVersion: '3.1.102'

steps:
- task: UseDotNet@2
  displayName: 'Use dotnet Core SDk $(dotnetSdkVersion)'
  inputs:
    version: '$(dotnetSdkVersion)'

- task: DotNetCoreCLI@2
  displayName: 'Restore sln dependencies'
  inputs:
    command: 'restore'
    projects: 'AgileTea.Persistence.sln'

- task: DotNetCoreCLI@2
  displayName: 'Build the solution'
  inputs:
    command: 'build'
    arguments: '--no-restore'
    configuration: $(configuration)
    projects: 'AgileTea.Persistence.sln'   

- task: DotNetCoreCLI@2
  displayName: 'Install Report Generator tool'
  inputs:
    command: custom
    custom: tool
    arguments: 'install --global dotnet-reportgenerator-globaltool'
    
- task: DotNetCoreCLI@2
  displayName: 'Run unit tests on Tests - $(configuration)'
  inputs:
    command: test
    arguments: '--no-build /p:CollectCoverage=true /p:CoverletOutput=../TestResults/Coverage/'
    projects: '**/*[Tt]ests.csproj'
    configuration: $(configuration)

- task: DotNetCoreCLI@2
  displayName: 'Run unit tests on Common Tests - $(configuration)'
  inputs:
    command: test
    arguments: '--no-build /p:CollectCoverage=true /p:CoverletOutput=../TestResults/Coverage/'
    publishTestResults: true
    projects: 'src/AgileTea.Persistence.Common.Tests/AgileTea.Persistence.Common.Tests.csproj'
    configuration: $(configuration)

- task: DotNetCoreCLI@2
  displayName: 'Run unit tests on Mongo Tests - $(configuration)'
  inputs:
    command: test
    arguments: '--no-build /p:CollectCoverage=true /p:CoverletOutput=../TestResults/Coverage/ /p:MergeWith="../TestResults/Coverage/coverage.json" /p:CoverletOutputFormat="cobertura"'
    publishTestResults: true
    projects: 'src/AgileTea.Persistence.Mongo.Tests/AgileTea.Persistence.Mongo.Tests.csproj'
    configuration: $(configuration)

- script: |
    reportgenerator -reports:"$(Build.SourcesDirectory)/**/coverage.cobertura.xml" -targetdir:$(Build.SourcesDirectory)/CodeCoverage -reporttypes:HtmlInline_AzurePipelines
  displayName: 'Create code coverage report'

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage report'
  inputs:
    codeCoverageTool: 'cobertura'
    summaryFileLocation: '$(Build.SourcesDirectory)/**/coverage.cobertura.xml'

Note the 2 dotnet test tasks.

The first one targets the Common project and sets no coverlet output format which effectively allows the result to create a coverage.json file (this is the default coverlet report format).

The second dotnet test targets the Mongo project and sets the report format to cobertura. It also uses a MergeWith option to pull in the results from the previous step and adds the publishTestResults option. The result is that the combined coverage report is created and published to your pipeline analysis:

Sonar Analysis

To add in the steps to perform sonar analysis on your build, you need the following:

1. CI steps to setup, run and publish the Sonar Analysis using your Sonar organisation and project keys

2. A Service Connection to Sonar setup in Azure DevOps

3. Pull request analysis enabled in Sonar

Let’s take the first of these:

CI Setup

On the my Sonar project page, near the bottom left you can view (and copy) the organisation and project keys – grab these:

Sonar currently cannot utilise the dotnet core 3.x sdk to perform analysis so you need to ensure you have the 2.1 version installed within your pipeline.

Underneath the UseDotNet@2 task add another one but hard-coding the version to 2.1.505:

# sonar requirement - cannot analyse projects built using the dotnet core 3.x sdk so need to downgrade the sdk
- task: UseDotNet@2
  displayName: 'Dropping .net core sdk to 2.1.505 for SonarCloud'
  inputs:
    version: '2.1.505'

Add these into the following step within your CI pipeline before the dotnet BUILD step

- task: SonarCloudPrepare@1
  inputs:
    SonarCloud: 'AgileTea Document Persistence'
    organization: 'agiletea'
    projectKey: 'agiletea_AgileTea.DocumentDb.Persistence'
    projectName: 'AgileTea.DocumentDb.Persistence'
    extraProperties: |
      sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/**/coverage.opencover.xml 

Modify the second dotnet test task to include the opencover format as follows (note the /p:CoverletOutputFormat=”cobertura%2copencover” amended argument):

- task: DotNetCoreCLI@2
  displayName: 'Run unit tests on Mongo Tests - $(configuration)'
  inputs:
    command: test
    arguments: '--no-build /p:CollectCoverage=true /p:CoverletOutput=../TestResults/Coverage/ /p:MergeWith="../TestResults/Coverage/coverage.json" /p:CoverletOutputFormat="cobertura%2copencover"'
    publishTestResults: true
    projects: 'src/AgileTea.Persistence.Mongo.Tests/AgileTea.Persistence.Mongo.Tests.csproj'
    configuration: $(configuration)

Finally, before the PublishCodeCoverageResults task, immediately after the report generator script, add the following tasks:

- task: SonarCloudAnalyze@1
  displayName: 'Run SonarCloud code analysis'

- task: SonarCloudPublish@1
  displayName: 'Publish SonarCloud analysis results'

Before you run this amended Pipeline, you’ll need to set up a new Service Connection to Sonar within your Azure DevOps porject settings. First, grab a PAT from Sonar as mentioned in my previous post, then within Azure DevOps, bottom left you should see a Project Settings button. Click on this then select Service Connections from the next page:

Click on New service connection (top right) and scroll down to the Sonar Cloud option. Paste in your PAT and give it the same name from the SonarCloud argument above (for me this was ‘AgileTea Document Persistence’).

If you run the Pipeline now again (make sure to target your current branch and not the master branch), you should get to see how these new tasks execute. If successful, the Run Sonar Cloud Analysis task log will give you a link towards the bottom that can allow you to view the result of that scan:

Pull Request Access

The sonar analysis has automatic access to your master branch but you’ll want to analyse new branches as they work their way through the Pull Request process. Depending on where your code is stored (and therefore where your Pull Request come from) you’ll need to configure Sonar accordingly.

For GitHub repos, where my code is stored, you simply need to open up Administration settings, go to Pull Requests and fill in the necessary details for your GitHub project:

Back in my Azure DevOps I need to make 2 changes. Firstly, the default names for my pipelines now seem too similar and unhelpful so I’ll rename each to be more obvious:

I also want my CI pipeline to now trigger on a PR request so I’ll add a PR trigger against the master branch underneath the ‘none’ trigger:

# need to block CI pipeline running on simple branch pushes
trigger: none

pr:
 - master

I then need to allow my branch to be merged with the master before this can take effect (bit of a chicken and egg issue). Once I have completed this merge I can then create a new branch to test it out.

I also want GitHub to include my CI build as part of the Pull request check. This is done in Settings > Branches > Rules

I’ve now merged by CI pipeline changes into master so let’s see what happens when I create a new branch, make some changes and submit a new Pull request.

Once the Pull request has been created, you should now notice a new check has been applied that blocks the completion of the Pull Check

Over in my Azure DevOps CI pipelines I can see a new build has been kicked off automatically:

The GitHub check is smart enough to recognise that it not only requires the CI pipeline to complete successfully but also requires the Sonar Quality Gate to be passed successfully. After a few minutes, the Pull Request updates it status with the checks completed and a handy SonarCloud review:

I’ve set up my branch protection on GitHub to require 2 reviewers on Pull Requests hence the other blocker but with admin rights I can override this.

Once the Pull request has been completed, the merge to master is done and, at that point, my original Master pipeline is kicked off which processed as expected but with the addition of packing my code as a Nuget package and publishing the artifacts ready for release.

For completion’s sake, here’s my final CI yaml:

# ASP.NET Core
# Build and test ASP.NET Core projects targeting .NET Core.
# Add steps that run tests, create a NuGet package, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/dotnet-core

# need to block CI pipeline running on simple branch pushes
trigger: none

pr:
 - master

# the build will run on a Microsoft hosted agent, using the lastest Windows VM Image
pool:
  vmImage: 'windows-latest'

variables:
  configuration: 'Release'
  dotnetSdkVersion: '3.1.102'

steps:
- task: UseDotNet@2
  displayName: 'Use dotnet Core SDk $(dotnetSdkVersion)'
  inputs:
    version: '$(dotnetSdkVersion)'

# sonar requirement - cannot analyse projects built using the dotnet core 3.x sdk so need to include the 2.1 sdk
- task: UseDotNet@2
  displayName: 'Dropping .net core sdk to 2.1.505 for SonarCloud'
  inputs:
    version: '2.1.505'

- task: DotNetCoreCLI@2
  displayName: 'Restore sln dependencies'
  inputs:
    command: 'restore'
    projects: 'AgileTea.Persistence.sln'

- task: SonarCloudPrepare@1
  inputs:
    SonarCloud: 'AgileTea Document Persistence'
    organization: 'agiletea'
    projectKey: 'agiletea_AgileTea.DocumentDb.Persistence'
    projectName: 'AgileTea.DocumentDb.Persistence'
    extraProperties: |
      sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/**/coverage.opencover.xml

- task: DotNetCoreCLI@2
  displayName: 'Build the solution'
  inputs:
    command: 'build'
    arguments: '--no-restore'
    configuration: $(configuration)
    projects: 'AgileTea.Persistence.sln'   

- task: DotNetCoreCLI@2
  displayName: 'Install Report Generator tool'
  inputs:
    command: custom
    custom: tool
    arguments: 'install --global dotnet-reportgenerator-globaltool'

# need to split out the test projects to enable merging of the coverage results - the first should use the default coverlet json output format
- task: DotNetCoreCLI@2
  displayName: 'Run unit tests on Common Tests - $(configuration)'
  inputs:
    command: test
    arguments: '--no-build /p:CollectCoverage=true /p:CoverletOutput=../TestResults/Coverage/'
    publishTestResults: true
    projects: 'src/AgileTea.Persistence.Common.Tests/AgileTea.Persistence.Common.Tests.csproj'
    configuration: $(configuration)

# this test task needs to switch the format to cobertura AND perform a MERGE with the coverage.json created in the last step
- task: DotNetCoreCLI@2
  displayName: 'Run unit tests on Mongo Tests - $(configuration)'
  inputs:
    command: test
    arguments: '--no-build /p:CollectCoverage=true /p:CoverletOutput=../TestResults/Coverage/ /p:MergeWith="../TestResults/Coverage/coverage.json" /p:CoverletOutputFormat="cobertura%2copencover"'
    publishTestResults: true
    projects: 'src/AgileTea.Persistence.Mongo.Tests/AgileTea.Persistence.Mongo.Tests.csproj'
    configuration: $(configuration)

- script: |
    reportgenerator -reports:"$(Build.SourcesDirectory)/**/coverage.cobertura.xml" -targetdir:$(Build.SourcesDirectory)/CodeCoverage -reporttypes:HtmlInline_AzurePipelines
  displayName: 'Create code coverage report'

- task: SonarCloudAnalyze@1
  displayName: 'Run SonarCloud code analysis'

- task: SonarCloudPublish@1
  displayName: 'Publish SonarCloud analysis results'

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage report'
  inputs:
    codeCoverageTool: 'cobertura'
    summaryFileLocation: '$(Build.SourcesDirectory)/**/coverage.cobertura.xml'

In conclusion, I cannot think of a good reason not to employ the use of CI, Pull Requests, Code Reviews and Test Coverage for greenfield projects at the very least. The setup is cheap (Sonar charges 10 euros a month for analysing a private repository of up to 100K lines of code), relatively quick and pays dividends in terms of reducing the build up of technical debt. For me, it’s a no brainer.