This post will show how to automatically analyse all third-party dependencies in my codebase via an automated pipeline that will be scheduled to run weekly.
The uptake of DevOps tooling is increasing faster than ever. More teams are leveraging automation to deploy their code more efficiently and with greater quality.
Every cloud provider has DevOps automation available, with lots of tutorials out there on the Internet on how to write functioning CI and CD pipelines from scratch. These examples almost always consist of a build stage, a unit test stage and a deployment stage.
However, one area that is often overlooked is security. We have started to bake more and more of the traditional QA processes into our DevOps, such as automated UI and end-to-end testing, yet security tends to have less focus. DevSecOps is the practice of integrating this security throughout the software delivery lifecycle (SDLC).
Why DevSecOps?
With software continuing to run more and more of our world, the amount of data captured by these systems continues to increase. Our dependence and faith in these systems are more critical than ever. It might surprise you (or not) to know that even with the importance of keeping these systems secure, security experts are already vastly outnumbered within their organizations at a ratio of 1 security professional for every 100 developers.
This imbalance, along with the modern trend of breaking systems up into multiple services, drives an environment where there are now more attack vectors than ever before and fewer people focused solely on guarding these targets. It’s not reasonable to expect the under-pressure security experts in these companies to be involved in the day-to-day implementation and testing of these systems, nor would it be sustainable.
The driving motivation behind DevSecOps is to allow security teams to define the security strategy and to then automate and codify the enforcement as much as possible into the SDLC.
Word documents don’t break builds - Scott Hanselman
An organisation can have the most well-defined security programme in the industry, but without automated governance, the security team become the bottleneck to enforcing it, or worse, it is not enforced at all.
Where can we start?
When starting to think about DevSecOps and how to implement it, often looking at the DevOps steps you currently have can give good clues on where to go. If you begin to think about each step and how you can secure it, ideas can quickly begin to emerge.
Third-party libraries are one of the largest attack vectors in an application and yet are often overlooked. Unless your security program includes a strict whitelist of dependencies, there’s a good chance that over time, more and more libraries have been added to your codebase, because who wants to reinvent the wheel?
The risk with all of these dependencies is that it’s unlikely that the end-users have taken the time, or have the expertise, to read the full code base and check for any security vulnerabilities. Couple that with the tendency to forget to update packages over time and that typically 30-80% of an application’s code is made up of third-party code, and you can quickly see where problems might begin.
To introduce DevSecOps to my workflow and try to protect myself and my users from these unsafe packages, I am going to automatically analyse all of the third-party dependencies in my codebase via a new automated pipeline that will be scheduled to run weekly.
While I already have workflows that trigger on Pull Requests and Deployments, adding the third-party package scanning into them would only allow me to find vulnerabilities when I write or deploy new code. With a frequently scheduled run, I can find security issues with third-party dependencies I have already deployed, as they are reported.
While I will focus on Azure DevOps pipelines in this post, the concepts can be applied to any automated build system.
The trigger
To begin, I place the following YAML at the top of my blank pipeline file, which creates a schedule to fire every Sunday at midnight. This pipeline will run against my develop branch which is my continuously deployed branch and will run regardless of whether there have been changes to the code during the week. Note that I also use trigger: none because if you do not specify a trigger in your pipeline, it is run on each push on all branches.
trigger: none
schedules:
- cron: "0 0 * * 0"
branches:
include:
- develop
always: true
This trigger should be tuned based on your security needs and risk tolerance in partnership with your security team’s strategy. You may opt to run it more frequently and on multiple branches, which is made easy in Azure with support for branch name wildcards.
Scanning Nuget Packages for Vulnerabilities
At the start of March 2021, Microsoft announced that the .NET 5 SDK would include a new dotnet CLI command to scan nuget packages for vulnerabilities. This tool gets its information directly from both the Common Vulnerabilities and Exposures (CVE) database and the centralised GitHub Advisory Database.
By running the CLI command dotnet list package --vulnerable you can list any known vulnerabilities in the dependencies of your projects and solutions. If you would like to also scan the dependencies of your dependencies, you can append the --include-transitive flag.
Unfortunately, the tool does not yet support meaningful exit codes, nor does it have the option to output the results as JSON, and so to run it as part of a CI build we have to do a little scripting. I have opted for PowerShell here, but you can use any framework you like.
($output = dotnet list package --vulnerable)
$errors = $output | Select-String '>'
if ($errors.Count -gt 0) {
foreach ($err in $errors) {
Write-Host "##vso[task.logissue type=error]Found vulnerable NuGet package $err"
}
exit 1
}
exit 0
In this script (adapted from an answer on StackOverflow), I am running the tool which by default picks up the solution file in the working directory, writing the result to the task’s output, and then parsing that output to find any occurrences of the > character, which only appears next to the name of each vulnerable package. I write the name of each of these vulnerable packages to the task’s error stream.
I call this script from my YAML pipeline using the code below. Note that to use this tool, you must first restore your referenced nuget packages. Writing to the task’s error log didn’t fail the task as expected so I also specify the exit code in the script above.
steps:
- task: DotNetCoreCLI@2
displayName: Restore Nuget Packages
inputs:
command: restore
projects: '**/*.csproj'
- task: PowerShell@2
displayName: Scan Nuget Packages for Vulnerabilities
continueOnError: true
inputs:
filePath: 'build/scan-nuget-vulnerabilities.ps1'
failOnStderr: true
Scanning Yarn Packages for Vulnerabilities
As well as scanning my .NET API server for issues, I want to scan the packages used by my web UI. I am using Yarn for my package manager, but NPM provides comparable audit functionality.
In a similar fashion to the nuget script, I want to parse out each vulnerable package so I can report it directly to the build and see the package names on the dashboard. Here I use the yarn npm audit tool.
yarn npm audit --severity $env:YARNMINSEVERITYLEVEL
$jsonObject = yarn npm audit --json --severity $env:YARNMINSEVERITYLEVEL | ConvertFrom-Json
$vulnerablePackageFound = $false
$jsonObject.advisories.PSObject.Properties | ForEach-Object {
Write-Host "##vso[task.logissue type=error]Found yarn package '$($_.Value.module_name)' with $($_.Value.severity) vulnerability severity. $($_.Value.recommendation)"
$vulnerablePackageFound = $true
}
if ($vulnerablePackageFound) {
exit 1
}
else {
exit 0
}
I run the command twice because I want to output the formatted table to the console and also use the --json flag to get the results into a variable I can parse. I then loop over each advisory (sadly the advisories property is not an array), logging it to the tasks error console.
This script uses an environment variable provided by the Azure pipeline to determine the minimum severity level that I want my build step to fail for. The possible options are info, low, moderate, high, critical.
variables:
YarnMinSeverityLevel: moderate
I can then execute the script with a new task, in the same way as the above Nuget script.
The Result
This is my full YAML file (I opt to run on Linux):
schedules:
- cron: "0 0 * * 0"
branches:
include:
- develop
always: true
pool:
vmImage: 'ubuntu-latest'
variables:
- name: YarnMinVulnerabilityLevel
value: 8
steps:
- task: DotNetCoreCLI@2
displayName: Restore Nuget Packages
inputs:
command: restore
projects: '**/*.csproj'
- task: PowerShell@2
displayName: Scan Nuget Packages for Vulnerabilities
continueOnError: true
inputs:
filePath: 'build/scan-nuget-vulnerabilities.ps1'
failOnStderr: true
- task: PowerShell@2
displayName: Scan Yarn Packages for Vulnerabilities
inputs:
filePath: 'build/scan-yarn-vulnerabilities.ps1'
failOnStderr: true
workingDirectory: 'src/UI'
Below is an example of the errors shown on the build dashboard when a vulnerable package is detected that meets my minimum severity limit.
Conclusion
In this post, I introduced the concept of DevSecOps and why it is such an essential part of the SDLC (hopefully it will soon be just another natural concern for DevOps and we can cut the extra term). I then explained just how risky unaudited third-party libraries can be, particularly if they are not kept up-to-date, and demonstrated how to begin to automate the process of minimising this threat vector in your applications. I am looking forward to bolstering this pipeline with more gates soon.
