Overview

Azure DevOps is a comprehensive set of services and tools designed to support the software development process and foster a culture of collaboration. By bringing developers, project managers, and contributors together, organizations can build products faster than traditional software development approaches. Azure DevOps is available both in the cloud as Azure DevOps Services and on-premises as Azure DevOps Server.

Azure DevOps Overview
Azure DevOps Overview

Azure DevOps YAML

Azure DevOps YAML is a configuration language used to describe continuous integration (CI) and continuous delivery (CD) operations in Azure Pipelines. YAML stands for "YAML Ain't Markup Language" and is used for data serialization. Azure DevOps uses YAML files to create pipelines that define how code is built, tested, and deployed.

YAML files specify the phases, jobs, steps, and resources used in a pipeline. This allows any aspect of your pipeline to be expressed as code and stored in a version control system. This enables you to track and revert changes to your pipeline configuration, just as you do with your code.

YAML Syntax Skeleton Structure

Azure DevOps YAML syntax defines the structure of configuration files used to automate CI/CD operations with Azure Pipelines. The skeleton structure determines the different parts of the pipeline and the order in which they work. Here are the main components and their functions:

  • Stages: Used to divide the pipeline into logical sections. For example, you can create a build stage, a test stage, and a deployment stage. Each stage can contain one or more jobs.
  • Jobs: Identifies the job within the stage. Each job consists of steps that perform a specific task or sequence of tasks. Jobs can be executed in parallel or sequentially.
  • Steps: These are the smallest units of tasks that a pipeline performs. They can include actions such as commands, scripts, and tasks.
  • Pool: Specify the agent or agent pool on which to run the job. For example, you can define a virtual machine image such as "vmImage: ubuntu-latest".
  • Triggers: Determines when the pipeline is triggered. These triggers can be code changes, scheduled triggers, or external events.
  • Resources: Define the external resources used by the pipeline. These can be other repositories, packages.
  • Variables: Define the variables used in the pipeline. These variables can be reused at different stages of the pipeline.
  • Parameters: Define the parameters that are imported when the pipeline is run. This makes the pipeline more flexible.

A basic skeleton structure of an Azure DevOps YAML file might look like this:

trigger:
- main

stages:
- stage: Build
  jobs:
  - job: BuildJob
    pool:
      vmImage: 'ubuntu-latest'
    steps:
    - script: echo Building the project...
      displayName: 'Build'

- stage: Test
  jobs:
  - job: TestJob
    pool:
      vmImage: 'ubuntu-latest'
    steps:
    - script: echo Running tests...
      displayName: 'Test'

- stage: Deploy
  jobs:
  - job: DeployJob
    pool:
      vmImage: 'ubuntu-latest'
    steps:
    - script: echo Deploying to production...
      displayName: 'Deploy'

In this example, the trigger keyword defines which branch changes the pipeline will run on. stages defines the different stages and each stage contains jobs. The agent environment where each job will run is specified with pool and the tasks it will perform are specified with steps.

Simple Syntax Examples

Trigger and Pool Example

In this example, any change in the main branch triggers the pipeline and runs in the ubuntu-latest virtual machine image.

trigger:
- main

pool:
  vmImage: 'ubuntu-latest'

Step Example

Here, a script is executed and then a task is used in the next step.

steps:
- script: echo Hello, Azure DevOps!
  displayName: 'Greet Azure DevOps'

- task: CopyFiles@2
  inputs:
    SourceFolder: 'source'
    TargetFolder: 'target'

Job and Steps Example

In this example, a job is defined and contains two steps: running a script and copying a file.

jobs:
- job: ExampleJob
  pool:
    vmImage: 'ubuntu-latest'
  steps:
  - script: echo Compiling the code...
    displayName: 'Compile Code'

  - task: CopyFiles@2
    inputs:
      SourceFolder: 'bin'
      TargetFolder: 'artifact'

Stages and Jobs Example

This example defines a pipeline with two stages: Build and Deploy. Each phase contains its own jobs.

stages:
- stage: Build
  jobs:
  - job: BuildJob
    pool:
      vmImage: 'ubuntu-latest'
    steps:
    - script: echo Building the project...
      displayName: 'Build Project'

- stage: Deploy
  jobs:
  - job: DeployJob
    pool:
      vmImage: 'ubuntu-latest'
    steps:
    - script: echo Deploying to production...
      displayName: 'Deploy Project'

Advanced Syntax Examples

This example defines a pipeline with multiple phases, jobs, and conditional steps. Also, parallel jobs are run for different configurations using the matrix strategy.

trigger:
- main
- feature/*

variables:
  buildConfiguration: 'Release'
  buildPlatform: 'Any CPU'

stages:
- stage: Build
  displayName: 'Build stage'
  jobs:
  - job: BuildJob
    displayName: 'Build'
    pool:
      vmImage: 'windows-latest'
    strategy:
      matrix:
        Debug:
          buildConfiguration: 'Debug'
        Release:
          buildConfiguration: 'Release'
    steps:
    - script: dotnet build --configuration $(buildConfiguration)
      displayName: 'Dotnet Build $(buildConfiguration)'

- stage: Test
  displayName: 'Test stage'
  jobs:
  - job: TestJob
    displayName: 'Run Tests'
    pool:
      vmImage: 'windows-latest'
    steps:
    - script: dotnet test --configuration $(buildConfiguration) --collect "Code coverage"
      displayName: 'Dotnet Test'

- stage: Deploy
  displayName: 'Deploy stage'
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
  jobs:
  - deployment: DeployJob
    displayName: 'Deploy to Production'
    environment: 'production'
    strategy:
      runOnce:
        deploy:
          steps:
          - script: echo Deploying to production...
            displayName: 'Deploy Script'
          - task: AzureRmWebAppDeployment@4
            inputs:
              azureSubscription: 'AzureSubscription'
              appType: 'webApp'
              appName: 'MyProductionApp'
              package: '$(Pipeline.Workspace)/drop/app.zip'

In this example, the trigger keyword is used to trigger the pipeline on changes in the main and feature/* branches. variables is used to define global variables. stages keyword is used to define three stages (Build, Test, Deploy) and each stage contains jobs. In the Build phase, strategy defines parallel jobs for Debug and Release configurations.

Multiple Self-Hosted Agents Usage

This example shows how to implement the use of multiple agents in a yml file.

stages:
- stage: Build
  displayName: 'Build Application'
  jobs:
  - job: BuildJob
    pool:
      name: WindowsAgents
    steps:
    - script: build.cmd
      displayName: 'Build'

- stage: Test
  displayName: 'Run Tests'
  jobs:
  - job: TestJob
    pool:
      name: LinuxAgents
    steps:
    - script: test.sh
      displayName: 'Test'

- stage: Deploy
  displayName: 'Deploy to Staging'
  jobs:
  - job: DeployJob
    pool:
      name: MacAgents
    steps:
    - script: deploy.sh
      displayName: 'Deploy'

How Flexible is Azure DevOps?

Advanced examples that show that Azure DevOps YAML configurations are dynamic and flexible are as follows.

  1. Dynamic Variables and Parameters: In this example, dynamic configurations are provided by using pipeline parameters and conditional expressions.
parameters:
- name: deployEnvironment
  displayName: 'Deploy Environment'
  type: string
  default: 'staging'
  values:
  - staging
  - production

stages:
- stage: Build
  jobs:
  - job: BuildJob
    steps:
    - script: echo Building the project...
      displayName: 'Build'

- ${{ if eq(parameters.deployEnvironment, 'staging') }}:
  - stage: DeployStaging
    jobs:
    - deployment: StagingJob
      environment: staging
      steps:
      - script: echo Deploying to staging...
        displayName: 'Deploy to Staging'

- ${{ if eq(parameters.deployEnvironment, 'production') }}:
  - stage: DeployProduction
    jobs:
    - deployment: ProductionJob
      environment: production
      steps:
      - script: echo Deploying to production...
        displayName: 'Deploy to Production'
  1. Template Usage: This configuration uses templates to reduce repetitive configurations.
jobs:
- template: build-template.yml  # Template reference
  parameters:
    solution: '**/*.sln'
    buildPlatform: 'Any CPU'
    buildConfiguration: 'Release'

- template: test-template.yml  # Template reference
  parameters:
    testFramework: 'NUnit'
    testFiles: '**/*Tests.dll'
  1. Conditional Triggers and Strategies: In this example, conditional triggers and deployment strategies are defined for different branches and tags.
trigger:
  branches:
    include:
    - main
    - feature/*
  tags:
    include:
    - v*

pr:
  branches:
    include:
    - main

stages:
- stage: Build
  jobs:
  - job: BuildJob
    steps:
    - script: echo Building the project...
      displayName: 'Build'

- stage: Deploy
  condition: startsWith(variables['Build.SourceBranch'], 'refs/tags/v')
  jobs:
  - job: DeployJob
    steps:
    - script: echo Deploying the project...
      displayName: 'Deploy'

Final (End-To-End DevSecOps Pipeline)

End-to-End DevSecOps Pipeline
End-to-End DevSecOps Pipeline

This comprehensive example demonstrates a complete DevSecOps pipeline with multiple security scanning stages:

trigger:
- master

stages:
- stage: sonatype
  pool: centos

  jobs:
    - job: sonatype
      steps:
      - task: NexusIqPipelineTask@1
        inputs:
          nexusIqService: 'sonatype'
          applicationId: 'webgoat'
          stage: 'release'
          scanTargets: '**/*.*'

- stage: fortify
  pool: batuhan-local
  
  jobs:
    - job: fortify
      steps: 
      
      - task: FortifySCA@7
        inputs:
           applicationType: 'java'
           buildSourceVersion: '11'
           fortifyBuildId: 'webgoatjava'
           fortifyScanType: 'LocalScan'
           runFortifyUpload: true
           fortifyServerName: 'ssc test'
           fortifyApplicationName: 'webgoatjava'
           fortifyApplicationVersion: 'webgoatjava'

- stage: trivy_file_scan
  pool: centos

  jobs: 
    - job: trivy 
      steps:
        
      - task: Bash@3
        inputs:
          targetType: 'inline'
          script: 'trivy repo https://github.com/nullx3d/WebGoat'

- stage: docker_up
  pool: centos

  jobs: 
    - job: docker
      steps:

      - task: CmdLine@2
        inputs:
         script: 'docker-compose up -d'
         workingDirectory: '/home/sancak/Downloads/WebGoat'

- stage: trivy_image_scan
  pool: centos

  jobs:
     - job: 
       steps:
         - task: Bash@3
           inputs:
            targetType: 'inline'
            script: 'trivy image webgoat/webgoat-8.0'

- stage: invicti
  
  jobs: 
    - job: invicti
      steps:

        - task: netsparker-cloud@1
          inputs:
           apiConnection: 'invicti'
           scanTypes: '0'
           scanWebSites: 'e15b8874-5cff-4dd7-02a7-ae43025f6459'
           scanWebSitesProfile: '5d21265b-4372-49fc-c209-b0ff051e9f2f'
           hasReport: true
           reportType: 'ScanDetail'

This end-to-end pipeline demonstrates:

  • Sonatype Nexus IQ: Software composition analysis for open source vulnerabilities
  • Fortify SCA: Static code analysis for security vulnerabilities
  • Trivy: Container and repository vulnerability scanning
  • Docker: Container orchestration and deployment
  • Invicti: Dynamic application security testing (DAST)

Conclusion

Azure DevOps YAML provides a powerful and flexible way to define CI/CD pipelines as code. From simple single-stage pipelines to complex multi-stage DevSecOps workflows, YAML syntax offers the flexibility needed for modern software development practices.

Key benefits include:

  • Version control for pipeline configurations
  • Reusable templates and components
  • Conditional execution and branching
  • Multi-agent support for different platforms
  • Integration with security scanning tools
  • Scalable and maintainable pipeline definitions

For more information about Azure DevOps YAML syntax and advanced features, visit the Microsoft Learn documentation and explore the official Azure Pipelines YAML repository.