https://discord.umbraco.com logo
Join Discord
Powered by
Azure devops yml optimization for building Umbraco...
# help-with-other
j
Has any had any experience with setting up a yml devops pipeline for building and deploying to an azure app service before ? I have succesfully made one for building and deploying my Umbraco 15 website (takes around 20-25 min with a staging deployment slots), i feel the time is worse than other projects i have made in the past and could use some inspiration or insight to trim bottlenecks. One thing i have investigated is why my dotnet publish produce a "runtimes" folder with all thinkable platforms (its actually 200 mb, where my project is around 100mb), i tried researching why they are included, and even at one point removed some of the runtime folders i was sure was not in use to only see my app service not being able to run anymore. Does any have insight why this is and Is it really not possible to reduce this ? I takes forever deploying to my app service and afterwards swapping takes everywhere from 5-10 minutes (which feels weird).
Current build stage:
Copy code
yaml
stages:
- stage: Build
  jobs:
  - job:
    steps:
      # Set node version
      - task: NodeTool@0
        displayName: 'Use Node 20.10.0'
        inputs:
          versionSpec: 20.10.0

      # Install npm packages
      - task: Npm@1
        displayName: 'npm install'
        inputs:
          workingDir: ${{ parameters.workingDir }}
          verbose: false

      # Install npm packages (ui)
      - task: Npm@1
        displayName: 'npm install (ui)'
        inputs:
          workingDir: ${{ parameters.uiFolder }}
          verbose: false

      # Caching NuGet packages
      - task: Cache@2
        inputs:
          key: 'nuget | "$(Agent.OS)" | nuget.config,**/*.csproj'
          restoreKeys: |
            nuget | "$(Agent.OS)"
          path: $(Pipeline.Workspace)/.nuget

      # Authenticate with NuGet
      - task: NuGetAuthenticate@1

      # Restore project
      - task: DotNetCoreCLI@2
        displayName: Restore
        inputs:
          command: restore
          projects: '${{ parameters.restoreBuildProjects }}'
          feedsToUse: config
          nugetConfigPath: 'nuget.config'
          restoreDirectory: $(Pipeline.Workspace)/.nuget

      # Build project 
      - task: DotNetCoreCLI@2
        displayName: Build
        inputs:
          projects: '${{ parameters.restoreBuildProjects }}'
          arguments: '--configuration ${{ parameters.buildConfiguration }} --no-restore'
          verbosityRestore: Detailed
          verbosityPack: Detailed

      # Build project 
      - task: DotNetCoreCLI@2
        displayName: Publish
        inputs:
          command: publish
          publishWebProjects: false
          zipAfterPublish: true
          projects: '${{ parameters.restoreBuildProjects }}'
          arguments: '--configuration ${{ parameters.buildConfiguration }} --output ${{ parameters.outputDirectory }} --no-restore --no-build'
Current build stage - continued:
Copy code
yaml
      - task: PublishBuildArtifacts@1
        displayName: 'Publish Artifact: drop'
        inputs:
          PathtoPublish: '${{ parameters.outputDirectory }}'
          ArtifactName: 'drop'
        condition: succeeded()
Current deploy stage:
Copy code
yaml
stages:
- stage: Deploy
  dependsOn:
    - Build
  jobs:
    - job: Deploy
      steps:
      - checkout: none # This prevents the automatic checkout of the repository, it is not needed for this job
      - download: current
        artifact: drop
        patterns: '**/*.zip'

      - task: AzureRmWebAppDeployment@4
        displayName: 'Deploy Azure App Service to staging slot'
        inputs:
          azureSubscription: '${{ parameters.azureServiceSubscription }}'
          appType: 'webApp'
          WebAppName: '${{ parameters.appServiceName }}'
          packageForLinux: '${{ parameters.packageDir }}'
          enableCustomDeployment: true
          ExcludeFilesFromAppDataFlag: false
          DeploymentType: 'webDeploy'
          SlotName: staging
          deployToSlotOrASE: true
          ResourceGroupName: '${{ parameters.resourceGroupName }}'
          RemoveAdditionalFilesFlag: true
      
      - task: AzureAppServiceManage@0
        displayName: 'Swap staging slot to production'
        inputs:
          azureSubscription: '${{ parameters.azureServiceSubscription }}'
          WebAppName: '${{ parameters.appServiceName }}'
          ResourceGroupName: '${{ parameters.resourceGroupName }}'
          SourceSlot: staging
          SwapWithProduction: true
m
I am using Github Actions but these suggestions should still be relevant for you. You will need to research a few things to figure out what works best for you. You can setup concurrency with your pipeline where two tasks happen side by side on separate agents to reduce real-world time. So in your case running the npm commands and dotnet commands on different agents so they happen concurrently. Then combine the build output into one artifact for deployment. I found this blog post that might help you go in the right direction: https://samlearnsazure.blog/2020/02/11/parallel-azure-pipelines-jobs/ but googling concurrency with azure devops should give you lots of information. Screenshot is a complex project we have that has multiple jobs but has been created in a way to do things at the same time if they are independent. So here the dotnet build and npm commands run at the same time. Swaps usually take some time to run. You will find it takes some time if you manually swap in the Azure portal. https://cdn.discordapp.com/attachments/1341442490472861726/1341792848307032074/image.png?ex=67b74978&is=67b5f7f8&hm=51742f9a7805ba6b7ad3f88fe11714ae09941daf1df32014eea5ce59b8fdeae2&
Also try using ubuntu agents where possible. Ubuntu is much faster than windows. You don't need windows to build Umbraco netcore.
j
Thats a very interesting read, and clean looking pipeline 🙂 I actually already use it for my IaC (bicep) that runs in parallel to my build stage, i have just removed it from my example as i thought my bottleneck for built times is located elsewhere. Is you setup by any chance an Umbraco website ? as i am looking at your pipeline i see you have a separate deploy for "backoffice" that looks interesting. Could that by any chance be whats get placed in wwwroot/umbraco as i just realised it contains a whopping number of files (13.306 Files, 307 Folders), and wondering if this needs to be handled better and separte somehow. I think its the reasons its slowing deployment to my app service as the log is crazy long 😅 , before it proceeds to swapping deployment slots.
m
Its an Umbraco load balanced site. So the deploy back office is to a web app for the back office that the client uses and the frontend is a multi instance web app that has the same umbraco site but you don't access /umbraco from that web app its what the public uses. Are you using Ubuntu for your agent or windows?
j
I am using windows agents.
m
one of the tests can't be ran in ubuntu so its ran separately
j
So i am actually overwriting the default agent that was designed to handle yaml pipelines, i did not know that. Could be interesting to try and changing it to ubuntu-latest to see the difference.
m
pretty confident you will see a big improvement. this solution has 23 projects inside and dotnet build in ubuntu completes in 1m 6s Here is some stats on the build solution process. https://cdn.discordapp.com/attachments/1341442490472861726/1341814370321305630/image.png?ex=67b75d83&is=67b60c03&hm=5173dce99a64f1940ac2f6ed14e965dced00757f4238b42eb7e3ac13a30afdd7&
m
We moved all ours to unbuntu for the same reason 🙂
j
I have not ran though the ubuntu agents list of things it has installed, but it should have the same toolsets as windows-latest right ? (i havent used it before, coming from classic pipelines, which is properly why i just used windows-latest)
m
hard to say but there's no harm in creating another branch and running your changes in that branch to figure things out. you might find simply changing to ubuntu-latest is enough to make a difference and everything else still works as expected.
m
glancing over what you shared you should be fine 🙂 even powershell has a version that works on linux now
m
I haven't tried this since I manually swap my slots. But you might be able to remove the swap slot step completely by enabling auto swap on your web app: https://learn.microsoft.com/en-us/azure/app-service/deploy-staging-slots?tabs=portal#Auto-Swap
Will save you the build minutes that way
j
So i would just need to configure it with my bicep instead once. As i only have a staging slot i deploy to it only needs to swap with my production once the website answer. Differently going to check out that feature, as that would save me a lot of time (hopefully) 😆
m
Looks like you can yeah, you need to specify the name of the slot to auto swap with in autoSwapSlotName. Not tried it so not 100% this is the right place. https://learn.microsoft.com/en-us/azure/templates/microsoft.web/sites/slots?pivots=deployment-language-bicep
custom warm up is interesting too you can specify the paths azure should check to determine if its safe to swap or not
j
i have tried to set always on on my deployment slot, to see if it speeds up the process of swapping.
m
These are some really old blog posts but it reveals that a lot of stuff is going on in the background when you hit that swap button https://ruslany.net/2015/09/how-to-warm-up-azure-web-app-during-deployment-slots-swap/ https://ruslany.net/2019/06/azure-app-service-deployment-slots-tips-and-tricks/
j
I see i can't switch to "ubuntu-latest" image without doing something about some of my "built events" i have. They use cmd or mac terminal commands (like set or export for setting variables) based on the os you are running locally. i guess i would need to find something similar to ubuntu.
So my built step is failing as a result. Restore step works fine.
m
You could try running it on mac-latest but they are a bit naf in Azure (so much so we use our dev macs for some builds) 😄
j
trying it just for fun 🙂
And actually a great fast way for me to test my cache step.
it seem its takes longer for it to zip the publish output folder in the "publish" step.
And fails afterwards in my deploy stage, as i am using some task macos image does not have appearently 😅 worth a try.
I think my next step would be to look into the auto-swap Matthew mentioned. Hope it will save me some minutes in the deploy stage.
m
Here is everything installed on mac-latest - https://github.com/actions/runner-images/blob/main/images/macos/macos-14-arm64-Readme.md 🙂 you can find all the images as well
j
i mean its not that bad. i actually figured out how to cut the "runtimes" folder entirely and get the website to run on my app service. (saving me around 220 mb worth of files).
i am guessing that the publish job takes a long time as it zip the umbraco folder located under wwwroot/umbraco (13.306 Files, 307 Folders) which in turn also makes my deploy to my app service take forever as it needs to unzip all these files again and match it with the change history.
m
yeah the unavoidable unzip 🙂
j
I managed to get build and deploy down to 6-8 minutes now. I did a number of things to get here, one of them was changing to auto swap and removing the deployment slot swap yaml step altogether. https://cdn.discordapp.com/attachments/1341442490472861726/1342080460670631987/image.png?ex=67c03e54&is=67beecd4&hm=4b85d219913621b33537b4a81ad94dd2e1b9ec17d71bf27ef945dfcf60f7fd61&
I made the option RemoveAdditionalFilesFlag: '${{ parameters.removeAdditionalFiles }}' on my app service deployment optional (with false as default now) instead of "true". Meaning i keep files between deployments. (hopefully it can figure out changes on files that already exist on app service between deployments).
and the last change that made it very fast was changing my publish arguments to the following arguments: '--configuration ${{ parameters.buildConfiguration }} --output ${{ parameters.outputDirectory }} --no-restore --no-build --runtime win-x64 --self-contained false -p:StaticWebAssetsEnabled=${{ parameters.updateBackofficeAssets }}' I found out that when setting -p:StaticWebAssetsEnabled=${{ parameters.updateBackofficeAssets }}' to false, the wwwroot/umbraco backoffice folders does not get included in my output publish folder. (as the assets lives in a RCL project, which basicly turns off the inclusion of these files) So now it is an optional step, that we can enable only at times where we have migrated to a newer version of Umbraco or have made backoffice extensions in our own RCL project. Else the files just lives on the app service, and we dont delete and copy them each time we deploy 🙂
it does require a deployment at first where the option is enabled in order for the backoffice files to be transferred once, but after that we only need to deploy our code changes 🥳 🚀
Overall it brings the zip package to deploy from around 200 mb down to 72mb 😍 (but will increase at times when we need to migrate to newer Umbraco versions).
m
So how many deployments till it pays for itself 😄
j
I think i have used around 2 work days time tweaking and researching possibilities 😃 . Me and my other colleagues actually deploy multiple times each day 2-3 times to production each. Roughly 1-2 weeks from now it should have been paid off in time saved (as we now save around 10-15 minute each time vs the old pipeline setup). 🥳 I appreciate all the input, i still wanna try the latest-ubuntu agent image at some point to compare build times. From here i have hard time seeing where it could improve. The auto swap properly needs some more love, with the combination of ensuring my solution is warm before the swap. (health checks or something ?)
m
We have a templated script for checking something is up
Copy code
parameters:
  appUrl: ''
  
steps:
- task: Bash@3
  displayName: "Verify application URL has started"
  inputs:
    targetType: 'inline'
    script: |
      retry_count=0
      max_retries=5
      while [ $retry_count -lt $max_retries ]; do
        if curl -f -s -o /dev/null "${{ parameters.appUrl }}"; then
          echo "Application URL responded successfully."
          exit 0
        else
          echo "Attempt $((retry_count + 1)) failed. Retrying in 5 seconds..."
          retry_count=$((retry_count + 1))
          sleep 5
        fi
      done
      echo "Application URL did not respond successfully after $max_retries attempts."
      exit 1
  condition: succeeded()
j
Do you know if it speeds up the swapping ?
m
We dont swap currently as the CMS is headless and everything is cache we can afford it dying
its also why my build takes 4mins 😄
m
You may be able to speed up "a swap" by not swapping at all but adjusting the weight of traffic from one slot to the other https://learn.microsoft.com/en-us/azure/app-service/deploy-staging-slots?tabs=portal#route-production-traffic-automatically On the same page if you add this into your web config azure will check the paths you specify to verify if the site is warmed up or not. https://learn.microsoft.com/en-us/azure/app-service/deploy-staging-slots?tabs=portal#specify-custom-warm-up
j
Something hit me today. I use the same database between my 2 deployment slots (staging and production). Won't i get in trouble down the road, as i am also using uSync ? As i deploy to my staging slot usync file changes are deployed to it, as the website starts up database changes are being made, when it gets ready it swaps. However production now become the staging slot and it will contain old uSync files which could protentional effect the database. 😅
m
j
As of now lucene index files could also be different, against the same database.
Maybe i should clear out the folder umbraco/Data/TEMP as a prestep to my deployment, where the index files are placed. Just to make sure they always gets built accordingly to the database.
What are your thoughts on the uSync.CommandLine tool Matt ? I had not heard or seen of it before, but it looks to contain a lot of cool utility commands, that could inform me if the website is up and or used for rebuilding the index 🙂
m
Honestly I swear to let you do uSync imports so you didnt have to do them on startup
k
The v15 version is slightly diffrent (still does it all) https://github.com/Jumoo/uSync.CommandLine/tree/v15/main because its using the mangement API - and built in Auth for API users in umbraco.
j
I am considering dropping deployment slots, as they seem to inflict more problems than first anticipated 🧐 As far as i understand now, a lot of the problems ends up being tied to the fact that lucene and nucache stores physical files. Also experiencing problems with a mainlock file being set sometimes, but not removed again, that essentially locks editors from creating content altogether. Happens during the swap. Does any have experience with a running example of a Umbraco site that use deployment slots with success ? I am still fairly new in developing Umbraco sites, so i might have missed out on some essential stuff 😅
m
It's something I have done rarely, with the odd pain point needing a restore but usually a code fix is enough
s
We have been using deployment slots for quite some time. At the moment, I am planning to move away from them as we see some weird problems from time to time when we swap slots.
l
Maybe this helpt? https://discord.com/channels/869656431308189746/1316698816468226109 (or non Discord link: https://discord.com/channels/869656431308189746/1316698816468226109) I really want to start using deployment slots, because Umbraco boots way too slow in many instances, taking up to 20-30 minutes sometimes.
j
There are for sure some interesting key points from the discussion/post you mention Luuk, like setting nuchache to load in memory, with the cost of startup time but with the benefit of maybe solving the lock problem i mentioned before that can happen between swaps. I am still wondering about why Umbraco statup time is so different and slow compared to running the solution locally. What tier of app service plan are you Umbracians running (if you are deploying with Azure) ? Currently i am just at the bare minimum of what is required to use deployment slots "S1"
m
Hybrid cache in 15 and beyond will solve that problem by replacing nucache 🙂
l
We're usually running on P1V3 or P2V3 and database on 10-50 dtu (think that is considered S0 - S2). Most of the time, this performance is more than enough and doesn't even get close to getting bogged down. But as far as I know, the main culprit for slow starting Umbraco sites is the loading of the NuCache on startup and especially on Linux this is just a slow process, even with the UsePagedSql setting set to false (although it helps). Hybrid Cache should solve a lot of this issue. The difference between development and production is probably down to the number of nodes. But also, maybe some of our custom code or packages are slowing things down. I'm going to look into it soon, together with indexes that get corrupted way too often.
m
Did you see Shannon released a new version of Examine, which has a fix for index corruptions
l
It's implemented in the latest versions of Umbraco right? Indexes are still a horror and get corrupted regularly here (running in Azure web apps)
m
I am not sure to be honest, if not it should be update able
l
You mean this one? It's not in Umbraco 13.7.2 (which has 3.5.0): https://github.com/Shazwazza/Examine/releases/tag/v3.7.0
m
That's the one
j
Can the version of lucene run with the latest version of Umbraco 15.3.0 ? it seem like it still uses 3.6.0 when i look at used package version.
It does looks like you can install it manually along side Umbraco with no noticeable problem so far.
l
Assuming that there are not breaking changes (which it should have, because it's a minor update) and Umbraco dependencies allow for a newer version, then I don't see a problem 🙂 Let us know if you notice a difference
We had two sites going down in the middle of the night again because somehow index files got corrupted again and locking issues
j
I will properly be able to give a report sometime next week if i notice far less lock issues, after we have deployed the upgrade 🙂
Been running with Examine 3.7.0 for some time now. The lock still occurs on weekly basis sadly 😅 i am not sure if it is less time than before to be honest.
I mean corruption of index. But it sort of locks the editor from creating new content.
l
Here, 3.7.0 makes all the difference. Its so much better! If you are hosting in Azure, are you sure you checked these settings? https://docs.umbraco.com/umbraco-cms/13.latest/fundamentals/setup/server-setup/azure-web-apps
j
Yeps, i already have the settings up and running, and are using 3.7.0. But i am using deployment slots in Azure which i am still suspecting is corrupting the indexes 😅
We are not scaling out, so i do not think i need to look into the Load Balancing doc 🧐 But everything else is "by the book" regarding the doc, we also use minimum recommended Azure SQL Tier is "S2". Just today i upgraded our App Service Plan from S1 to P1V3 which reduced the swaping time between deployment slots from taking 4 minutes down to ~1 min 30 sec, and also gave the backoffice a much needed performance boost.
k
also got the maindomdiscriminator and sitename set to unique values on slots?
UMBRACO__CMS__GLOBAL__MAINDOMKEYDISCRIMINATOR
Umbraco__CMS__Hosting__SiteName
j
Just back from vacation 🙂 kows thanks for the input i will look into it.
14 Views
PreviousNext
Powered by