Alternative build agents for Azure Pipelines

Non-technical overview

Azure DevOps is a Microsoft service developed prior to their acquisition of GitHub. There’s a lot of overlap between the two services, and – even though it seems inevitable that GitHub is the future, there are a lot of organizations still using Azure DevOps.

Azure Pipelines is a part of Azure DevOps where you can build and test code projects. (When programmers use the word “build”, they mean to convert code written by a human programmer into something that can run on a computer.) “CI/CD” has recently become a buzzword in the software development community. It stands for Continuous Integration/Continuous Deployment. Azure Pipelines is the part of Azure DevOps that provides CI/CD services.

The “building software” part of CI/CD requires build agents. A build agent is a computer used to build software, but a it’s a different computer than one used by a human programmer to write code. Azure Pipelines maintains a pool of VMs that can serve as your build agent. When a build process is triggered, one of these is temporarily allocated to your process. It does your build and then reverts back to the pool. (CI/CD is all about automating processes. All of these happens “auto-magically” as directed by a script.)

The VMs provided by Microsoft will meet most needs, but not all. I’m working on project for a client where the amount of RAM provided in the Microsoft-hosted agents wasn’t enough and the build process was failing because it ran out of memory.

When the Microsoft-hosted agents won’t meet your needs, you can plug you own “Self-hosted agents” into the process. This can either be physical or virtual machines that you set up, configure and maintain, or – you can take advantage of Azure virtual machine scale set agents. This lets you specify a “size” of Azure VM (allowing you to specify the processor, amount of RAM, amount of storage, etc.) and tells Azure DevOps to use that for your build agent.

This option is more expensive and requires more administration on your part, but if you have needs beyond what the Microsoft-hosted agents can handle – it’s nice to have this option. The rest of this post will recap some of the technical issues I ran into setting this up recently.

Technical details

This is a good starting point for background on Azure Pipelines Agents. If you follow the link in the Microsoft-hosted agents section, you can find out about the specific hardware allocated to these VMs. As of this writing, the amount of RAM is 7GB. This client has a particularly large Angular project that required more RAM than that.

I didn’t want to actually “self-host” in a physical sense, so I went with the Azure virtual machine scale set option. Here is the page with the specific instructions. There’s a lot of information here, but basically it consisted of three steps:

  1. In the Azure portal, create the scale set.
  2. In the Azure DevOps portal, set up an agent pool pointing to what you created in #1.
  3. Change your pipelines to use the new agent pool.

Create the Scale Set in Azure

The instructions walk you through using the Azure Cloud Shell to create the scale set. The az vmss create statement is the one to focus on. The vm-sku setting is probably the most important one. This is what specifies the “size” of the VM. Start here to determine which size will work best for your needs. (Don’t agonize over this too long at the beginning. It’s pretty simple to change this setting on your scale set.) Make sure to pay attention to the usage price of your chosen size. (Here’s the pricing table for Linux VMs.)

The instance-count setting only specifies the number of instances initially created. Once you “hand control of the set over to Azure DevOps” (in step 2). It will override this instance count.

Once you’ve successfully run the az vmss create statement. You’re done with step 1. Azure DevOps will handle everything else (including installing required software on the VMs).

Create the Agent Pool in Azure DevOps

The instructions in the article above are pretty good. Here are some notes of things I ran into:

  • You will need to be an Agent Pool Administrator to create the new Agent Pool. If you are missing this, you will get to the end of the form and get an error when you submit saying you need “Manage permission”. If you get this, have the owner of the organization use the Security button on the Agent Pools page (top right – next to blue Add Pool button) to grant this permission to you.
  • The Maximum number of virtual machines in the scale set setting overrides the instance-count setting from step 1.

Change your Pipelines to use the new Agent Pool

Here is the relevant part of the Azure Pipelines YAML schema. Basically, if you are using a Microsoft-hosted agent pool, you only specify the vmImage value.

If you want to use a self-hosted agent pool, you need to specify the name. This is the name you provided in step 2 and the name that shows up in the list of Agent Pools in Azure DevOps.

In the case shown above, your YAML would contain this:

pool: Self Hosted Scale Set pool

Final Considerations

The 3 main settings on the Azure DevOps Agent Pool will probably involve some trial and error depending on your specific needs.

  • Maximum number of virtual machines in the scale set
  • Number of agents to keep on standby
  • Delay in minutes before deleting the excess idle agents

If you set the Number of agents to keep on standby to 0, you won’t be paying for instances to sit idle, but your Pipeline runs will sit in the Queued state for a little while to give Azure time to “spin up” an image for you. (In my case, this was about 5 minutes.) You can monitor the number (and status) of instances in the Azure portal by finding the Virtual machine scale set and clicking on Instances on the left-side menu. (Hint – if you can’t find it, look at the Resource Group you specified in the az vmss create statement.)

“No!” to the return of mask mandates

This afternoon UAB announced that masks are required indoors, regardless of vaccination status. This one has me mad. The incentive to getting the vaccine was – “get the vaccine and get back to normal”.

What’s changed? Numbers are going up for unvaccinated people.

(By the way, if you’re a media organization – please include vaccination status in any COVID numbers you report.)

Before I go on, let me mention a couple of things:

  1. I have a couple of clients that are hospitals. I was on-site at one today. I wore a mask. It was uncomfortable, but I understand that that’s the thing to do in a hospital.
  2. This fall, I’ll be volunteering at a school to help teach computer science. I’m resigned to the fact that I’m going to need to wear a mask. I’m not thrilled about it, but I understand that a lot of the kids there are under 12 and can’t get the vaccine yet.

Having said that, at this point – we generally have a “pandemic of the unvaccinated”. Here’s a quote from Dr. Karen Landers from an article posted today on “If you look at the outcomes of COVID-19 and persons who have been fully vaccinated and then you look at the outcomes of COVID-19 for persons who haven’t been vaccinated. The persons who have not been vaccinated, right now, are making up almost all of the hospitalizations that we’ve had.”

If you choose to not get vaccinated, that’s your right. But it’s also your responsibility to accept the consequences of your decision.

I understand there are “breakthrough cases” of vaccinated people getting infected, but – their experience seems on par with getting the flu in previous years.

I have a child starting at UAB this fall. UAB decided earlier in the year that freshman orientation would have to be virtual via Zoom. (Auburn and Alabama decided to return to in-person orientation.) But later, UAB decided that in addition to Zoom sessions, they would offer “Bonus Days” to let students come on campus. I was on campus with my son on Monday. I was grateful that they decided to offer some on-campus experience, but it seems to me like they jumped the gun on the original decision.

Today’s announcement feels like that. It also feels like decision makers are making the calculation that people have been vaccinated are less likely to be “unruly” about further inconveniences imposed on them and that the vaccine skeptics are more likely to yell and scream.

Ironically, yesterday my son had to go the doctor to get his “required vaccination” paperwork. From UAB’s website: “All entering freshmen must show proof of the following immunizations: MMR (measles, mumps, and rubella), Tdap (tetanus and diphtheria), meningococcal, and varicella.”

I’m opposed to government’s issuing vaccine mandates. But – here’s an institution that already has vaccine requirements in place and they’re going to make a decision that requires vaccinated people to be inconvenienced to accommodate non-vaccinated people. Why not just add the COVID vaccine to the list for on-campus students?

If I’m a non-vaccinated person, the logical take-away from this is “Why should I get vaccinated? Apparently it doesn’t make any difference and I would still have to wear a mask anyway.”

This isn’t just about UAB, but they are a convenient target at the moment. I fear that many other institutions and businesses will follow in their footsteps with logic-defying decisions and I want to preemptively yell “Don’t do it!”. We have been generally compliant, but we’re about to start getting unruly.

Addendum added on July 30, 2021 – I didn’t realize that Alabama schools are prohibited by law from adding the COVID vaccine to their list of requirements. SB 267 prohibits creating vaccine requirements that weren’t already in place on January 1, 2021. Personally, I think this may need to be re-considered.

Inching Back to Normal

About a year ago, when the COVID lockdowns were first starting – I think we thought the end was going to be different. I know I did. I remember hearing commentators talk about pent-up demand and I guess I kind of subconsciously envisioned something like the famous scene where the sailor kisses the nurse in Times Square on V-J day. But now that we’re getting here, it’s not like that at all.

I’m lucky enough to be fully vaccinated. I live in a state where there is no longer a mask mandate. Most of the businesses around me still have “masks required” signs on their door and I comply. But – there’s one store close to me that doesn’t. It feels strange to go in and not put on a mask, but I do. Not to make a political statement, but to start inching back to normal.

I think it’s going to be harder to get back to normal than I thought it would be at the beginning of all of this. At this point, it’s easier to work at home and meet via Zoom. (I realize I’m very lucky to have that choice.) But – I think it’s important to start deciding to get dressed (really dressed, not t-shirts and sweats), and go back to the office and meeting with people in person.

I’m realizing that, unlike World War II, there’s not going to be a definitive end to the pandemic. There’s not going to be a time where they (whoever they is) declare “All clear – resume your normal lives now.” Everyone is going to have decide what’s safe on their own.

A New South

Watching the riots in the wake of the tragic killing of George Floyd have led me to put down some thoughts that I’ve had for awhile.

For background, I’m a white man in my early 50s. I’ve lived in Alabama just about my entire life.

A few years ago, a man came to speak at our church. He was a white man on staff at a small church in the middle of an urban, inner-city Birmingham area church. He told us about some of the struggles his church and his members (predominantly black) were going through and how we could help.

Later, I had coffee with him to find out more. He told me “before you start, I want to give you some homework”. This surprised me a little, but – OK. He asked me to read a book, “Divided by Faith”, and watch a video documentary, The African Americans: Many Rivers to Cross.

The documentary was hosted by Henry Louis Gates. I knew he was a professor and had been in the news a little before over an incident that resulted in the “Obama Beer Summit”. I’ll admit that based on what I had heard about that incident – I had a slightly negative opinion/perception of him.

I’ll also admit to being pleasantly surprised at the documentary. I think I was expecting to be lectured about “white privilege”, but it really wasn’t like that. I found it very educational.

It was also disturbing. Since one of the topics was slavery, this isn’t surprising. While watching, I had a thought which really made the evil of slavery hit home for me.

A few years before (in 2009), America was shocked to find out that Jaycee Dugard, who had been kidnapped at the age of 11 in 1991, was still alive and had been in captivity for 18 years. She had been raped repeatedly and had two children whose father was her kidnapper.

As awful as that was, when the public and law enforcement found out about her – she was rescued. What occurred to me while watching the documentary was that slavery was this same crime, but instead of a shocking outlier – it was a “protected-by-law” institution. Imagine Jaycee Dugard escaping and when the police found her, returning her to her captors. That (over and over) was slavery.

I mentioned I’ve lived in Alabama almost my whole life. I did go to college in Boston for a couple of years. I’m embarrassed now to say – for part of that time a Confederate flag hung on my bedroom door like a poster. It was an ignorant, insensitive act with no malice intended. In my mind, it was a “Southern pride” thing – part of a friendly regional rivalry with some of my Yankee friends. (I was very outnumbered.)

Despite our protests that the Confederacy was about “states rights” and “regional pride”, I feel safe in saying that – for our black brothers and sisters, it means “you are not as good as us, and you are not part of us”.

I think that Birmingham’s mayor did the right thing by telling the protestors and rioters who wanted to tear down the Confederate memorial “Let me handle this” and he did. It was time for it to go. Now it’s time for a lot more things to change.

I still like the idea of Southern pride, but one that all of us – black and white (and any other color) – can share and be a part of. Confederate imagery divides us. Let’s come up with some “New South” imagery that we can all display proudly.

Offline Mode

It’s Week 4 of “social distancing” mandated by COVID-19. During uncertain times (a bit of an understatement for our current situation), it’s easy to worry about the future.

In our connected world, our computers, phones, and many other devices have Internet connections. A lot of them have an “offline mode” to be used when we’re on an airplane or no connection is available. Offline mode isn’t as good as being connected.

A few days ago, during a moment of worry – I said a quick prayer and was struck by how quickly my outlook changed. It occurred to me that my default setting with God was “offline mode”. I connected occasionally (church on Sundays, devotional time, etc.) but was offline most of the time.

1 Thessalonians 5:17 tells us to “pray without ceasing”. A modern day paraphrase of that could be – “switch your default setting to connected mode”. Connected mode is better than offline mode.

So What?

A lot of our current education is based on answering either Who, What or How:

  • Who was the seventh President of the United States?
  • What is photosynthesis?
  • How do you factor this polynomial?

I think we would benefit from spending more time on Why, as in “Why do I need to know this?” Or to be more concise – “So What?”

I’m not recommending that students are disrespectful to their teachers. I’m suggesting that the adults need to spend more time asking these kinds of questions about our curriculum.

As I write this, we’re in week three of “social distancing” due to COVID-19. Many schools are about to come back “online” in remote learning mode. It seems pretty obvious to me that – at least at first – this isn’t going to be as efficient as the in-person methods we’ve been using for over 100 years. This is a good time to focus on the material that we think is really important for students to learn.

I suspect we’re going to learn that the curriculum starts from “How many days are in the school year?” as opposed to “What’s important to learn?” I don’t think this is on purpose, but I think inertia has brought us pretty far down this road.

One Size Fits None

As I’m writing this, we’re in Week 2 of the COVID-19 “social distancing” phase. All the schools are closed and trying to transition to online instruction. I feel like this time will help us evaluate separately the different services that schools are providing in society. This crisis has really brought attention to the fact that schools provide, in addition to education, child care services for working parents and meals for students in need. (It’s also the hub of social interaction for most kids and teenagers.) I think these are all needed services, but I feel like our current “delivery mechanism” evolved without much of a plan and this may be an opportunity to evaluate and improve.

This year, I got to volunteer to help teach a Computer Science class at a local high school. There are more than 20 students in the class, and I’d guess that about 4 are interested in being there and learning the material. The remaining students are there because they need a credit and have to be somewhere during that period. These students have needs that aren’t being met by sitting a class that they have no interest in, and – their presence is reducing the effectiveness for the 4 who could benefit from it.

There are a students for whom high school is effectively a minimum security prison term and they are just doing their time until its over. I’d argue that – once you reach about 16, if you’re not getting anything out of school – you’d be better off getting a job. I’d love a world where older teenagers could get some “life experience” but then had the ability to easily re-enter an education system once they are motivated to learn. (Also – I’m guessing we lose a lot of good teachers who thought their job was to teach, but become disillusioned after realizing that their students aren’t interested in learning and they really just need to keep them out of trouble and from being a disruption. )

Remote learning has the potential to provide an unlimited course catalog of subjects to offer, which increases the likelihood that students can learn about something they are actually interested in. It also has potential to help us transition to a culture of lifetime learning, not just until you’re 18.

There are plenty of needs for local schools and personnel, but maybe some teachers move into more of a “mentoring” role, helping older students navigate the process as opposed to providing actual instruction (that may be delivered via YouTube).

I’ll be the first to confess that I don’t know exactly how this would work, but – while we’re all disrupted, this may be a good time to begin the conversation.


No Stupid Questions

People always say “there’s no such thing as a stupid question”, but most of the time – the people they say that too don’t believe them and decide “I’m not going to ask that because I’ll sound stupid”.

I think that most of the time, when someone “doesn’t understand” – it really means “I’ve come up with a mental model of how this works, and the new information you’re giving me doesn’t fit into that model”. The best way for a teacher to help get you on track is to understand your current mental model. The best way for the teacher to figure this out is for you to ask questions.

It’s like being lost and calling and asking someone for directions (before we had GPS and mapping apps). They can only give you directions if they know where you are coming from. If you tell them “I’m looking at a Dairy Queen right next to a Taco Bell” (and they’re familiar with the area) – they can say “Oh – you turned left instead of right. Here’s what you want to do…”

If you just keep saying “I’m lost” (or worse, not saying anything) – it’s much harder for someone to help get you back on track.

Remote Learning

As I’m writing this, we’re about a week into the “social distancing” phase of the fight against COVID-19. At this point, pretty much every school and college in the country has been shut down and either has or is trying to transition to remote instruction.

Even though this transition (like everything) is chaotic right now, once we get this going – I can’t help but think that we are going to realize that remote instruction is a valuable “arrow in our quiver”. I’m not saying that this will work for every situation, but this is the perfect opportunity to put in place the idea of transitioning in-person lectures to YouTube videos.

Instead of the same information being delivered thousands of times across the country by different teachers with different presentation abilities, let the best lecturers present the material and let local teachers answer questions and help students process and apply the information. Khan Academy is just one example of an online resource that has many great online lessons. (This isn’t an original idea. This is one of the many interesting ideas Seth Godin makes in “Stop Stealing Dreams”. Video here, manifesto here.)

Don’t misunderstand. We still need local teachers and local schools. We just don’t need them to do the same thing they’ve done for years.

You can lead a student to class but you can’t make them learn

Horse water drink
Photo by Jon Tyson on Unsplash

Borrowing from the old proverb about forcing horses to drink, I think that all too frequently – this is exactly what we’re trying to do in education.

As soon as you think about it like this, I think it’s pretty obvious that force feeding knowledge to uninterested students isn’t a path to success.

I’ll confess, as a parent – I’ve frequently gotten frustrated with some of my son’s teachers. This year, I’ve been volunteering with TEALS to teach Computer Science in a high school and I have gained some empathy with teachers. It’s hard to teach – much less inspire – students who have absolutely no interest in being there.

When kids are younger, I think they will “learn” out of obedience. Some (a minority, I’m guessing) become genuinely interested in subjects and learn because they are curious. I think the rest reach the point of “I’m bored and I don’t care” at different ages. (Personally, I hit this point in college.) Could we transition from our current “drink from this firehouse until you’re 18” model to more of a “work for awhile, then learn for awhile” model?

It’s odd that we spend of so much money, time, and effort on compulsory education until students are 18 (when most of them have no appreciation or interest in what they are being taught), but once you’re an adult and have “been out in the world” and have a better feel for what you’d like and need to learn, it’s a bit of an uphill climb to obtain education at that time.

In general, I’m not an advocate of government spending. In this case, we’re already spending a lot of money and I think it’s worth asking if we could “spend smarter”. I’m also not suggesting this would be easy. Making changes to the idea of “high school” – an institution that’s so embedded into our culture – would be a huge shift and unintended consequences are a very real risk. But – the more I think about it, I can’t help but feel “this is broken”.