Incrementally Moving to Trunk Based Development and CI - Part Two

December 7, 2018    DevOps Development BDD AutomatedTesting

Incrementally Moving to Trunk Based Development and CI - Part Two

In Part One I laid out my thoughts on Trunk Based Development (TBD) and how it interacts with aspects of DevOps, Automation, Separation of concerns, Clean Architecture and more.

Here are some questions you might ask, a hypothetical and resources to reference.

What about X?

I’m sure there are a lot of questions floating around in your head. “Why should I change my process?” “Why do you think this is the approach to use?” I’ll attempt to answer some of them. Add more in the comments and let’s discuss more. Remember to read the recommended books for a better/in-depth argument and answers to your questions.

Feature flags for the full stack?

I’d try having a DB table called FeatureFlags. Cache this value on startup and use this throughout. Have a way in the UI to turn this value on and off and require a login/logout. You should think about having this per user or per group or for the full app. Having for the full app would be the easiest first attempt.

Won’t more commits/pushes = more chaos?

“My concern is that I think that there would be a noticeable drop in productivity as we change processes. We’re already constantly behind, so I’m not sure it it will be accepted by management.” ~ a knowledgeable and experienced co-worker

This question is very valid. I’d point you back to The DevOps Handbook and the Accelerate research book that points out that teams have gotten over this hump and that you probably will have a drop in “productivity”, but if TBD is achieved, you’ll gain that time back quickly.

You need automated tests and builds to reduce the chaos.

How does QA keep up with things changing?

  • Isolate in progress with feature flags
    • make sure to remove flags after work is done
    • talk about combination of flags
    • Have flags toggleable in a UI/script so QA can change when needed
  • Automation will reduce the manual testing time and manual regression testing needs
  • Automate the ability to create an environment, allow QA to each have an isolated testing environment if necessary
  • They are still essential
  • Embed them in the team, work together

How do I monitor with things moving?

  • Use AppInsights or similar tool (Stackify, etc)
  • Create a page that shows the version of all the pieces (pull of the dll version)
  • create ways to choose versions of components to add into the overall installation
  • maybe component y needs a quick fix, but z hasn’t changed

My team isn’t ready or doesn’t seem to have desire to change or learn new things

  • Most developers I work with have a desire to learn new things. Sometimes there are too many deadlines or pressure for them to spend energy
  • Make sure that changes can be implemented, there’s not much that is more demoralizing then identifying issues and not getting a chance to change things.

How is my marketing/tech writing/training group going to keep up?

  • embed a Tech writer in the team
  • involve them with planning before sprints

How do I know what’s released in each version if it is always changing?

  • Create a version page to easily view it
  • Use Azure DevOps work items and require association with a check in. Coupling this with a release pipeline provides good visibility to this.

How do I fix issues in production?

  • Label the source code (in the build) with each artifact (the output of the build). Then you have to know what’s released.
  • When a bug is found, create a branch, fix the bug, do a good pull request/code review, push that version out with only that change.
  • merge that bug fix commit into master.
  • See Git-scm Chapter 3.2

How do I handle database schema changes?

  • Step #1. Get your database schema into source control with DB Project or other tool
  • Check in the DB changes before or with other changes.
  • Make additive changes with default values.
  • if you need to remove a column, declare them as obsolete and notify consumers that they have a few release/months/what makes sense.
  • Here’s a more in depth article

How do I handle deploying to multiple environments/clients that are hosted in different data centers?

It’s a lot easier to do TBD and CD when you have a simple web application, but it is definitely more complicated when you can’t just use Azure DevOps builds and pipelines to push things out when you want. You’ll have multiple versions and production environments to handle.

  • Is is possible to move to a Software As Service (SaS) model and add in users/auth to distinguish between the clients?
  • Successful example: VSTS (now Azure DevOps) was releasing every 3 weeks to their Azure hosted and TFS versions every quarter or so.
  • I suggest a goal of having only 2 or 3 versions out. Make it easy and safe to upgrade and deploy. Then the Ops people will take your updates. Make it automated so that they can schedule it on off hours. After a dozen or so non-event upgrades, they might even trust you to push the changes remotely, without all hands on deck. Work to build that trust.
    • Don’t forget a rollback plan, in case something goes wrong.
  • As you “move the pain forward” and deploy more often, you’ll get better as you improve your process and automation.

What about the activations, marketing and tech writer teams?

Changing to Continuos Delivery won’t just affect the development cycle, but also the release teams. The tech writers will ask, “How do we keep up with all the changes?”. The marketing people will ask, “How do know what’s new and when to share them?”, especially when somethings are released, but just hidden behind a feature flag. The people in charge of installing the software (unless you’re just deploying to the cloud) will ask, “What do I deploy? How often?”.

One way to approach this is to “shift left”. Embed tech writers into the team, instead of waiting to hand it off when it’s all done.

Help marketing to see the value of releasing more often and work with them to have a regular what’s new announcement or switch to announcing the big/exciting stuff.

Train activations to just take the latest. After a streak of successful, even boring releases, they’ll get used to the idea quickly. If they aren’t afraid and can run the updates quickly and easily, they’ll like that a lot. Then go further and completely automate the process with them.

Why you should

My friend, slarson gave this feedback. “For the process discussion and friction around having dev and master be separate and they pain it introduces I like to remind people: 10 minutes of your day is 1 week of your year. It is super counter to a lot of dev ops cultural posts, but I think getting flags around changes, some form of test automation on a build, linting, and an easy way to publish a metric are more important than the cultural side because the standard argument is how “that’d be nice, but we don’t have X Y Z so we can’t do that””

A hypothetical

Let’s say you have a MVC .Net Framework application and multiple WebApi projects that is several years old. There are 3 TFVC branches DEV/QA/RC in your on-premise TFS. There are no builds (or existing XAML builds that take way too long) in your TFS setup at the moment.

A standard process is to write code locally, check in to main with a goal of one change-set, deploy to Dev, test in Dev and review with the team. Once approved, merge to QA and QA can start testing. Repeat as necessary for any issues QA finds. Once QA verifies the changes, then merged to RC.

How do you incrementally improve?

  1. Learn
    1. Buy the Phoenix Project and The DevOps Handbook for everyone. Commit to reading as a team weekly book lunch discussions.
    2. Watch the Spotify videos
  2. Identity
    1. We need builds to make sure the code builds
    2. There aren’t many unit tests, if any
    3. Does the deploy to environment happen manually through VS or through the XAML builds?
    4. How stable are things? Is QA getting interrupted often
    5. We need to get the database into source and version it with our code.
    6. We have complex production environment with multiple servers with CRM and other dependencies.
    7. Branching issues
      1. Are they adding value?
      2. We spend several hours per developer merging to QA and still have manual merge issues
      3. Do you have errors due to manual merging issues?
      4. Do you know what version of code is in what branch?
      5. Can you reduce 2 branches into 1, then eventually go TBD?
    8. Merging to RC isn’t happening as often as it should and getting a release out is very difficult (2 weeks on average, with frequent delays when bugs are found)
    9. Do a value stream mapping exercise to identify work ( Making Work Visible looks like a great reference book)
    10. How can you minimize work in progress (WIP)?
    11. Where is the technical debt in your system?
  3. Make a plan/goals - Month 1
    1. Get a gated build working that build the solution and run tests (use the “vNext” builds and convert any XAML builds to the task based builds)
    2. Write unit tests on new features, any new bugs found and identify critical areas for tests to be added
    3. Create a SQL Server Database Project and use it to deploy DB changes
    4. Investigate moving to each developer having their own database.
    5. Identify minimal seed data needed for each database.
    6. Move to 2 week sprints with the goal of something to demo and release at the end of each sprint
    7. Pay down technical debt in x area each sprint.
      1. Work on reducing technical debt and coupling in your code.
        1. Clean Code by Bob Martin and @ardalis at Weekly Dev Tips has great tips on this.
  4. Goals for beyond (split into pieces that into 1 or more sprints)
    1. Investigate migrating to the Azure DevOps Platform
    2. Create scripts to create your environments (Automate everything you can)
    3. Does moving to Azure AppService and other tools on Azure make sense for our company?
    4. Create a Continuous Integration/Release Pipeline to take the built files and deploy to the environments
      1. Deploy the web application
      2. Deploy the Apis
      3. Update the Database
    5. Start writing UI tests with Selenium to cover common use cases and reduce manual regression testing
    6. Move more code into a “platform API” so that other applications can use the same endpoint.
    7. Make environment creation easy
      1. testing a branch should be easy
      2. Azure and virtualization makes this easy in some cases (but watch your costs)
      3. Containers might be an option here
      4. This includes DB creation from scripts with the minimum amount of seed data
      5. make it throw-awayable
    8. Add AppInsights (if hosted on Azure/internet out accessible) or Stackify for logging and monitoring
    9. Show progress to the company
    10. Move to Git
      1. Git helps a lot with branches
      2. You don’t have to move from TFVC, but after moving to Git myself, I think it does help a lot.
      3. Be prepared to have a lot of questions and some messed up branches. It’s good to have a git “guru” available for these questions for a few weeks as people get used to the change.
      4. Here’s my help on the topic
    11. Feature flags/toggles
      1. hide in-progress work
      2. different then hide/show features from users based on user/feature set
  5. Evaluate (Retrospective)
    1. How have things improved?
      1. Are we getting changes from dev start to tested complete out to our users faster?
      2. Deploying faster?
      3. Less time merging?
      4. Less confusion on what is in what environments?
    2. Anything not going well?
    3. What do we need to change/tweak to continue improving?
  6. Celebrate, share what you learned and continue improving.

Recap

In order to reduce the friction of getting value (completed features/fixes) to our customers, we want to incrementally move from processes that cause the friction. One of these areas is having multiple branches to isolate changes that require merging. We need to move incrementally, because in most circumstances the business is still demanding new features. There are certain “capabilities” that are required to move towards trunk based development.

I’ve proposed some common things that will help move towards TBD.

  1. Learn Together

  2. Identify

  3. Make goals together

  4. Choose where to start to incrementally improve

  5. Get management buy-in and ensure time to work on this

  6. Evaluate often (Retrospective)

  7. Start moving and improving

    “Trunk Based Development requires teamwork, empathy, and openness. All of which are best achieved as a team, and not as an individual.”

~ https://levelup.gitconnected.com/why-are-so-many-people-against-trunk-based-development-a785d9322584

More Resources

This is my collection of links. Of course, there are many more.

Side Note

I considered naming this “I want to be done merging code manually, but I don’t know how to get there” to mimic my I know I should be Unit Testing, but I don’t know how or where to start - presentation :-).

A Grand Adventure

I think it’s worth repeating this :-).

via GIPHY

As Bilbo said to Frodo “He often used to say there was only one Road; that it was like a great river: its springs were at every doorstep and every path was its tributary. “It’s a dangerous business, Frodo, going out of your door,” he used to say. “You step into the Road, and if you don’t keep your feet, there is no telling where you might be swept off to.” Frodo Baggins about Bilbo, The Fellowship of the Ring, Three is Company” ~ Found with Bing on

Keep evaluating and improving and I think you’ll end up in a much more productive and happy place for your team and company.

If you find value in this post, please re-tweet my conversation and share this with others.

Related Posts
April 9, 2024
Use .Net CancellationTokens
December 19, 2023
.Net Aspire
October 25, 2023
Clean Architecture Example
Watch the Story for Good News
I gladly accept BTC Lightning Network tips at [email protected]

Please consider using Brave and adding me to your BAT payment ledger. Then you won't have to see ads! (when I get to $100 in Google Ads for a payout, I pledge to turn off ads)

Use Brave

Also check out my Resources Page for referrals that would help me.

Swan logo
Use Swan Bitcoin to onramp with low fees and automatic daily cost averaging and get $10 in BTC when you sign up.