Incrementally Moving to Continuous Integration and Trunk Based Development

November 23, 2018    DevOps Development BDD AutomatedTesting

Incrementally Moving to Continuous Integration and Trunk Based Development


I was eating lunch a few weeks ago and the leader of development at the company I was on contract for sat down and asked me how things were going. I had not slept the best and had just spent a lot of time merging from our Dev branch to our Main branch. We started talking and I was attempting to extoll the virtues of trunk based development and feature flags.

He challenged that by pointing out that it’s hard to find a path to move from TFVC (TFS Version Control) branches to a trunk based development flow. He’s tried trainings and other things with his distributed team in the USA and the world (10-20 developers). Unit testing hasn’t stuck with this team yet. He also has insightful questions about how to have a feature flag that spans the full stack (DB, APIs, UI, services, etc). This all got me thinking and lead to this article.

This company has a common approach of using TFVC with a Dev branch, Main branch and RC branch. They have a nicely laid out development process that requires reviews, demos and testing before merging it to Main. They have dependencies on MS Dynamics CRM and Sharepoint that make isolating code this way necessary.

There are already 1000s of articles on Continuous Integration DevOps, but I needed to take the time to write out what I’ve learned from reading, watching and observing. Hopefully, this brings value to the community and you the reader.

What are we trying to accomplish?

Back to the Future tracks Back to the Future tracks

  • Reduce friction
  • Deliver value faster with increased quality
  • Win new customers and improve current relationships => help our business succeed and excel in the marketplace
  • Increase Net Promoter Score (NPS) as discussed in the Accelerate book, attract and keep talent in house
  • improve everyone’s working lives, reduce stress

This transformation will take time (years), but we are looking for an incremental approach and continuous improvement.

How does DevOps fit in this?

I’ll use Donovan Brown’s definition “DevOps is the union of people, process, and products to enable continuous delivery of value to our end users.” and “I choose value over software”.

I use these thoughts as guiding principles to improvement. The Accelerate Book highlights what they’ve learned from years of State of DevOps Reports research and TBD along with the DevOps concepts are indicators of higher performing teams.

Focusing in on Trunk Based Development (TBD)

I see TBD as a core aspect of delivering software with less friction. Thoughtworks has an excellent article about TBD. Take some time and read this article. I won’t re-hash TBD more here.

What are the capabilities/pre-requirements needed to do TBD?

My pre-req capabilities Here’s a higher res version of my drawing

Pardon my handwriting and Surface Book drawing skills, but I visualized the capabilities something like this. I don’t think you have to move from the center out in a linear fashion, but communication aka a part of DevOps is definitely at the center of moving towards TBD (and DevOps).

Is there an incremental path to TBD? / Where do I start?

On page 145 of the DevOps Handbook Mr. Gruver shares his experience moving to TBD.

“However, trunk-based development required them to build more effective automated testing. Gruver observed, ‘Without automated testing, CI is the fastest way to get a big pile of junk that never compiles or runs correctly.”

You should definitely not start by deleting all your branches and getting rid of your process. You need to do some preparation work first.

Thankfully, The DevOps Handbook has part 2 - Where to start? to give us guidance. Here are my thoughts, many from this book, other resources, stories and experiences over the years.

Start with building the team and an open/no blame culture

We are a team, let’s stop blaming each other (even if we don’t say this out loud) and work together. That means getting managers, developers, Ops, QA and more people together and talking. Buy some pizza have some fun and identify how to improve together.

Learn together

3 DevOpsBooks

I highly recommend starting with The Phoenix Project as an introduction to the DevOps ideas. Then get The DevOps Handbook to use as a reference and read together. Accelerate is also a great reference. I’m thankful for Gene Kim and others for putting these books out in the last several years. Continuous Delivery by Jez Humble (at least the first 5 chapters) came out before the others and is a foundational book as well. There are many articles, talks and videos as well, that I’ve referenced at the end of this article.

Watch how Spotify organized their teams and delivers software. This was eye-opening for me when I watched it the first time.

Don’t get stuck in learning paralysis. You can do while you learn. Create a hypothesis, do small experiments, record and learn.


What are the company goals? Do we know why we are doing what we are doing?

As a developer, I can easily lose sight of the company goals while I’m focused on creating a feature, debugging problems that are reported, helping others, etc. Does you team know why you are building each piece of software and how it helps your company succeed in the marketplace (and/or serve others)?

What are the pain points?

Start by identifying the pain points you know you have. Ask your team. I’d bet you’ll have a number of things your team is aware of, but never have time to do it.

Chapter 6 of The DevOps Handbook says to “gain a sufficient understanding of how value is delivered to the customer”. “Conduct a workshop with all the major stakeholders and perform a value stream mapping exercise”. Figure out all the steps required to create value and identify where there are delays and wait times. Page 65 has an example diagram.

Can you put metrics on the delays? Examples: We spend 10% merging code. We spend 10 days of manual testing and 10 more days fixing issues before we can create a build to release. (see some good examples on pg 144 in the DevOps Handbook)

Put a number from 0 to 10 on each of the capabilities.

Where can we automate (environment creation, database seed data, database schema updates, unit testing, UI testing)?

Where can I start adding tests? I have a full talk about this that I named I know I should be Unit Testing, but I don’t know how or where to start

What are the major bottlenecks in the flow of your work? In the Phoenix Project everything seems to always come back to the same guy. Phoenix Project - 90 “Improvements anywhere besides the constraint is just an illusion” another great quote is “improving your daily work is more important than daily work”

Make some goals together

Jeremy Likeness has a great article about goals. Where do you want to be in each of these capabilities in 1 week, 1 month, 1 year?

Choose where to start to incrementally improve

Long Road - where to start

First: Do you have your code in source control? If not do this ASAP!

Get a gated check in build running?

Move your code to Azure DevOps?

Consider moving to git? (Note: TFVC is still supported and still a good option, I’ve just found Git to be much more enjoyable, pull requests much better then TFS code reviews and productive after getting over the learning curve. I recommend having someone available who has experience to help others out when they mess up a branch)

Is there a new project or small team that you could experiment with these concepts? Then you’ll have stories to share about wins and what went well and able to spread this knowledge to other teams.

Merge a Dev branch with a Main branch? How do you isolate in progress code and your QA from things changing all the time? - I think you’d have feature flags, and versioning in a build (installers can help with this, versioning dlls is a must as well for support) that QA can choose the version set they want deployed

Can your developers run the part of the system they are working on locally?

  • Local Database
    • You need to be able to create the db with seed data quickly and reliably
  • Do you have a complex environment?
    • Multiple services?
    • External dependencies (CRM, Sharepoint, Redis Cache, others)
    • Can you simplify?
  • How long does it take to get a new developer up and running.
  • Do you need to invest in more documentation of the system and setup?

Move DB into source control with dacpack?

Can you create environments with scripts?

  • Local development (get new developers going quickly, not taking days and having different setup then each other/environments)
  • QA/Test/Demo/Prod

Breaking up monolithic systems into smaller pieces?

What makes the most sense for your team?

How do you achieve always being deployable?

There are many options, again avoid paralysis and start with some small experiments or where you’d most help the company’s goals.

My generic suggestion

It really depends on your team/application/environment, but here is my blind recommendation to you.

  1. Create a Gated Check-in build
    1. No one can check-in if it doesn’t build
    2. require work items
    3. require Pull Requests - get people reviewing each others’ work
    4. Run unit tests
    5. Create an artifact to pass through the build pipeline
    6. 10 minutes or less
    7. Keep the code always deployable
  2. Create a Pipeline
    1. Publish to a test environment
      1. Run integration tests
      2. Run UI tests
      3. pass the artifact along if all passes
    2. Integrate all the services or pieces together
  3. Get the environment running locally easily
    1. local databases
    2. script all that is necessary
    3. add in DB to source
  4. Make environment creation easy
    1. testing a branch should be easy
    2. Azure and virtualization makes this easy in some cases (but watch your costs)
    3. Containers might be an option here
    4. This includes DB creation from scripts with the minimum amount of seed data
    5. make it throw-awayable
  5. Evaluate your usages of branches
    1. Are they adding value?
    2. How much time is spent in merging?
    3. Do you have errors due to manual merging issues?
    4. Do you know what version of code is in what branch?
    5. Can you reduce 2 branches into 1, then eventually go TBD
  6. Add unit tests here’s help
  7. Move to Git
    1. Git helps a lot with branches
  8. Feature flags/toggable
    1. hide in-progress work
    2. different then hide/show features from users based on user/feature set
  9. Integration tests and UI tests

Ensure time to work on this

All this identifying will be for naught if you don’t get time to work on things. It’s necessary to have management’s help in securing time. You may have to make a case in terms of technical debt (how is the current situation slowing everything down?). Show your goals and plans to improve.

What about X?

Feature flags for the full stack?

more commits/pushes = more chaos?


“My concern is that I think that there would be a noticeable drop in productivity as we change processes. We’re already constantly behind, so I’m not sure it it will be accepted by management.”

My team isn’t ready or doesn’t seem to have desire to change or learn new things

A hypothetical

More Resources

This is my collection of links. Of course, there are many more.

Side Note

I considered naming this “I want to be done merging code manually, but I don’t know how to get there” to mimic my I know I should be Unit Testing, but I don’t know how or where to start - presentation :-).

comments powered by Disqus

Please consider using Brave and adding me to your payment ledger. Then you won't have to see ads!

Support me and download Brave!

Use Brave