I was eating lunch a few weeks ago and the leader of development at the company I was on contract for sat down and asked me how things were going. I had not slept the best and had just spent a lot of time merging from our Dev branch to our Main branch. We started talking and I was attempting to extoll the virtues of trunk based development and feature flags.
He challenged that by pointing out that it’s hard to find a path to move from TFVC (TFS Version Control) branches to a trunk based development flow. He’s tried trainings and other things with his distributed team in the USA and the world (10-20 developers). Unit testing hasn’t stuck with this team yet. He also has insightful questions about how to have a feature flag that spans the full stack (DB, APIs, UI, services, etc). This all got me thinking and lead to this article.
This company has a common approach of using TFVC with a Dev branch, Main branch and RC branch. They have a nicely laid out development process that requires reviews, demos and testing before merging it to Main. They have dependencies on MS Dynamics CRM and Sharepoint that make isolating code this way necessary.
There are already 1000s of articles on Continuous Integration DevOps, but I needed to take the time to write out what I’ve learned from reading, watching and observing. Hopefully, this brings value to the community and you the reader.
This transformation will take time (years), but we are looking for an incremental approach and continuous improvement.
I’ll use Donovan Brown’s definition “DevOps is the union of people, process, and products to enable continuous delivery of value to our end users.” and “I choose value over software”.
I use these thoughts as guiding principles to improvement. The Accelerate Book highlights what they’ve learned from years of State of DevOps Reports research and TBD along with the DevOps concepts are indicators of higher performing teams.
I see TBD as a core aspect of delivering software with less friction. Thoughtworks has an excellent article about TBD. Take some time and read this article. I won’t re-hash TBD more here.
Pardon my handwriting and Surface Book drawing skills, but I visualized the capabilities something like this. I don’t think you have to move from the center out in a linear fashion, but communication aka a part of DevOps is definitely at the center of moving towards TBD (and DevOps).
On page 145 of the DevOps Handbook Mr. Gruver shares his experience moving to TBD.
“However, trunk-based development required them to build more effective automated testing. Gruver observed, ‘Without automated testing, CI is the fastest way to get a big pile of junk that never compiles or runs correctly.”
You should definitely not start by deleting all your branches and getting rid of your process. You need to do some preparation work first.
Thankfully, The DevOps Handbook has part 2 - Where to start? to give us guidance. Here are my thoughts, many from this book, other resources, stories and experiences over the years.
We are a team, let’s stop blaming each other (even if we don’t say this out loud) and work together. That means getting managers, developers, Ops, QA and more people together and talking. Buy some pizza have some fun and identify how to improve together.
I highly recommend starting with The Phoenix Project as an introduction to the DevOps ideas. Then get The DevOps Handbook to use as a reference and read together. Accelerate is also a great reference. I’m thankful for Gene Kim and others for putting these books out in the last several years. Continuous Delivery by Jez Humble (at least the first 5 chapters) came out before the others and is a foundational book as well. There are many articles, talks and videos as well, that I’ve referenced at the end of this article.
Watch how Spotify organized their teams and delivers software. This was eye-opening for me when I watched it the first time.
Don’t get stuck in learning paralysis. You can do while you learn. Create a hypothesis, do small experiments, record and learn.
As a developer, I can easily lose sight of the company goals while I’m focused on creating a feature, debugging problems that are reported, helping others, etc. Does you team know why you are building each piece of software and how it helps your company succeed in the marketplace (and/or serve others)?
Start by identifying the pain points you know you have. Ask your team. I’d bet you’ll have a number of things your team is aware of, but never have time to do it.
Chapter 6 of The DevOps Handbook says to “gain a sufficient understanding of how value is delivered to the customer”. “Conduct a workshop with all the major stakeholders and perform a value stream mapping exercise”. Figure out all the steps required to create value and identify where there are delays and wait times. Page 65 has an example diagram.
Can you put metrics on the delays? Examples: We spend 10% merging code. We spend 10 days of manual testing and 10 more days fixing issues before we can create a build to release. (see some good examples on pg 144 in the DevOps Handbook)
Put a number from 0 to 10 on each of the capabilities.
Where can we automate (environment creation, database seed data, database schema updates, unit testing, UI testing)?
Where can I start adding tests? I have a full talk about this that I named I know I should be Unit Testing, but I don’t know how or where to start
What are the major bottlenecks in the flow of your work? In the Phoenix Project everything seems to always come back to the same guy. Phoenix Project - 90 “Improvements anywhere besides the constraint is just an illusion” another great quote is “improving your daily work is more important than daily work”
Jeremy Likeness has a great article about goals. Where do you want to be in each of these capabilities in 1 week, 1 month, 1 year?
First: Do you have your code in source control? If not do this ASAP!
Get a gated check in build running?
Move your code to Azure DevOps?
Consider moving to git? (Note: TFVC is still supported and still a good option, I’ve just found Git to be much more enjoyable, pull requests much better then TFS code reviews and productive after getting over the learning curve. I recommend having someone available who has experience to help others out when they mess up a branch)
Is there a new project or small team that you could experiment with these concepts? Then you’ll have stories to share about wins and what went well and able to spread this knowledge to other teams.
Merge a Dev branch with a Main branch? How do you isolate in progress code and your QA from things changing all the time? - I think you’d have feature flags, and versioning in a build (installers can help with this, versioning dlls is a must as well for support) that QA can choose the version set they want deployed
Can your developers run the part of the system they are working on locally?
Move DB into source control with dacpack?
Can you create environments with scripts?
Breaking up monolithic systems into smaller pieces?
What makes the most sense for your team?
How do you achieve always being deployable?
There are many options, again avoid paralysis and start with some small experiments or where you’d most help the company’s goals.
It really depends on your team/application/environment, but here is my blind recommendation to you.
All this identifying will be for naught if you don’t get time to work on things. It’s necessary to have management’s help in securing time. You may have to make a case in terms of technical debt (how is the current situation slowing everything down?). Show your goals and plans to improve.
Feature flags for the full stack?
more commits/pushes = more chaos?
“My concern is that I think that there would be a noticeable drop in productivity as we change processes. We’re already constantly behind, so I’m not sure it it will be accepted by management.”
My team isn’t ready or doesn’t seem to have desire to change or learn new things
This is my collection of links. Of course, there are many more.
I considered naming this “I want to be done merging code manually, but I don’t know how to get there” to mimic my I know I should be Unit Testing, but I don’t know how or where to start - presentation :-).
Please consider using Brave and adding me to your payment ledger. Then you won't have to see ads!