Improving Organizations with a consistent flow

October 1, 2021    Process DevOps Agile Improving Organizations Productivity

Improving Organizations - Moving to a Consistent Flow of work

I’m working with an organization that has many clients with on-premise installations. This has been difficult to deal with as the various clients have different upgrade thinking. There are many different versions out there. The current upgrade process takes a lot of time and has a lot of risk. That pushes the organization to have less frequent, big bang deployments and “drops”. This is far from the ideals of DevOps and Better Value, Sooner Safer Happier of smaller batch sizes, deploy more often, “if it hurts, do it more often”.

We are working on Sooner, Safer and Happier Upgrades, but that is going to take a big investment. In the meantime, I helped lead discussions about moving towards a consistent flow of work to be able to have a release created after every sprint and validated by the end of the next. We hope to reduce that over time. Our clients might not always take the latest release, but that’s a people problem and we need to build trust, automate more tests, simplify and making the upgrades Sooner, Safer and Happier.

We are also reducing long lived branches, moving towards Trunk Based Development and other continual improvements that you can read about in my past articles.

Watching how Spotify organized their teams and delivers software was eye-opening for me when I watched it the first couple times years ago and their release train was an inspiration.

I created some diagrams with Mermaid Markdown Editor and have asked for permission to post them.

Hopefully this gives you some ideas, but this will depend on your team, organization, culture, complexity of your system, etc. Keep in mind that this is an improvement step for us at the same time as setting a vision for where we want to go.

Never stop improving! Find the next weakest link (biggest pain) and improve that next and keep on going.

Goals

  • have a QA validated release on a consistent schedule and reduce that time as we get better.
  • Improve flow time (idea/bug to providing value and used by the client)
  • Improve Quality and reduce regression issues
  • Reduce the need for manual drops/hot fixes
  • Reduce work and confusion about all the different versions and when the next release will come out
  • We still need to learn more about feature flag and use them.

Viewing the diagrams

If you’d like to view a higher resolution version, uou can copy paste my Markdown code into the Mermaid JS online editor. I’ve also had success using an extension inside of VS code to preview it.

Development Flow

This is the normal flow of work.

development Flow

Here’s the markdown for Mermaid JS

graph TD
subgraph Preparation
    A[New Feature or Bug - State: New] --> B[Investigation] --> US1
%% Product Owner PO investigates idea for a feature, gathers client feedback
%% Determine feasability, how to apply to the product beyond a single client
%% Creates User Story or Stories with requirements
   US1{Sizing: Multiple User Stories?} --> |Yes| US2[Create Epic] 
   --> US3[Assign a Feature Team and Channel] --> US4[Feature Flag or Feature Branch Needed?] -->C
%% bring work to the team, not people to the work
%% change our feature teams around to support this?   
   US1 --> |No| C
   C[Priorization and HLOE - State: Approved] --> D
%% High Level Of Effort - HLOE, with developer input
%% will it fit in a sprint, are they small enough pieces of work?
%% what are the deadlines and expectations?
   D[Sprint Planning -> Add to sprint] --> D2[Assign work - Dev and Testing]
   D2 --> PreImp[Pre-Implementation Meeting]
%% 1 or multiple, PO, Dev(s), QA, Activations/Support
   PreImp --> E{Enough Details?}
   E --> |No| F[Investigation] --> PreImp
end   
   E --> |Yes| Dev1[Development Starts - State: Active] -->
   Dev2{Feature Branch - FB?} --> |No| Dev3
subgraph Normal   
   Dev3[New local branch from master] -->
   Dev4[Rebase From master often] --> Dev4 
   --> Dev5[Coding, Testing, Documentation] -->
   DevPR[Create PR] --> DevPR1[Feedback, Complete PR]
   DevPR1 --> DevPR3[Post Imp Meeting] --> DevPR2
%% all involved with the item should be in this meeting
   DevPR1 --> DevPRDeploy[Deply to DEV01 & QA01] --> DevPRDeploy1[Run Integration Tests] -->
   DevPR2{More to do?} --> |Yes| Dev3
   DevPR2 --> |No| DevDone[State: Resolve, Test in QA01] -->
   DevDone1{QA verify in QA01} --> |Good| Verified1
   DevDone1 --> |Needs Work| DevDone3[State: Active, Record steps, Communicate with Dev] --> Dev3
   Verified1[Close] --> Verified2
   Verified1 --> Verified3
   Verified2[Add UI Automation] --> 
   Verified3[Deploy to Int05] -->
   Verified4[Run UI Automation Tests] --> 
   UITests{Passed?} --> |No| UITests1[Investigate]
   UITests1 --> |Regression| UITests2[Record, Activate] --> Dev3
   UITests1 --> |Not related| UITests3[Fix Test] --> Verified4
   
   Done --> Done3[Follow Release Process and deliver the value]
   
   Verified1 --> Done[Release Notes]
   Verified1 --> Done1[Demo or Highlight reel?]
  
end
subgraph Feature Branch - FB
%% move to different diagram?   
   Dev2 --> |Yes| DevFB[Create FB branch from master] --> DevFB1
%% FB = feature branch this needs more refinement and instructions
%% don't rebase if multiple developers
   DevFB1[New local branch from FB] --> 
   DevFB2[Merge from master to FB often] --> DevFB2 --> DevFB3
%% often = daily or more   
   DevFB3[Merge from FB to local branch often] --> DevFB3 -->
   DevFB4[Development, automated tests] -->
   DevFB5[Merge into FB from local branch for each working piece, feedback] -->
   ReadyForPR{Done with working piece?} --> |Yes| ReadyForPR1
%% smaller PRs = easier reviews and quicker feedback
   ReadyForPR1{Feature Flags in Place?} --> |Yes| PR
   ReadyForPR1 --> |No| DevFB2
   PR[Create PR] --> PR1[Feedback, Complete PR] --> PR2{More to do?} --> DevFB1
end

End of Sprint

end of sprint


graph TD
    A[End Of Sprint] --> B
    B[Create Release Branch] --> B1
    B1{Any in progress work?} ---> |Not Finished| Hide
    B1 --> |All Complete| C
    Hide[Feature Flag, Complete or Revert from release branch] --> B1
%% for large features, Feature flags or features branches can be created at the start of the work 
    C[Build and Deploy to Test - QA approval] --> D[Run Sanity Automated Tests] --> G
    C  --> ReleaseNotes[Create Release Notes and details]
    C --> E[Exploratory Testing] --> G
    C --> F[Test changes since last release] --> G
    G{Issues Found?} --> |Yes| Issues1a
    G --> |No| Stable
    Issues1a{Time to fix and test?} --> |Yes| Issues
    Issues1a --> |No| H[Feature Flag, Complete or Revert - Next release]
    Issues[Analysis, Fix with automated tests in master] --> Issues1    
    Issues1[Verify in QA01] --> Issues2
    Issues2[Cherry-pick from master into release] --> C
%% can we fix and test before the end of the testing period?
%% if too close to needing a relase, then     
subgraph Approved Release    
    Stable[Publish release] --> Stable2
    Stable2[Notify Activations/Support] --> Stable3
    
    Stable3[Upgrade all clients ASAP] --> Released[Delete feature flag code]
    %% even if clients can't update this often, the flow still works
    %% they may choose to release less often, but smaller releases => smaller batch size
    Stable3 --> Released2[Reflect, Learn, Celebrate]
end

Timelines

timeline

  journey
    title Development Timeline    
    section Sprint 1
        Develop, Test, Automate on the master branch : -100
        Create release branch : -100
        Start new sprint - Handle in progress items, help test : -100
    section Sprint 2
        Update INT Test Environment: -100
        Test Changes And Data differences: -100
        Stamp of approval & Publish installers: -100
        Create Release Notes with change notices : -100
    section Sprint 3
        Goal - Upgrade all clients : -100 

timeline moonshot

journey
    title Development Timeline - Ideal 1
    section Before Sprint
        Planning, Grooming : -100
    section Sprint 1
        Pre-Imp, Develop, Feedback, Test : -100
        Automate on the master branch : -100
        Create release branch : -100
        Finish in progress items, help test : -100
    section Sprint 2 - one week or less
        Update INT Test Environment: -100
        Test Changes And Data differences: -100
        Stamp of approval & Publish installers: -100
        Create Release Notes with change notices : -100
    section Every 4 weeks
        Goal - Upgrade all clients : -100 

Hot fixes and Drops

I leave this to the end, because I hope you don’t do this. Our team doesn’t trust the build/installers with past and present problems, it takes too much effort to upgrade, we have too many versions, the batch size is too large between builds, etc, that it’s become normal to create drops (build the code, copy paste dlls to servers or sql scripts). We really need to get out of this. It creates Frankenstein environments, is really hard to troubleshoot and a lot of coordination overhead.

We will improve and the consistent release + more automated tests and better quality are a few of many steps to reduce the need for drops. I hope they become a rare exception and that you never have to do one this way!

hot fixes and drops

graph TD
    A[Hot Fix Needed] --> B
    B[Analysis] --> BB
    B --> Drop
    %% reproduction, meetings, talking with Support/Activations/Pete/QA/developers
    Drop{Do we need a drop?} --> |No| I
    Drop --> |Yes| BB
    I[It's not a hotfix, move to backlog]
    BB{Reproduce in QA01?} --> |Yes|C
    BB --> |No|NoReproA
    NoReproA[Are you sure? Is it already fixed? Config/Data?] --> NoReproB1
    NoReproB1[Investigate. Why did this happen?] --> NoReproB2    
    NoReproB2{Start in master?} --> |Yes| C
    NoReproB2 --> |No| NoReproB3
    %% repo in environment with the client data
    NoReproB3[Find correct branch] --> NoReproC
    NoReproC[Development, automated tests in branch] -->NoReproD
    NoReproD[PR with feedback]-->NoReproE
    NoReproE[Cherry-pick to master] --> D
    C[Development, automated tests in master] --> D
    D[PR] --> E
    E[Verify in QA01] --> F
    F{What release branches need this?} --> G
    G[Cherry-pick to release branches] --> J
    J[Create drops] --> K
    K[Create Instructions] --> L
    L[Apply drop to internal test environment *no builds*] --> M
    M{Verify} --> |Yes| Client1
    M --> |No| Z(Reflect, Learn, start over)
    
    Client1[Drop on client test] --> Client2[Verify] -->
    Client3[CIPs, Apply to client prod] --> 
    O[Retrospective ]


comments powered by Disqus