DevOps will fail!….Unless.
‘Is it a bird?…..Is it a plane?……NO. It’s DevOps’!
The location: The high tech looking Evoluon centre in Eindhoven. At least it was high tech looking in the 60’s with its Spaceship design. But technology changes rapidly, and what was high tech quickly becomes out of date. Just as our old traditional models of ‘waterfall projects’ and ‘SILO based IT structures’ was good in one era, but has now become out of date as we enter the era of digital transformation.
The Topics: The fast pace of technology change and the need for speed has prompted a shift in the traditional SILO focused IT capabilities and a shift in the way we manage ‘projects’, ‘teams’ and ‘skills’. DevOps is the latest buzzword. But what is DevOps and what does it mean to the way we work? In a DevOps world ‘Team Work’, one of the main topics of the conference, becomes even more important. It means breaking down traditional barriers and fostering a culture of end-to-end working.
DevOps will fail…..unless!
This was the theme of the workshop organized by ImproveQS and GamingWorks at the ‘Competence Development Day’ for a large manufacturing organization in the technology branch. ‘Why will we fail’? And ‘What is the ‘Unless’’? These were the questions the delegates wanted answers to in the workshop. We used the Phoenix Project business simulation game to ‘walk through’ some scenarios to explore issues many organizations currently face and how DevOps principles can be used to address these challenges. We also showed how a business simulation game is a powerful way of bringing end-to-end stakeholders together to explore these topics and challenges, and to agree concrete improvement steps in the DevOps journey.
We had one and 1 half hours to create some practical learning takeaways!
These quotes at the end of the session sum up the team findings:
‘It’s AGILE!…’
‘It’s LEAN!….’ ‘
NO!….It’s a MINDSET!…’
‘It’s a question of pick & choose what works’!
‘…Let go of all the framework dogmas and JUST DO IT!’ ‘………TOGETHER’!
The Phoenix Project business simulation
The Phoenix Project business simulation is a dynamic, classroom based, business game based upon the book ‘The Phoenix Project’. The simulation is based upon ‘learning-by-doing’ or ‘experiential learning’. A team of players are assigned roles and responsibilities and placed in a realistic environment in which they must effectively communicate and collaborate and apply DevOps best practices in order to succeed.
What happened?
As we walked through a scenario in the simulation we created situations and asked the teams if they recognized these in real life. These were their recognized discoveries:
- It was unclear to the team how work ‘entered the system’ (some business leads went to Application teams, others went to the IT manager to get things on the change calendar, others go directly to specialists to get something done quickly).
- It was also unclear how work ‘flowed through the system’. There was ping-ponging back and forward seeking information, seeking authorization, seeking resources. Questions were going back up stream, work was waiting for decisions to be made.
- The IT teams were all busy ‘doing work’ – but it was unclear how all the changes and WIP (Work-In-Progress) related back to the business initiatives.
- It was unclear how to prioritize the work and who decided priorities between ‘Issues’ and ‘New features’. When WIP was filled up and choices needed to be made. The business was just focused on prioritizing ‘new features’ and had little time or attention for issues, whilst IT gave priority to fixing issues.
- There were continual disruptions to people doing their work because of lack of insight. ‘How far are we’? ‘Who is working on what’? ‘What do I need to test’? ‘Who has changed something’?
- The business threw their requirements over the wall and did not ask if and when they were needed for questions, clarification, testing or prioritization of work. The business just assumed everything would be done.
- The IT director did not have insight into what everybody was working on. New business work was accepted. Suddenly specialists were saying ‘NO’ to work, or ignoring things, or dropping existing work for other higher priority work – it seemed that priority was based on who shouts the loudest.
- The tester and the CISO had no insight into what needed doing and were involved too late. The CISO had no insight into security related issues or who was changing (or not changing) things that were vital to security concerns.
- Finally a business project was delivered, not all the work had been done, other work had been dropped, without notifying the business. Mistakes were made. The CEO was unable to report to the market his quarterly achievements. Targets were not realized. Nobody in IT knew what these goals were nor how what they were doing related to these goals!
Reflection:
After the scenario walk through we used the 3 ways of DevOps to explore what had happened and how we could use DevOps to solve these issues.
It was clear there was no flow. Defects were being passed downstream. At the start of the simulation the split between planned work and unplanned work was 50/50. Based upon their discoveries the team realized that If we were to play the next round the split would be MORE unplanned rework rather than new planned business work.
‘A key takeaway is to look at the split between planned and unplanned work in each of our teams and see how we can use DevOps principles to reduce the unplanned work’. Said one delegate.
The team discussed and experimented with visualization, value stream mapping, exploring how ‘feedback’ and ‘building quality in’ could be achieved. ‘Integrated testing’ and ‘test driven development’ with everybody assuming responsibilities for ensuring they never passed a defect downstream.
At the end of the fast paced workshop we ask delegates ‘what have we now discovered about DevOps principles that we want to take away and apply in our own work’?
Key learning – takeaways
I have clustered the takeaways in terms of the 3 ways of DevOps
First way – flow
- Collaboration is needed from all end-to-end stakeholder teams. BizDevsecQAOps.
- Important to go back and identify constraints in the flow, especially skills. The importance of 3:1 (3 people should be able to do the same task). 1:3 (1 person should be able to do 3 tasks in the team).
- Importance to visualize WIP and WIP limits as well as the amount of time for various activities, by various people to also identify the need for creating multi-functional teams and for sharing knowledge between highly skills and less skilled team members.
- The need for priority based decision making.
(IT Governance = Value creation. Value creation is ‘Benefits realization’, ‘Risk optimization’, ‘Resource optimization’. There is always too much work for the resources available. Resources must be balanced between (benefits: new business features) and (risks: technical debt & issues). The product owner and the team must understand the value to be realized and impact on value of risks to be able to effectively prioritize. - Value stream mapping to understand the end-to-end flow and to identify impediments and constraints to flow.
- A Visual management board is crucial, and must be made BY the team, FOR the team. It must enable the team to manage and prioritize their work AND their improvements. Visualize also the backlog of improvements.
Second way – Feedback
- Identify for each team the split between planned work and unplanned work. Identify where the unplanned work is coming from and agree how to make improvements to reduce this. (e.g integrated testing, direct feedback).
- Talk to the business to get a better understanding of value and to convince the business of need to change behavior.
- Sharing information end-to-end to help identify and understand goals, priorities, status, issues. Ask up and down stream ‘what do you need from me’?
- Visual management board needs to be built by the end-to-end team to support and enable their work and their decision making.
- Agree shared responsibility (Business & IT) for both value as well as technical debt. Prioritization of resources must be aligned to an agreed set of priorities.
- Stakeholder management is critical. End-to-end stakeholders.
- Start measuring capacity and resource usage to identify real WIP and WIP limits.
The third way – continual experimentation and learning
- It is clearly about culture. Communication & collaboration and the need to bring stakeholders together to agree, prioritize, improve, identify and agree how to deal with impediments, how to increase flow, how to remove waste.
- The visual management system should also show team impediments and improvement needs. These also need to be prioritized by the team and time assigned for making improvements.
- Mix approaches – Agile, Lean, ITSM. Choose what is needed from the various frameworks.
- Understand the Value stream and all the various stakeholders then map their tools onto the value stream. Are the tools aligned? What can the tools do to automate the testing and flow, and how can the tools be used to speed up flow and reduce hand-offs and mistakes.
- Just do it! Common sense approach to pick and choose what is needed for team from Lean, Agile, ITSM – avoid dogmatic approach to ‘framework as the goal’.
- All need to be involved in experimenting and agreeing a DevOps way of working. Getting people together in 1 room face to face to agree flow, constraints and iterative improvements.
- It isn’t an ‘implementation’ – it is about small, short continual iterative improvements that the teams are ‘able’ to adopt and apply.
- Start ‘base-lining’ testing. What tests are needed and how to ensure the appropriate test scripts are built. Test driven development and start integrating testing into everybody’s role.
The simulation session had helped bring end-to-end stakeholders together to explore common issues they recognize and to explore how DevOps principles could be used to help change both ‘mindsets’ and ‘behavior’. The team had captured a set of concrete improvements they wanted to take away……the question is, how will they now create buy-in within their own teams back at work? And how will they engage and involve the business product owners, confronting them as well with the need to transform.