Agile Project Management with SCRUM - US

Project Methodology History

Thinking back over the history of software
- What is different about software
  - more malleable during development compared to items in the physical world
  - Fewer set rules on how things work (Ctrl-Atl-Del, Ctrl-x, Ctrl-c, Ctrl-v)
  - Closer to thought processes than to physical processes
  - Different limitations (gravity matters, programs should not be slow)
  - Different type of documentation

Examples from years ago
- Waterfall for developing operating systems
  - For a substantial time hold meetings with users group (Banks, large installations)
    - Decide on changes to be made in next version
    - Develop the requirements
  - Design all needed changes for a substantial time period
    - Approve the design document causing it to be (somewhat) unchangeable
  - Develop the new features
    - maybe 600 developers working over a substantial time period
  - Test the system
    - In some cases find out that there are problems with the system as designed
  - Release the new version to beta testers
    - Beta test users may find bugs that must be corrected
  - Release the software to all users
    - Go back and correct any bugs that are found
- Software Consulting
  - Price is given to customer perhaps through a bidding process
  - Customer and consulting requirements team draw up the requirements
    - Requirements are signed off by customer (with a signature on a document)
    - This defines what is to be built
  - Design team creates a design document
    - Customer reads (or not) the design document
    - Customer approves design document with a signature on a document
  - Development teams develops the software
  - Testing team tests the software
  - Product is released to customer
- These are called Waterfall methodologies

Originally project management in software
- Reflected manufacturing or construction project management
  - New physical products or buildings are envisioned
  - Requirements are gathered
  - Products are designed (by draftsmen, engineers, scientists)
    - Models may be built
    - various parts may be machined or assembled to try them out
  - Product testing takes place if in manufacturing (usually this would be final testing)
  - Products put into production (might be on some type of assembly line)
- This give rise to the waterfall concept in project management

Waterfall Project Management

The Waterfall project management methodology

Problems with the Waterfall for software
- Software is not a manufacturing or building construction process
  - In particular there is no manufacturing or assembly line
  - It is more like everything up to final testing
- Software development allows for a more malleable process
  - You cannot add a new story to a building
  - You cannot easily make a bridge 200 feet longer
  - You cannot easily add a extra wheel to a car
  - You can add a new feature to software
    - If the change is seen soon
- Complexity of software
  - Design may not be understood until software is partially built
  - Partially built software may cause the requirements to change
    - The user did not really understand what a requirement might entail
    - The user finds that a new requirement is needed
      - There will be 12000 items in a list and not 20
- Team structure and collaboration
  - Does design need a special team that is separate form development
    - Development takes the software architect's designs, looks at them
      - Places them in the round basket on the floor next to their desk
      - Merrily goes on developing what they think is needed

References

Wikipedia on the Waterfall Methodology

Unified Process

Recognizing problems with the Waterfall method the Unified Process is an alternative

Unified Process is Iterative and Incremental - Time boxed iterations that incrementally build the product
It became important partially because of Rational Rose (UML) and IBM's Rational Unified Process
Consists of a set of phases Inception, Elaboration, Construction, and Transition
- Inception
  - Establish justification
  - Project Scope
  - Use cases
  - outline architecture
  - Find risks
  - Preliminary project schedule
- Elaboration
  - Capture a large portion of system requirements
  - Address risks
  - Create UML use case, class, and package diagrams
  - Deliver a plan for the construction phase
- Construction
  - build the system using a set of time boxed iterations
  - Use UML diagrams in defining the system
    - sequence, activity, collaboration, state, and interaction
- Transition
  - Deploy to users
  - Get feedback
  - User training

Unified process
- similar in some ways to waterfall
  - Designs are made mostly up front in the process
  - Users see the product at the end during transition
  - The phases are similar to a Waterfall Process (but slightly different)
- Differences from Waterfall
  - The disciplines overlap unlike Waterfall i.e. business modeling and implementation
- UML is used extensively for designing and documentation
  - Documentation consists of several types of UML diagrams
    - In particular the sequence, state, activity, class, and package diagrams

References

Wikipedia on the Unified Process

Agile and SCRUM

Agile was proposed as an alternative to the Waterfall method. It employs some of the concepts used in the Unified Method.

Agile - These are software development methods in which the software solutions are arrived at through collaborative cross-functional, self-organizing teams. It features iterative development, early delivery, rapid response to changing requirements, and continuous improvement of processes.

Scrum - Refers to a particular Agile methodology as shown in the image below. It places a number of named concepts within the Agile methodology. The word scrum refers to scrum in Rugby where the team attempts to move the ball forward through a rugged collaborative effort (bruises and all).

Agile SCRUM can be said to have a number of organizing principles. These have developed over a number of years through experimentation with the processes of Agile. This list comes from the book Essential Scrum by Kenneth Rubin.

Prediction and Adaptation - Waterfall is a predictive methodology (predict up front). Scrum is an Adaptive methodology (adapt to changing requirements)
Validated Learning - validate important assumptions fast, learn through many learning loops
Work in Progress (WIP) - implement small sized slices of the project so that a large amount of work in progress is not disrupted by changes in requirements or designs. Focus on idle work and not idle workers.
Progress - measure progress by what has been delivered and validated not by how far we are along a predefined plan
Performance
- build in quality by moving forward but not hurrying.
- If quality is built in then there will be less costly rework required.
- Value is built in to a high level of confidence validated by testing.
- Have the minimal sufficient amount of ceremony, amount of documentation, and amount of sign off ceremony
  - The purpose of documentation
    - Part of the product, user manual, user guide, etc.
    - Capture important discussions, decisions, or agreements
    - Help new team members come on board
    - Regulatory requirements

In Class Exercise

Discuss project management methodologies that you are aware of. Where do they fit in terms of their characteristics of Prediction vs Adaptation, Validated Learning, WIP, Measurement of Progress, and Performance of the team.
Discuss any projects that you have worked on that you are willing to share and any problems and solutions that were encountered.
Note that Projects to be discussed can be either software, hardware, construction, or mechanical

References

Wikipedia on Scrum Software Development)

Wikipedia on Agile Software Development

Essential Scrum by Kenneth Rubin, Addison Wesley, January, 2015

Sample Project for use in the class

There is a company that wants to have an internal website to keep their employees and representatives apprised of the company news. They also want to encourage networking among employees so there will be sharing and forums on the website where employees can gather into networks, share information, and make friends.

Build a web based system that will:

Have the company news on a main page
- Allow editors to create and approve news
Have the CEO statement on a page
- Allow the CEO to have a weekly statement
- checked by someone as an editor
- Allow users to comment on the CEOs statement
Users can get other users as friends
- Must have a search methods to find friends
Users can put in their profile
- Name
- Profession
- Personal Statement
- Image
- Other??
Users can contribute to various forums
- have an approval process for the forums
- Forums have a structure that is easy to use so people can join the forums they would enjoy
Users and editors can add articles to the company news
- have an approval process for the news
- Put in a bad word recognizer so that inappropriate articles or forum entries are stopped
The website should not be overly dependent upon commercial off the shelf (COTS) products
- So it will unique to the company
Anything else that would encourage the company to be a good place to work

Agile Scrum Core Concepts

The concepts here are taken from experience, web site discussions, and books see the references for more details on website and book references.

Product Backlog - a set of user stories
- At the start there are Epic Stories that suggest the software to be developed
Sprints - Scrum uses short duration iterations to complete work
Sprint Backlog - Developer chosen Short Stories that are decomposed from Epics that can be finished in a sprint
Daily Scrum meeting - a short collaborative meeting of the team to plan the day's work
Users and stakeholders
Product Owner
Scrum Master
Development Team

Scrum Roles

Scrum does not follow the roles below that have been common during the last decades
- Managers
- business analysts for gathering requirements
- Software architecture team
- Development team
- Testing team
- Database administrators
- Network and System Administrators
These roles originated in the Waterfall method
- It is usually the hiring pattern for companies
- It usually indicates the personnel hierarchy and salary levels in a company
Scrum has the roles given below

Users and Stakeholders

Users are those who will eventually use the products
Stakeholders are those internally who have a vested interest in the product
- Stakeholders might also be representatives of those with a vested interest

Product Owner

the empowered central point of authority over the product's development
The person or team that interacts with the users and stakeholders and is their voice
- Similar to a Product Manager
The person or team that communicates to the development team what to build
- Ensures that the acceptance criteria are specified
- Their might be business analysts working on the acceptance criteria
  - The product owner ensures that those acceptance criteria are used in testing the product
Responsibilities of the Product Owner
- Economics - Make sure that good economic decisions are made
  - For any releases
  - For each Sprint
  - For the Product Backlog
- Planning participation
  - For the Product (i.e. the User Stories and Product Backlog)
  - For Sprint planning (Which Stories are to be in the Sprint Backlog)
  - For Releases (is the release ready what is the release date)
  - For the company portfolio (how does this product fit in the company portfolio)
- Groom the Product Backlog
  - Removing and adding items to the Product Backlog
- Define acceptance criteria and verify they are met
- Collaborate with all people involved
  - Collaboration being the keyword
  - It is easy to misuse the Sprint Backlog to force more work from the development team
Characteristics of the Product Owner
- Domain Skills
- People Skills
  - Collaborative
  - Communicator
  - motivator
  - Negotiator
  - Builds consensus
- Decision maker
- Accountable

In Class Exercise

Assume that the Product Owner is in the management hierarchy of the company, say at a fairly high level. Those in the hierarchy above that position set the dates for releases, the budget for the project, and the number of developers that can be hired. What can be the effects on the project and the Agile method? How can any problems be mitigated?

Scrum Master

Gets everyone to understand and embrace the Scrum values, and principles
Provides process leadership
Helps the organization develop their own Scrum approach
Coaches the Product Owner and the Development Team on process
Responsibilities
- Develop a solid relationship with and trust of the Product Owner
  - The Product Owner is often a high level manager with authority
- Understand the Scrum process well
- Remove impediments that inhibit the team
  - May need to use the authority of the high level manager
- Coach the Product Owner and Development Team
- Protect the development team
- Change agent
  - Needs to be able to change minds
Characteristics
- Knowledgeable
- Calm and patient
- Collaborative
- Transparent
  - No hidden agendas
  - Upfront and straight forward
- Communicative

In Class Exercise

Discuss problems that you can see with this position and solutions for those problems
What can happen when the Scrum Master is managed by the Product Owner?

Development Team

Plans each Sprint
Performs Sprint execution until "Done"
Helps Product Owner groom the Product Backlog
Holds daily Scrum
Helps in adapting the Scrum process

Team characteristics
- Cross functional
  - Members will have different skills
    - Software architecture
    - Testing
    - Development
    - Database
    - UI design
    - Network and system admin
- Diverse
  - Senior and Junior members
  - many viewpoints
  - Accepting of others
- Collaborative and self organizing
- Sufficient for completing the entire set of Sprint tasks
- Each member performs a broad set of tasks and also has a specialty
  - Software architecture might be a specialty
  - Testing, coding, and database work might be other assigned tasks
  - Teams members move between task types
- Long Lived
  - The team is such that members want to continue on the team
  - The organizational structure is devoted to keeping teams together
  - Loss of team members is a loss of domain and process knowledge

In Class Exercise

Discuss the issues (for and against) the separation of Software Architecture, Testing, and Development teams

Discuss the issues related to members of the team in terms of collaboration ability, self organization, testing, database administration, and system/network administration. How would any issues be mitigated?

Managers

Managers have concerns with implementing Agile
- 30% of respondents listed Loss of Management Control as an issue
- 30% of respondents listed Management Opposition as an issue
- VersionOne 8th Annual State of Agile Report

Does Agile effect the manager role? Yes and no
Functional Managers responsibilities in Agile
- Team formation, including setting goals, empowerment, and appointing team members
- Addressing team issues, such as badly behaving team members
- Nurturing, energizing
- Developing competence
- Functional leadership
- Coaching
- Team integrity
Scrum for small to medium projects does not use a Project Manager
- The duties are spread between the Product Owner and the ScrumMaster
- One thing to remember is that the Project Manager role varies company to company

In Class Exercise

Discuss the role of the Project and Functional Managers in both an Agile and non-Agile environment

What are the similarities and the differences?

Explain the role of the Project Manager as you know it

Product Backlog

The product backlog is a prioritized listing of the user stories that make up the product requirements

User Stories at the highest priority are of finer granularity

Image from heliosobject.com

Requirements and User Stories

In Waterfall requirements are detailed up front and stand alone
In Scrum requirements are negotiated through conversations
- Conversations happen throughout development
In Scrum requirements are often created as a set of User Stories
User Stories start out as Epics
- Epics are very high level descriptions of the product
- Epics are broken down before they are included in Sprints
- Epics usually would take longer than a Sprint's time interval
- Epics become on decomposition Themes, and then User Stories
  - Themes represent a partial breakdown of the Epics that is not at the level of a Story
  - Stories are of small duration that can be included in a Sprint, two to four weeks of effort

User Workshop

Often the Epics, Themes, or User Stories are written on 3 by 5 cards or on Post-it-notes
- Sometimes this is done in a User Workshop
Epics that are of the highest priority are broken down into Themes.
Themes are broken down into User Stories that can be implemented in a Sprint

For each card there is a:

Conversation among the development team, stakeholders, and product owners
- During the user workshop to bring clarity to the item
- During grooming of the Product Backlog
  - Break Epics into Themes and Themes into User Stories
- During sprint planning
- During development
Set of conditions of satisfaction created
- Allows Product Owner and Developers to say story is "Done"
- Can be placed on back of card during user workshop

Characteristics of User Stories

independent - they do not depend on each other
negotiable - they are place holders for later conversations
value - the stories are valuable to the users or customers
Can be Estimated - the story can be estimated by the development team
Sized to Sprints - can be implemented and "Done" during a Sprint
Testing - the story has satisfaction criteria that can be tested

Backlog Grooming

Grooming consists of
- User stories are updated as the Sprints occur
- New stories can be created as Sprints occur
- Epics are broken down into Themes and User Stories
- Stories priority can be updated
Grooming is done by
- the Product Owner
- With input by
  - Stakeholders
  - Developers
  - ScrumMaster

In Class Exercise

Assume that we are in a user workshop. Take the sample project given in the table of contents and create a Product Backlog using its description. This will be done as a group effort. Appoint individuals to act as the Product Owner, Stakeholders, ScrumMaster, and Developers

First create a set of Epic Stories
Prioritize the stories
Create a set of Themes from the first two highest priority Epics
Create a set of User Stories from several of the Themes

Discuss any problems encountered during this effort and propose solutions

Discuss the times that you have seen user specify requirements upfront only to later say

"This is not correct"

"I didn't really mean what I said"

"Read my mind and not my words"

"It is not what is needed"

References

Description of the Product Backlog

Images of Scrum processes

Estimation and Velocity

The question - How long with product development take?
- Requires estimation of size of the Epics, Themes, and Stories
- Requires knowing how much is done per Sprint
  - Velocity tells the story points or effort hours accomplished in a Sprint
    - Just a matter of record keeping
- Requires the length of each Sprint, i.e. two weeks

Estimate Size of the development effort

By T-shirt sizes (small, medium, large, extra-large)
By Story Points
- Some arbitrary scale is used
  - Fibonacci numbers 1, 2, 3, 5, 8, 13
    - Recognition that the larger the effort the harder it is to assign a size
  - Series of numbers 1 to 10 for example
    - Standard method of rating something
By ideal hours sometimes called effort hours
- This is the method used most often in the past
  - That is estimate number of person hours project will take
  - Account for time off
  - Account for meetings
  - Account for Scrum related activities, i.e. grooming the Product Backlog
  - An ideal hour is not the same as actual time spent

T-shirt sizes is very rough but could be applied to a portfolio of products
- Product A is extra large, Product B is small so lets work on Product B
Story points are good for Epics, Themes, and User Stories
Effort hours are usually used for the tasks of a User Story

Planning Poker for Estimating

First choose a scale to represent size, for example a Fibonacci sequence
The full Scrum team plays the "game" while estimating
Product Owner presents, describes, and clarifies Product Backlog Items
ScrumMaster coaches the team to help it apply planning poker
Development team generates the estimates
- Each team member has a set of planning poker cards
- For an item chosen by the Product Owner
  - Development team discusses the item and ask questions
  - Each team member selects a card representing their size estimate
  - All estimate cards are exposed at once
  - If all cards are the same that size is assigned to the item
  - If the cards are not the same the team discusses the issues and re-votes until consensus

In Class Exercise

Come up with at least two User Stories that pertain to development efforts that you are aware of

Decide whether they are Epics, Themes, or Stories
Decompose if they are Epics or Themes into Stories
Play planning poker to determine the size of the stories.
- Use the Fibonacci playing cards given in the zip file at the bottom of this document
  - They might also be handed out in class

Velocity

Velocity gives the size of the effort accomplished per Sprint
- Prior history is the best way to determine velocity
- Otherwise velocity is an educated guess
  - What most development teams do is guess at least at the start of Product Development
  - Often ranges are used early in the development process rather than single figure estimates

Questions to be asked

Does the organization have a record keeping method that can give velocity?
- Timekeeping system buckets in which developers put their hours?
- Hours kept track of by some other system?

The advantages of Sprints for determining Velocity
- User Stories are assigned to Sprints
- Completed Stories are kept track of
- Each story had a up front estimate
- Therefore Velocity during the Sprint can be calculated

Hard Problems In Development:

Getting accurate recording of development time. Sprints automatically give these.
Accounting for all of the other time: meetings, planing, breaks, education, mentoring
Applying these times in estimating future development efforts. Planning poker is intended to help solve this problem by using multiple opinions from developers and consensus among the developers.

Misusing Velocity

Velocity is a planning tool
It is not intended to be a performance metric
- Which team has the highest velocity is not a good metric.
- Telling a team that their velocity is not high enough is not appropriate.
- Misuse of Velocity may cause the teams to inflate their numbers
- Misuse of Velocity may cause the teams to write poorly structured code

Sprints

Image from http://www.jaftalks.com

Sprint Planning

Sprint planning occurs before each Sprint
Participants are:
- Product Owner
  - Gives a goal for the Sprint
  - Listens and gives input
  - Does not determine what is to be delivered
- ScrumMaster
  - Observes, facilitates, and asks questions
- Development team
  - Determines what it can deliver
  - Makes a commitment to deliver
  - ScrumMaster can challenge the items to be delivered

Development team activities

Determine capacity
- Capacity indicates the amount of work the team can do in the Sprint
  - Story Point methodology - dependent upon prior team story point velocity
  - Effort Hour methodology - dependent upon prior team effort hour velocity
Use a method such as Poker Planning to choose Product Backlog Items
- Backlog items chosen until capacity is filled
Decompose Product Backlog Items into development/programming tasks
Estimate the tasks in Effort hours
- Check that effort hours also match the capacity

In Class Exercise

Use the Product Backlog developed in prior activities to choose items for the Sprint Backlog

Chose how long a Sprint will be
Come up with a figure for available effort hours for each developer
- Usually these figures would come from prior effort hour velocity
- This gives the capacity for the Sprint in effort hours
Chose Backlog Stories by priority
Use the Story point estimates assigned to the story during grooming to fill capacity
Decompose Backlog items into development tasks
User planning poker with effort hours to determine task effort hours
- Other methods can be used

Sprint Execution

During Execution development team self-organizes
ScrumMaster participates as coach, facilitator, and impediment remover
Product Owner clarifies questions
Development team
- Manages flow of work
  - What work can run in parallel
  - What are dependencies between tasks
  - Who should do which tasks
  - Are there skills constraints
Daily Scrum
- 15 minute timeboxed activity
  - Are there issues blocking implementation
  - Discussion of possible dependencies
  - How to organize the work
Team uses best practices
- Automated testing
- Automated builds
- Code reuse
- Coding standards
- Common coding environment

Sprint Burndown and Burnup Charts

Image from timberwolfconsulting.com

Burndown Charts

give the amount of effort remaining in the Sprint
- either effort hours or story points per day
The chart shows if the Sprint is on track to finish correctly

Burnup Charts

These are the opposite
- Shows the amount of effort expended in the Sprint

Sprint Review

Focuses on the product
In a meeting the product is inspected and reviewed
Participants are:
- Product owner
- ScrumMaster
- Development team
- Stakeholders
Development team demonstrates the results of the Sprint
Discuss and adapt to changes needed

Sprint Retrospective

Sprint Retrospective focuses on the process
Product Owner, ScrumMaster, and Development team participate
Activities in Retrospective
- Set an appropriate atmosphere
- Identify insights
- Determine actions to adapt the process

References

Agile Scrum and Sprint

Sprint Artifacts

Technical Dept

The amount of time needed to correct faults/problems in a software system.
Causes of technical dept
- Development was hurried so code is poorly structured
- APIs to other systems change over time
- Design makes no sense over time
- Code duplication instead of reuse
- Insufficient testing
- Defects in the software
- Inexperienced team members
- Insufficient time, budget or human resources causing shortcuts to be taken

Most system gradually degrade in functionality as new features are added
Most organizations do not account for time needed to address technical dept
- Code rewrites
- Design changes
- Spaghetti code
- Developers using the shotgun approach to development (doesn't work try shooting it again)
- Object Oriented Code with too many classes and too much inheritance

Issues and Impediments with Agile

Concerns with implementing an Agile Initiative From VersionOne 8th Annual State of Agile Report

Lack of up-front planning 30%
Loss of management control 30%
Management opposition 28%
Lack of documentation 24%
Lack of predictability 23%
Lack of engineering discipline 20%
Dev team opposed to change 17%
No concerns 19%
Inability to scale 18%
Regulatory compliance 15%
Quality of engineering talent 13%
Reduced software quality 9%

References

Top 10 issues with Agile VersionOne 8th Annual State of Agile Report

References

Essential Scrum by Kenneth Rubin - this is a very good resource that paints a fairly complete picture of the Agile Scrum methodology. It can be found on Amazon.com

International Scrum Institute

Agile in Mechanical Engineering and Product Design

Lean and Agile Mechanical

Agile in Medical Device Development

Agile and Mechanical Engineers

Scrum in Mechanical Product Development

Challenges of Becoming Agile

Improve competitiveness with Agile