The beginning of software development

At the beginning of software development, there was no good experience or thought to guide the operation of a development project. In the beginning, people identified some of the key assumptions of software development and mapped them to the understandable processes that existed at that time.

The initial assumptions are as follows:

Software development takes a long time

Software releases are not frequent, and it is difficult to make changes after the software is built, so make sure to do things right the first time. Software development requires many different and similar assumptions in the construction industry. It takes a long time for the building to be completed, and one cannot simply add a layer or expand the area after completion. Construction also involves many different professions, from designers to developers, to quality supervision and workers, electricians and plumbers. It can be seen from the naming of these roles that software development has borrowed a lot from the construction industry. The process followed by the construction industry is to divide the end-to-end project into different stages, and each stage of the process is handled by different professionals. This practice of assigning roles at each stage is conducive to making full use of costly human resources.

Agile development

People don't want to see their work being abandoned, so one of the initial thoughts is to decompose large projects into smaller parts and gradually iterate out solutions. This method allows people to have a more intuitive impression of the progress of the product over time, and allows the team to collect feedback to see whether the product has solved the user's problem. This early feedback allows the team to correct as needed.

But it is difficult to transition from a waterfall model to iterative delivery. Many organizations interpret iterative development as delivering solutions incrementally over time. The term agile is very flexible. By design, there is no agile process-its focus is on small iterations, continuous learning, and reactivity. Although the process is easy to follow and implement, these soft skills are difficult to master. Insufficient description of the process will allow people to bypass some of the more difficult changes and implement agile processes that are "effective for them" and give them a compelling name, such as Wagile or WaterScrumFall. The better transparency of agile development limits the risk of severe overruns, but it does not solve the fundamental problem that leads to the high failure rate of waterfall projects. There are still projects that will take longer than expected to build, and some projects will be cancelled because they cannot deliver the expected value of the product. In order to achieve the leap to iterative delivery, we need to change our assumptions about software delivery. We have to admit that we don't know what the product to be built is like in the early stage, so we need to replace the Big Design Up Front (BDUF) stage with incremental design. This gives the team the ability to learn during the project and change the design when needed. The challenge here is that the design phase is a key phase to ensure coordination within and outside the project team, and to summarize the various time estimates required to implement the solution. The processes of a company are not isolated, so changing one process will affect another process that seems to be unrelated. For example, the estimates obtained from BDUF will be required by other supporting processes, such as funding, supervision, resource allocation, and environmental allocation. The problem with adopting these "agile that works for us" is that we fail to recognize or resolve the conflict between our assumptions about software delivery and the existing situation.

Hypothesis

Software development takes a long time

Software releases are not frequent

After the software is built, it is difficult to make changes, so make sure to do things right the first time. It is impossible to know exactly what kind of product to build in the early stage. Therefore, the large-scale development is broken down into small parts, and the project is constantly learning, software development Many different and costly skill sets are needed. If we don’t know exactly what product to build in the early stage, how can we do things right at once? There are often big discrepancies between the large-scale software design and the final product-this creates an unexpected challenge that requires the design to be adjusted according to new and emerging needs. This shows that, unlike the construction industry, software changes are not difficult after construction, and there is no need for meticulous design in the early stage like physical buildings. We can simply add (software) the number of layers, change the height or enlarge the area, without having to overthrow and start over. Therefore, Hypothesis 2 is actually incorrect.

True agile development

If the design can be changed with the learning in the iteration process, then only doing the design required for each iteration will help reduce possible rework.

This is actually the most difficult step, because it not only requires changes in the working methods of some functional departments in the project team, but also changes in the working methods of the support departments.

We do not have enough architects to equip each team with one to guide them in continuous architectural choices. Architects need to define good practices, principles, and enforcement measures to ensure that the design meets the specified constraints, rather than defining a fixed architecture. Development engineers will be given more responsibility. They have to complete the task of giving the best solution to the problem, not just building the software according to the requirements specification. Architects no longer face the challenges of delivery, so their assumptions are often incorrect. The front-line developers often come up with better solutions when facing challenges. Without extensive design in advance, it is difficult to estimate time and cost. In the waterfall project, time and cost are determined based on estimates at the start of the project. In incremental development, the scope of the project will be adjusted according to the actual new situation encountered. Time and cost estimates can still be based on a rough understanding of delivery and team size, but because the scope of the project will change, the return on investment (ROI) and the benefits realisation schedule cannot be given in the early stage. These are not tasks that can be easily solved, but they can still be solved. I will discuss in detail how to solve these difficulties in a follow-up article. Now, let's update our assumptions.

Hypothesis

Software development takes a long time

Software releases are not frequent, and it is impossible to know exactly what kind of product to build in the early stage. Therefore, large-scale development is broken down into small parts and continuous learning during the project. Software development requires many different and costly skill sets. The team is responsible for designing under certain boundary conditions. The benefit of iterative delivery is largely because it can get early feedback on the product. In order to obtain these feedbacks, a working version of the software is required, which solves the customer’s problems, can perform demonstrations, and obtain actual feedback. In order to deliver such a software version, there are two main challenges to overcome-how to prioritize software development and how to test. In the traditional waterfall project, the most effective method is to build the product according to the components-build the perfect tire, perfect chassis, and finally assemble it into the car. Context switching is time consuming, so the fewer the switching, the more efficient the development time can be used. This component development method is deeply ingrained in the development team, even if they adopt agile development, the actual construction method is still the same. The problem is, if the customer wants to get from point A to point B, and what we show the customer is a completed wheel, the customer cannot give any valuable feedback. This will not solve their problem. We need to switch from component-based development to iterative development. This requires a ideological transformation, that is, focusing on customer problems rather than solutions. Only by not doing a lot of pre-design and acknowledging that we were not very clear about what needs to be built in the early stage, can this transformation be realized. But even so, the team still needs training, because it will affect the writing and development of the requirements document. Developer time is a high cost in software development. Is the product definition change worthwhile? In addition to the 31% failure rate, Microsoft's analysis pointed out that up to 66% of the functions did not achieve the intended commercial value. Considering the failure rate, ensuring that we do things right is far more important than development efficiency. We will come back to this point soon, because it is about to affect our next hypothesis. The second challenge is about testing. In order to get valuable feedback, we need to make the software work in front of the customer, not just as a demonstration. But this requires testing, a job that takes a lot of time and effort to complete. In the past, it was not cost-effective to automate test cases if only one test was performed on each major release. But there are two things that changed the test situation. First, the current software is required by all aspects of business, and the situation that used to take many days to release once a year has disappeared. Second, like the developer's time, the biggest waste in a project is developing the wrong thing. Automated testing allows us to test the customer's ideas more frequently, and the benefits it brings outweigh the costs. The automated test cycle allows the team to continue to develop "releaseable" code throughout the development process. 100% automated testing is a difficult goal to achieve, so there will be a small amount of manual acceptance testing before the software is released, but the quality is enough to verify the solution.

Continuous delivery/deployment (DevOps)

The advantage of continuous integration is that the code is available for deployment at any stage. In this way, even if the project is about to be suspended, we still have deployable code. However, like the aforementioned tests, deploying code is costly and time-consuming. Even if the software is ready, people still need some time to move the code between environments and complete the deployment steps. And the software may also encounter new problems, such as poor load performance. So you think the software is ready to be released, but it is not. The earlier a software problem is discovered, the lower the cost of solving it. So ideally, the team should find out whether there are non-functional requirements that need to be resolved after each iteration. Manual deployment takes a long time, so deployment should also be automated. Continuous delivery reduces the time it takes to release the code, but it may not be necessary to release it sometimes. For example, if you have only completed half of a feature, it is reasonable to release it after the second iteration. Understanding the entire deployment phase helps us find problems, so we should deploy more frequently. This introduces feature release control (feature toggles/feature flags). Develop new features behind the logo so that we can partially release the completed features, but leave them in a closed state. Users will not even be aware of these new features, but we can test that these codes will not cause damage to other parts. Feature release control has other benefits. Putting the function into a dynamic conversion, we can start A/B testing and multiple testing in the test environment, and then more accurately quantify the effect of the change, which allows us to more accurately verify our assumptions about customers. Make corrections if necessary. We can dynamically turn on or turn off a function, so user acceptance testing can be performed in the production environment, thereby saving time and cost. After these changes, let's review our assumptions. The team reduced the delivery time from a few months to a few days, to several times a day. We can't really say that software development takes a long time. If it doesn't take long, can we distribute professionals to different stages-how to manage multiple stages every day? We have not reduced the required skill set, so we still need architects, business analysts, manual test engineers, automated test engineers, developers, operations personnel, security, pipeline developers, and so on. Each person only contributes a small part, so we don't need to keep these people in a team, so the cost is too high. But again, we can't let them be independent of the team, because the response will be too slow. Like the architects described earlier, after determining the boundaries and restrictions, some roles no longer need to be deeply involved in the project. However, the rest of the project will have to take on more responsibilities. Such people are called T-shaped talents. They have deep professional knowledge in one field, and a broad understanding of other fields. Their extensive knowledge allows them to solve related problems in areas where full-time staff are not involved.

Hypothesis

Software releases can be fast and frequent

In the early stage, it is impossible to know exactly what kind of product to build, so the large-scale development is broken down into small parts, and the project is continuously learning. In continuous delivery, T-shaped talents are the most effective

The implementation team needs to be responsible for design, development and release under the defined boundaries. If we release quickly and iteratively, how to define the completion? If we have a set scope and time, the problem is simple. Now, we have to deploy the code at any time, what will prevent us from doing this? How do we know when the product is good enough? If software delivery no longer takes a long time, it opens up a world of experimentation and verification for us. Correspondingly, this also allows us to confirm whether the effect of each small iteration can meet the needs of users and our bottom line of business.

Business processes are replaced by mutually coordinated teams that must deliver the products required by the business. The team experimented and iterated the product concepts that needed to be developed to ensure that the set goals could be achieved.

This structure ensures continuous delivery to verify our attempts, but there are still some inefficiencies, because there may be a lack of consistency between the product/user experience/design and implementation teams. This isolation is usually classified as "business" (defining the scope of the product to be developed) and "IT" (doing the actual delivery work). This means that only half of the team is truly responsible for accomplishing the goal, and the other half is used as a delivery tool. We still need to change our thinking from deliverables to delivering business results.

Hypothesis

Software releases can be fast and frequent

It is impossible to know exactly what kind of product to build in the early stage, so the large-scale development is broken down into small parts, and the project is constantly learning, in continuous delivery, cross-team T-shaped talents are the most effective, and the team is delivered. Responsible for the product results, we will have a true cross-functional product team, mining needs, delivering products, and creating results for customers and business. Let the development engineers step in at the concept stage, brainstorming ideas, and providing input on the complexity of delivering them. Designers participate in the development team from start to finish, and can better understand the impact of design on customers. In order to fully migrate from the defined phase of the waterfall project to the continuous flow of the product team, continuous experimentation and innovation are required to eliminate possible obstacles. But everything is changing. The team continues to grow with product demand, including market, customer service, and operation and maintenance. The evolution of the process never stops. I will add one last assumption: the best process comes from continuous feedback and experimentation.

Software delivery implementation process

The process is amazing. When doing some repetitive work, a clear process can help us clarify what needs to be done and those known errors that need to be avoided. In large companies, process institutionalization is a more effective way of doing things. But sometimes this can be problematic. "Good processes serve us, and in turn, serve customers. However, if we are not vigilant, the process will become a problem. The process becomes a substitute for the results you want to achieve." The common problem with all processes is that they include process establishment. Initial assumptions and limitations. Over time, those initial assumptions that have been embedded in the process are forgotten. The problem is that when assumptions and restrictions no longer make sense, the process hinders our work. It’s not easy to make this transition, I will go into it in the next article