One of the most requested tasks we are called into enterprises for is to help them adopt an agile methodology. Often these organizations are giants in their industry and have reached these positions using a cocktail of waterfall, V-model, and self-built processes that have resulted in billions of dollars in revenue. The end products they’ve produced range from sub-par to breathtaking excellence. Therefore the internal push often does not come from the executive level but instead mid-level management who are neck deep in trying to bring software development into the organization to embrace the internet of things revolution. They are not only working to change the methodology but also the culture of the company and transition from, one and done, to a platform that is consistently updated week over week, month over month, and year over year.
Historically automotive, home appliance, aerospace, and other industries that produce products with high price tags have operated in a multi-year long product cycle. Companies often did and still do customer feedback reviews at specific milestones, like 90 days after initial purchase. Outside of these engagements with a sampling of customers, mechanic visits, customer trips to the dealership, or other infrequent touch points, there wasn’t much of a feedback loop. Customers expected this, and due to the logistical, manufacturing, and regulatory wonders that had to be executed to deliver these products, companies developed their workflows around conservative practices that introduced minimal levels of risk from year to year.
Manufacturers focused on mechanical and electrical systems with little to no software integrated into their products. Then once development was completed, aside from recalls, or extremely minor tweaks once the product shipped, it wasn’t updated. It resided with a customer for 5–20 years and never saw a security update, feature enhancement, or optimization. Did they sell without any of these? Yup. Did they get positive reviews from customers? Yup. Did customers expect to pay for ongoing updates and enhancements? Nope.
This is where we often see the disconnect between the two primary segments we interface with at customers who are looking to transition to agile. On one side are mid-level managers and their software teams who often have a background in consumer electronics or web apps and have a culture of building, releasing, breaking, and iterating, continually bringing new features to customers. On the other are upper executives who often have a background from within the organization itself and have come up through a time where mechanical and electrical were king to the customers and the company. They know full well the complications that can come from breaking anything in a regulatory heavy industry and rightly hold concerns ranging from security to recalls. Both sides are intelligent and capable and are trying to make their products the best they can be.
Adopting The New Process
This is where we often enter the fold. Many of our customers believe the tool choices or the process definition will be difficult and it can be. We use our experience from rolling out implementations at various size organizations as well as in our internal development activities to help pick the right tools for the teams or organization we’re assisting. We then assist with training and getting people up to speed on the selected toolchains. Teaching them the discipline necessary to ensure complete traceability, from requirements, to the lines of code that fulfill them and the unit tests that validate them. The real difficulty comes with redefining much of the existing culture, one that is built around old methodologies and compliance requirements. For example, in automotive, there’s a safety rating process that is used to assign a ranking to each system in the vehicle. Automotive Safety Integrity Level B (ASIL B) basically dictates that there must be a well-defined development process (which we cover with our end to end traceability solution) and code must have 100% test coverage.
This creates headaches out of the gate. We understand the reasoning behind ASIL B (and the other requirements to the various ASIL ratings) but far too often find teams that are using this as the guiding light for their culture. If you look at it historically, standards like this make sense. When first bringing software into the vehicle many organizations were attempting to define blanket requirements where no previous standard had existed. So management needs to ensure ASIL B is met to meet certain requirements and of course who doesn’t like seeing 100%. On paper, it just looks good. This butts up against software engineers who laugh at the requirement knowing that 100% test coverage means nothing. Then the advocacy commences and we hear many of the same things:
One Side: We are only being paid/tracked to reach 100% test coverage and obtain ASIL B. If we meet the requirements set by the customer we’re golden.
The Other: 100% test coverage doesn’t mean 100% validation and will not guarantee fewer issues or security holes occur in production. It’s short-sighted to use this as a metric.
Both are valid points. If you’re not paid based on the proper criteria, then it pushes your culture to advocate for the wrong things. Although this can result in short-term gains for a company or an industry, it can have negative long-term consequences. Such as an industry-wide culture that promotes:
- Writing tests to only test happy path scenarios
- Writing tests only to the requirements that were defined before kickoff of the project
- Not preparing to push updates, monitor the performance of your application in production, and iterate on your tests and functionality
- A higher percentage of manual tests executed by validation teams
- Higher cost when an issue is found or worse when a recall occurs
Yes, surprisingly all of this relates to being agile. The point is if upper management doesn’t place funding or structure their proposals to customers properly and shift what they are marketing or advocating on, the mid-level management and engineering teams are incentivized not to follow an agile methodology and fall more and more into a waterfall method. An example would be in a proposal stating something like:
- Shall be ASIL B Compliant
As opposed to:
- Shall continuously iterate over the lifetime of the project to identify and validate against happy path, malicious inputs, expected outputs, known scenarios, and issues identified through the usage of real customers based on what can be gathered by application performance monitoring solutions.
In the first example, revenue correlates to only achieving 100% coverage for the initial requirements defined in the specification. In the second we shift to an iterative process that will require ongoing funding for the given solution and sets the expectation that there will need to be development beyond the initial launch of the product. If you’re a supplier, making these shifts might mean you need to work with customers to educate them on why doing this will be advantages to them. If you’re a mid-level manager in an organization building software, you may have to coach upper management, purchasing, and sales on these new needs. Otherwise, leadership will continue to see these elements as binary objects that have either been accomplished or not. Remember software is not a widget.
Benefits of Shifting
Once these expectations shift, it becomes much easier to facilitate an agile development process. By incentivizing to prepare for managing product updates over its lifecycle as opposed to when it goes into production, it pushes teams to:
- Build a platform that can be reused and updated over time rather than throw away code that only meets the existing requirements
- Create tests for the platform that continuously improve on letting fewer issues or security holes into production
- Not write tests for the sole purpose of increasing coverage percentages and checking boxes
All of which helps all parties by:
- Setting the groundwork for customers to be more involved throughout the process rather than defining a fixed scope up front that cannot change without a significant amount of overhead
- Facilitating a better end product for end users and improving relationships
- Reducing the need for costly recalls
- Reducing negative customer reviews
There are always exceptions, and this scenario doesn’t apply to every company. Some are entirely on board and ready to jump in head first. Whether you’re at a company that is working through this now or thinking about doing it in the future and need help with choosing tools, building out pipelines, or need an outsider to help be an unbiased set of eyes we would love to help (our contact). Our goal at Auklet is to help you solve problems with your software and it all starts by creating a robust foundational process. If you’re already agile and need an over the air update solution and APM tool for your continuous integration pipe check out my post on OTA solutions and our edge first APM solution, Auklet.