I have a 1966 Chevy truck that I’m working on restoring. Its electrical system includes 12 volt lighting, an AM/FM radio, and ignition system with battery, generator, and mechanical distributor with a CAM-driven set of points to control spark plug timing all connected with ~150 yards of wire. Compare that with today’s vehicles (electrical or otherwise):
- 2-3 miles of wiring
- Hundreds of electronic control units
- Hundreds of millions of lines of code
…a dramatic change in 50 years. In the past, integration between these electrical, mechanical, safety, and other systems was performed by a few experienced lead “systems engineers” who relied on their experience interacting with lead domain engineers to make sure everything worked together. They typically documented the product with system and subsystem spec’s to communicate with each other across the domain silos generating large amounts of paper in the process. The resulting product structure was predictable by Conway’s Law, with the finished product mirroring their organization structures.
Conways Law: Organizations build products that match their organization/communication structures.
“Systems engineering is a transdisciplinary and integrative approach to enable the successful realization, use, and retirement of engineered systems, using systems principles and concepts, and scientific, technological, and management methods.” –INCOSE
As the number of multi-domain products increased, INCOSE (International Council on Systems Engineering) and other professional societies formed and started defining “What is systems engineering?” (in my own layman terms its defined as the engineering of the combinations of electrical, mechanics, software, etc.) and documented a standardized systems engineering process in various systems development standards like ISO-15288, EIA632, IEEE1220, the Systems Engineering Handbook, etc. which organizations implemented inside their existing organization silos with each silo automating their part of the process. This created islands/silos of information that automated the “spec-build-integrate” product development process but didn’t solve the cross-silo communication problem. So, you are still discovering integration problems when you attempt to bring all the physical pieces together late in development. If they escape, you have a recall and headlines like this announcement from the aerospace industry in the Guardian Newspaper about the Japanese aerospace industry canceling the Spacejet Regional Aircraft program.
“After years of delays, Mitsubishi Heavy Industries admits building Japan’s first homegrown passenger jet was too difficult and probably not viable” (Feb. 8, 2023) The Guardian
“Some test flights were aborted because of air conditioning defects and other software problems, and the delays meant revisions to the original design were required.”
They will write off $7.6B on the failed program
Or this example from the automotive Industry:
This infographic from RecallMasters.com (yes there is an industry built around managing recalls) summarizes the recalls from 2022. Per National Highway Traffic Safety Administration (NHTSA.gov) there were ~35 million vehicle recalls in 2023 which according to AlexPartners consultancy cost ~$17.5 billion (at ~$500/recall/vehicle). If you look at annual reports of automotive companies you will find they are carrying $113 billion in warranty reserves (2.5% of revenue) on their books to deal with the problem.
It’s not just in the aerospace or automotive industries that this is happening, if you’re working in any complex product industry, you can relate as we continue to develop in silos and integrate on the right-side of the development process, discover the integration problems late in the development cycle (hopefully before shipping) and start the cycle over. This is why you actually plan for and will spend half or more of your project money, time, and resources (if you’re lucky) on getting everything working together. (Don’t believe me? Go ask your program manager how much pad they put into their program plans for integration.) In the past you could survive the “heroic integration” experience by putting more resources on the job to hold schedule, but with today’s complex products (thousands of ECUs, millions of lines of code, et al…) you can no longer afford that.
We’ve done the research in nearly all industries; everyone has the same late-cycle system integration problem and is spending half or more of their program resources on getting everything working together.
So what do we do about it?
I’ll pick up this thread in my next blog post. Please join me for the conversation.
Mark Sampson
Systems Engineering Evangelist