Why Software Won’t Save You
Why do your phone and your computer receive software “updates” every week or two but nothing works better?
The discipline of engineering is all about solving problems — and problems abound. The Earth is warming at an alarming rate making vast stretches of land uninhabitable. Leaving the Earth for places unknown is impractical for most earthlings. Data is more readily available than ever before and yet the truth is harder to come by. A cell phone costs a thousand dollars and only works for a few years. There are lots of problems and we may logically deduce that the best way to address such problems would be by solving them.
Solving a problem requires that it first be understood. Understanding a problem is something we can get from a scientist; but, when we want to solve the problem, we call an engineer. We use science to better understand our world. Scientific method begins with a confused puzzlement and refines it into a clear explanation. Engineering also does this in order to understand the root cause of the problem. Then, using a method that goes beyond science, the engineer will divine the resolution to the problem and implement it.
Scientific method is of great value and has yielded an understanding of our world that goes beyond mere reason. String theory, thermodynamics, organic synthesis and set theory have contributed to our understanding of the nature of reality itself. Scientific method is well understood and while the actual execution is very difficult, the method is fairly straightforward: observe, hypothesize, test, repeat. If repeated attempts to test the hypothesis fail to disprove it, the hypothesis may become a theory. A theory is a hypothesis that is true without likely dispute.
Engineering method always starts with scientific method; but, with the root cause assessed, we are done with science. The engineer now imagines the structure of worlds without the problem, relates those worlds back to the problem as understood and muses. The musing is a key part of the engineering discipline but it is also very personal. The engineer must assess the wide array of actions or devices that may resolve the root cause and solve the problem. This is not a procedure. It is a method. There are three common manifestations of this method.
- Integration: Everyone knew about the parts, but the engineer puts them together to derive a new capability. Example: the diesel engine
- Revelation: Detailed research and imagination reveals something no one else knew. That knowledge is then exploited. Example: the airfoil
- Persistence: Out of many possibilities, every one was tried until finally something worked. Not as noble as the others but sometimes appropriate. Example: the light bulb
In the field of software development we have similar examples. In 1984 engineers at Sun Microsystems invented Network File System. They realized that the new network protocols and the new disk storage systems and the new inexpensive memory could all come together to allow a user to read and write a storage system a thousand miles away as if it were plugged right into their desktop computer. They integrated these apparently unrelated technologies and proved the Sun motto: The network is the computer. In that same year, advanced math and ingenious analysis allowed Radia Perlman to develop the Spanning Tree Protocol (STP). Her revelation and the resulting algorithm is now used to flawlessly move information on the Internet from the source to the correct destination. The inverted index and the trigram search are elusively simple algorithms without which Wikipedia and dictionary.com would not be practical.
There is no procedure defining the process with which these technologies were developed. The method is what we call “engineering.” Scientific method was used to trace the appearance of the problem back to the root cause. With that known, engineering method kicked in. There was musing and puzzling and testing and detailed analysis of options occupying days, nights and weekends until finally the solution was discovered and the world saw one more problem resolved.
And Then It Stopped
These engineering solutions that are the basis of the Internet, TCP/IP, HTTP, mass storage and clustered computers, were developed prior to the year 2000. What has been produced since, with a few exceptions, are fairly simple variations on those themes. Amazon and the last online vendor you visited used the inverted index and map-reduce combining information about you and your recent purchases to recommend you purchase the same thing you just purchased, over and over again for months. Your cell phone company sends you repeated updates, not to add new capabilities but to assure that your cell phone will not be useful in another few years. Dr. Perlman’s STP has not been much improved in the twenty-first century.
Economists who pay attention to such things tell us that innovation in the United States hit a peak in the 1970s and has declined steadily since. Both hardware and software have stalled. I can guess about hardware, but I am a direct witness to the failure of software. Software isn’t a minor player in this system. Almost every interesting device we use is driven fundamentally by software: software developed using a common step-by-step procedure — a procedure known throughout the industry as “Agile.”
Agile is an attempt by the software industry to eliminate the strange unpredictable profit-threatening uncertainty of musing by pretending that engineering is actually just a simple step-by-step procedure. By being a simple step-by-step procedure, the concern that the problem might be insoluble never arises. The possibility that profits may be threatened by unseen variables vanishes behind the illusion of the predictable schedule. The threat of failure against one’s competitors diminishes because all of the competitors are enjoying the same delusion that innovation is just not needed since engineering is simple. It also means that the workers developing the software are cheaper since they require no special skills, just the ability to follow instructions. This is why Google is training elementary school children to write software. They will be the cheap software labor of the simple procedural future.
The most popular procedure by far in the Agile universe is called Scrum. In a Scrum shop, all workers toil in a single large room without intervening walls. The open-plan office, a throw-back to the crowded desk farms of the 1930s, is proudly touted as a modern innovation improving communication and quality. Serious studies of such office environments demonstrate the opposite. In Scrum, there are three types of workers. A product owner communes with the customer to find out what they want. Their desires are translated into short documents called stories which are written on recipe cards or their equivalent. A scrum master runs daily short meetings with the developers who each take a story and within no more than two weeks, write the software and the tests that satisfy the request in the story. That is how simple modern software “engineering” is.
Any story that cannot be resolved in two weeks (or cannot be broken into multiple stories each resolvable within two weeks) will not be attempted. A story like “Come up with the protocol upon which we can build an Internet” will not be attempted. A story like “Come up with a way to make a storage device in California act like it is on your desktop in Cambridge” will be reviewed in disbelief. These are not solutions that customers are asking for! Why, the customer isn’t even aware that there is a problem!
Ubiquitous Agile
Scan the job openings in the local paper or on LinkedIn. Every software opening brags that they are Agile. Let us be clear, agility is always good. Agility understands that the specifics of any problem may change over time and the solution must be designed to address the unpredictable variations common to the real world. This simple fact of real engineering was corrupted by a team of software celebrities in 2001 when they produced their Manifesto for Agile Software Development also known as The Agile Manifesto, a document describing the noble endeavor of engineering as (guess what…) a step-by-step procedure.
This was an amazing boon to timid executives who had spent the previous decades begging engineers for solutions to impossible problems only to receive iffy, wishy-washy and hopeful claims that the solution would be forthcoming only to eventually realize that the solution required a lot of esoteric musing with no certain completion date. Suddenly, by simply implementing Agile, the results would be certain and on a predictable schedule. Unfortunately this is the case simply because the Agile Manifesto extracts engineering from the system. It presents a pitiful simulation of a child’s view of engineering.
Disciplined engineering is fundamentally agile. It understands that the solution to the impossible problem (the only kind of problem requiring an engineer) cannot be resolved on an arbitrary schedule. In a disciplined engineering environment, the schedule is necessarily flexible. The engineer is given sufficient leeway to determine the root cause of the problem and then muse until the inevitable revelation identifies the solution. The developers are participating by testing theories and verifying assumptions. Once the solution becomes clear, the process really does become somewhat predictable. Even a process similar to Scrum could be used for the implementation of the solution. The wise manager learns to have patience with the unpredictables and confidence in her people. Eventually the discussion, whiteboard scribbles and musing will become a well-documented plan: an actionable plan which will gently enter the realm of predictable step-by-step implementation. Engineers and developers will turn to their keyboards and write the software. Yes, even within engineering method, there is eventually a procedure that yields the predictable result.
Unfortunately, this is not possible in the modern Agile world of software. In that world, an impossible problem is set aside in an instrument they call an epic. The epic is a no-mans-land. In the world of modern Agile, the real problem is not to be confronted. Instead, it is set aside for later — much later — probably never. Top management likes Agile because the purpose of their company is not to do good for the world or provide what the customer needs. The only purpose of their company is to keep them wealthy.
The Burden of Competition
After extensive experimentation in the technology of Neoliberalism, the wealthy have found that innovation has a negative impact on their bottom line. It is risky and in the long run it is much easier to routinely deliver simple software updates to their existing software which keeps the customers believing they are getting value without actually delivering any value. No innovation and no problem-solving means continued income with very little outlay. Such companies no longer need to compete because, since Reagan, no U.S. President has enforced the laws that require companies to compete. Instead, if a small company innovates in such a way as to threaten another company, that competitor is simply purchased and their innovation neutralized — frozen in patents and non-disclosure agreements.
This is why software will not save you. Unless we reintroduce engineering as a discipline, you will continue to see pointless trivial updates to existing marginally useful software without limit. As long as you think the latest mandatory update is helping you, the software moguls will continue to be wealthy without actually providing you with anything that you need. You may wonder what you need that is not currently being provided using modern Agile. Here are a few possibilities.
Lots of countries count national votes by hand from paper ballots. The U.S. has switched to largely unverifiable voting machines bought from highly partisan third-parties. Why don’t we move to an online voting scheme that anyone with access to a computer or cell phone could use to cast their ballot? We are told that it would be impossible to implement such a system due to security and access control concerns. Wait, it’s impossible?! This sounds like a job for engineering. It isn’t until everyone thinks it can’t be done that engineers start to get really excited.
People’s identities are being stolen all the time. Their credit cards are being hacked and their credit ratings ruined while they go through the time-consuming task of restoring their proper credit history. What if U.S. passports were provided to all citizens of voting age with a built-in ten-year elliptic curve security token. A central system managed by or regulated by the U.S. State Department would serve as the clearing house for all transactions based upon the citizen’s six digit one-time security token. When your credit card is used, your security token must also be entered. Without physical possession of the citizen’s passport, the transaction would be invalid. This would make identity theft fantastically complicated requiring the transfer of actual physical documents and the software and protocols required to assure reliable authorization would challenge any good software engineer.
With the addition of numerous small solar and wind power sources, the complications involved in delivering power for peak use and compensating for losses in old transmission lines make efficient routing of surplus power from areas of high production to areas of high need represents a supreme networking and algorithmic challenge. For instance, in an emergency could a brace of charged electric vehicles be tapped during a sudden summer heatwave? With the owners’ permission and the right software, yes.
We have information coming at us from all sides and sources; and yet, much of what is said or printed is balderdash. Any rational person could test any one of the random statements made on any network, podcast or blog by locating two or three independent authoritative sources, assessing their take on the issue and comparing that against the statement in question. The problem is that that could take an hour or so and there are a hundred other such statements to assess and that rational person still has to put in ten hours each day earning a paycheck. Fortunately, these statements tend to be formulaic — often primitive copies of instructions released to the news drones from their overseers. The sentence structure tends to be elementary for the target audience. It could therefore be parsed by software which could divine the essential claim and then undertake an automated search for evidence that could yield an assessment of the truth value of that statement. This could be done quickly and backed with evidence for any statement on a public medium.
An Innovative Future
So we have reviewed a few obvious cases where truly innovative software could greatly improve the world. Unfortunately, software won’t save us unless we reintroduce actual innovation into the system. We must supplement the step-by-step procedure with engineering method: the method that gave us the steam engine, solid state electronics, the personal computer, the Internet and all of the classic technologies that modern procedures are merely repainting and rebranding. Yes, people we consider “special” were behind these inventions, but these people were merely engineers and there are scores of engineers in the software industry trapped in the procedural cage of Agile. We must strive to encourage engineering discipline and true agility. The campaign to bring innovation back to software will be resisted by complacent capitalists who are happy raking in money while dishing out pablum to unsuspecting users and so this will not be easy.
There are rare companies using procedure for simple implementation and engineering method to actually compete and prevail. There are software engineers heading up teams of skilled developers who will implement the elegant solution to the problem everyone thought was impossible. Unfortunately, the Agile myth has been promoted so effectively that companies with excellent engineering records will still be tempted to give all of that up in order to follow the common wisdom so as not to lag behind such “innovators” as Google and Facebook. This ignores, of course, that most of their “innovation” comes from buying small clever competitors.
Keep this in mind from now on: Software that is updated more than once per quarter is not agile, it is broken. If software is to be part of the solution to the problems we face, we will all need to participate in the campaign to promote software to an engineering discipline. This will require a change to our economic culture. We must elect people to our government who will require competition by enforcing anti-monopoly laws and raising the top marginal tax rate to that of the 1950s. Under those circumstances, wealthy owners must cycle profits back into their companies so that they may surpass their active competition. This need to produce new and useful products for consumers who will again have a choice between active rivals will drive the need for innovation. Panicky management confronting real risks will rediscover engineering because that is how you beat your competition.
Buy products from worker-owned coops (http://usworker.coop) because, unwilling to outsource their labor, they innovate to survive. Purchase software and software-driven devices from small creative companies whenever possible, especially when those companies market themselves as “Innovative” as opposed to “Agile.” Demand stable software. Get a geek friend to install open source software (like LibreOffice on Linux) to your computer and learn how to use it. There is even ongoing development on an open-source cell phone (http://postmarketos.org) that is expected to operate flawlessly for up to ten years.
We have decided to shackle everything we own with software for reasons that are not always clear. That’s why your television and your telephone and your car have to “boot up” before any of the controls work. Believe it or not, software isn’t really required for everything; in fact, a lot of electronic devices work better without software but that is not the route we have chosen. For that reason, any electronically-driven hardware will require large amounts of software. For any problem to actually be solved, then, innovative software will be required. By this I mean, software is important. Encourage the innovation that is now in place and demand alternatives to the mundane software options you now have. When you find yourself becoming complacent: “Oh, but it’s so easy to buy everything from that one online store,” or “My smart phone does everything for me,” start rethinking your life, for God’s sake. Go to a bookstore and see what real browsing is like. Call your software provider and demand that they explain why the hell they insist upon weekly updates to your software. You should repeat this phrase in every call, “Why is your software so broken? Shouldn’t I get software that isn’t broken?”
Fight your complacency and change the culture. Join with me and others to make software part of the engineered and progressive future.
Julian S. Taylor is the author of Famine in the Bullpen a book about bringing innovation back to software engineering.
Available at or orderable from your local bookstore.
Rediscover real browsing at your local bookstore.
Also available in ebook and audio formats at Sockwood Press.
