Organizing “spaghetti” Software So It Can Be Easily Modified

As a software engineer, Dan Sturtevant SM ’08, PhD ’13 had jobs where making a small change to a codebase was easy — and jobs where a similarly small change would cause other, seemingly random parts of the codebase to break down or malfunction. Making these changes could remind Sturtevant what he liked about being a programmer, or make him feel like an idiot.

That experience still puzzled Sturtevant when he arrived at MIT in 2006, first as a master’s and later as a PhD student. Then, he stumbled onto a series of papers written by former MIT Sloan School of Management Visiting Professor Alan MacCormack that he thought explained his dilemma.

The papers described a new way to analyze the structure of large software applications to find problems not with single lines of code, but with the way code files interact and overlap. The idea formed the basis of a new approach to evaluating and improving code, and Sturtevant would go on to team up with the paper’s authors to found Silverthread.

In the ensuing years, Silverthread has gone from diagnosing hidden structural problems in codebases to solving those problems by making some of the world’s largest codebases more modular.

“If you print out a 100-million-line codebase and stack it, it goes about a mile and a half into the air,” Sturtevant says. “It’s impossible for any one developer to understand everything going on in this system — all of the interconnections and all of the ways if you change one thing it messes other things up. It’s incomprehensible, and a lot of organizations and companies fail because their software accumulates what we call technical, or architectural, debt.”

Technical debt is one of the more professional-sounding names developers have given to a problem that Silverthread’s research has shown plagues companies for years and leads to lower developer productivity, higher turnover, and lower employee morale.

Now, Sturtevant delights in going into organizations like the ones he struggled in, and solving problems developers knew they had but couldn’t quite put their finger on.

“The teams working in the parts of the code that have these problems are miserable,” Sturtevant says. “They are unproductive, fighting bugs all the time, and demoralized, working until midnight on Friday just trying to get stuff working — and the problem isn’t their fault.”

A lightbulb goes on

Sturtevant earned his master’s degree from MIT’s System Design and Management (SDM) program just as the economy was collapsing in 2008, so he decided to stick around.

“I said, I’ve enjoyed my 18 months at MIT so much that I don’t know what I’ll get out of a PhD, but I know if I’m in this environment for another few years it’ll be worthwhile,” Sturtevant recalls. “The community is so good, the opportunities are so good, that I knew it would lead to great things.”

Sturevant was a PhD student when he found MacCormack’s papers, which were co-authored with former Harvard Business School professor and Silverthread co-founder Carliss Baldwin.

“I read those papers and a lightbulb went off,” Sturtevant says. “As a software developer, in the companies where my life was hell, maybe I was working inside a part of a codebase that could be identified as problematic.”

As part of his PhD thesis, Sturtevant decided to go back to one of the companies where he struggled with the new approach, and discovered he’d been working in sections of poorly structured code — what he sometimes refers to as spaghetti architecture — all along. He also worked to quantify the impact of the problem, finding that not only were the software bugs worse in the places with spaghetti architecture, but also that those bugs were more likely to hurt customer experience. Developers working on that part of the codebase were 10 times as likely to quit or be fired.

When Sturtevant earned his PhD, he presented the results as part of a webinar series hosted by SDM. More than 300 people tuned in.

“Apparently it was a problem a lot of people had,” Sturtevant recalls. “Attendees were executives from huge companies. Afterward, my phone is ringing off the hook. I’m getting calls from large companies with a lot of MIT graduates who had come through the SDM program and were listening. They were all saying, ‘I have this problem too. Do you have a product I can buy?’”

Sturtevant founded Silverthread with MacCormack, Baldwin, and Michael Davies, an MIT senior lecturer and tech entrepreneur. One of the first MIT alumni Silverthread connected with worked at a large government agency.

“He said, ‘I bet this explains a lot of the problems with our code, I bet this explains why entire government programs fail, and I bet this explains a lot of the waste that goes on in software development for the government,’” Sturtevant recalls. “The government has some of the oldest and largest codebases in the world.”

Silverthread’s early work with customers diagnosed their problems using the algorithms developed by MacCormack and Baldwin. Today the company works to make spaghetti architecture more modular so it can be changed without creating insurmountable complexity and inexplicable problems.

Silverthread calls its tools the CodeMRI Suite, which includes the CodeMRI Portfolio and CodeMRI Diagnostic to assess the technical health and economic impact of software systems, and CodeMRI Care and CodeMRI Modernize to fix codebases.

Sturtevant describes good code architecture like LEGO blocks. Pieces can be combined, stacked, or otherwise moved around in a modular fashion. He thinks of spaghetti architecture as LEGO blocks that have melted together.

“If you think about a massive codebase with tens of thousands of source code files, large applications like Microsoft Windows or Google, many people work to create different algorithms and encode domain knowledge in the software,” Sturtevant says. “It’s the purest form of knowledge work, and it’s like collective poetry writing — except that a lot of the poetry is crap and it’s been built over decades and expired and rotted in different ways. By using automated techniques, we can uncover the hidden structure in this massive thing that no one can understand and then correct those issues.”

Modernizing software platforms

Silverthread is growing rapidly as it works with banks, biotech startups, automotive companies, and government agencies. Sturtevant says Silverthread’s tools are industry-agnostic and hold value for any large codebases that have been built up over time.

Ultimately, Sturtevant says the codebases Silverthread helps build are more dynamic, allowing companies to reap the full rewards of innovations like cloud computing.

“When your software is like LEGO bricks, you can understand the relationships between each part and each part can evolve independently,” Sturtevant says. “That’s a healthy system.”

Reprinted with permission of MIT News (http://news.mit.edu/)

Source: cyberpogo.com