Untangling the Knot: Enabling Rapid Software Evolution
Created February 2022
To quickly deliver new capabilities or take advantage of new technologies, organizations often need to first improve the modularity of existing software by isolating existing capabilities from surrounding code. Our automated refactoring solution recommends ways to refactor existing software, significantly increasing the efficiency of software evolution.
Software Is Never Done
Software-reliant systems need to evolve over time to meet new requirements, take advantage of new technology, or accommodate changes in everything from the deployment environment to business priorities. Many evolution projects start with a common problem—isolating software capability from its tangle of dependencies. Isolating software is key to working more efficiently with software and deploying it in new ways or to new environments. Common examples include
- decomposing monolithic applications into independently evolvable, deployable units (e.g., microservices)
- reusing capability in a different system or on a different platform
- improving migration of capability to the cloud (e.g., going cloud native instead of relying on lift and shift)
- isolating custom capabilities for easier replacement with third-party or open source alternatives
But all too often the structure of software becomes too complicated to allow rapid and cost-effective improvements. This is common in long-lived systems and even occurs frequently in newer systems, hampering the ability to independently evolve, migrate, or replace business and mission capabilities. Software refactoring is a known technique for improving the structure of software, but it typically relies heavily on slow, manual efforts when applied at scale.
To understand how large-scale refactoring is performed today, we surveyed industry practitioners. We found that large-scale refactoring is common and efforts often take hundreds of staff days to complete. Organizations also commonly forego desired refactoring efforts because they prioritize new features more highly or have concerns about the cost involved. However, foregoing refactoring often leads to a slower pace of capability delivery.
One of the contributing factors to expensive large-scale refactoring efforts is the lack of tools designed to solve that problem. To address this problem, the SEI built an automated refactoring assistant for developers that helps development teams improve the software structure for several common forms of change that involve software isolation. It solves project-specific problems with a semi-automated approach. Our goal is to enable refactoring to be completed in less than one-third of the time required by manual approaches.
Automating Software Isolation
The SEI's refactoring assistant analyzes code written in C#—and soon, Java—and recommends specific refactorings that isolate its functionality from its tangle of dependencies. In one proprietary example, a team estimated 14,000 hours of software development work alone—excluding integration and testing—to isolate a mission capability from the underlying hardware platform. If successful, our work would reduce the development time required to less than 5,000 hours.
Our prototype combines advances in search-based software engineering with static code analysis and refactoring knowledge. It is unique in its focus on project-specific goals as opposed to improving general software metrics. This goal is incorporated in genetic algorithms through fitness functions that guide the search to solutions for the project-specific isolation goal. In practice, our prototype recommends solutions that solve more than 85% of the problems on typical projects, suggesting that our effort-reduction goal is obtainable. The search algorithm relies on a representation derived from static code analysis and uses formalizations of refactorings as operators to explore many different combinations of code changes during search.
This work addresses a widespread, recurring need in software organizations. Because requirements and technology are never frozen in time, the need to adapt working software to new contexts will remain a common need across many software systems. Our solution has broad implications for moving existing software to modern architectures and infrastructures, such as service-based architectures, microservices, serverless architectures, cloud environments, and containers. It also addresses a pervasive research challenge in improving automated support for developers to perform architecture refactoring tasks.
Our prototype currently works on C# codebases as large as 1.2 million source lines of code with no problem. Support for Java code is expected in early 2022. We are also actively working on research to integrate additional preferences from users and bias recommendations toward solutions favored by developers.
April 11, 2022 Conference Paper
This paper introduces an industry survey that assessed which tools developers use in large-scale refactoring efforts and how well those tools support refactoring.read
March 15, 2022 Article
This paper describes a search-based algorithm that recommends a series of refactorings that isolate specified software from its architectural dependencies.read
November 07, 2021 Presentation
Our prototype tool combines advances in search-based software engineering with static code analysis and refactoring knowledge for cost-effective software improvement.read
December 15, 2020 Video
This project uses AI techniques to recommend refactorings that can improve the structure of software in significantly less time than it takes to manually refactor.watch
November 30, 2020 Conference Paper
This paper describes a vision for creating the next generation of industry-relevant tools for automating software evolution and automated refactoring at scale.read