Arcade Game Maker Pedagogical Product Line: Architecture Evaluation Report
This report presents the results of an architecture evaluation of the Arcade Game Maker (AGM) product line software architecture, which took place in Clemson, SC, on February 5, 2004. This evaluation was performed by Luminary Software and followed the SEI Architecture Tradeoff Analysis Method® (ATAM®), a method for evaluating a software system's architectural decisions in light of desired system quality attributes.
ATAM evaluations have two main phases. In Phase 1, the evaluation team interacts with the architect and a few other key stakeholders (e.g., a project manager or customer/marketing representative). The evaluation team and the system stakeholders walk through all the steps of the ATAM, gathering information about the system, its important quality attributes, and its architecture. We begin analyzing architectural decisions in light of the quality attributes to uncover risks and tradeoffs. The evaluation team continues the analysis after the Phase 1 meeting, interacting with the architect as necessary to elicit the necessary information. This interaction typically takes several weeks.
In Phase 2, the evaluation team walks a larger group of stakeholders through all the steps of the ATAM and the output from Phase 1. With this larger group, important quality attributes are illuminated, and the architecture's ability to support those goals continues.
Due to the simplicity of the AGM product line architecture, Phase 1 and Phase 2 were combined. No non-technical stakeholders were involved at the latter phase.
The system stakeholders (architects, managers, developers, testers, integrators, etc.) participating in the AGM product line ATAM evaluation are listed in the following table.
|System Stakeholders Participating in the AGM Product Line Evaluation|
The ATAM evaluation team members and their assigned roles in the AGM ATAM evaluation are listed in the next table.
|Evaluation Team for the AGM Product Line Evaluation|
The Architecture Tradeoff Analysis Method (ATAM)
The ATAM depends on an architecture being suitable (or not) only in the context of specific quality attributes that it must impart to the system. The ATAM uses stakeholder perspectives to produce scenarios that define the qualities of interest for the system under consideration. Scenarios give specific instances of usage, performance requirements, growth requirements, types of failures, possible threats, and likely modifications. After important quality attributes are identified in detail, the architectural decisions relevant to each one can be illuminated and analyzed with respect to their appropriateness.
The ATAM steps are carried out in two phases. In Phase 1, the evaluation team interacts with the system's primary decision makers: architects, managers, and perhaps a marketing or customer representative. During Phase 2, a larger group of stakeholders is assembled, including developers, testers, maintainers, administrators, and players. The two-phase approach insures that the analysis is based on a broad and appropriate range of perspectives.
- Present the ATAM. Evaluators explain the method.
- Present business drivers. Appropriate system representatives present an overview of the system, its requirements, business goals, context, and the architectural quality drivers.
- Present architecture. The system or software architect (or another lead technical person) presents the architecture.
- Identify architectural approaches. The system or software architects present general architectural approaches to achieve specific qualities. The evaluation team creates a list and adds approaches gleaned during Step 3 or learned during their pre-exercise review of the architecture documentation (e.g., "a cyclic executive is used to ensure real-time performance"). Known architectural approaches have known quality attribute properties, and these approaches will help carry out the analysis steps.
- Generate quality attribute utility tree. Participants build a utility tree, which is a prioritized set of detailed statements about what quality attributes are most important for the architecture to support (e.g., performance, modifiability, reliability, or security) and specific scenarios that express these attributes.
- Analyze architectural approaches. The evaluators and the architects map the utility tree scenarios to the architecture to see how it responds to each important scenario.
- Present the ATAM. Evaluators explain the method.
- Recap of Steps 2 - 6 of Phase 1. Evaluators recap Phase 1.
- Brainstorm and prioritize scenarios. The stakeholders brainstorm additional scenarios that express specific quality concerns. After brainstorming, the group chooses the most important ones using a facilitated voting process.
- Analyze architectural approaches. As in Step 6 of Phase 1, the evaluators and architects map the high-priority brainstormed scenarios to the architecture.
- Present results. A presentation and final report are produced that capture the results of the process and summarize the key findings.
Scenario analysis produces the following results:
- A collection of sensitivity and tradeoff points. A sensitivity point is an architectural decision that affects the achievement of a particular quality. A tradeoff point is an architectural decision that affects more than one quality attribute (possibly in opposite ways).
- A collection of risks and non-risks. A risk is an architectural decision that is problematic in light of the quality attributes that it affects. A non-risk is an architectural decision that is appropriate in the context of the quality attributes that it affects.
- A list of issues (or decisions not yet made). During an evaluation, issues not directly related to the architecture may arise. They may have to do with an organization's processes, personnel, or other special circumstances. The ATAM process records them so that they can be addressed by other means. The list of decisions not yet made arises from the stage of the evaluation's life cycle. An architecture represents a collection of decisions. Not all relevent decisions may have been made at the time of the evaluation, even when designing the architecture. The development team knows about some of these decisions and has put them on a list for further consideration. Others are news to the development team and stakeholders.
Results of the overall exercise also include the summary of the business drivers, the architecture, the utility tree, and the analysis of each chosen scenario. All these results are recorded so that all stakeholders can verify that the results have been identified correctly.
The number of scenarios analyzed during the evaluation is controlled by the amount of time allowed for the evaluation, but the process insures that the most important ones are addressed.
After the evaluation, the evaluators document it, uncovered information, and the framework for ongoing analysis that they discovered. This documentation is the report in your hands. A detailed description of the ATAM process is provided by Clements, Northrop, Kazman, and Klein [Clements 02, Kazman 00].
Business Drivers Presentation
The AGM product line manager described AGM's business objectives for the AGM product line and identified the following product and architectural goals:
- Maintain low cost. Many of the products are intended either for AGM to give away via the Web or for customers to give aways at conventions.
- Offer the player an interesting game-playing experience. People who play games are easily frustrated and quick to scorn games that they believe to be inferior. The game must be sufficiently fast and realistic.
For a complete list of driving business requirements, see Arcade Game Maker Pedagogical Product Line: Business Case.
AGM's lead software architect presented the architecture of the AGM product line and identified the following approaches:
- a modified model-view-controller (MVC) front end
- a component-container structure for the model
- an event-driven error-handling mechanism
Several views are presented below. For more architectural details, see Arcade Game Maker Pedagogical Product Line: Architecture Documentation, Volume 1: Beyond Views and Volume 2: Software Architecture Views.
In the following figure, the GameBoard is the central feature of the architecture. It is a container that includes all the game's action. The StationarySprites are the obstacles into which the MovableSprites collide. Movement and collision are the two major actions that occur in the games. The second figure illustrates how obstacles are defined.
Specialization Hierarchy for StationarySprites
The utility tree (shown in the next two tables) provides a vehicle for translating the quality attribute goals articulated in the business drivers presentation to "testable" quality attribute scenarios. The tree contains Utility as the root node, which is an expression of the overall "goodness" of the system. For the AGM product line, the second level nodes were performance and modifiability.
Specific concerns are included under each quality attribute. These concerns arise from considering the quality-attribute-specific stimuli and responses that the architecture must address. For example, for the AGM product line, performance was broken into two concerns: (1) the total number of Sprites on the screen and (2) how many Sprites have to be checked for collisions after each tick of the game clock.
Finally, these concerns are represented by a small number of scenarios that are leaves (or levels) of the utility tree.
A scenario represents a use or modification of the architecture, applied not only to determine if the architecture meets a functional requirement, but also (and more significantly) for predicting system qualities such as performance, reliability, modifiability, and so forth.
The scenarios at the leaves of the utility tree are prioritized along two dimensions: (1) importance to the system and (2) perceived risk in achieving this goal. These scenarios are prioritized relative to each other as high, medium, or low risk.
|Phase 1: Quality Attribute Utility Tree - Performance|
|Attribute Concerns||A. The growing number of StationarySprites may degrade game performance.|
|Scenarios||1. With each tick of the simulation clock, each StationarySprite is checked to determine whether it has been hit by one of the MovableSprites. The next tick cannot be handled until all current Sprites have been checked for collisions.||(H,H)|
|2. All StationarySprites are created at the start of a new match for a game. As the number of Sprites increases, so does the startup time for a game, which may become unacceptable.||(H,H)|
|Attribute Concerns||B. The rate of ticks generated by the timer may be too slow for smooth animation.|
|Scenarios||1. The player sees the animation as jerky motion.||(H,L)|
|Attribute Concerns||C. The rate of ticks generated by the timer may be too fast for the player to react.|
|Scenarios||1. The player cannot move the paddle sufficiently fast to intercept the puck.||(H,L)|
|Phase 1: Quality Attribute Utility Tree - Modifiability|
|Attribute Concerns||A. The games' range of scoring procedures may be too broad to fit within the architectural approach.|
|Scenarios||1. Pong and Brickles use simple counting schemes while Bowling requires more complex logic to compute the score.||(H,H)|
|Attribute Concerns||B. The modified MVC approach we are using may not be sufficiently flexible to accommodate later additions to the product line.|
|Scenarios||1. The addition of Pinball may require so many more display attributes that a full-blown MVC would be a better choice.||(H,H)|
In addition to the scenarios at the leaves of the utility tree, a scenario elicitation process allows stakeholders to contribute additional scenarios that reflect their concerns and understanding of how the architecture will accommodate their needs. A particular scenario may have implications for many stakeholders: for a modification, one stakeholder may be concerned with the difficulty of a change and its performance impact, while another may be interested in how the change will affect the integrability of the architecture. The following table shows scenarios collected by a round-robin brainstorming activity. After the scenarios were generated, they were prioritized using a voting process in which participants were given individual votes that they could allocate to any scenario or scenarios.
|Phase 2: Brainstormed Scenarios|
|1.||A player downloads a game executable to his computer. When he attempts to play the game, his computer does not have sufficient memory and the operating system reboots. He thinks that our game caused the problem and we lose his goodwill.||5|
Analysis of Architectural Approaches
A-1 from the Phase 1 utility tree and the only scenario from Phase 2's generation/prioritization process were examined in detail vis-à-vis the AGM architecture.
Performance A-1: With each tick of the simulation clock, each StationarySprite is checked to determine whether it has been hit by one of the MovableSprites. The next tick cannot be handled until all current Sprites have been checked for collisions. This is part of the main action loop. The current implementation uses a simple approach that searches all Sprites in the system. If the product does not meet performance goals, consider implementing a more sophisticated scheme (e.g., a screen grid system).
Modifiability A-1: Pong and Brickles use simple counting schemes, while Bowling requires more complex logic to compute the score. The architecture contains an interface so the scoring scheme can be readily changed.
Phase 2: A player downloads a game executable to his computer. When he attempts to use the game, his computer does not have sufficient memory and th operating system reboots. He thinks that our game caused the problem and we lose his goodwill. The architecture does not contain a start-up module that requires minimal memory and checks for the required amount of memory so that it can issue a message to the player before terminating.
Risks, Sensitivities, and Tradeoffs
The analyses of architectural approaches by applying selected scenarios and the ensuing discussions uncovered a set of risks, sensitivities, and tradeoffs.
The collected risks are as follows:
Risk 1: The highest risk is the potential for player dissatisfaction. The current architecture supports a very simple graphical model. The graphics are not as realistic as other available games. If players are dissatisfied, they may have a negative impression of the company, and the goodwill expected from the free games will not accrue.
The following items were identified as areas in which no risk is involved:
Non-Risk 1: Scoring and accurately representing the games. Each game is sufficiently simple that the scoring is clear and easily implemented.
The collected sensitivities are as follows:
Sensitivity 1: The number of graphical elements in a game. If games were added to the product line that used a larger number of elements, many of the quality attributes would degrade.
The collected tradeoffs are as follows:
Tradeoff 1: There has been a tradeoff between memory use and modifiability. The full MVC architecture results in a duplication of data between the model and the views. Some pieces of data might be in every view plus the model. This duplication is not a problem for the free versions but would have been one for the wireless device versions. The modified MVC architecture only stores the data in one place, but it is much harder to add additional views. Because this is a fundamental architecture element, it was not made a variation.
Tradeoff 2: There has been a tradeoff between performance and simplicity. As stated previously in this report, the graphical area of the game interface could have been divided into regions to speed collision checking. To reduce development costs, a single region was used.
Risk, Sensitivity, and Tradeoff Themes
The set of identified risks, sensitivities, and tradeoffs prompted a discussion of architecture themes. The following themes represent the key architectural issues that pose potential problems for the success of the venture:
Scalability. The architecture's ability to accommodate additional games is questionable. Because the architecture is very simple to keep costs low, certain techniques (e.g., dividing the graphics field into regions to speed collision checking) have not been used, and the architecture's ability to support a large number of screen objects while meeting performance goals is uncertain. The modified MVC architectural pattern will not easily support additional windows for displaying different information.
Realism. The realism of the games is questionable. The architecture does not support the sophisticated input and output devices that make interaction more natural. The graphics are very simple to allow the games to run on almost any hardware.
Conclusions and Next Steps
The results of the ATAM evaluation showed the need to modify the existing architecture to include user and system models that allow the player to control more aspects of the game. The detailed design process will assess whether this should be done through menu selections or a configuration file. The next step is to revise the architecture to accommodate these models.
The performance of the current design is adequate (just barely). We will not do anything explicitly to improve performance, but the work described above cannot degrade performance.
If this product line proves successful, we will revisit the architecture and modify it to support a second product line.