Verification and Validation by Four-Fold Docking: A Case Study of Agent-based Models of Anopheles gambiae

S. M. Niaz Arifin (1), Gregory J. Davis, Steve Kurtz, James E. Gentile, Ying Zhou and Gregory R. Madey

Department of Computer Science and Engineering University of Notre Dame Notre Dame, IN 46556

TITLE: Verification & Validation by Four-Fold Docking: A Case Study of Agent-Based Models of Anopheles gambiae

ABSTRACT: Assessing the credibility of complex simulation models can be challenging. In agent-based modeling (ABM), verification & validation (V&V) are used to determine that the model is an accurate representation of the real system. Verification compares multiple models with each other in order to ensure that the implementations are correct. Validation, on the other hand, has the goal to ensure that the model is semantically right, and it is indeed modeling the phenomena of the real system being simulated. Docking, also known as `alignment' or `replication', is a form of V&V that tries to align multiple models in order to investigate whether they yield similar results. Docking is useful to confirm whether the claimed results of a given simulation are reliable, and can be reproduced by someone starting from scratch.

Malaria is one of the top three pathogen-specific causes of global mortality, causing an estimated two million deaths per year, mainly in children. Human malaria is transmitted only by females of the genus Anopheles, which are regarded as the primary vector for transmission. Anopheles gambiae, which refers to a complex in the genus Anopheles, contains the most important malaria vectors in Sub-Saharan Africa, and the most efficient malaria vectors in the world.

Using the technique of docking, we perform V&V of agent-based simulation models that simulate the life cycle of Anopheles gambiae. We develop four separate implementations that sprung from a simplified, fixed version of a core malaria entomology model. The core model is primarily governed by the biology underlying Anopheles gambiae.

The earliest implementation of the core model, CPP1, is developed in C++. The second implementation, Java1, is built over three phases using Java. The re-factored implementations, CPP2 and Java2, are developed in C++ and Java, respectively. For the purpose of docking, CPP2 and Java2 are verified with respect to each other (internal verification), and a single output is compared with those of CPP1 and Java1.

In a complex ABM, synergies arising from seemingly insignificant differences in separate implementations may lead to significant mismatch in overall model output. Hence, it is crucial, and sometimes necessary, to `compartmentalize' the artificial simulation world, i.e. to separate it into isolated compartments so that errors in one specific compartment are not propagated and thus cannot in

uence the discovery (and correction) of errors in other compartments. We compartmentalize the mosquito world with respect to the adult and aquatic populations, remove all potential randomness from the model, and perform some experiments to identify and fix the sources of output mismatches. Following the `Divide and Conquer' paradigm, we first ensure that the pieces are working as intended. Then, we combine the compartments, put back all randomness, and perform more complex experiments.

We describe the design of these experiments, analyze their results, and show how they lead to a successful dock between all four implementations. We compare output abundance (population) graphs from different implementations, and discuss the findings obtained throughout the docking process. We also describe how these findings help to clarify concepts, eliminate ambiguities, and achieve a complete four-fold docking of the core model.

The complete four-fold docking encompasses verification between the four implementations, as well as validation against the core model with respect to these implementations. Results show that seemingly insignificant difierences in separate implementations may lead to significant mismatch in overall model output, suggesting that the entire docking process should be iterative in nature and should involve well-planned feedback from earlier implementations.

The iterative docking process produces incremental agreement in model output. This also serves as a case-study to illustrate the importance of docking complex simulations, since even the best simulation programs may imply dubious assumptions, leading to erroneous results. Rigorous docking can detect such problems, uncovering these incorrect assumptions. As the core model is continually being refined, and new features are added to it, the need to verify all implementations, and to validate these against the core model, warrants a continuous V&V process to be carried on.

(1) sarifin@nd.edu

Back to Swarmfest2010_program