The first paragraph should summarize the reading. The second paragraph should highlight an specific point and/or briefly explore something that interested you (e.g., you may wish to focus on one aspect of the paper in more depth, you may wish to discuss something in the reading that you disagree with). Each paragraph represents one point of the assignment. During the lecture, we will draw upon your reports for some group discussion.
You should submit a summary paragraph and idea highlight paragraphper each separate article.
Very simple task
rsta.royalsocietypublishing.org Research Article submitted to journal Subject Areas: computer modeling and simulation, computational physics Keywords: reproducibility, validation, replication, nanophysics Author for correspondence: L.A. Barba e-mail:
[email protected] Reproducible Validation and Replication Studies in Nanoscale Physics N. C. Clementi1, L. A. Barba1 1Department of Mechanical and Aerospace Engineering, The George Washington University, Washington D.C., USA Credibility building activities in computational research include verification and validation, reproducibility and replication, and uncertainty quantification. Though orthogonal to each other, they are related. This paper presents validation and replication studies in electromagnetic excitations on nanoscale structures, where the quantity of interest is the wavelength at which resonance peaks occur. The study uses the open-source software PyGBe: a boundary element solver with trecode acceleration and GPU capability. We replicate a result by Rockstuhl et al. (2005, doi:10/dsxw9d) with a two-dimensional boundary element method on silicon carbide particles, despite differences in our method. The second replication case from Ellis et al. (2016, doi:10/f83zcb) looks at aspect ratio effects on high-order modes of localized surface phonon-polariton nanostructures. The results partially replicate: the wavenumber position of some mode match, but for other modes they differ. With virtually no information about the original simulations, explaining the discrepancies is not possible. A comparison with experiments that measured polarized reflectance of silicon carbide nano pillars provides a validation case. The wavenumber of the dominant mode and two more do match, but differences remain in other minor modes. Results in this paper were produced with strict reproducibility practices, and we share reproducibility packages for all, including input files, execution scripts, secondary data, post-processing code and plotting scripts, and the figures (deposited in Zenodo). In view of the many challenges faced, we propose that reproducible practices make replication and validation more feasible. c© The Author(s) Published by the Royal Society. All rights reserved. ar X iv :2 00 8. 05 41 4v 1 [ ph ys ic s. co m p- ph ] 1 2 A ug 2 02 0 http://crossmark.crossref.org/dialog/?doi=10.1098/rsta.&domain=pdf&date_stamp= mailto:
[email protected] 2 rsta.royalsocietypublishing.org P hil. Trans. R .S oc. A 0000000 .................................................................. 1. Introduction Some fields of research, particularly solid mechanics and computational fluid dynamics, have a long tradition of community consensus building and established practices for verification and validation of computational models. Such practices are uncommon in other fields of science, especially if they have more recently become computationally intensive. Verification and validation also become increasingly difficult when the computational models arise from many levels of mathematical and physical modeling, representing a complex system. In recent years, science as a whole has come to be concerned with reproducibility and replication as a new front in the continual campaign to build confidence on published findings. Together with formal processes of uncertainty quantification, we have now three complementary “axes” for building trust in science. The lengths to which research communities should go to conduct activities in verification and validation, reproducibility and replication, and uncertainty quantification, are highly debated. Some journals require articles reporting on computational results to include proof of these activities, while most do not consider these aspects at all in their review criteria. In this paper, we tackle a sub-field of computational physics where tradition for these confidence-building activities is scant. The physical setting, excitation of resonance modes in nanostructures under an electromagnetic field, relies on multiple levels of modeling, while the experimental methods are complicated by the small length scales. We previously developed a computational model and software (called PyGBe) that has undergone code and solution verification, but a validation opportunity had remained elusive. Here, we present replication studies and a validation case based on published simulation and experimental results. Moreover, the studies in this paper were conducted under rigorous reproducibility practices, and all digital artifacts needed to reproduce every figure are shared in reproducibility packages available in a GitHub repository and archival services. 2. Background and methods (a) Verification, validation, reproducibility and replication Verification and validation of computational models—often abbreviated V&V and viewed in concert—have developed into a mature subject with more than two decades of organized efforts to standardize it, dedicated conferences, and a journal. The American Society of Mechanical Engineers (ASME), a standards-developing organization, formed its first Verification and Validation committee (known as V&V 10) in 2001, with the charter: “to develop standards for assessing the correctness and credibility of modeling and simulation in computational solid mechanics.” It approved its first document in 2006: The Guide for Verification and Validation in Computational Solid Mechanics (known as V&V 10-2006). The fact that this guide was five years in the making illustrates just how complex the subject matter, and building consensus about it, can be. Since that first effort, six additional standards sub-committees have tackled V&V in a variety of contexts. V&V 70 is the latest, focused on machine-learning models. The key principles laid out in the first V&V standard persevere through the many subsequent efforts that have operated to this day. They are: . Verification must precede validation. . The need for validation experiments and the associated accuracy requirements for computational model predictions are based on the intended use of the model. . Validation of a complex system should be pursued in a hierarchical fashion from the component level to the system level. . Validation is specific to a particular computational model for a particular intended use. 3 rsta.royalsocietypublishing.org P hil. Trans. R .S oc. A 0000000 .................................................................. . Validation must assess the predictive capability of the model in the physical realm of interest, and it must address uncertainties that arise from both simulation results and experimental data. The process of verification establishes that a computational model correctly describes the intended mathematical equations and their solutions. It encompasses both code correctness, and solution accuracy. Validation, on the other hand, seeks to determine to which measure a computational model represents the physical world. We like to say that “verification is solving the equations right, and validation is solving the right equations” [1]. But in reality the exercise can be much more complicated than this sounds. Computational models in most cases are built in a hierarchy of simplifications and approximations, and comparing with the physical world means conducting experiments, which themselves carry uncertainties. As we will discuss in this paper, verification and validation in contexts that involve complex physics at less tractable scales (either very small, or very large), or where experimental methods are nascent, proceeds in a tangle of researcher judgements and path finding. In practice, validation activities reported in the scholarly literature often concentrate on using a stylized benchmark, and comparing experimental measurements with the results from computational models on that benchmark. Seldom do these activities address the key principles of pursuing validation in a hierarchical fashion from the component to the system level, and of assessing the predictive capability of the computational model accounting for various sources of uncertainties. Comprehensive validation studies are difficult, expensive, and time consuming. Often, they are severely limited by practical constraints, and the conclusions equivocal. Yet the computational models still provide useful insights into the research or engineering question at hand, and we build trust on them little by little. Verification and validation align on one axis of the multi-dimensional question of when are claims to knowledge arising from modeling and simulation justified, credible, true [2]. Two other axes of this question are: reproducibility and replication, and uncertainty quantification (UQ). Uncertainty quantification uses statistical methods to give objective confidence levels for the results of simulations. Uncertainties typically stem from input data, modeling errors, genuine physical uncertainties, random processes, and so on. A scientific study may be reproducible, the simulations within it undergone V&V, yet the results are still uncertain. Building confidence in scientific findings obtained through computational modeling and simulation entails efforts in the three “axes of truth” described here. Reproducibility and replication (we could call it R&R) preoccupy scientific communities more recently. Agreement on the terminology, to begin with, has been elusive [3]. The National Academies of Science, Engineering and Medicine (NASEM) released in May 2019 a consensus study report on Replicability and Reproducibility in Science [4] with definitions as follows. “Reproducibility is obtaining consistent results using the same input data, computational steps, methods, and code, and conditions of analysis. Replicability is obtaining consistent measurements or results, or drawing consistent conclusions using new data, methods, or conditions, in a study aimed at the same scientific question.” According to these definitions, reproducibility demands full transparency of the computational workflow, which at the very least means open code and open data, where ‘open‘ means shared at time of publication (or earlier) under a standard public license. This condition is infrequently satisfied. The NASEM report describes how a number of systematic efforts to reproduce computational results have failed in more than half of the attempts made, mainly due to inadequately specified or unavailable data, code and computational workflow [5–8]. Recommendation 4-1 of the NASEM report states that “to help ensure reproducibility of computational results, researchers should convey clear, specific, and complete information about any computational methods and data products that support their published results in order to enable other researchers to repeat the analysis, unless such information is restricted by nonpublic data policies. That information should include the data, study methods, and computational environment” [4]. 4 rsta.royalsocietypublishing.org P hil. Trans. R .S oc. A 0000000 .................................................................. Although it may seem evident that running an analysis with identical inputs would result in identical outputs, this is sometimes not true. For example, the combination of floating-point representation of numbers and parallel processing means that running