The Question-and-Answer Approach Metrics of this type rely on a network of risk management professionals to monitor progress in meeting some sweeping organizational objective such as compliance with regulatory reporting requirements. The risk management professionals themselves are not usually responsible for meeting the objectives, just for identifying the right set of individuals who should address them, organizing a reporting framework, and periodically polling those accountable to see whether they have achieved compliance. These types of metrics are often created using online surveys. The major challenge in this approach is devising methods of identifying organizational structure and individuals within them that can be held accountable for survey answers. The state of the art in survey metrics collection utilities are systems wherein the accountable person can be linked to the control they are reporting on in such a way that when they log in to the risk application, their login is treated as a digital signature. This, combined with strict management policy that questionnaires must be answered within strict timelines, allow the risk managers to report the results with firmwide authority. Figure 6A-5 provides an example of the type of online surveys an accountable manager may encounter. From such a survey, metrics such as Figure 6A-6 may be derived. Risk management professionals reduce accountability to self-reported task completion. From Figure 6A-6, it may be inferred that all of the new applications introduced into operation in the first quarter are using the firm’s single sign-on system for authentication, and are compliant with security policy, design, and change control standards. However, there were a few introduced in the second quarter that did not make use of single sign-on and were not policy or standards compliant. The pie charts in Figure 6A-7 show this line of business (LOB) in comparison with the
other LOBs in the firm. By comparison, LOB1 seems to have superior security deployment processes. However, given that the data is based, not on automated measures, independent audit, or repeatable processes, but instead on data collection from accountable managers, all that can be inferred from Figure 6A-7 is that LOB1 claims to have fully compliant applications. The risk manager running the report is reduced to clerk status. These scenarios exist because the management objective on which the metric is modeled does not easily conform to available security measurement techniques. Suppose instead, there were automated verification procedures that could check to see if an application was in compliance, and there was a complete and accurate inventory of machines deployed in support of a given application. The inventory, in combination with automated verification procedures and/or technical configuration audits, could be used to establish that applications met firmwide security objectives. However, pressure for results in short timeframes sometimes require even the most technical of metrics professionals to resort to a question-and-answer method of data gathering. These scenarios almost always suffer from real or perceived ambiguity in the way the “compliance” officer phrases overall goals and statements to be affirmed. For example, suppose an executive of a given organization is asked to affirm whether or not that organization has a policy to achieve compliance with a given regulation. The executive may sincerely respond in the affirmative, yet be thinking of a policy that states that the organization is fully committed to apply to all applicable regulation. At the same time, the executive may omit to mention that he or she does not believe that the regulation in question applies to that organization. Even when specific technology questions seem easy to verify, a respondent may find a question ambiguous enough for a misleading answer. For example, note the question in Figure 6A-5 that reads: “Does the application encrypt PCIS9 data?” A manager who encrypts some but not all such data may honestly and misleadingly answer “yes.” These examples and others like them leave a loophole of plausible deniability among those accountable for answering surveys while allowing the compliance officer to publish remediation metrics showing 100% compliance.