52 | T+D | JUNE 2008 Photo by iStockphoto.com This latest form of measurement may be evolving in importance. ORGANIZATIONAL INTELLIGENCE SURVEYS Surveys in general are commonly used for varied purposes in the context of workplace learning and performance and human capital management. Uses may include assessing training and learning needs, evaluating programs and solutions, measuring employee perceptions and attitudes, and conducting organizational research. But organizational intelligence surveys are an entirely different form of survey that account for strategic factors that enable or inhibit employee engagement, and other important organizational outcomes. By Salvatore V. Falletta THE ORGANIZATIONAL SURVEY IS ONE OF THE MOST PREVALENT AND WIDELY USED METHODS FOR COLLECTING DATA AND INFORMATION ABOUT EMPLOYEE THOUGHTS, FEELINGS, AND BEHAVIORS IN ORGANIZATIONAL SETTINGS. JUNE 2008 | T+D | 53 LI STEN TO THI S FEATURE at www.astd.org/TD/TDpodcasts.htm ORGANIZATIONAL LEADERSHIP CULTURE 54 | T+D | JUNE 2008 The evolving nature of organizational surveys For decades, traditional employee satisfaction surveys were the norm. These surveys were lengthy employee opinion questionnaires (100 to 150 items) that attempted to measure job satisfaction and general satisfaction with organizational-sponsored programs—the extent to which employees were satisfied with various programs, benefits, and services. By the early 1990s, more targeted employee pulse surveys began to emerge. These were typically administered on a quarterly or biannual basis. They measured employee perceptions and reactions to organizational change efforts and popular management trends, such as quality management initiatives, restructuring, and system implementations. The dot-com era from roughly 1995 to 2000, coupled with the war for talent, ushered in the concept of employee engagement. This led to the development and validation of a number of branded and competing definitions of engagement, survey instruments, and concomitant items and questions by consulting firms and research consortia and think tanks. Unfortunately, these varying definitions and measurement tools limited the extent to which research on employee engagement can be generalized beyond specific firms’ practices. Moreover, many of the survey instruments available comprise merely a few items related to employee motivation, commitment, and retention. They omit important strategic levers and primary drivers that ultimately affect employee engagement. Hence, the lack of a standard definition and reliable measurement tools has left practitioners dazed and confused as to what employee Environmental inputs Leadership Strategy Culture Structure and adaptablity Information and technology Direct manager Employee engagement Measures and rewards Growth and development Performance outputs Organizational capability and execution: key indices ORGANIZATIONAL INTELLIGENCE MODEL JUNE 2008 | T+D | 55 engagement actually is, and how to accurately measure it. How do we leap ahead? Survey consultants and practitioners are continuing to extol the value of employee engagement. However, the means of measuring and demonstrating its impact continues to lag behind. A comprehensive approach for measuring employee engagement at the cognitive, affective, and behavioral levels is sorely needed. Cognitive engagement refers to what employees think—their rational commitment to and beliefs about the organization. Engagement, at the affective level, refers to how employees feel about their organizations—their emotional attachment and connection to their jobs, direct managers, co-workers, and the organization. The behavioral domain refers to how employees act—the discretionary energy and effort employees exert on behalf of the organizations they serve. Organizational intelligence surveys measure employee engagement at each of these levels and more. They are broader than employee engagement surveys, yet concise and more focused than antiquated employee satisfaction surveys. In general, there are three tenets that underlie organizational intelligence surveys, making them distinct from traditional employee and organizational surveys—they are evidence-based, model-driven, and focused on action planning and real change. Evidence-based. Organizational intelligence surveys are grounded in theory and empirical research and are tested for validity and reliability in different settings over time. Validity refers to the extent to which the survey items and questions truly represent the factor or variable of interest. In other words, they measure what they are supposed to measure. Reliability refers to the extent to which the survey instrument consistently measures the same characteristic or attributes over time. So, does the instrument give the same results when the measurement is repeated? Research on the relationships and dynamics among survey factors and variables has led to the development and refinement of models of organizational effectiveness and performance. Moreover, many of these factors and relationships have been determined through research to drive, and in some cases, predict specific organizational outcomes such as employee engagement and retention. This is often done through some type of driver analysis. Here, survey data is used to pinpoint the specific factors and variables that are highly correlated with employee engagement and other organizational outcomes. Linkage research is another approach that is typically used in this context, where perceptual data from various surveys and feedback instruments is matched to hard performance measures. This yields employee survey results that are linked to customer survey results as well as revenue. Model-driven. Organizational intelligence surveys are based on a specific model or conceptual framework surrounding how people and organizations function. Model-driven survey efforts have been the mainstay in organization development circles for many years. Yet many of the models used today lack predictive utility in terms of measurement validity and reliability through which meaningful causal assertions can be made. The organizational intelligence model on page 54 serves as a useful framework to facilitate the design and interpretation of most employee and organizational survey efforts. The model includes 11 factors that affect employee engagement and performance. It depicts a top-down causal chain, making some tentative assertions with respect to cause and effect. In many ways, the organizational intelligence model can be thought of as a representation of an organization. The variables in the upper part of the model (such as environmental inputs) affect the organization from the outside. Within the organization, 56 | T+D | JUNE 2008 all strategic levers affect organization capability and execution. This includes the organization’s adaptability, management practices, and rewards and growth opportunities, among other factors. These latter internal factors in turn influence employee engagement and performance. In short, the organizational intelligence model defines important factors and relationships to consider in designing and developing survey instruments and items. Focused on action planning and change. Organizational intelligence surveys focus on action planning and change, and are implemented for that sole purpose. The action planning process involves identifying important issues for the organization to address. Through it, ideas and solutions are generated, and appropriate solutions and best approaches to implementation are selected. This enables actually making the change happen while monitoring the results and effectiveness of the change. For lasting change to occur, all levels of the organization—corporate, geographic regions, business units, functions, teams, and individual line managers—must participate in developing, implementing, and assuming ownership for continuous improvement. Meaningful action planning, “action doing,” and follow-up with all levels of stakeholders are key to ensuring execution. Survey design and development Again, the organizational model serves as the framework to guide the design and development of the survey. A survey based on the organizational intelligence model would be organized into 11 categories based on the factors in the model. The survey items representing each of these categories would be developed based on the definition of each and the specific needs of stakeholders. Three to five survey items are a good rule of thumb for adequately covering each factor, while minimizing item redundancy. Broader constructs such as leadership and culture may require more items for adequate measurement. The final step in designing an organizational intelligence survey is the selection of the item-response alternatives and scales. There are many scales from which to choose. Typically, the five-point Likert scale is used. The table above illustrates the most common response alternatives used with a five-point Likert scale. 5-POINT RATING SCALES 1 2 3 4 5 Strongly disagree Disagree Neither disagree nor agree Agree Strongly agree Very dissatisfied Dissatisfied Neither dissatisfied nor satisfied Satisfied Very satisfied Not at all To a little extent To some extent To a great extent To a very great extent Almost never Not very often Some of the time Most of the time Nearly always Very poor Poor Average Good Very good 1-20% effectiveness 21-40% effectiveness 41-60% effectiveness 61-80% effectiveness 81-100% effectiveness Definitely false False Don’t know True Definitely true None Very mild Mild Moderate Severe Not important Somewhat important Important Very important Extremely important Very low Low Moderate High Very high Substantially worse Somewhat worse About the same Somewhat better Substantially better JUNE 2008 | T+D | 57 Survey deployment Less than a decade ago, online survey technology was considered an alternative to the traditional paper and pencil survey. Today, online technology is arguably the norm, and provides a quick and efficient means for deploying organizational intelligence surveys. For some organizations and industries where employees do not have Internet access, such as retail and manufacturing environments, online survey technology may not be feasible. But where practical, there are important advantages to using online surveys: s global reach—the ability to distribute surveys to a geographically dispersed population in real time s real-time response tracking—the ability to monitor survey response rates in real time s instant feedback and reporting— ability to provide immediate online feedback and reporting to stakeholders s customization for specific audiences—ability to skip ahead of irrelevant items based on previous responses s high-quality open-ended responses due to typed, as opposed to handwritten, responses—ability to edit and spell check comments prior to survey submission s integrated online action planning tools as part of the overall online survey platform. Advances have also been made in information security and the means through which survey confidentiality is assured. For example, many online survey vendors and providers utilize data encryption, firewall technology, and password protection to ensure data security for employees and organizations. For lasting change to occur, all levels of the organization— corporate, geographic regions, business units, functions, teams, and individual line managers— must participate in developing, implementing, and assuming ownership for continuous improvement. 58 | T+D | JUNE 2008 Survey analytics Organizational intelligence surveys can be analyzed through a number of techniques. These include item analysis, conceptual analysis, comparative analysis, and content analysis. Item, conceptual, and comparative analyses are all used when working with quantitative survey data; content analysis is used for qualitative, or open-ended, survey data. Item analysis involves describing data in terms of frequencies, means, standard deviations, ranges, and percentages, and is the simplest type of analysis. This method is an important first step in identifying relative highs and lows in the data set. Since item analysis is applicable to individual survey items only (the unit of analysis is at the item level alone), the more complex relationships evident in the data will not be uncovered without further analysis. Conceptual analysis is more advanced and involves testing relationships among the factors in the survey, as identified by the organizational model. This analysis relies on inferential statistics to test the relationships among the factors in the model. One trend that is gaining popularity in all realms of survey practice is identifying key factors, variables, and indices that drive desired outcomes such as employee engagement. Advanced statistical techniques such as correlation and regression, and causal and predictive modeling are used as part of conceptual analysis. If done with the appropriate level of rigor, accounting for sufficient validity and reliability, the technique adds more power, utility, and credibility to any survey process. Another more common type of analysis is comparative analysis. As the name suggests, it involves comparing the survey results of one group or organization to another, comparing the results over time, or comparing the results of one company to another or to the industry (benchmarking). These comparisons may involve item or construct comparisons, as well as demographic comparisons. Best-in-class companies in the high-tech industry, such as Dell, Google, Microsoft, Nokia, IBM, and SAP, benchmark their survey results through employee research membership consortium the Information Technology Survey Group. The forth type of analysis is performed on open-ended or write-in survey questions. Content analysis involves categorizing open-ended responses into major themes. This data is a rich complement to quantitative data, and often helps to add context to the quantitative data because it is far more descriptive in nature. Moving beyond the proverbial data dump Today, more effort and emphasis is placed on the development of an executive summary of the survey results. The executive summary is crafted to tell a compelling story about the organization. This summary brings interpretation and meaning to the large amounts of data found in standard survey reports. Such data include percent favorable, neutral, and unfavorable; high and low ratings; and diagnostic inferences. Whenever possible, the data-based diagnostic inferences and insights should include advanced analytical procedures such as driver analysis, linkage research, and causal modeling procedures that demonstrate cause and effect. This is a powerful step in the overall organizational intelligence survey process. It differs significantly from traditional employee and organizational survey efforts that tend to focus exclusively on data and information rather than survey intelligence. Organizational intelligence surveys that are grounded in science and that are model-driven can greatly enhance the analysis and interpretation of the survey results, and can provide a valuable framework from which to act and make organizational changes. T+D Salvatore V. Falletta is president and CEO of Leadersphere Inc., an HR intelligence consulting firm in Sacramento, California; [email protected]. Think COGNITIVE Act AFFECTIVE BEHAVIORAL Feel Engagement A MULTIDIMENSIONAL CONCEPT