The key concepts which we have covered and may be applicable to this assignment are; Means, standard deviations, pivot tables, standard errors, correlations, z scores, multiple regression (we have been told that we would need this to analyse the teaching data), standard errors, T-statistic, confidence interval, NPS as well as other excel functions and simple formulas
ASSIGNMENT – PERFORMANCE APPRAISSAL It is February 2019. Dr. Angus Krystos is Dean of Faculty at the Darwin International Business School (DIBS). DIBS is an independently financed school of the publicly owned and funded University of Northern Australia (UNA) The most essential resource for DIBS is their academics. Business academics are in very short supply and have other attractive options, including full time consultancy. An important part of Angus’s role it to recruit and retain quality academics, who can both teach and research at a high level. For the past three years, a new formal appraisal system for academics had been in place, linked to remuneration. All academics sign a contract which requires them to be formally appraised each year. The contract states that “equal weight will be given to teaching and research”. There is a yearly bonus pool and this bonus pool is divided up amongst all faculty, but a larger bonus is given to those who receive a higher appraisal, how much larger being at the Dean’s discretion. Angus’s task is to appraise the 36 full time academics currently employed at DIBS, to communicate and explain the results of this exercise to them and to possibly make revisions to the appraisal system to be rolled out in the future, if appropriate. Industry background. If you think that tertiary education is somehow not the ‘real world” then think again. The Australian university system employs some 70,000 academics who both conduct research and deliver education to over 1.5 million students yearly. Many of these are full fee paying overseas students. In fact, university education is Australia’s second largest export earner, eclipsed only by mining. It is as essential to the Australian economy as call centres and software development are to India. Organisation background. DIBS has a suite of popular and lucrative executive programs and a distance delivered MBA course. However, the flagship degree is their full time 12 month MBA and it is largely on this degree that the school’s reputation and credibility rests. Two thirds of these students are from overseas with most students paying full fees of around AUD$80,000. The local students are eligible for government loans to support their studies. While DIBS does not receive any funding from UNA, its degrees have been accredited by the university since the school’s founding in 1975. All DIBS academics are appointed according to standard university criteria and are given honorary academic appointments at UNA so long as they are employed by DIBS. Professorial appointments must go through the usual UNA appointment procedures. In return for paying its own way, the university does not impose any “tax” on the fees from students. Universities would typically tax at least 50% of full fee paying students. DIBS also enjoys exclusive use of a complex of university owned buildings though the cost of maintenance and any improvements are borne by DIBS. They also have premises in the central business district as well as an executive education site on the coast, some 10 km west of the campus. What does UNA get out of this relationship? First, the reputation of having a very high profile business school without any of the risk. Second, all research generated by DIBS is attributed to UNA who receive block research funding from the federal government which is directly linked to their research output. DIBS receive none of this research funding even though they are doing the research. Faculty performance management Central to Angus’s role is recruiting new faculty as well as managing and developing their careers. Business academics are hard to attract and he is constantly reminded that faculty have other options. This manifests as a rather high turnover rate even for tenured faculty – much higher than in a conventional academic department. Teaching. Academics need to perform at a high level if the school is to maintain its reputation. Performance in the classroom is absolutely critical and high paying MBA students are not at all tolerant of courses that are poorly organised, badly delivered or not focused on business relevant problems. The school surveys students at the end of each course who give a 1-5 rating on 9 different dimensions. For instance they are asked “Was the lecturer well organised?” “Were the assessment tasks appropriate?” These questions are mainly there to give the academic detailed feedback on areas for improvement. But the critical question is question 8 which asks “Overall, how do you rate this course?” This question is on a 5 point scale, with 5 being good. So the school has information on how well the academic is performing overall in teaching, at least from the student’s point of view. Historically, a lecturer’s performance in the class room has been measured by the sample mean of student responses to Q8 across all courses taught that year. Typical means are from 3.8 to 4.4, where the highest possible mean would be 5.0. For the past three years though, a different measure to the mean has been used, namely the % of responses in the range 3-5. This move was controversial as someone could get 100% rating even if every student rated their course at 3 which would be a poor mean historically. The rationale was that the school especially wants to avoid students being actively dissatisfied with a course. Some faculty members have had satisfaction ratings above 95%. Some are as low as 70% which means that 3 out of 10 students were actively dis-satisfied with the course. Research. On the other hand, academics need to do research in order to stay at the forefront of their field. Most academics actively want to do research; they are interested in their subject and see themselves as part of a larger academic community that contributes to knowledge. From the school’s point of view, there are two reasons to care about research – even though they make no money directly from it. First, research ability is a proxy for the academic’s depth of understanding. This interest and engagement with their subject will surely spill over into other activities that more directly affect the school’s bottom line, not to mention their reputation. Second, academics value a highly productive and vibrant research environment, for their own stimulation and engagement. This can directly increase their research productivity. So if a school’s research standing starts to fall it becomes harder to attract good academics to work there. For research organisations, success truly breeds success and mediocrity breeds mediocrity. Thus, research output is also a performance measure but one that is valued by a quite different pool of stakeholders – potential academics, which remember are in short supply. It is also indirectly valued by UNA who benefit from the earlier mentioned block research funding. Research is difficult to measure. Not all research is of the same quality. Moreover, it is harder to publish research in some disciplines than others. The balance. All academics sign a contract which requires them to be formally appraised each year. The contract also states that “equal weight will be given to teaching and research”. The folklore is that better researchers will tend to make better teachers because of their deep understanding and enthusiasm for the subject. So one might expect that the best researchers would also be amongst the best teachers if it were not for the fact that there are only 24 hours in a day. Academics who expend a larger effort on teaching and developing up-to-date and innovative course materials will necessarily spend less time on research. This suggests a countervailing negative relationship between these two performance metrics. Angus wondered whether the data could tell him whether the better researchers tend to be better or worse teachers. There is a yearly bonus pool depending on the financial success of the school and this bonus pool is divided up amongst all faculty, but a larger bonus is given to those who receive a higher appraisal. The 2017-2018 data provided to Angus. Angus has collected information on the 36 academics that he is to appraise. The performance output measures are for 2017 and 2018 i.e. the previous two years.2 The data is objective in the sense that it has not been subject to any judgement and is automatically collected by the school. For each academic he has the % of satisfied students calculated for all students taught by that academic over the previous two years. This has been used to measure teaching performance. He also has a count of the number of research publications of each academic over the previous two years. The outlet for the research has to be on a mandated list of journals maintained by Department of Education, Training and Youth Affairs (DETYA). This is the measure of research output used by DETYA in evaluating total university research output. So the university, of which DIBS is a part, already requires this information to be collected and it directly affects their funding through block research grants from the federal government. The task. It is too late to change the basic performance measures used for this year. The two metrics measuring teaching and research success have already been agreed by the faculty and approved by the board two years ago. However, the appraisal results last year which were based on the present system were … controversial. Several faculty felt hardly done by and had good arguments for making the claim. So, moving forward Angus wants to design as fair and robust a system as possible. This 2017-2018 appraisal For the present, Angus has to come up with a way to rank the academics from highest to lowest based on the existing teaching and research metrics only. The Dean will make the final decision on how to distribute the bonus pool. But certainly higher ranked academics will receive more financial reward than lower ranked academics. Just as important as the money however will be the reputation and kudos the flows from the ranking as well as the signal the school is sending to the academic. In the past, the exact rankings have not been published within the school or even to the academics involved. Rather, they are divided into four groups. The top group receive the highest 2 The two year window is the standard period of appraisal and written into the standard contract. There are some academics who arrived more recently but they are not formally appraised, since their appointment already involved a very thorough appraisal. Their salary increments are set at the school average for their first two years. salary increment and their success is formally announced and celebrated by the school. Typically only a small number – say 5 at most – academics fall into the top group. Academics are a fractious species and Angus recalls the old adage that managing academics is like herding cats. No matter what system he comes up with, some