Online Tutorial

The excellent rating distributions of Selex Success are due in large part to our online Rater Tutorial. The Tutorial provides key information on rating accuracy in a concise manner. The Tutorial presents the two most common errors, halo and leniency, and describes why it is important to avoid these and how to avoid these.

The raters must view the Tutorial on their first entrance to the Rating website. The Tutorial is then available on every page in case the rater needs the information at a later point.

Administration Website

The Administration website makes organizing and tracking the data collection simple. The website provides up-to-the minute completion statistics. The reporting facilities enable Administrators to determine at a key stroke which participants have completed the rating process. The email facility makes follow-up easy. It allows the Administrator to email all incomplete participants at once or send individual emails. This immediate access to information and follow-up has a large impact on shortening the length of the data collection.

These data were collected online through the Selex Success website. Data collection locations included multiple field and corporate offices for a large financial institution at offices across the United States and the United Kingdom. Positions were Customer Service Representatives (N = 1,586). Participating supervisors and managers completed an online Rater Tutorial to improve accuracy and to avoid common rater errors such as leniency and halo. The raters used a two-phase judgment process for each rating item where the first judgment was whether or not the rating item described a critical success outcome that is essential to the position. The second judgment was performed only for items judged to be essential, and used this five point scale.

         

Overall Performance Rating Distribution

		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	Weak	88	5.5	5.6	5.6
	Fair	270	17.0	17.1	22.7
	Good	504	31.8	32.0	54.7
	Strong	469	29.6	29.8	84.5
	Best	245	15.4	15.5	100.0
	Total	1576	99.4	100.0
Missing	System	10	.6
Total		1586	100.0

The ratings generally conformed to the suggested distribution with slightly less than 10% of the ratings in the bottom category and slightly more than 10% in the top category. Nonetheless, mean ratings were near 3 on the 5 point scale, and standard deviations were near 1.0. Overall the ratings show excellent distributional properties. The ratings of overall performance (R33) had a mean of 3.33 and a standard deviation of 1.096.

The table to the right provides the distribution of supervisor ratings for the overall performance item.

Descriptive Statistics

	Mean	Std. Deviation(a)	Analysis N(a)
R1	3.19	1.153	1586
R2	3.43	1.056	1586
R3	3.40	1.079	1586
R4	3.20	1.060	1586
R5	3.18	1.070	1586
R6	3.23	.963	1586
R7	3.00	1.089	1586
R8	3.07	1.039	1586
R9	3.23	1.103	1586
R10	2.87	.952	1586
R11	3.06	1.004	1586
R12	3.28	1.103	1586
R13	3.18	1.008	1586
R14	3.35	.982	1586
R15	3.17	1.132	1586
R16	3.32	.888	1586
R17	3.36	1.050	1586
R18	3.33	1.305	1586
R19	3.39	.871	1586
R20	3.19	1.084	1586
R21	2.97	1.076	1586
R22	3.36	.895	1586
R23	3.59	1.094	1586
R24	2.64	1.173	1586
R25	2.95	1.257	1586
R26	3.21	1.158	1586
R27	3.58	1.108	1586
R28	3.38	.962	1586
R29	3.10	1.099	1586
R30	3.15	1.332	1586
R31	3.43	1.139	1586
R32	3.14	.996	1586
R33	3.33	1.096	1586
a For each variable, missing values are replaced with the variable mean.

Factor Analysis

A factor analysis is used to discover the structure of independent variables that underlie the ratings. The resulting factor scores represent the true set of performance factors that underlie the larger set of performance ratings. The factor scores have the advantages of interpretable construct validity and greatly enhanced reliability compared to the raw ratings.

The table to the right shows the excellent statistical properties of the item level data, with item means ranging from 2.64 to 3.59 and standard deviations ranging from .871 to 1.305.

Rotated Component Matrix(a)

	Component
	1	2	3	4	5
R13	.792
R17	.751
R16	.748
R14	.729
R22	.616
R15	.589
R1	.532
R33	.512
R8
R9		.803
R20		.758
R12		.740
R3		.691
R11		.674
R2		.541
R25			.792
R27			.773
R26			.721
R5			.664
R28			.600
R4			.585
R32			.526
R18				.795
R30				.778
R19				.673
R29				.556
R31				.534
R23				.514
R21					.809
R10					.794
R6					.595
R7					.576
R24
Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization.
Rotation converged in 7 iterations.

This is called a component analysis because the initial estimates of commonality are set at unity or 1.0. The advantage of this method is that the component scores can be calculated reliably.

The Varimax rotation is an orthogonal rotation that applies a criterion of factor independence so that the resulting factors are uncorrelated.

The resulting five criterion factors are shown below, with the rating items that had the highest factor loadings.

e-Selex.com validates against factors, rather than items. We use data reduction techniques, as described on previous pages, that develop these orthogonal factors. Using factor scores as the criterion provides tremendous advantages in understanding what is being measured.

These charts show the distributions of the factor scores for each of the five (5) factors. The distributional properties are excellent, as shown by the bell-shaped curves in each chart.

These five factor scores are the primary results of the criterion analysis. The factor scores with these bell-shaped distributions provide a criteria that are ideal for validation purposes and other statistical uses. The variability around the mean in these factor scores provides optimal measurement properties for researching relationships between these construct-valid metrics and predictor data.

Click on any graph for a larger version.