Assessing Cognitive Competencies at Work
An Hr's Handbook for Understanding How Cognitive Skills Evolve with Job Roles and Levels
THE SCIENCE BEHIND HUMAN INTELLIGENCE
Human intelligence is the mental quality that consists of the abilities to learn from experience, adapt to new situations, understand and handle abstract concepts, and use knowledge to manipulate one’s environment. In simpler words, it is the ability to acquire and apply knowledge and skills.
Cognitive Intelligence + Emotional Intelligence
Cognitive intelligence is defined as the combination of verbal, numerical and spatial abilities which includes visualizing, use of memory, word fluency, verbal relations, perceptual speed, induction and deduction.
Emotional intelligence is a form of intelligence involving the ability to read one’s own and others’ emotions and use this reading to manage one’s thoughts and behaviours.
Charles Spearman in the early 20th century established that there must be one central factor that influences our cognitive abilities. He termed this central factor as general intelligence.
‘g’ factor: also known as general intelligence factor is a variable that is required in all tasks that require intelligence. ‘g’ is also responsible for achievement of positive correlations among different cognitive tasks, reflecting the fact that an individual's performance on one type of cognitive task tends to be comparable to that person's performance on other kinds of cognitive tasks.
G is powered by four core cognitive brain processes that are closely associated with and directly impact cognitive intelligence. These are – attention, speed, memory and visualization.
The definition of ‘g’ was further expanded and bifurcated into two by Raymond Cattell, who proposed that ‘g’ consists of two parts: Fluid and crystallized Intelligence.
Fluid intelligence can be defined as the ability to solve unfamiliar problems, by making use of logical reasoning. This ability is not dependent on prior learning, life experience or education.
Example: The true use of fluid intelligence is exemplified by early men who used instincts, logical reasoning and problem solving skills to survive. They did not have any education or access to any form of knowledge that would help them to meet their day to day needs of food, shelter and protection from predators.
MEASURING COGNITIVE INTELLIGENCE
In order to measure intelligence, we need to identify the cluster of abilities or skills, in the form of which intelligence manifests itself. These skills and abilities cluster is called a ‘competency’.
We have identified the key competencies that help in accurate measurement of our fluid intelligence, crystallized intelligence and core cognitive brain functions.
Core Competencies to Measure Cognitive Intelligence
USING COGNITIVE INTELLIGENCE AT WORKPLACE
Is it required to create a cognitive framework for each job role?
We cannot possibly create a competency framework for each job role in existence. So we decided to explore if there are certain job roles that require the same cognitive abilities and would therefore have the same competency framework.
Our research yielded that yes there in fact are job roles that require the same cognitive abilities.
For example Customer Service, Receptionist and Sales Executive all require basic communication skills and performing the same kind of task day after day.
Mettl’s Job Categorization Framework
Any job in the world involves carrying out a task. The type and level of cognitive competencies required to complete a task in a particular job role, is dependent on the nature of the task.
The task can be:
The next question that comes to mind is-
How can we derive the type and level of competencies required in a job role on the basis of the nature of the job?
Mettl’s Framework for job categorization takes into account three core competencies along with two factors pertaining to each core competency.
We identified three core competencies that are required in any job role. These three core competencies are:
Information Processing: Refers to a person’s proficiency is carrying out numerical, analytical and critical reasoning.
Solution Generation: Refers to a person’s ability to think out of the box by making use of creativity and abstract reasoning competencies.
Decision Making: Refers to a person’s ability to solve problems and take well thought out, rational and accurate decisions.
Two factors pertaining to each core competencies are:
Frequency: How many times in a day does an employee need to make use of a particular competency
Level: How difficult are the tasks involving use of a particular competency, which will determine if an employee needs to have basic knowledge of a competency or be proficient in it.
Cognitive Competency Framework for Job Families
Mettl’s Job role categorisation framework has identified five different pools of job roles that differ from each other in the nature of the job and the cognitive competencies required. We call this pool of job role as a ‘job family’. Each job family contains job roles that are similar and require the same set of cognitive competencies to perform them. In other words, there is one cognitive competency framework for each job family.
CALCULATING EFFECTIVENESS OF COGNITIVE ASSESSMENTS
To make a cognitive assessment effective in choosing the right talent for a job role, it needs to accurately identify candidates from the talent pool who are likely to succeed at performing that job role. Once the assessment has been created for a particular job role, its effectiveness in selecting the right talent in an organization needs to be tested.
To have a cognitive assessment adept in identifying the right talent we fine-tune the three parameters identified earlier: the assessment mean score ,difficulty index and discrimination index.
Mean Score is the average score that candidate pool giving the same cognitive assessment achieved.
Low mean score signifies that less number of candidates from the talent pool were able to score high on the aptitude test.
Medium mean score means that moderate number of candidates were able to score high on the aptitude test.
High mean score means that majority of candidates from the talent pool were able to score high on the aptitude test.
The ideal mean score for you depends upon what level of candidates you are looking for. Example, Out of your talent pool, If you want to select candidates with the highest cognitive intelligence, your cognitive assessment should have a very low mean score.
This is how you can adjust your mean score to control the number and intelligence level of candidates selected:
Note: You can adjust the mean score to adjust the number and IQ level of candidates. For example, if you want to select the top 5 percentile of the talent pool, you should aim for an aptitude test with mean score of 5%. It is a tool used for benchmarking an aptitude test as per an organization’s requirement.
Disclaimer: The kind of talent you end up with depends on the quality of the talent pool in general. If your talent pool quality in general is poor, selecting even the top 1 percentile may not provide you with high IQ candidates. Vise-a -versa, if your talent pool quality is too high, even lower percentile candidates may turn out to be high in IQ as the average IQ of the talent pool is high.
The Difficulty Index is the probability that candidates will answer a test question correctly. Every question that is included in a cognitive assessment is benchmarked via its difficulty index.
For example, if out of 100 students, 20 students were able to answer a particular question correctly, the difficulty index for that question would be: 20/100 = 0.5.
The difficulty level of a question is determined by its index range. Lower the difficulty index higher is the difficulty level of the question. It is a tool used for benchmarking an aptitude test as per an organization’s requirement.
Difficulty level is inversely correlated with mean score. High difficulty index signifies that majority of the candidates will perform poorly/fail the test, whereas high mean score signifies that most candidates were able to score high and hence passed the test.
LEVEL OF DIFFICULTY
|INDEX RANGE||DIFFICULTY LEVEL|
This is how you can adjust your difficulty level to control the number and intelligence level of candidates selected:
If the average difficulty level of all the questions in a cognitive assessment is high, it means that on an average the questions included in the assessment are difficult. This means that candidates, on average, will fail the test. However, those few who pass will have high IQ as they were able to clear a test deemed difficult by most. So a cognitive assessment with high difficulty level questions will churn out a pool of candidates who are low in number and high in intelligence.
If the average difficulty level of all the questions in a cognitive assessment is low, it means that on an average the questions included in the assessment are easy. This means that candidates, on average, will pass the test. However, the IQ of these candidates will not necessarily be very high as they were able to clear a test deemed easy by most. So a cognitive assessment with low difficulty level questions will churn out a pool of candidates who are high in number and low in intelligence.
Discrimination is the degree to which candidates with high overall score in a cognitive assessment also answered a particular question correct. It is an index of a question’s effectiveness at discriminating those who performed well in the test from those who did not. It determines if a cognitive assessment is valid or not.
In layman terms, if a question is answered correctly by the same proportion of high intelligence candidates and low intelligence candidates, then that question is not valid as it is not able to discriminate between the two.
Note: Unlike mean score and difficulty index, discrimination index is not a benchmarking tool. It's a validation tool which is critical to determine if an aptitude test is usable or not.
LEVEL OF DIFFICULTY
|INDEX RANGE||DISCRIMINATION LEVEL||ACTION|
|0.19 AND BELOW||POOR ITEM||SHOULD BE ELIMINATED OR NEED TO BE REVISED|
|0.20-0.29||MARGINAL ITEM||NEEDS SOME REVISION|
|0.30-0.39||REASONABLY GOOD ITEM||FOR IMPROVEMENT|
|0.40 AND ABOVE||VERY GOOD ITEM||RETAIN|
Ideal discriminatory index: The ideal discriminatory index for a question should be 0.4 and above. A cognitive assessment where each question has a discriminatory index of 0.4 or above, will be able to effectively filter our undesirable talent and select desirable talent accurately suited for a job role.
The Mettl Advantage: Addressing Special Circumstances
How does Mettl customize and benchmark assessments to ensure accurate talent assessment?
By matching the talent pool available with the job role requirement.
Case 1: Overqualified Talent : Step up the game (Quadrant 1)
If the talent pool possess higher intelligence than the job role demands, chances are that the cognitive test ideal for that job role/level will be too easy for the talent pool, which means-Higher number of candidates will ace the test The company will end up with a large pool of screened candidates.
Problem: More number of candidates than available job openings.
Solution: Pick the best, increase the difficulty level of the assessment, so that lesser number of candidates will be able to ace it. The company will go away with the cream-de la-cream of the talent.
Case 2: Underqualified talent : Mellow it down (Quadrant 2)
If the talent pool possess lower intelligence than the job role demands, chances are that the cognitive test ideal for that job role/level will be too difficult for the talent pool, which means-
Lower number of candidates will ace the test. The company will end up with a very small, compromised pool of screened candidates.
Problem: Less number of candidates than available job openings.
Solution: Make the best of a bad situation, decrease the difficulty level of the assessment, so that more number of candidates will be able to ace it. The company will still be able to fill its openings, even if with slightly compromised talent.
Suggestions: Candidates that lie in this quadrant are ideal participants for learning and development programs to upskill them.
Case 3: Special Case
In case of Quadrant 2 and 4, the cognitive abilities possessed by the candidate pool coincides with those required by the job role and hence will result in good job performance, higher job satisfaction and better work-life balance.
COGNITIVE INTELLIGENCE ASSESSMENT: USE CASES
The correlation between cognitive intelligence and work performance is 51% (minimum) and it can go as high as 84%.
High intelligence is an indicator of success on the job which is measured on the basis of:
• Evaluation of performance on tasks similar to those encountered on the job
• Performance ratings by supervisors
• Position in the occupational hierarchy
Why can measuring cognitive intelligence help in predicting success at work?
Cognitive intelligence mainly influences performance through the rate at which people learn knowledge relevant to the job – people with higher cognitive intelligence learn faster. But cognitive intelligence predicts success even when you take account of job knowledge. With high cognitive intelligence, people are more able to go beyond existing job knowledge and make judgements in unfamiliar situations.
Use of cognitive assessments/aptitude tests has increased by 29% overall
Its usage during hiring has increased by 28% and during L&D by 101%. The increased usage during L&D is driven by the fact that high cognitive intelligence equals to high learning agility and as IT sector is constantly evolving, its important that the employees in this field are highly agile in learning to keep up with new technologies
Use of cognitive has increased by 411% in L&D
Cognitive assessments/aptitude tests are the most commonly used assessments in pharma industry with 40% usage among all assessment tools. They have taken precedence over technical and psychometric with companies focusing more on subject matter knowledge and pharma manufacturing skills, which requires candidates/employees to have high cognitive abilities.
Usage of Cognitive assessments/aptitude tests has increased by 207% overall
Increasing demand for financial analysts and wealth managers from insurance industry has led to a sudden rise in usage of cognitive assessments/aptitude tests during hiring and L&D. These roles require high data processing power and analytics skills which can be assessed via cognitive assessments.
Usage of cognitive assessments/aptitude tests has increased by 99% overall
The surge is usage is led by increase in demand for consultants that can work across various industries and learn the tricks of the trade fast. This requires high brain processing power which can be easily tested via cognitive assessments.
Employee knowledge and skills are the backbones of an organisation. The ability to learn and apply knowledge and skills to succeed at any job role is what cognitive assessments measure. The more accurately they can measure a candidate’s cognitive intelligence, the better quality talent an organisation will end up with.
Top talent is spoiled for choice. They are limited, and they are wanted. The faster you identify and onboard them, the better, as they are not available in the market for long.
Effective cognitive assessments provide organisations with the power to quickly and effectively differentiate the real talent from the fake one.