How to grow the pool of talent for data and AI

Finding talent for the insight industry has changed enormously over the last few years. There are new and different clients, different technical skills and an increasingly diverse range of roles across a wide range of company cultures.

The job hasn’t got any easier. The industry’s recent growth has led to a shortage of skilled talent, large variations in salaries, unclear career paths and a lack of understanding among graduates of what a career in the sector can offer. Wherever in the world we recruit, there is a shortage of talent across a very segmented industry.

We used to be asked for proven questionnaire-writing skills and extensive experience of moderating groups. Now it’s just as likely to be experience of programming languages like Python and R, or software like Hadoop. Once, we looked for candidates with a first-class degree in any subject; today it is increasingly likely to be a degree in computer science or maths. Continuous and tracking data skills are gradually giving way to experience in machine learning or social listening.

What stays the same

It isn’t all different. What remains is the need for insight, whatever the intelligence or data source. The technical skills have changed, but many of the competencies are the same: seeing patterns in numbers, a qualitative mindset, storytelling, influencing, seeing the big picture, commercial aptitude and the ability to draw insight from a range of research approaches.

Skills and experience can be taught; competencies are much harder to teach. The research industry is not good at blowing its own trumpet, but many people in it have the competencies needed to provide insight, whatever data it is based on. That matters, because other backgrounds don’t necessarily offer them. Data scientists are used to working with one very specific area of data and getting the detail right, which can make it harder to work across different intelligence sources and see the big picture.

New competencies, and a pay gap

Researchers who want to move into the newer areas need some additional competencies. Data is a dynamic, fast-changing field that needs people who are flexible, fast-paced and prepared to take the occasional leap into the unknown. Not everyone will make the move, but many could. More movement between market research and data would help ease the skills shortage, and might also help even out salaries.

Data scientists are in short supply and command high salaries. In our experience, people with data and AI experience who can work autonomously and support others earn around 30% more than researchers at a similar level, and the gap widens at senior level. That’s a real problem for consultancies trying to attract people with the new skills, and for insight departments whose strategy draws on both kinds of data. How do you justify paying some team members more simply because their experience is with different intelligence sources?

To increase the supply of people with data and AI skills, it needs to be easier to move from one area of intelligence to another. Plenty of researchers have the competencies to build great careers in insight based on data and AI, with additional training and experience.

Mindset and mobility

In this new world, researchers need to be open to new methods, those recruiting need to be more open to different applicants, and there needs to be a better understanding of what the new world of insight requires. Greater mobility is possible with the right investment.

Moving between countries would help too, as different markets have different strengths, but that kind of mobility is becoming much harder.

Mobility from research into data science isn’t the whole answer. In the longer term, if the research industry wants to keep the people it has invested in and trained in data skills, it needs to address the salary gap. It also needs to compete with the culture of purely digital and data organisations, many of which work more flexibly on location and hours.

But greater mobility would really help an industry struggling to find people with all the skills it needs. It would certainly make my job easier. I believe the research industry should be more confident about its strengths, and about what its people can offer in this exciting new world of data.

This article first appeared in Research World.