Table of Contents
- Reduce costs and frustration by considering the roles of analysts, data engineers, and software engineers in data teams
- people unfit for work
- If you probably want an analyst
- If you probably want a data engineer
- Perhaps if you want a software engineer
- Conclusion
Reduce costs and frustration by considering the roles of analysts, data engineers, and software engineers in data teams
Don’t Hire a Data Scientist – You May Not Need It (Image Source: Nadine Shaabana on Unsplash )
people unfit for work
I’ve seen it in many companies I’ve worked for or consulted with. It’s not hiring the right people for the job!
We still hear a lot of bad stories about underperforming data science teams, management losing confidence in data science in general, and data scientists unhappy with their roles. If you think deeply, you can see a common pattern in these stories.
We keep hiring data scientists, but we don’t really understand where they are most valuable.
In many organizations, analysts and data engineers are what they really need. Sometimes the job isn’t about data and insights, and software engineers and developers are a much better fit for the job.
Data scientists have a dangerous mix of attributes (skills-job mismatch) that exacerbate this problem:
- A data scientist with a broad skill set has the foundational competencies to tackle most problems, but it takes longer and costs more than some experts.
- Because [data scientists] are careers for curious people, many of them end up working on issues that would otherwise be outside the scope of their role.
- To get a data science job, you have to compete , so for some people, becoming a data scientist means being happy to take on jobs that don’t suit them in order to satisfy others and be a good citizen. do.
- The inexperience of early-career data scientists and management , combined with a confusing and inadequate definition of the role (because no one seems to agree on what a data scientist is/isn’t) , misallocate resources to appropriate tasks.
The examples above are endless. I wouldn’t be surprised to see a data scientist who has been doing BI and basic database development for the past year and a half. These jobs are better suited to specialists than data scientists.
There is a huge mismatch between what an organization should provide for a job and the skills it allocates to the job.
If you probably want an analyst
Are you experimenting to steer and transform your business, or are you supporting your day-to-day business decisions? (Image source: Luke Chesser on Unsplash )
Knowing the right combination of skills is difficult. There is no simple formula for solving this problem, as there are many intertwined factors such as organization, industry, function, team and individual personality. In general, we need more analysts. Hiring a data scientist to create basic reports on business metrics is not the best use of your skills and resources.
Statisticians are a rarity these days. There are many statisticians out there doing great work, and organizations are in desperate need of these skill sets. According to sites like Glassdoor and Stack Overflow, the average salary of a data scientist is about 21% higher than that of a statistician . Many data scientists are jacks-of-all-trades, so it’s fairly easy for someone with a strong background in statistics to pick up the basics of data management and processing and get a big pay raise.Glassdoor is
an American website
where companies are reviewed by employees or former employees . It was acquired by Japan’s Recruit Holdings for $1.2 billion in 2018, but has been granted management independence. At the time of writing the translation of this article at the end of November 2021, the average annual salary of a UK data scientist listed on the site (where the author of this article, Adam Sroka, lives) was £46,953, according to statistics. For academics it was £39,376, the former about 19% higher
than .
But getting a new job is only half the problem. The statistician-turned-data-scientist finds himself in a completely different world. There’s a reason many data scientists have such a wide range of skill sets. I used to spend most of my day running statistical models, interpreting data, and writing reports to gain insights from company data. But now that my salary has increased by 20%, collecting data, building data pipelines, creating simple reports and slide decks, writing SQL to get the basic information I need to start analyzing, etc. You will spend almost all your time. These tasks are not the heart of a data scientist’s skill set. I became a data scientist and now find myself in an entry-level data analyst job supporting day-to-day business functions with simple analysis and reporting. Because we don’t have the right people to do the job.
When you compare the value you get from an analyst to the value you get from a statistician or a data scientist, the difference is huge. Analysts are essential to delivering the right insights and information to the right people at the right time. They provide analysis day in and day out. Most organizations are in hell surrounded by Excel spreadsheets, and analysts can bring those people back to their senses.
Data scientists and statisticians, on the other hand, should offer longer-term transformative services. Conduct experiments to derive new knowledge, test new hypotheses, and develop predictive capabilities and new models as part of a long-term research plan. These abilities make data science jobs more rewarding. There is no doubt that these skills require a great deal of time and practice. And it is precisely because of their specialized analytical skills that data scientists are useful to organizations without a deep understanding of the industry.
It’s not that one (data scientist or data analyst) is better than the other, they all have their place and serve different purposes.
The convergence of so many skills and nuanced roles into the data science murky waters has left many organizations in a predicament.
Workers with these data scientist skills may find themselves in less fulfilling jobs, even if they earn more. Additionally, organizations may be paying above-market prices for the skills they really need, while hiring less-skilled people for certain tasks.
If you probably want a data engineer
If you want to connect pipes, a plumber is a good choice (Image source: Crystal Kwok on Unsplash )
When I started data science, the term data engineer was not often heard. Looking at the Google Trends chart below, you can see the difference in popularity (hype) between the two occupations and the word “data engineer” in red versus ” data scientist ” in blue did not become popular until around 2014. There was certainly a lot of talk about data engineering, but many data scientists (myself included) had to do all the work themselves.
Changes in interest over time obtained from Google Trends for the search keywords “data scientist” in blue and “data engineer” in red ( Image source )
It’s no longer 2014. However, it is rather surprising that there is such a gap even now (the difference in popularity between data scientists and data engineers).
Most companies would rather hire data engineers than data scientists.
Hiring data scientists helps shape your needs for data and analytical capabilities early on, as they are often involved in different areas of your business. They usually excel at loose scoping and soft requirements. However, once the requirements become clear, a large amount of data management and processing becomes necessary very quickly.
Laying the groundwork for a data platform can take months, even for the simplest requirements. Also, as the platform grows, new requirements will emerge and maintenance costs will increase. I highly recommend hiring a good data engineer as early as possible. Many data scientists would make good data engineers, but they need both skills and knowledge of best practices when building and designing data solutions. Additionally, data engineers tend to be good at CI/CD, building tests, and all the good engineering practices we’ve relied on to grow our tech stack. Data scientists, on the other hand, should focus on single-shot experiments to deliver the most value.
There is also a lot of value to be gained from strong data engineering that supports functions across the business. Many data engineers have BI experience, so they have the advantage of having a detailed understanding of the data needs of business users and analysts.
Perhaps if you want a software engineer
Leave building scalable software to the experts (Image source: Yancy Min on Unsplash )
Advanced analytics and machine learning are increasingly central to large-scale applications. As such, the data team is central to the development process, shaping the application path and working with other teams to deliver the application.
However, many organizations fail here.
Just because you have machine learning in your application doesn’t mean you need a team of just data scientists.
Data scientists aren’t (at least they shouldn’t be) developing robust, scalable, interactive, and shippable software. The most value you can get from a data scientist is to give them a question with a clear, measurable deliverable and some data, and let them iterate on that question and come up with some answer. Of course, data scientists also play an important role in translating high-level proof-of-concept solutions and handing them over to other teams. But to include them in the project-wide dependencies would be a gross misallocation of resources.(*2) Regarding the role of the data scientist, who translates the needs of corporate management for data science projects into the software requirements that software engineers need, see the AINOW translated article, “
3 Every Data Science Team Should Hire. ” Two Often Overlooked Roles ”, headline
1. Data Science Translators .
What you really need is a strong team of software engineers. A data scientist can create an interactive GUI, but that job is much better left to a front-end developer. And while data scientists will be able to build pipelines that deliver data from source to sink, backend developers, data engineers, and testers will build more robust and scalable ones.
Even if a software engineer doesn’t understand how an AI model works, it can work well with the rest of the application to be part of a shippable, scalable piece of software.
Let the data scientists focus on the question-driven, scientific side of things, and leave the engineering to the engineers.
Engage data scientists better in projects, feel happier, write higher quality code, work on more edge cases, and reduce their frustration and frustration. let’s do it They are essential in some situations, but can be a total hindrance in others.
Conclusion
With so many people wanting to become a data scientist, the nuances of other data-related jobs are lost. Furthermore, many recruiters don’t understand what good data scientists can contribute, when they’re needed, and don’t have the experience to learn more about them. So many people ended up in the wrong roles and overall frustration.
The next time you want to advertise a data scientist job, think about what the core of that role is. Will the people you want to hire work in research and science, shaping the future direction of the company’s journey, or will they do basic engineering and support business as usual?
