Pivoting to Data Science or Machine Learning: Navigating Your Skill-Building Journey from IT role - A simple guide for you
Kicking Off Your 2024 Learning Adventure: Choosing Between Data Science, Machine Learning, Analytics, and the World of Generative AI LLMs – Where to Dive In?
Since October last year, I have been writing about the mid-career transition from existing IT roles to the modern digital stack especially AI,ML and data.
By now I have come across 100+ professionals with whom I have spoken over LinkedIn about mid-career transition. I think it will be really cool to playback with my audience at “Purple Slingshot” about some common questions that were asked me during these interesting interactions.
One of the common questions was that if anyone who wishes to even make the first move is to fix the direction to where to look at with respect to modern technology like Data Science, AI or ML?!
“Do I start with a Data Science course or Machine Learning course in 2024?
I hear a lot about large language models with many confusing terms – that makes me think that Generative AI is the next thing I should focus on. But again, “it is so cluttered!”
These are the exact lines I wrote in my black diary after a LinkedIn interview that I took in December 2023.
This newsletter is all about my take on re-looking at all these areas, from a skilling perspective, and that too with pros and cons. My conclusion (of course, a clear winner) is at the end of this newsletter.
So let us start! We will start thinking about the highest project roles (billable) and then see what the options for non-billable job roles.
Data Science – The end goal of a data science professional is to draw data-driven insights for business. The work here as the name states is with “data”. This is an excellent area to point your reskilling compass to if you have a BI, Database, data modeling, SQL, Data Warehouse, ETL, Data analyst, or business analyst background.
For a senior IT professional, the highest non-billable role in this segment is “Director Data Science” and the highest project role is that of a “Principal Data Scientist”.
Non-technical roles may also move into directions such as sales, pre-sales, and product management. I am not talking or discussing more about non-technical or functional roles as these are true for all the stacks that I am discussing here.
The key tech stack here is – SQL, Data analysis using Python/R, Tableau/PowerBI for data analysis, Data mining techniques, and Data engineering, especially with data preparation & cleaning.
Higher skills include planning and executing a Data Science market Strategy if you are working for a firm that is a project/services based organization or may involve setting up a data science culture and team as part of enterprise change if you work in a functional organization or GCC (Global capability center).
If data especially in case of messy data and a business presentation on data sounds boring to you, avoid skilling in data science direction, or the best is to revisit your decision.
Machine Learning – The confusion here is mainly with the Machine Learning roles when compared with data science roles. This is true within the company’s HR departments as well!
I have provided consulting to HR services to lay a clear boundary between ML and Data Science job roles and job suites in an enterprise.
The right way to look at Machine Learning is to think of scaling and engineering.
Data science is a science, whereas today in 2024 Machine Learning is more of engineering with use of massive, big data datasets and using large compute (machines) helping engineers to make predictions.
Let us see this with an example use case and how the same use case maps in a data science vs machine learning role-
Here, a model is a machine learning code that is trained over labeled data (generally) and is asked to make some business prediction. In this case, the prediction is about predicting whether a customer will run away eventually or not.
Since 2012, Machine Learning has changed.
Today Machine Learning is more of an engineering marvel than a science. Most work is around getting such models into production and managing them over a data cycle called Machine learning Operations (MLOps). Yes, it is the machine learning’s alter ego for DevOps!
For a senior IT professional, the highest project role is – Principal Machine Learning Engineer (pays very high). The highest non-billing role is – “Head of Machine Learning”.
Smaller companies with less headcount will/may merge roles from data science and machine learning both at project levels and for non-billing roles as well.
On the other side, enterprises will spin a variant over the enterprise stacks as Databricks, and Google Tensorflow Framework and will also get the mapped skills in Machine Learning in Hyperscalers like AWS, Azure, and GCP. Ex- Being able to deliver MLOps using Azure.
Go for Machine Learning if you like the engineering side of things and if you have a future plan of moving towards AI in the long-run. You will also like these roles if you like SDLC lifecycle and Agile, and if you have a bit of awareness of coding and high school Maths.
Today most machine learning roles do not include high-end Maths in the day job, as it has moved to R/D or algo teams.
Avoid this direction if you do not like coding or have very little appetite for understanding the Maths behind data and forecasting. You will also not like this side of things if engineering/building using open source does not sound cool to you.
On the contrary, I think the demand for mid-career IT professionals is high in Machine Learning Ops because IT professionals know how enterprise systems work at these scales. As I said, today the challenge in machine learning is handling the prediction at scales using the right data and not mastering the stats behind them.
Analytics – This one is the easiest to understand in 2024 as this is an old area now. The Analytics team is an “all-human team,” that makes predictions and analyses without the help of machines or big data sets. These are the typical T-2 to T-n days forecasts and reports.
You may want to think about this role as a combination of Data Science and Machine Learning but at a very small scale. Most of the roles are junior for you and limited to some kind of analysis like “data analysis”.
As of now, this role is limited to niche markets and I do not recommend going for any analytics courses if you have crossed 15 years of experience. Instead, focus on getting machine learning or/and data science skills.
Generative AI – It is about the new blue-eyed lad, and is at the highest of any hype cycle! Gen AI is a type of neural network known as Transformers.
It works with massive words corpus by using a technique that includes Maths and is called vector Embedding. The current trend is to use a base model like GPT or train with your own enterprise data. Training an LLM with their own enterprise data is called fine-tuning and is massively expensive work even for large enterprises.
Today, the roles are very limited for AI engineers who can create end-to-end LLMs or incorporate a base model in current applications.
I believe this technology will evolve a lot and with time will be more cost and energy-efficient. The energy-efficient term is included here as LLM training is conducted using GPU or specialized CPU which consumes electricity.
Today, the work scope in LLM and Generative AI is emerging for experts who can deliver working with a very small team. Roles in such projects include – LLM Principal Engineer and AI Consultant. Non-project roles include working as an AI Strategy Head, Head of AI, AI Product Head, etc.
You can go LLM and you may consider it if you have a very strong applications background as cloud native, full stack etc. There is a great demand to get LLM merged with application stacks but this is a developing story.
Having the experience of working in the field of either Data Science and/or Machine Learning – will set up the right stepping stones for you to work as a Generative AI.
Conclusion:
Machine Learning or Data Science is the right pick for multiple reasons as I discussed above. If you have experience of 15+ years in IT, you by now would be an expert in some sort of software engineering, architecture, coding, and data management in general.
So, choose one of the two – i.e data science (especially if you have liked the era of T-n of reporting) or machine learning (especially MLOps if you have an IT background with project lifecycle).
Between, the two my personal pick is Machine Learning, as I have seen that in modern digital projects organizations lack thoughtful leadership to handle large-scale software projects, and a person with an IT background brings some experience of scale to the table. This is with a caveat that the person has the right skill sets in Machine Learning as well.
What are your thoughts? Please share in the comments, below.