How to get your ML models into production faster in 2022?

A Short Guide to Implementing ML Models Faster and Creating New Competitive Advantages

Through various AI projects, I’ve realized how building and implementing efficient machine learning ML models can quickly become a source of competitive advantage. Decision makers are learning that managing the full lifecycle of building, implementing, and debugging models within existing technology stacks, beyond data collection, can be tricky and pose unexpected challenges. .

In my experience, data scientists often spend time analyzing datasets, then looking for suitable algorithms, training new models, and handing them off to data engineers to run in production.

In this situation, the problem can arise that the data scientist is unaware of the challenges of running the model in production and the data engineer is unaware of how the model is structured. I’ve seen data scientists write applications that don’t scale in production.

Implementing ML models can be even more difficult for small companies that don’t have data engineers. The result is a poorly built ML implementation pipeline that keeps updating ML models and not making them work reliably in production.

Only a small percentage of machine learning and deep learning projects make it to production due to implementation complexity, lack of governance tools, and data issues .

Put your ML model into production

I have noticed that many companies start AI projects without a clearly defined product plan. A lot of the time no one really thinks about what happens after a proof of concept. This lack of strategy often causes significant problems when teams have to integrate models into production.

Depending on company size, experience managing AI projects, budget, and team, decision makers may decide to focus solely on training models for proof-of-concept without thinking about productivity or scalability. .

In such cases, the data science team may decide to wrap the model in an API and ship it directly to production. However, in these cases, they lack a scalable and reproducible plan for updating and debugging models in production. Therefore, engineers struggle to get systems running reliably, which can result in high ongoing support costs.

The lack of automated tasks in the machine learning lifecycle often leaves models with custom-built data pipelines and implementation environments for each model .

As described above, the lack of a consistent methodology (from model development to implementation in the production environment) leads to project fragmentation, which increases the cost of improving the ecosystem and slows down development. Additionally, teams write models in their preferred language (often R or Python) and rewrite them in another language to run in production frameworks (built with Java /Spark, etc.) is also common.

Also, an ML model consists of several elements such as datasets, parameters, and hyperparameters. These components should be carefully stored and versioned throughout the development pipeline to ensure reproducibility.

What you can do to get your ML model into production faster

MLOps are a hot topic in machine learning. To summarize, MLOps is a logical methodology that seeks to standardize and automate most data science tasks while building a collaborative environment for continuous development and rapid implementation of ML models.

MLOps makes it easy to implement models written in any open source language, with production-quality REST APIs to support real-time predictions. It has undergone a similar transition (leading to MLOps) like DevOps, which is well known in software engineering.

Successful companies leveraging MLops solutions employ similar best practices. These practices include standardized central pipelines for data preparation/training and updatable, reproducible workflows for future implementations.

Next to MLOps is a model management solution that makes it easy to compare different or new models on various metrics. Last but not least is the MLOps flow, which is based on continuous integration and delivery (CI/CD) of data and ML applications and supports all frameworks used in the enterprise, such as Torch and Tensorflow. ing.

A list of factors to consider to go one step further:

• Version control to keep track of all algorithms , trained models, datasets , metadata, and hyperparameters used. Containers (Docker, Kubernetes, etc.) are a great way to isolate and version your models, so consider using them.
• Automatic safeguards to avoid relying on manual or inconsistent processes.
• A model repository with a central repository where production-ready models can be stored, annotated and managed. This element is also relevant to setting up a data science team. What I’ve noticed is how difficult it is for employees to come in, see a project, and identify who is working on it and where the data is.
• For computer vision projects, you might consider using an AI toolkit with pretrained models. Specifically, you can bring your own data to fine-tune pretrained models for your specific use case. For example, use one of NVIDIA’s versatile, production-quality models for common AI tasks, or use one of over 100 combinations of neural network architectures such as ResNet, VGG, FasterRCNN, RetinaNet, YOLOv3/v4, and more. You could use one.
• Dashboard for monitoring resource management (GPU/CPU)
• Automation to monitor the performance of deployed models and roll back when new models perform poorly. Many models experience drift that degrades performance over time. Introduced models must be monitored and refitted. Each implementation model should log all inputs, outputs, and exceptions.
• Provide model builders with a way to self-implement models that have been approved by the internal model governance process.
• Prepare a development environment for testing and debugging ML models before deploying them to production.
• Preparing elastic clusters for testing different hyperparameter combinations in production.

These elements will help your company handle many repetitive tasks and lead to faster release cycles. What is another essential aspect of the ML development lifecycle that can benefit from MLOps? It’s data.

Data problem

As readers familiar with ML projects may know, manual data preparation and wrangling accounts for 80% of the time spent on ML projects.

Data scientists need tools for data wrangling, preparation, prototyping, and visualization to reduce this time. By leveraging MLOps pipelines, data science teams can accomplish these tasks faster and more efficiently. Advanced MLOps solutions can support ML projects from data selection and annotation to model optimization.

In computer vision projects, this need for data optimization becomes very important. Some companies have brought their own expertise in optimizing MLOps pipelines and training data.

Some companies are building end-to-end learning data platforms that outsource the entire learning data workflow, from new data selection, annotation, and model monitoring. At these companies, data scientists can focus on model selection and evaluation.

Some edge cases require high accuracy and need to be addressed before going into production. Some platforms proactively identify edge cases by anticipating all kinds of errors that may be present in a task and assigning them numerical penalty values ​​before the annotation process begins. This forward-looking approach empowers data science teams to provide comprehensive, actionable direction to get to production faster.

Additionally, the data science team should continue to receive sample results based on the initial guidelines. By doing so, you can prioritize and target which data should be annotated, and iteratively dictate faster model development.

Implementation strategy

Finally, I strongly advise companies not to just vaguely put their models into production. Companies should explore different ways to deploy AI-based solutions. Implementations of ‘shadow mode’ and ‘canary’ as described by Christopher Samullah are particularly useful for ML applications. In “Shadow Mode,” you can see the inputs to a new productized model and the predictions that the model outputs without actually putting it into production. In this mode, you are free to analyze the results without much impact if bugs are detected in exchange for not implementing them.

ML projects are no longer based solely on data science. It starts with clearly defining goals (e.g. acceptable level of accuracy) and fully embracing agile methodologies (e.g. iterating until you find the right algorithm, model, and accuracy).

Companies should seriously consider the time it takes to put models into production. Competitors gain a competitive advantage by getting models into production faster and more efficiently.

In addition, it is necessary to confirm that the accuracy of the model is maintained after it goes into production. Oversight of the implemented model can lead to bad surprises with respect to model performance, which surprises may require model retraining (eg, concept drift).