Faster value from machine learning projects – a collection of best practices learned from experience

The value of machine learning (ML) projects is found in the productionalization that results from them. Even though most of this business value centers on people and good business cases with tight scoping, much of the discussion still revolves around the technical aspects of machine learning projects.

Thanks to their complicated nature, it’s common for these projects to drag out from one or two months time-to-value up to a year or longer, with most of that time spent developing data infrastructure. In this blog post, I’ll discuss what I’ve learned through my experience of multiple idea-to-production ML projects; as you’ll see, only a few of these learnings relate to the technical aspects of the projects themselves.

Break the business case down into the smallest chunks possible

To create an actionable machine learning case, the business cases and the related activities should be very clearly defined. This is because an ML model answers the specific questions for which it is created; these questions can be broadly categorized as scoring, classification, and prediction.

Read more - how to find machine learning use cases?

The following business case provides a clear example of why such a breakdown is needed: we want to improve employee satisfaction as less commitment decreases employee performance. Here we have identified a driver and an outcome, but we’re still missing the actual action we want to trigger to improve satisfaction. In this case, the tasks for ML could be to create a categorization for employee satisfaction (data exploration to understand broader trends), predict when changes will occur between the categories (identify when to take action) then based on these changes recommend actions that can lead to a positive state change (find causes and recommend action).

Attach projects to a value stream

Attaching a project to a value stream raises its potential to result in a usable product with real-world impact. Many projects have potential, but because they miss any direct link to the business’s value stream, they can fall into a space where they are not being driven forwards.

Ask, don't assume

When it comes to data, it’s much better to ask a domain expert than to assume that some data is an outlier that can be dismissed. These insights can often become precious and even become business cases in their own right. In particular, manually inputted data can often be flawed, and missing values are expected.

Avoid analyzing the business case to death

It’s so much better to start implementing a project with partial information than to continue the business case analysis for too long – after all, you’ll never have all the details. Although good planning makes sense in the long run, with machine learning solutions, the project can change direction dramatically during the early data exploration phases, so there is a strong incentive to start early even if some details are still unknown.

Be clear on the drivers you’re trying to influence

For many companies, ML projects are not inherently value-adding. This is why all projects should be driving business results and focused either on core metrics like conversion rate improvement or relating to customer experience such as net promoter score (NPS). Regardless of what the business target is, the metrics that the project aims to impact should be clearly defined in order to identify the sub-drivers for the projects. For example, improved customer experience measured through NPS is based on multiple drivers, including ease of use, findability, logistics, and customer service.

Read more - six inspirational ways of making money with data

Scope – solve one problem at a time

Fast business value is best achieved with a laser focus on one single problem at a time. Particularly with ML projects, we should be very clear about the scope; this is especially important in an ML context because these cases often involve the first really deep dive into some data assets that can, in turn give rise to new ideas and questions. Rather than shifting focus mid-project, these ideas and questions should be acknowledged and recorded as backlog items to be assessed once the original scope is fulfilled – it’s better to deliver on the original scope than to fail to deliver on five new ideas.

Identify the right data and assign the right people at the right time

ML projects should be near the value-creating core of the business; it’s necessary to attach the right people to the project. Business and technical owners are needed with a vested interest in driving the project forward and clearing obstacles from the development team’s path. Business owners should take the role of internal champion for the project, driving the adoption of the solution and helping to overcome any technical infrastructure or access issues. ML projects are also slightly more involved for stakeholders. With many solutions being so close to the value-creating core, a lot of process work and training may be needed within the organization.

An optimally structured development team would be able to deliver end-to-end development. ML projects are often delivered in fast development sprints of around one to two weeks to allow a short feedback loop.

An optimally structured team might look like this:

• Multidisciplinary team
• Service design
• Business design
• Data scientist
• Data engineer X2
• Business owner
• Technical owner (can be the same person as the business owner)

Include machine learning within individual business functions

In many companies, machine learning solutions are developed by centralized teams that rotate from project to project, working in different business units with only short stints in each business context. This leads to shallow point solutions that just answer the business unit’s specific problems. This kind of approach can be very successful if the business units and the problem spaces share many similarities. Still, it can also be detrimental to the full adoption of the ML solutions produced.

In order to get the most out of machine learning, it is highly beneficial for machine learning specialists to be integrated into teams. This integration allows them to build deeper domain understanding and familiarise themselves with the intricacies of the value creation process. It also makes the team more committed to the long-term success of the ML projects (actual value creation) and can improve the identification and proactive pitching of new machine learning solutions for the business.

Another benefit often missed is the inclusion of data professionals in solution planning and architecture, which allows data and machine learning to be taken into account in the early phases of planning a solution rather than being an afterthought. This enables faster and more varied data business case development, as these aspects are noted in the initial solution development. Ultimately, any solutions should always be owned by the business.

Start simple

Experience has shown me that around 80% of the value of ML cases can be captured with relatively simplistic models and architecture. This means that at the outset of your project, you probably won't need to account for the four-yearly seasonal effect of the Olympics on your weekly forecasting model, for example. This simplified approach also enables faster development and testing cycles. Doing a quick and dirty test run can save you many headaches down the line. In this sense, the same applies in ML as in development in general: “premature optimization is the root of all evil” – any technical sophistication and integrations can follow at a later stage.

Beware of scope creep – stay focused

When data exploration starts, the project team and stakeholders often start finding new avenues based on the data. This can often lead to a new round of ideation around the topic, which can cause branching of focus that will most likely be detrimental to the original goal. Ideation can be very beneficial but it should be handled in a highly structured manner by adding new avenues to the backlog and investigating further only when the original goal has been achieved or there is a need to pivot the original goal due to model or data limitations. You should handle new ideas with care, as shutting down people’s views and ideas can leave them thinking that you’re fixated on progressing with your own. It is essential to communicate clearly that new ideas cannot be pursued due to the limitations of the current project scope but that they will be included in the backlog from which new business cases can be built.

Communicate constantly

In comparison with traditional software development projects, ML projects have more uncertainty built-in due to data quality, integrations, modeling work, and so on. It’s essential to communicate with all relevant stakeholders about any unsuccessful results or poor performance, as there may be domain-specific data issues or other anomalies in the data collection process that need to be fixed. Any communication should be addressed to a broad audience, and during the project, you should keep every end-user informed about how the solution will affect their workflow.

For this reason, it is recommended that you communicate more during the data exploration phase than you would in traditional software projects to build trust among stakeholders and identify any domain-specific data issues early on.

During the modeling phase, the following themes should be communicated, although there is no need to go too deep into the technical aspects:

• the data and features
• the model and how it works
• the important features
• the output of the model
• the performance of the model

It’s also recommended that meetings are held at least weekly, or even short ones each day.

A/B test everything

If the business case and technical setup allow, I recommend that you run testing in a rapid cycle to form a development-business feedback loop. Depending on the case, backtesting, simulations and live A/B testing should also be run on limited exposure to testing for any interactions or anomalies that have not yet been accounted for. An excellent example of this is that while a recommender model may perform very well in offline testing when run with users, we may notice an adverse effect on some metrics that were not the targets of the optimization.

Encourage fast development cycles

Achieving a fast feedback cycle between data professionals and business owners can be the difference between success and failure. Machine learning solutions benefit significantly from a short feedback cycle. This is especially true in the early data exploration phase, where assumptions about the data sources are validated. My recommendation is to hold weekly meetings, but daily sessions should be considered during the most intensive data exploration phase.

A fast feedback cycle is also relevant to the following key point. If the data or methodologies do not support the initial business cases, pivot your machine learning projects quickly and move to some adjacent aspect of the problem. Pivoting quickly can save much time and allow for the creation of backlog items for further data collection, or infrastructure can be created for the original idea. The purpose of the fast feedback cycle is to speed up the idea–MVP–production–backlog cycle to enable the rapid testing of ideas, if not in online systems at least offline.

Aim for continuous improvement and plan for maintainability

Because most machine learning projects deal with complex processes and are never truly finished, two major themes need to be addressed in every ML solution: continuous improvement and maintainability.

Continuous improvement plays a big part in machine learning projects, and it can be achieved through two intertwining routes:

1. The first iteration of the machine learning solution might bring only modest results as there is only partial data available for training the models – and perhaps some data will even be gathered through the first iteration itself.
2. As the solution produces more results and the functionality is better understood, new ideas for improvement are generated, which can multiply the original solution's effectiveness.

For these reasons, when planning and budgeting for an ML project it’s a good idea to factor in at least one or two smaller sprints after the solution has been productionised.

Maintainability is the second central theme that must be addressed when planning an ML solution. Machine learning solutions are often built and maintained by small, dedicated teams, creating the challenge of navigating a trade-off between new development, continuous improvement, and maintenance work. This trade-off should not be taken lightly, as any development throughput will eventually lead to more maintenance work, which can come at the expense of new development or continuous improvement. This is why, when productionising the solution, the maintenance aspect should be specifically addressed and the solution should be created in a way that requires only low-level maintenance that any team member can perform in order to avoid locking people into maintenance roles.

Use existing tools wherever possible

A core concept that has greatly helped with the final push in machine learning solution projects is using the organization’s existing tools. This is important because the project value is often generated in cooperation with human end users (human-in-the-loop solutions), which in turn leads to process, workflow, and organisational changes. The use of existing tools increases the odds of successful process change as it eliminates one more moving part from the productionisation of the solution. In addition, end-users are more likely to use the solution if the tool is already familiar.

Lean architecture

To create fast value with ML, do not start the project with the aim of building a data platform for all possible past, current, and future business cases. Your project can be part of a process where solutions are migrated to a new, better-managed data platform one use case at a time, without blocking the opportunity to start creating value from your data. This will allow the project team to extract more value from selected source system data rather than wasting time waiting for ETLs and API access to be built onto a centralized data platform that contains all possible data starting from the 1990s.

Remember the people – plan rollout in plenty of time and follow up on the solution

Many machine learning projects produce solutions designed to help experts or end users work, and real value is only generated when the solution is integrated with the process and end-user workflow. As the projects often create human-in-the-loop solutions, communication, testing, and materials are required for end-users. This means that the rollout of a solution needs to be well planned out, especially in terms of how it changes the value creation process.

If anything, communication and materials should be sent too frequently to end-users and stakeholders, starting right from the early phases of the project. This ensures that information about any workflow and business process changes that the solution will cause (or require) is disseminated sufficiently. If the solution changes the workflow, training should also be organized to ensure that users fully understand the solution and its intended use. Including actual end-users throughout this process makes it possible to establish a complete chain feedback loop and ensure that any process intricacies are considered in the final product.

Be mindful of the importance of monitoring

Monitoring can be looked at from two perspectives. The more important of these is monitoring whether the ML solution is creating the desired impact – and if not, why not. Secondly, as ML solutions are never fully “ready”, the model’s performance should be monitored closely to identify model drift and look out for changes in data or underlying behavior.

It’s so much easier to improve an existing venture than to launch a new one

In some companies, there is still some disconnect between operations and data. Even if some visualization dashboards and ML are running, the organization may still not be running together with its data. In this kind of situation, it can be tempting to build ML adoption around some new data-centered business idea when starting on your ML journey. In most cases, I would strongly advise against this; I’ve never seen a company where all of the core value streams are so data-intensive that there are no new ML business cases to explore that can improve operations. You should be wary about building ML adoption around new business models because, with so many other factors beyond ML project success influencing overall success, the ML case is often one of the smallest elements that need to succeed.