Delivering Impactful AI/ML Services: Handing Off More than a Codebase

Many potential roadblocks stand between the conception of a machine learning project and its ultimate success. The majority of ML and AI projects never fully make it into production, even despite the best efforts of developers, scientists, engineers, and consultants. When the team at Valkyrie takes on a project, we know that success depends on more than delivering a first-class ML model or AI solution. We ensure that our clients have the resources to get the most out of the work, an understanding of any risks or limitations, and the organizational support needed to carry the project into the future.

Sharing Knowledge to Expedite Success

Scientists put a lot of thought and time into the AI and ML models they develop, and it can be disheartening to watch a project stall before the solution can be deployed to end users. The most rewarding aspect of the job is usually not the statistical accuracy of a model, but rather the positive effects of a solution that has been deployed successfully. Valkyrie scientists don’t just want to get our checks and sign-off for the day; we want to see our ideas turn into tangible benefits for clients and stakeholders. Through our work, we’ve developed solutions that have stopped greywater from spilling into the ocean, advised policymakers on neighborhoods at risk for experiencing homelessness, and predicted the spread of COVID-19 during the height of the pandemic.

Consultants often participate in the initial stages of ideation, development, and first deployment, but the longer-term processes of final productionalization and ongoing maintenance often fall outside the scope of a consulting contract. A crucial component of delivering impactful AI is providing a roadmap between a proof of concept and a truly operational product in order to equip the client with what they need to successfully maintain and grow the AI solution. At the end of an engagement, we never want a client to ask themselves, “So now what do I do with this thing?”.

Knowledge transfer means so much more than documented code and a final readout. Complex designs take time to digest, and questions may not occur to stakeholders until to work through parts of the system on their own. Important details about computing resources, maintenance schedules, staff skill requirements, and other operational concerns should be discussed early in a project and revisited throughout development. By the time clients receive the final deliverables, they should be ready to take the wheel instead of opening an instruction manual. Successful knowledge transfer means that client stakeholders not only know the next steps to take with the deployment of an AI model, but they are prepared to take those steps as soon as possible and with minimal help from external teams.

Understanding the Responsibility to Avoid Misuse

In addition to sharing positive aspects of a model such as expected performance and ROI, a good team should make sure that client stakeholders understand the limits of an AI or ML implementation. As the saying goes, “All models are wrong, but some are useful.” The usefulness of a model quickly vanishes when it is stretched to an ill-suited application. Client stakeholders must be aware of the assumptions baked into a model, and the ranges beyond which those assumptions will fail. For example, an AI navigation system developed for shipping logistics might be a poor fit for routing emergency vehicles. Consultants should also clarify any technical restrictions of the particular approach used, such as hardware requirements or scale constraints. Helping clients avoid potential traps and pitfalls can go a long way toward ensuring the successful deployment of an AI system with maximum impact.

Valkyrie places a lot of emphasis on ethics and accountability in our company culture, right down to the way we design our algorithms. We implement formal processes to consider what issues could appear if our models fail, and we take care to look out for unintended consequences that could occur even when the models work as designed. A complete knowledge transfer process includes sharing ethical considerations with clients, and consultants have a duty to educate clients about avoiding misuse or negative side effects. Scientists want to deploy impactful AI, but more importantly, we want to deploy AI with a positive impact. Achieving that goal requires that client stakeholders understand the responsibility that powerful AI models demand.

Recognizing the Importance of Invested Stakeholders

No matter how sophisticated an ML model or AI implementation may be, no project can survive without the key ingredient of invested stakeholders. From the beginning of an engagement, a consulting team should look to identify two essential types of client stakeholders: technical individual contributors to keep the system running smoothly and leadership champions who will advocate for the project’s success. Throughout the development process, consultants must provide client technical stakeholders with the knowledge and skills that they will need as the eventual operators of the AI/ML codebase. At the same time, the consulting team should meet with client leadership to ensure that the delivered solution will align with the leadership vision. Investing in these relationships is a critical factor in leaving the client in a strong position once the models are in their hands.

In some circumstances, a company may lack the appropriate technical roles to operate an ML model, and external teams occasionally encounter internal stakeholders who have general reservations about AI or question how it can contribute to their business. In the case of missing technical resources, consultants can offer guidance about the type of science or engineering staff that a client may need to bring on board in order to deploy their solution successfully. With reluctant or doubtful stakeholders, open a dialogue about their workflow and pain points, and present them with concrete examples of how prior work has added value in similar circumstances. Most importantly, listen to their concerns and answer their questions, so that the client can make informed decisions as they incorporate an AI project into their organization.

Conclusion

No ML or AI project will prevail without a high-quality model and a thorough implementation. However, for a project to really thrive, additional information must be incorporated into an operation and integrated into its workflow. Stakeholders must possess critical knowledge about how to keep a model running smoothly and how to steer the project away from potential hazards, and leadership must understand the resources and attention required to maintain the functionality of their AI system and to allow it to grow along with their needs. Whether you are a consultant from an outside group, an internal data scientist working on a prototype, or a graduate student sharing a proof of concept, a key to deploying impactful AI is communicating to other stakeholders the essential information that will make the project successful in the long term.

About the Author:

Austin Hilberg, PhD is a Senior Data Scientist and Engineer at Valkyrie, where he is a member of the Algorithm Accountability Task Force and provides intern mentorship. He earned his PhD in Biomedical Engineering from the University of California, Los Angeles. Since 2018 Austin has applied data science and machine learning to many different domains, from small academic labs and startups to large conglomerates and government agencies.