Practical tips for AI development

Over four years, researchers from the AI@Work group studied how AI is developed and applied within organizations. Based on in this in-depth ethnographic research, the team designed practical recommendations for AI development.

The AIKnow, an NWO funded Open Competition project, followed AI from development in the lab to its use on the work floor. The team, consisting of Marleen Huysman, Ella Hafermalz, Anastasia Sergeeva, Tomislav Karacic, Mario de Sosa and Anne Mayer, studied AI development and its use in organizational contexts.

Together with practitioners the team developed practical recommendations for organizations and launched Managing AIWISEly — a new program designed to train professionals as AI Polymaths, equipped to co-produce data, co-explain, and co-deploy AI.

Click to download the flyer with practical recommendations

The research is supported by the NWO Open Competition Grant “AIKnow: Is AI outsmarting us? The impact of AI on knowledge work” (file number 406.18.E8.030). The flyer is licensed under CC BY-NC-ND 4.0

Discover AI@Work research group

Meet the community of researchers who study AI and its consequences for organizations.

Managing AI WISEly Program

Interested in managing AI effectively in your organization?

Check out our new EE program

In this video, we dive into the case of developing AI technology for hiring, unpack what went wrong, and—more importantly—what we can learn from it.

In one project, a complex matching algorithm for recruiting job candidates failed because it didn’t align with the recruiters' existing workflow. In contrast, a simpler tool that spotlighted overlooked candidates succeeded by addressing a specific, practical need. Lesson learned: instead of developing ambitious cutting-edge tools, focus on supporting the everyday workflows of experts.

In this video, we dive into the case of using AI to identify cucumber traits for breeding, unpack what went wrong, and—more importantly—what we can learn from it.

Pitfall 1: Treating domain experts as users instead of co-designers

🙅 Unproductive Question:
How can we get experts to trust and use our models?
🙌 Productive Question:
How can we involve experts and decision makers in deciding what our model should focus on from the beginning
Pitfall 2: Aiming to produce “unbiased” predictions that experts will follow

🙅 Unproductive Question:
How can we produce predictions that are better and less biased than those currently offered by experts?
🙌 Productive Question:
What new insights, not currently available to experts but helpful for their decisions, can we realistically offer?
Pitfall 3: Underestimating the collective effort required to organize and label data

🙅 Unproductive Question:
What goals do we want to optimize for and what datasets are available for building new models?
🙌 Productive Question:
What specific goal can we agree on together with experts and decision makers, so that we can focus our data sourcing and labeling efforts on the most critical parameters?

Tip 1: Co-design with experts

💡Involve experts early to ensure the model addresses real needs.
Tip 2: Give experts a new perspective

💡 Deliver new insights, not just accuracy.
Tip 3: Plan for data challenges

💡 Ensure aims are narrow enough to be operationalized.

AIKNOW Project team

These recommendations are based on ethnographic research conducted by the reserchers from the AI@Work research group at the KIN Center for Digital Innovation.

dr. Wendy Günther

Assistant Professor

Find out more