Researching 37,000 Companies on Human-Level Quality at Scale

Lead research is a critical component of sales and marketing strategies. It often starts with Google searches, a powerful tool when used effectively. However, as essential as this process is, it is time-consuming and labor-intensive, especially when scaled to thousands of companies.

One of our customers, Vergleich.org (VGL), was facing that challenge – for 37,000 rows of data – and we automated the process for them.

This case study explores how we used our AI to automate data collection, reducing costs, and freeing up human resources for more creative tasks.

The Traditional Approach to Lead Research

VGL’s lead research typically started with a Google search: Enter the product name and off they went. This process required researchers to make numerous assumptions, guiding them like "Google machines" toward the information they need.

For instance, if tasked with finding contact information for a company in Asia but only knows the product name, a person might start by searching for the product and filtering through trademark registers, product pages, and company profiles.

This method relies heavily on the researcher's knowledge, context understanding, and ability to navigate vast amounts of information to identify relevant details. While effective, this approach is not scalable when dealing with thousands of products or companies. Or 37,000 in this particular case.

The repetitive nature of the task, combined with the time and effort required, makes it a prime candidate for automation which is what we are going to examine next.

Automating and Scaling High-Quality Research

The primary task at hand was to streamline the data collection process for company research using AI. The challenge lies in scaling this process to handle large volumes. Manually executing this task for 37,000 companies would be time-consuming, monotonous, and hence prohibitively expensive.

Our goal: Develop an AI solution that can mimic the human research process, making intelligent decisions and evaluations while navigating through information online – which is what we did for Vergleich.org.

Developing the AI Solution

Step 1: Analyze the Human Research Process

Understand human search behavior: The first step involves analyzing how humans approach the search process. This includes how they choose search paths, navigate websites, and make decisions based on context and world knowledge.
Identify implicit evaluations: Humans make split-second decisions when selecting search results. These implicit evaluations—such as determining whether a product name is generic or identifying aggregator sites—need to be explicitly understood and replicated in the AI.

Step 2: Develop an AI-Supported Solution

Teach our AI the human thought processes: The AI needs to be trained to mimic human thought processes and evaluations. This involves implementing planning and evaluation capabilities similar to those humans use.
Explicit formulation of criteria: Evaluation criteria must be clearly defined for the AI. For example, understanding whether a search result is contextually relevant or whether a website link is the correct one to follow.

Step 3: Structure the Research Process for AI

Break down complex steps: The human research process is broken down into clear, defined actions that the AI can follow. For example, the AI might be instructed to "Choose the right result in the given context and follow this path along N steps with a specific intention."
Path planning for AI: The AI must understand the intention behind each step, enabling it to make informed decisions at every stage of the research process.

Step 4: Avoiding Pitfalls

Guiding the AI with explicit instructions: Modern language models often struggle with understanding intention. To counter this, the AI must be provided with explicit instructions and evaluation criteria to ensure it stays "on track" during the research process.

Result: Enhanced Efficiency and Quality in Lead Research

The result of these actions is a sophisticated automation system that can efficiently handle complex research tasks. Key outcomes include:

Scalability: The process is now scalable, capable of handling lead research for thousands of companies without the need for extensive human intervention.
Cost reduction: Automation significantly reduces the costs associated with manual research, including personnel expenses.
Increased speed: AI operates 24/7, collecting data at a pace that far exceeds human capabilities.
Consistent quality: By adhering to explicitly defined criteria, the AI ensures a consistent quality of research, free from subjective biases.
Employee satisfaction: By automating repetitive tasks, human researchers are freed to focus on more creative and complex challenges.

Valeriy Leibert, the company's B2B Sales Manager commented:

Narratic's AI research enabled us to find contact details of our target customers at half the cost. The significantly shorter research time is also a major advantage. Leads at the touch of a button!

Conclusion

Automating lead research with AI not only addresses the challenge of scaling data collection but also transforms how organizations approach this essential task. By leveraging AI to mimic human thought processes, businesses can achieve more efficient, cost-effective, and high-quality outcomes.

If you are facing a similar issue, we’d be happy to discuss your case in more detail.