Dark Matter: Reimagining online search with…

Online search is critical for people researching and deciding their next moves. But while search engines produce results quickly, they only answer the questions that users ask and they’re influenced by previous searches, making them biased. Working with a leading global drinks company, we’ve developed a new kind of artificial intelligence (AI) search tool, named Dark Matter, that goes a step further by also answering questions users didn’t know they could ask – but that could unlock valuable opportunities. We’ve used Dark Matter to uncover new possibilities for several multinational organisations – from discovering future technologies to finding new acquisition opportunities.

Key successes

Pioneered a unique AI online search tool by bringing together expertise in machine learning, deep learning and natural language processing
Ensured online search is more valuable than it is today, unlocking important insights and opportunities for organisations around the world

Facing up to the limitation of today's search

Organisations are constantly looking for ways to stand out in crowded markets or serve their stakeholders better. It might mean looking into the future to anticipate trends. Or it might mean spotting unmet needs in today’s world. Either way, searching the web features heavily in their research. And that’s not surprising. Search engines are the quickest route into the world’s information.

There’s just one snag. Web search only gives you information about topics you seek out. They’re also biased as they’re based on attributes the browser already knows about you, like your previous searches, your location and where you work. In a fraction of a second, search engines produce answers to almost any question you care to ask. But your next breakthrough could depend on asking a completely different question you haven’t thought of. It might mean searching in a place you don’t know about – yet. And conventional web search can’t help with that.

That’s about to change. Our team, with expertise in machine learning and natural language processing, a subfield of computer science that uses computer-based methods to analyse language in text and speech, have developed a new online search tool to open up areas people don’t know about yet. And it’s already leading some into undiscovered territory.

Discovering a smarter kind of search

The idea to develop the search tool came into fruition while we were working with a leading global drinks company. We were collaborating with them to uncover new and disruptive technologies that would impact their business in the next 10 to 15 years.

They already had teams of their people dedicated to the research, so we needed a different method to be able to give them new findings. That’s when we hit on the idea of applying AI. While there are a lot of tools for AI innovation in the public domain, it’s knowing how to apply them that holds many organisations back. Our team understand how they work, what value they have and how to bring them together successfully.

Leveraging natural language processing

We set about creating a tool that uses natural language processing and machine learning to take searchers to information that’s related to their area of interest, even though they don’t explicitly ask for it. Users can tell search engines to disregard factors like their search history and location. But the results will still favour articles that contain the search term the user asks about, and which other sites have linked to. We needed to create a tool that isn’t constrained in the same way.

Our analytics team developed the tool from scripts in the programming language Python, combined with text analytics and natural language processing algorithms. We refined the scripts and, over about six weeks, developed a scalable product with a front end. We then went on to configure it with our client’s technical team.

We designed the tool so it can be used in a variety of ways. For example, a user can search for what they’re broadly interested in, for instance ‘Packaging’. They also give it a set of subjects within that topic, like ‘materials’, ‘smart packaging’ or ‘production’.

The tool then combs the web in the way standard search engines do, but it highlights subjects that the user might also find relevant. The user then chooses the results that look most interesting, and the tool takes these and the original results to create a ‘fingerprint’ to continue the search. The user can carry on in this way, with the tool using natural language processing techniques, like analysing sentence structure, to uncover relevant new subjects, even though the user doesn’t specifically ask for them.

The tool creates a semantic graph from the results, showing the topic and subjects you started with, and new subjects related to each. You can then click through all the documents associated with them and control how your search develops from there.”

Willem van Asperen PA data science expert

Bringing together technical expertise to validate findings

By leveraging the tool, we were able to deliver an analysis of disruptive technology and applications for the drinks company to consider. And we did three times faster than if they had relied on traditional search engines.

We then called on our technology experts at our Global Innovation and Technology Centre (GITC) with expertise in the different fields to validate and enrich our findings.

Through working sessions with the drinks company’s team, our team were able to provide their deep technical knowledge to test and adapt the technologies and applications the AI tool had uncovered around the company’s requirements. This enabled us to uncover the best possible opportunities for the organisation so they could incorporate them into their new innovation strategy.

Applying the tool in novel ways

After the success of this first pilot, our team has carried on developing it and leveraging it to uncover new opportunities for other organisations.

We developed the tool so it can be applied in a variety of ways – from discovering new technology and acquisition opportunities to strengthening PR and marketing strategies. We’re actively piloting it with clients across a range of sectors, including healthcare and life sciences, financial services, private equity and consumer and manufacturing.”

Peter de Vries PA growth strategy expert

For example, a leading global beer group wanted to know about possible applications for a waste product of their brewing process. We used our AI search tool to uncover a number of new and unexpected applications that will enable them to get more value from the waste product compared to existing solutions used by competitors.

In parallel, we used the tool with a global food corporation who wanted new acquisition targets that would enable them to get new sources of alternative proteins, besides the ones their analysts were spotting. Our tool uncovered new areas of alternative and novel proteins. Our team at the GITC then did an assessment of viability and scalability of each target to enable the corporation to select the most promising candidates.

Unlocking valuable insights and opportunities

The tool offers organisations the ability to subscribe to the tool in different ways, either using it themselves with technical support, or with consultancy to guide their searches and interpret and act on the results.

Users will also be able to keep searches going over a period of time. The tool learns continuously and updates at intervals, finding new documents and subjects, as well as producing email summaries and dashboards that plot trends.

It promises to make online search even more valuable than it is today, unlocking valuable insights and opportunities for organisations around the world.