For more videos from GraphConnect SF and to register for GraphConnect Europe, check out graphconnect.com.
Today, I’m going to give you an overview of how we work with graphs at the World Economic Forum. I’m sure you might be keen to see traversals through billions of nodes or gigabits of analytics with community detection.
In actuality, our graph is small but incredibly powerful. (It’s also still under construction.)
Today we’ll focus on the reasons we chose a graph database, as well as the unique business opportunities graphs present for complex organizations such as the World Economic Forum.
In this post, I’ll describe two key projects: transformation maps, and graph search and recommendations. Both of them started their lives with the recognition that leaders and experts need a broader view of both interconnected issues and the emergence of disruptive change.
Disruptive Change in the World Economic Forum
There are a number of industries that have experienced – or will soon experience – disruptive changes.
3D printing is changing manufacturing and supply chains. Self-driving cars have yet to disrupt the automotive industry, while diesel emission scandals can impact the industry overnight. The hotel industry is threatened by the sharing economy, and the media industry is constantly reinventing itself in attempts to increase readership and profits.
Many organizations, including the Forum, are experiencing radical changes. But the Forum is unique because it is a meta-organization – what we call a platform.
The Forum has a variety of roles:
- To filter out the noise around global issues
- To find the patterns in data
- To indicate what is on the horizon for other organizations
We gather change-makers from the public, business and civil sectors to build a consensus on how to move action forwards. To this end, we’re building technologies to facilitate such high-level collaboration and knowledge exchange.
The Davos Conference
But first and foremost, the Forum is most well-known for its annual meeting in Davos, a small town on the eastern tip of Switzerland. It brings together 2,500 of our members from the top 1,000 global businesses, along with government, NGO, academia and faith leaders.
In January 2015, the Forum’s annual meeting included a variety of prestigious speakers. John Kerry delivered a special address on freedom, democracy and the rule of law. Bill and Melinda Gates spoke about sustainable development. Angela Merkel addressed global responsibilities in a digital age. Even will.i.am was interviewed in one of our smaller rooms.
Discussions took place surrounding the future of the military and human versus artificial intelligence. And there were plenty of quotable moments.
But that overview just skims the surface of the real complexity that plays out in Davos. It’s much more than a conference. For us, as the organizer, it’s a solution of global complexity through network thinking.
The people who come to the conference are in large part the leaders of the world, from Fortune 500 CEOs to heads of state. Extraordinary logistical work takes place to plan for 2,500 high-profile participants, their accompanying military and private security, and the program’s 1,500 speaking roles.
We also have to manage speaker and participant expectations and requests. What do they want to talk about? What key messages do they wish to deliver? Will they be speaking about the most important issues of the time alongside the people they wish to appear with?
Our success is because of our program team, which works year-round to identify key issues and speakers for the conference. It is a marathon of matchmaking and diplomacy. It’s a chance to discuss both emerging and unknown issues with a variety of experts.
The annual meeting in Davos is not just a conference; it’s a solutions space. It’s where world leaders come to discover, understand and discuss the most pressing combinations of global issues.
Making Solutions Usable through Graph Databases
Helping leaders make informed decisions and take appropriate action requires an integrated flow of content analysis and opinion. Throughout the year, with our network of members and partners, we monitor, measure and identify trends and build an archive of valuable knowledge to navigate the complexity and change that feeds into our meetings.
Below is a rapid-fire overview of some of the products we’ve built and how they’ll be strengthened when connected to our graph.
The Forum produces 73 reports and white papers annually, the most important of which are delivered in digital format. But last year, the World Bank released a report stating most of their PDFs were never read.
Making sure your digital products can be found and consumed is incredibly important. There’s a lot of knowledge locked away in the data structure of PDFs, which is waiting to be released and used.
The most common issues we work with involve infrastructure, trust, new technologies in finance, employment, education and skills, the eurozone and global trends. And many teams within the Forum have a need for a digital platform because they’re creating datasets and data visualization as part of their analysis.
You can see below a sample of the Global Risks Report, which produces between 30-50 issues each year that evaluates their impact and likelihood, connected risks, clusters and centers of gravity. It is a graph analysis. They invite experts to discuss the findings and work with our partner organizations to perform the research.
This is all evidence-based data, the most significant of which are our eight global indexes. These include rankings across 140 economies comprised of sub-indexes, topic-driven pillars and hundreds of individual indicators with additions over time series.
We have a data API and indicator catalog pairing at the back with scorecard pages for each economy. The indicators are harvested, processed and normalized by our analytical methodology from a number of open sources, including the Organization for Economic Cooperation and Development (OECD) and the World Bank, and from our executive opinion survey.
Our global index includes Global Competitiveness, Africa Competitiveness, Travel and Tourism Competitiveness, Energy Architecture Performance, the Gender Gap, Human Capital, Information Technologies and Network Readiness.
Most recently, we have included a new index called the Inclusive Growth Report (below), which aims to facilitate a deeper debate and change around sustainable policy recommendations.
Translating Davos into Data
During all of our events, a team from Livestream films and streams the videos around the world and archives them on their platform. We built a video API with a configurable player that we and other organizations embed on our own website.
All the photographs I showed at the beginning are filmed and have accompanying transcripts, tagged topics, issues and speakers associated with them in their data model.
This archive of videos dealing with global issues is growing with every event that we host, and it becomes a time series of global issue change and the responses that leaders are having to them.
At Davos we also have a “Social Wall,” an 8-meter-tall display outside the plenary hall that tracks peoples’ responses to the Davos conference.
The Social Wall tracks the reach, or influence, of the people posting about topics relevant to the conference, as well as the reach of the topics themselves.
The Twitter API gives us details regarding which leader is most socially active and the most re-tweeted. Our Imagery API supplies a moment-by-moment feed of the world leaders delivering their ideas. The News API finds the articles that are being written about our proceedings. Custom D3-based data visualizations show them and nine experienced templates deliver them.
Also during our events, member extranet and apps receive most of their traffic as members plan their itineraries. It includes all the people, organizations and sessions associated with the Forum. In between events, it’s a community space for discussions around common interests and a planning space for action and global issues.
Graph and Product Integration
We are working to integrate the white papers and risk assessments I mentioned above with strategically valuable knowledge, awareness, discovery and collaboration tools. And it’s the platform that’s benefitting the most from the graph work that we’re doing.
Graphs provide huge potential for the Forum, and other organizations of its type. Upon examining the dataset, it’s clear how many entity relationships, subgraphs and traversals could be possible.
As a recap, the Forum can be defined as a content channel, trend spotter, global issues analysis machine, matchmaking service for group collaboration, benchmark dataset provider and recommender for knowledge about global transformations.
The problem with unequal distribution of the future is that leaders don’t know the weak signals from the noise. This is where the Forum comes in. We find and share patterns in the data, which helps characterize the nature of the discourse and act appropriately.
A graph is the best way to capture all of this complexity. In many ways, it’s been in development for over 40 years with paper and spreadsheets. However, the massive growth in data complexity has necessitated new technology.
Klaus Schwab, the Forum chairman, requested a global issues mapping project to ameliorate an increasing knowledge gap amongst senior leaders of unforeseen impacts. The result is what are known as transformation maps.
Building a Transformation Map of Global Issues
We’ve always had a database, but it never connected different issues. We didn’t even have a taxonomy to place a global issues layer on top of a people layer. It was time to come up with a solution to this problem.
Below is what you would expect the graph to look like, which depicts the actual data we’ve been putting into our graph.
But, if the data model is part of the product and is interacted with directly by the end user, you need to rethink the way it’s visualized.
When we released the transformation maps at Davos this year it was received exceptionally well. We had CEOs taking over the 82-inch touch screen (seen below), to perform demos to their colleagues.
In the example transformation map below, you can see how a key issue for an economy can be affected by a key issue for an industry, and between industries.
This is a graph traversal that steps through several degrees of separation to show the cause and effect between distant key issues.
The transformation maps became a key facilitation tool for group strategic planning. They were developed in response to crucial user experience time during an industry strategy meeting. We watched how our chief strategy officers discussed and explored the data structure – how they moved from key issue to key issue, insight area to insight area.
A number of questions came up:
- What if you want to find an expert from a network to talk to?
- What if you want to discover sessions of the event that you are attending, dealing with a specific subject?
- What if you want to join a foreign project?
Graphing the Global Issues Archive
The Forum is still going through the process of taking data from our older infrastructures and transferring it over to graph databases.
To begin this process, we started with what we knew best – experts and issue mapping – and we drew upon our internal issue-focused teams in our communities and networks. Together, we’ve mapped the key issues within 110 primary global issues, economies and industries, along with the drivers of change upon them and between them.
We have teams who coordinate our regional meetings in Africa, the Middle East, Latin America and East Asia. We have 21 industry teams – from automotive to telecommunications – and we have over 30 communities. We have a network of global agenda councils – the world’s biggest brainstorm – with 1,500 global experts in 80 councils who meet once a year to discuss the main global trends within their knowledge domain.
The data structure isn’t magic – it’s the result of a large amount of effort from a large number of experts. Over several months, we built our prototype content input environment, monitored the quality of the incoming data, verified the number of key issues that were being defined in certain issue areas and wrote tagging guidelines.
Our dataset is the collective wisdom of experts. If you could get everybody to talk to everybody else and then capture the resulting conversations in a graph, what would it look like? This is our first attempt at a model that demonstrates that.
This isn’t all to say that machine learning won’t have a huge role in scaling, modeling and improving the tagging and building its taxonomy – it will. Human expertise will increasingly be augmented by machine intelligence.
This also becomes a year-round data model that represents the things we and our members want and need to talk about. It has become a definition in data structure for our engagement and relationships with our members and their organizations.
It’s awareness-raising – it gives strategic focus, and it can evolve over time.
Using Graphs to Build Organizational Relevance
In order to utilize this graph tool most effectively, we needed a powerful search tool. We embarked on a graph search project to increase the findability and relevance of our data. It was important for our content to be surfaced in the right place, to the right people, at the right time.
The search tool needs to be more like a solutions architecture than a string retrieval. So while transformation maps are our taxonomic glue, our graph will become the central infrastructure that joins everything together as the metastore.
Developing a search tool would also solve some relevancy issues the Forum had experienced for many years. In 2012, after spending four months working on a search tool, we found we would need five full-time staff to continually classify and weigh the data in order for the search to function properly. This is when we realized that something was deeply wrong.
At that point, we decided to try relying on Google. But on a private platform, you can’t rely on page rankings and Google weighting algorithms to build relevance.
To address our relevancy problems, we took a number of steps.
The first step we’ve taken is to move to elasticsearch and rebuild our index structures with some expertise in hand. This is thanks to both Neo4j in Europe and GraphAware in London.
The second step was to engage in some relevance engineering, intercepting our search results with our graph. Graph traversals supply weightings, contacts and dependencies that just don’t exist in a standard index.
We wanted entity recognition as the entry point for search relevance, and the hierarchy of assumed needs for the ranking of the top ten results. We also want search and recommendations as an API infrastructure so we can put a permissions layer over it and use it across different platforms.
One of the first things we did was look at the way a recommender system works on a data model. However, a problem presented itself immediately – the reliance on similarity.
Consider the following example. I normally only buy work-related Kindle books. Then a year ago I bought a pair of shoes – these shoes:
Three months afterwards, Amazon kept recommending exactly the same shoes.
Perhaps it was because I had bought two pairs of shoes, or that the product was such an outlier from what I normally purchased, or that everyone who bought leadership and user experience books also bought Scarpa shoes. They knew it was significant but they didn’t know how to analyze it.
Traversal of person-to-person matchmaking similarity can reap rewards, but we wanted to explore how recommenders intersect with search. We instead began thinking more about recommendations like matchmaking for multiple entities, which is what we do during our meetings.
Again, it’s more of a solutions engine approach that we’re taking with search – matchmaking rather than recommendations. This is where the Forum finds itself in its graph journey at the moment, with a lot of really interesting ideas and possibilities to unpack.
New Technologies
However, all things are evolving, and ideas seem to flourish when the time is right. For the Forum, the time for new technologies was forced by the business changes we were going through. We needed to fix search as one of our primary issues.
We also came to realize that a standard database would timeout if we started doing the kinds of traversals that we wanted to perform. Our first graph planning was done in 2011, but it took from 2012 to 2014 to put an issues taxonomy together that we could all agree upon.
That was long enough for the graph world to start getting to a point of maturity. It’s also long enough that machine learning work as a service began to arrive, such as IBM Watson, AlchemyAPI and others.
The hard librarianship that we would have had to do in 2011 and 2012 has been replaced with entity and concept extraction work using Wikipedia-linked microservices. And that’s going to help us to scale our issues taxonomy supported by human-in-the-loop expertise.
This next generation of technologies and data augmentation will bring us new opportunities, which we are exploring and growing right now. It’s going to help us build more as yet unimagined products, finding the right patterns in the data to help our members focus their strategies on the global issues that matter to them.
Inspired by Scott’s talk? Register for GraphConnect Europe on 26 April 2016 at for more industry-leading presentations and workshops on the evolving world of graph database technology.
