Chapter 3 —
Measuring Interoperability

Comparing identifiers

A simplistic approach would be to classify relations between the identifiers (ids) of two datasets in four categories. One dataset can contain the same identifier as a dataset in the other. When this identifier identifies the same real-world thing in both datasets, we call this a correct id match. When this identifier identifies a different real-world thing, we can call this a false id match or an id conflict. In the same way, we have a correct different id and a false different id.

In Table 1, we give an example of an open dataset of public toilets in the city of Ghent. In Table 2, a similar dataset about the city of Antwerp is given. An identifier is each element that is not a literal value such as “3.73”. Identifiers may be elements of the data model, as well as real-world objects described within the dataset. In these two tables, we can label some identifiers as conflicting, other elements as correct id matches, others as correct different ids or false different ids. However, the labeling can happen differently depending on the domain expert. Certainly when “loose semantics” are utilized it becomes difficult to label an identifier as identifying “the same as” another identifier.

An initial metric

In our research paper in 2013 for the sake of simplicity, we assumed that domain experts would be able to tell whether two identifiers are “the same as” or “not the same as”. When we would have a list of statements, classifying these identifiers in four categories, we would be able to deduct a metric for the interoperability of these two datasets. The first metric we introduced was the id ratio.

We apply this formula on top of our two example datasets in Table 1 and Table 2. The correct identifier matches are: “id”, “long”, “lat”, “fee” (4). Conflicts would be the identifier “1” (1). Of course, this depends on our view of the world and our definitions of the things within this dataset. Real-world concept matches would be: “id”, “long”, “lat”, “type/type_sanit”, “fee”, “urinal/new_urinal” (6). An initial metric would thus score these two datasets as 66% interoperable, with 1 conflict.

In a simple experiment, we asked programmers to give a score for the interoperability between 5 different datasets and a reference dataset. The outcome revealed that there were clear design issues with this metric: when there would be a low amount of real-world matches, the score would be influenced quickly, as the number of samples to be tested is low. A full report, and the research data, on this experiment can be found on a Github repository at pietercolpaert/iiop-demo.

Identifier interoperability, relevance, and number of conflicts

There are two problems with id_%. First, The id_% may be calculated for 2 datasets which are not at all relevant to compare. This can lead to an inaccurate high or low interoperability score, as the number of real-world concept matches is low. Second, when different datasets are brought together, some identifiers are used more than others. An identifier which is used once has the same weight as an identifier that is used in almost all facts.

Instead of calculating an identifier ratio, we introduce a relevance (ρ) metric. A higher score on this metric would mean that two datasets are relevant matches to be merged. We now can introduce two types of relevance: the relevance of these two datasets as is (ρ_identifiers), and the relevance of these two datasets when all identifiers would be interoperable (ρ_real-world). We define the ρ as the number of occurences of an identifier when both datasets would be merged and would be expressed in triples. The ρ_identifiers in our example would become 8, as 2 times 4 statements can be extracted from a merged table. The ρ_real-world would become 12, as the number of occurences that would be left when the dataset would be 100% interoperable, would be 2 times 6. We then define the Identifier Interoperability (iiop), as the ratio between these two relevance numbers. For this dataset, the iiop becomes 66%. Coincidentally the same as the id_%, with more rows in the dataset, it would quickly be different. When repeating the same experiment as with the id_%, we now see a credible interoperability metric when comparing the ρ_real-world, the number of conflicts and the iiop.

While these three metrics may sound straightforward, it appears to be a tedious task to gather statements. What is the threshold to decide whether two identifiers are the same as or not the same as [2]? A philosophical discussion arrises – cfr. Theseus’s paradox – whether something identified by one party can truly be the same as something identified by someone else. It is up to researchers to be cautious with these statements, as a same as statement may be prone to interpretation.

The role of Linked Open Data

Conflicts are the easiest to resolve. Instead of using local identifiers when publishing the data, all identifiers can be preceded by a baseuri. E.g., instead of “1”, an identifier can contain http://{myhost}/{resources}/1. This simple trick will eleminate the number of conflicts to zero. In order to have persistentPersistent identifiers are identifiers that stand the test of time. Cool uris don’t change. identifiers, organizations introduce a uri strategy. This strategy contains the rules needed to build new identifiers for datasets maintained within this organization.

Another problem that arose, was that a third party cannot be entirely sure what was intended with a certain identifier. Linked Data documents these terms by providing a uniform interface for resolving the documentation of these identifiers: the http protocol. This way, a third party can be certain what the meaning is, and when the data is not linked, it can link the data itself more easily by comparing the definitions. The more relevant extra data is provided with this definition, the easier linking datasets should become.

Linked Data helps to solve the semantic interoperability, for which the identifier interoperability is just an indicator. By using http uris, it becomes possible to avoid conflicts, at least, when an authority does not create one identifier with multiple conflicting definitions and when this definition, resolvable through the uri, is clear enough for third parties not to missinterpret it. The effort it would take for data publishers to document their datasets better and to provide “same as” statements with other data sources is similar to just providing uris within your data and linking them with existing datasets from the start. However, the theoretical framework built, provides an extra motivation for Linked Data.

Studying interoperability indirectly

It is understandable that policy makers today invest time and money in raising interoperability, rather than measuring its current state. When however no globally unique identifiers are used, can we work on semantic interoperability? Even more generally, is interoperability a problem data publishers are worried about at all? Or is their foremost concern to comply to regulations to publish data as is? In this section, we are going to measure interoperability indirectly, by trying to find qualitative answers to these questions by studying the adoption by third parties, studying the technology of published datasets and finally studying the organizations themselves.

Making the cost-benefit analysis for data reuse, when more third party adoption can be seen, we can also conclude the datasets become more interoperable when the benefits did not change. We can perform interviews of market players and ask them how easy it is to adopt certain datasets today. This is an interesting post-assessment. In a study we executed within the domain of multimodal route planners [3], it appeared that only a limited amount of market players could be identified that reused the datasets. Each time, reusers would replicate the entire dataset locally. In this paper [3] we reported on the study we executed We concluded that still only companies with large resources can afford to reuse data. This is again evidence that the cost for adoption, and thus the interoperability problems need to lower.

When studying data policies today, we can observe a certain maturity on the five layers of data source interoperability. For example, when a well-known open license is used, we can assume the legal interoperability is higher than with other datasets. When the data is publicly shared on the Web and accessible through a url, we can say the technical interoperability is high, and when documents and server functionality are documented through hypermedia controls, we can assume the querying interoperability is high. At some level, in order to raise interoperability, we have to make a decision for a certain standard or technology together. From an academic perspective, we can only observe that such a decision does not hinder other interoperability layers. When studying the interoperability, we could then give a higher score to these technologies that are already well adopted. With the upcoming http/2.0 standard, all Web-communication will be secure by default (https). In the rest of this book, we will use http as an acronym for the Web’s protocol, yet we will assume the transport layer is secure. Today, these technologies would be http as a uniform interface, rdf as a framework to raise the semantic interoperability, and the popular Creative Commons licenses for the legal aspect. In the next chapter, we will study organizations based on the reasoning to set up a certain kind of service today. Epistemologically, new insights can be created by the data owner to publish data in a more interoperable way.

Qualitatively measuring and raising interoperability on the 5 layers

Aspects such as the organizational, cultural, or political environment may be enablers for a higher interoperability on several levels, but should as such not be taken into account for studying data source interoperability itself. In among others the European framework isa² and eif, also political interoperability is added. When quantitatively studying dataset interoperability, we can discuss each of the five layers seperately. In this section we give an overview of all layers and how they can be used in interviews. Depending on the quality that we want to reach in our information system, we may be more strict on each of the levels, and thus for every project, a different kind of scoring card can be created. This is in line with other interoperability studies [4], that overall agree that interoperability is notoriously difficult to measure, and thus qualitatively set the expectation for every project.

Conclusion

Raising interoperability entails making it easier for all user agents on the Web to discover, access and understand your data. We explored different ways to measure interoperability between two datasets. For Open Data however, we would then need to generalize – or scale up – these approaches to interoperability between a dataset and “all other possible open datasets”.

A first – not advised due to the investments needed – way is to compare identifiers between these two datasets or systems. When the same identifiers are used over the two datasets, we can assume a high interoperability. However, whether identifiers actually identify the same thing is prone to interpretation. We see this exercise mainly as proof that Linked Data is the right way forward: by using http uris, we use the uniform interface of the Web to document identifiers. Furthermore, using Web addresses instead of local identifiers will make sure another Web framework – rdf – can be used to discuss the relation between different real-world objects. Through comparing identifiers, we showed the importance of rdf to raise the semantic interoperability. Only time will tell whether using Web addresses for identifiers – and consequentially rdf – will become the norm for all aspects. Well established standards then will have to evolve to rdf as well. Also within the legal, technical, syntactical, and querying interoperability, rdf may play its role. Without rdf, we would have to rely on a different mechanism to retrieve machine readable license information, or we would have to rely on a specification that reintroduces syntax rules for hypermedia.

A second way to measure interoperability between datasets is to study the effects. The fact that Open Data by definition should allow data to be redistributed and mashed up with other datasets, means it becomes hard to automatically count each access to a data fact coming from your original dataset. A successful open data policy can thus be measured by the number of parties that declare that they reuse the dataset. Interviewing the parties that voluntarily declared this fact, may result in interesting insights on how to raise the interoperability. However, this is a post-assessment when an Open Data policy (or a data sharing policy) is in place.

A third way is to interview data owners within an organization on their own vision on Open Data. During the interviews, questions can be asked on why certain decisions have been taken, each time categorizing the answer at an interoperability layer. This is an epistemological approach, in which data owners will get new insights when explaining their vision within the context of the 5 interoperability layers. Depending on the interviewed organizations, different technologies can be assumed accepted. While some organizations will find it evident to use http as a uniform interface, others may still send data that should be public to all stake holders using a fax machine. For fewer cases – discussed in the next chapter – rdf was evident. Thefore, we need a good mix between desk research on the quality of the data and interviews with data maintainers in order to create a good overview of the interoperability.

In the next chapter we introduce our own context of the organizations we worked with and will choose a qualitative approach to studying interoperability.