From metadata and open knowledge to AI: revisiting twenty years of assumptions about how we share and use information.
In 2005, Professor Manolis Stavrakakis, teaching viticulture at the Agricultural University of Athens, asked me a question that ended up influencing much of my professional life.
He had spent more than forty years studying some of the world’s oldest grape varieties. Like many researchers, he and his colleagues were producing and collecting increasingly large amounts of scientific information. The problem was not a lack of knowledge. It was becoming difficult to manage what was already available, connect it with other research, and make good use of it.
Could technology help?
He also encouraged me to pursue a PhD at the university’s Informatics Laboratory and introduced me to the people there.
I was a computer scientist, interested in databases, information systems, and decision support. The question seemed like a particularly interesting problem to investigate. It also gave me an opportunity to apply my technical background to an area of research that had a much broader purpose.
My doctoral research eventually focused on information overload, metadata, and personalised access to electronic services. I explored how information could be described in ways that allowed different systems to understand it, and how recommendation algorithms could help people compare alternatives and select what was most relevant to them.
Looking back, what attracted me was not simply the possibility of organizing more information. It was the idea that technology could help people find and use knowledge that might otherwise remain difficult to discover or connect. It was the beginning of a much longer journey.
More than twenty years later, with AI changing how we discover, process, and use information, I find myself returning to remarkably similar questions. But I also realize that my understanding of the problem has changed considerably.
When I believed openness was the answer
During my early research years, I developed a strong conviction about the importance of open knowledge.
Much of the scientific and regulatory information available to us is produced by public institutions, universities, research organizations, and government agencies. A significant part is funded directly or indirectly by taxpayers. I believed, and still believe, that we have a responsibility to make this knowledge accessible and reusable wherever possible.
Consider a food safety incident published by a national authority. The information has been generated through a public process, often using public resources, to protect consumers and inform other stakeholders. Why shouldn’t it be available to researchers, companies, professionals, and citizens who may be able to use it to prevent similar problems elsewhere? A food incident in one country may contain information that is relevant to a food producer, importer, researcher, or authority operating in another. Making that information available should not require every organization to independently rediscover what has already been documented.
Of course, there can be legitimate reasons to restrict particular information, including privacy, confidentiality, and other legal obligations. But my starting assumption was that openness should be the default rather than the exception, especially for publicly funded information intended to serve the public interest.
At the same time, I was fascinated by metadata and interoperability.
We had been developing methods to describe information consistently, identify relationships between different resources, and allow independently developed information systems to exchange and reuse what they contained.
I was also influenced by the work that had been done in libraries and the cultural heritage sector to distinguish an original intellectual work from its different expressions, editions, and representations.Consider a book, for example. There is the original intellectual work, but there may also be different editions, translations, digital versions, and other representations associated with it. Describing these relationships helps people understand what a particular resource represents and how it relates to others.
I thought we could apply similar principles to scientific and agricultural information. A food safety finding might originate with a public authority, later appear in a translated record, become part of a harmonized collection, and eventually contribute to an analytical assessment. Could we describe these different representations and transformations in a way that preserved their origins and clarified the rights associated with each?
Together with licensing approaches such as Creative Commons, these ideas seemed to offer a promising answer to many of the challenges of sharing knowledge. For a period, I probably believed that if we selected the appropriate metadata models and licenses, we could resolve most of the difficulties. We could identify the original information, acknowledge its publisher, describe subsequent transformations, clarify how they could be reused, and allow different systems to work together.
I expected that transferring these principles from scientific repositories and cultural collections into agriculture and food would be largely a matter of applying the right methods and standards.
Perhaps I was too optimistic about how much of the problem this could solve.
Trying to connect knowledge across institutions
Over the following years, I had the opportunity to work with researchers, institutions, and organizations that shared similar ambitions.
Through Agroknow and collaborations involving organizations such as FAO, agricultural research institutions, the European Commission, and the World Bank’s Global Food Safety Partnership, we worked on ways to make scientific and agricultural information easier to discover, connect, and reuse. Projects such as agINFRA explored the idea of shared scientific infrastructures. Other initiatives focused on open agricultural data, interoperability, and the possibility of a global data ecosystem for agriculture and food.
I met people who had been working on these problems for many years, often from quite different perspectives. Some were researchers, others were technologists, policymakers, or people trying to develop practical services around shared information. It was an exciting period. We were not simply discussing new technologies. We were exploring whether institutions that had historically created and maintained their own information collections could become part of something larger.
I also became involved in conversations about how openness could support food safety knowledge and capacity development, including work with Paul Stacey from Creative Commons and Christine Geith from Michigan State University through the Global Food Safety Partnership. These collaborations helped me appreciate that making knowledge available was not only a technological ambition. It also required people and institutions to agree on common objectives, work together, and invest in maintaining what they were building.
But making independently developed systems work together turned out to be more complicated than agreeing that they should. Different organizations had their own priorities, resources, policies, and responsibilities. Information was published in different formats, languages, and classifications. Some resources were open, others were restricted, and many were available without being particularly easy to reuse.
There was another difficulty that became increasingly apparent. A project could demonstrate that connecting information was technically possible without necessarily establishing how the resulting infrastructure would be maintained after the project ended. Who would finance its continuing operation? Who would keep the information accurate and up to date? What would motivate organizations to contribute information or expertise when the benefits might be shared by others?
These were not secondary questions. They affected whether the original ambition could be sustained.
In 2016, I contributed to the GODAN discussion about a global data ecosystem for agriculture and food. We were already examining trust, incentives, interoperability, and sustainable business models.
A year later, in a presentation titled What Does Effective Data Sharing Mean?, I was asking participants to consider who owned shared data, who could access or reuse it, whether commercial use was allowed, whether the platforms involved could be trusted, and whether the proposed arrangements were fair. In other words, even then I was beginning to recognize that the problem could not be solved simply by publishing more information or improving technical standards.
In 2020, while editing Digital Science Recommendations for Food & Agriculture, I returned to the ambition of creating an interoperable, shared global information space. I also expressed my concern that much of our collective effort was still going toward making individual data collections larger, rather than connecting them effectively. We had made important progress. But we were still creating information silos, even as we became better at managing the information inside them.
Looking back, I recognize that the questions were becoming broader than the technical ones that had originally attracted me.And we were still a long way from answering them.
What I had underestimated
At the same time, building Agroknow and later developing commercial food safety intelligence services gave me another perspective. It made me appreciate the amount of work required to transform available information into something that people can actually use.
Take the general example of a publicly announced food recall. The original announcement may be available freely on an authority’s website. But bringing together similar announcements from different countries, translating them, identifying the companies, ingredients, products, and hazards involved, removing duplication, reconciling different terminology, and maintaining reliable historical records requires considerable effort.
One authority may classify a product according to its ingredients, another according to its commercial category. One may use a particular hazard terminology while another uses a different description. Dates may refer to different stages of an incident. Important elements may be recorded in narrative text rather than in structured fields. Making these records comparable requires more than collecting them. It requires people, technology, scientific and technical expertise, investment, and continuous work.
Even after the initial collection has been created, it needs to be maintained. Sources change, records are corrected, classifications evolve, and additional information becomes available. The resulting resource can be more complete, consistent, and useful for particular analytical purposes than the individual announcements from which it originated.
I gradually came to recognize that this additional work can create legitimate commercial value. Information made available by the public sector can become the foundation upon which companies build services that customers are willing to pay for. That does not necessarily mean that every transformation creates new intellectual property rights over the underlying facts. The legal rights associated with an enriched dataset, software, analysis, or other resulting work depend on the nature of what has been created, the applicable law, and the relevant contractual and licensing conditions.
But it does mean that free access to original information and commercial value created through its processing are not inherently incompatible. This changed how I thought about the economics of knowledge. I had initially focused heavily on how information could become accessible and reusable. I was now also appreciating the question of who would make the necessary investments, who would maintain the resulting services, and how this work could be financially sustainable.
The value of shared information does not eliminate the cost of turning it into something useful.
Nor does the existence of commercial value necessarily mean that the original publicly available information should become restricted. I believe both points matter.
When ownership becomes more complicated
An even more difficult realization concerned the ownership, use, and responsibility associated with information generated throughout the food supply chain. Consider two common situations. These are illustrative examples of the questions involved, rather than descriptions of particular customers, contracts, or company operations.
The first is a laboratory test. A food company pays a laboratory to analyze a sample. The laboratory performs the analysis using its scientific expertise, personnel, equipment, methods, and quality systems. Who has the right to use the resulting data? Can the laboratory reuse it for statistical analysis? Can the customer share it with suppliers or other laboratories? Under which conditions might the results become part of a larger collection used to study emerging risks? And who is responsible if a result is incorrect, incomplete, or interpreted outside the context in which it was generated? It is tempting to answer these questions by asking who paid for the test or who produced the result. But neither question necessarily gives us a complete answer. The laboratory may have contractual, professional, and regulatory obligations. The customer may have legitimate confidentiality and commercial interests. There may be restrictions concerning the use of the sample, the publication of findings, or the sharing of information with other parties. The result itself may also have scientific limitations that affect how it can be compared with other findings.
A second example is an audit report. An organization may commission and finance an assessment, while another organization conducts the work using trained auditors, technical expertise, and established procedures. Depending on the type of audit, applicable standards, and contractual arrangements, questions of independence, impartiality, confidentiality, and liability may be particularly important. The organization paying for an audit may have legitimate rights concerning the report, but payment does not automatically determine the professional responsibilities of the assessor or grant unrestricted control over the findings.Likewise, the organization performing the audit may have important responsibilities without necessarily possessing unrestricted rights to reuse or publish the information.
These examples illustrate why data sharing in food safety can be so complicated. The questions are not simply who owns the information or which license should be attached to a record. They concern who financed the activity, who generated the finding, under which scientific and legal conditions it was produced, who may use it for which purposes, and who remains responsible if something goes wrong.
There are also situations where the consequences of misinterpreting information may be considerably more serious than the commercial value of the information itself. A laboratory result or audit finding might influence decisions about product safety, supplier approval, market access, or regulatory compliance.
This is where I started to appreciate the limitations of my earlier assumptions. We can use metadata to describe the source of a record, its relationships with other records, and perhaps its permitted uses. We can apply licensing frameworks to clarify certain rights. But these mechanisms alone cannot determine whether information is scientifically reliable, whether it can legitimately be reused for another purpose, or who should be accountable for the consequences.
What I originally saw largely as a technical and licensing problem was also an institutional, economic, scientific, and legal one.
I haven’t abandoned the original ambition. But I no longer believe that selecting the right metadata model and license is enough to make it work.
©Phil Crow 2023
Why AI brings me back to these questions
Today, AI gives us capabilities that would have been difficult to imagine when I started my research. Systems can process enormous quantities of documents, extract information from different languages, identify relationships, assist with analysis, and generate recommendations at a scale and cost that were previously difficult to achieve. Some of the technical barriers that occupied us for years may become considerably easier to overcome. We may no longer need the same amount of manual work to translate information into common representations, identify relationships between records, or create the metadata needed to connect them. At least some of this work may become easier, faster, and less expensive.
But I am not convinced that this automatically brings us closer to the kind of shared and trustworthy knowledge environment we imagined. If anything, some of the unresolved questions are becoming more important. For much of my career, the challenge was helping people discover and use information that other people and organizations had generated.Now we increasingly have machines discovering information, interpreting its meaning, combining it with other information, and potentially using the results to recommend or initiate actions.
This introduces another level of complexity.
How does an AI system determine whether a particular record is authoritative, whether it can legitimately be reused, whether two apparently similar findings are actually comparable, or whether important context is missing?
Imagine a model combining laboratory test results produced by different organizations. The records may refer to the same microorganism or chemical hazard, but the laboratories may have used different analytical methods, sampling procedures, detection limits, or reporting conventions. The results may have been produced for different purposes and under different regulatory conditions. It may be technically possible to process all those records and identify apparent patterns. That does not necessarily make the resulting analysis scientifically meaningful.
Without understanding these differences, a model could combine findings that should not be treated as directly comparable or generate conclusions that appear more reliable than the underlying evidence justifies. The same applies to audit findings, regulatory decisions, and food safety incidents. A more capable algorithm does not automatically compensate for missing information, incompatible definitions, or unclear provenance. And even where the technical analysis is sound, questions remain about whether the information could legitimately be used for that particular purpose.
This makes me think that some of the principles we were exploring through metadata, interoperability, and knowledge representation may become even more relevant in the age of AI. Not necessarily because the same technical methods will remain appropriate. Many of them may be replaced or substantially improved. But because the underlying need remains. Information must carry enough context for other people and systems to understand what it represents, where it came from, under which conditions it was created, and what limitations or restrictions accompany its use.
There is also an economic question that I find particularly interesting.
If AI substantially reduces the cost of collecting, translating, classifying, and processing information, how does that change the value of the services built around it? Perhaps some activities that previously required significant investment will become relatively inexpensive. Others, particularly those involving verification, scientific interpretation, accountability, and integration into real decisions, may become more important. We may also see new commercial models emerging around the ability to provide trustworthy, well-governed intelligence rather than simply access to processed information.
These are possibilities, not conclusions. Their economic implications will depend on how AI capabilities develop, how reliable they become, and how organizations adopt them. But I think they deserve careful examination.
Food safety provides a particularly relevant environment in which to explore these questions because the consequences of getting an assessment or decision wrong can be substantial. The ultimate purpose of food safety intelligence is not to produce more information, better dashboards, or more sophisticated predictions. It is to help people and organizations make decisions that reduce risk and prevent harm.
The distance between a technically impressive analysis and an actual preventive decision can still be considerable.
This is one of the reasons I continue to be interested not only in what AI can produce, but in how it may change the way people interpret information, exercise professional judgment, and act upon evidence.
Returning to the commons
During the summer of 2026, I returned to David Bollier’s Think Like a Commoner, a book Paul Stacey had given me approximately a decade earlier. Rereading it, together with some reflections on stewardship during my visits to Canada, brought many of these earlier questions back into focus. I have been thinking again about what it means to create and maintain shared knowledge. About ownership, contribution, governance, and the responsibilities of those who benefit from resources created collectively.
The idea of a commons goes beyond making something freely accessible. It also raises questions about the community that uses a shared resource, the rules under which it is governed, and the work required to maintain it. This perspective feels particularly relevant when thinking about the information foundations that support increasingly sophisticated analytical and AI systems.
I still strongly believe that much of the knowledge generated through public resources should be accessible and reusable for public benefit. I also believe that the people and organizations investing in making this information reliable and useful can create legitimate commercial value. And I am increasingly interested in whether those benefiting from shared knowledge foundations should also accept some responsibility for sustaining them.
I find myself considering three propositions.
First, knowledge that is essential for protecting public health, particularly when generated through public resources, should be appropriately accessible and reusable.
Second, the work required to make that knowledge reliable, useful, and actionable can create legitimate commercial value and deserves appropriate reward.
Third, organizations benefiting from shared knowledge foundations may also have responsibilities to help maintain and strengthen those foundations.
These propositions seem compatible to me. But I do not think their practical implications are straightforward. >What exactly should be shared? Who should determine the conditions? How should rights and responsibilities be allocated among those who finance, generate, transform, and use information? What should happen when the information concerns a public safety risk but has been generated through a privately funded test or assessment? How can we encourage more organizations to share useful information without undermining their legitimate commercial interests, professional responsibilities, or legal obligations? And how can these arrangements work at the scale at which AI systems may increasingly be exchanging and using knowledge without a human examining every individual record?
I am conscious that openness is not always the right answer. Privacy, scientific independence, legitimate commercial interests, and the potential consequences of misuse cannot simply be set aside in pursuit of a more connected information environment. Nor can we expect that commercial incentives alone will sustain all the knowledge resources on which businesses and society depend.
There may be different answers for different kinds of information. Perhaps we need to start by understanding these differences much better. A more systematic approach to describing the rights, responsibilities, and conditions associated with information could facilitate much broader sharing and reuse. But I now see that this would require the participation of many different stakeholders, including those responsible for generating information, those who finance it, those who transform it, and those who depend upon it to make decisions.
Technology would be an important part of the answer. It would not be the complete answer.
An unfinished conversation
I do not have a complete solution to these questions. What I have is a growing sense that many of the problems we were working on twenty years ago are becoming relevant again, although the technologies, economic possibilities, and consequences have changed considerably.
I also recognize that some of the assumptions I held earlier in my career were incomplete. I underestimated how much effort it takes to maintain useful knowledge resources, how difficult it can be to align the interests of participating organizations, and how complicated ownership and responsibility become when information is generated and used in real operational environments. At the same time, I remain convinced that greater interoperability and appropriate sharing of knowledge could create substantial public and commercial value.
I do not see these as opposing ambitions. I see them as different aspects of a problem that deserves more attention, especially now that AI may allow us to process and connect information at a scale that was not previously practical.
This is also why I want to return to some of my earlier research and experiences, examine what has changed, and reconsider ideas that I once thought I understood reasonably well. I would like to explore how AI may change the relationship between shared knowledge, commercial intelligence, human judgment, and useful action. Food safety will remain my main area of reference. It is where I have spent much of my professional life and where I believe many of these questions become very concrete. But I suspect that similar challenges exist across other knowledge-intensive sectors.
I would like this to be an open conversation, drawing on published research, practical experience, and the perspectives of people working across science, technology, policy, and business. I am particularly interested in perspectives that challenge my own assumptions.
One question I would like to explore first is which types of food safety information should be more openly accessible and reusable, and what conditions would make that sharing responsible and sustainable. Because perhaps the most useful starting point is not asking what AI can now do with all the information available to us. It is asking what kind of knowledge foundations, rights, relationships, and responsibilities we need to make that intelligence genuinely useful.
Twenty years after that first question at the Agricultural University of Athens, I think this is still worth investigating.
These are personal reflections and do not represent the views or positions of SGS or FoodNexus.
