No Gold Standard: Measuring Success in Medical Affairs

To understand true publication impact and influence patient outcomes, Medical Affairs teams must have their own benchmarks

Compass Points: The Future of Medical Affairs is a series exploring the strategic challenges facing Medical Affairs teams in today’s communication landscape—and the tools that will help them get it right.

The best publication strategy is like a treatment plan: bespoke

In the past, journal citations served as the primary metric for measuring publication impact. Citations all but guaranteed a share of voice and influence amongst key opinion leaders and healthcare practitioners; they were also a simple, clear metric to share upward, proving research impact and justifying the allocation of resources.

Today, journals have come to occupy a different place in the Medical Affairs community. They still confer legitimacy, but they’re not the only way to have an impact; they aren’t even necessarily the most appropriate channel through which teams can or should disseminate information. 

How scientific information travels can be measured in both scientific impact and real world impact. Scientific impact comprises long-tail, more static measures such as citations and subsequent policy changes tracked over the course of months or years. But real-world impact—how a publication influences thought, conversation and even behavior—can be observed in how information ripples through other more immediate channels, like social or broadcast media and forums. 

Capturing an accurate picture of how a publication has performed requires a view of both. This holistic view allows teams to accurately benchmark performance, measure impact, and demonstrate value to stakeholders, supporting the overarching goal of improving patient outcomes.

The role of the journal has changed

While journals still heavily inform the provision of healthcare alongside clinical guidelines and regulatory bodies, they are not the only place members of the life sciences community can encounter and learn about new research.

As scientific information has come to travel on more horizontal, peer-to-peer channels, such as social media or podcasts hosted by trusted key opinion leaders, practitioners are able to learn about and interact with new research outside of the journal publication and conference cycle. This makes it easier for HCPs to stay on top of relevant research, and to quickly sift through the studies that are relevant to their clinical practice. This is why, depending on the therapeutic area and the goals of a given publication launch, Medical Affairs teams may find they gain more traction by diversifying to non-traditional channels. 

But in order for publication planners to take advantage of this reality—to optimize distribution across geographies, channels and a variety of timescales—requires dynamic, granular data that is consistently tracked through time. And as journals have come to form only part of the life sciences research diet, Medical Affairs teams have been left without a single, strategic reference point both for forward-planning and post-publication performance reporting. 

As a result, teams find themselves in a familiar position: unable to reliably demonstrate impact, defend decisions to stakeholders, or to quickly iterate for later distribution plans. This can undermine a team’s efficacy, and ultimately delay or limit influence on patient outcomes. 

What teams lose without a consistent benchmark

Benchmarking plays a critical role in publication planning. It allows teams to reference both the performance of earlier publications and the work of competitors, and to learn, in real time, what is working and what is not. Without benchmarks, it is impossible to know what is reasonable for research to achieve, and therefore, impossible to contextualize impact and prove a return on education. 

But benchmarking is also one of the most laborious parts of the publication planning cycle. The process of consistently benchmarking, tracking, reconciling and cleaning point-in-time data—from social media, journals, podcasts, conferences, magazine articles, and more—can take teams weeks of work. And because of the fragmentation inherent to the process, all of this work, ultimately, still may not be able to capture the nuance of a publication’s impact.

The knock-on effect is that teams are unable to design strategies which are optimized for a given therapeutic area and to meet specific performance goals, such as social media engagement. This undermines the team’s ability to demonstrate that strategic objectives are met and can limit the diffusion of information into communities that could benefit from it. 

This cycle repeats; without live, ongoing benchmarking, it’s impossible to see what’s changing in the competitive landscape and react to it. 

In recent years, the Medical Affairs community has matured dramatically with regards to its use of data. Teams rely heavily on analytics and data-driven decision-making. They know what data is available to them and how they can use it to inform publication strategies. 

But the tools available for assembling and parsing this data haven’t kept pace. Even as many teams embrace the use of general-purpose AI, the output is often unstandardized and non-reproducible—what AI surfaces today may be different from what it surfaces tomorrow, so teams can’t be sure they’re comparing like with like. In other words, teams gain speed, but not certainty.

This is what Compass by Dimensions was created to address. It brings together both traditional and alternative metrics so teams can easily benchmark against internal and competitor data, track publication performance through time and across channels in a standard and simple way, allowing to better demonstrate value, influence therapeutic behavior, meet education objectives, and trace real-world research impact. As a result, teams can move from publication strategies which are fundamentally reactive to those which are proactive, and as a result, better able to meet strategic objectives.

Figure 1: Workspace performance. Total number of attention events tracked across supported sources.

Tapping into the discussions that matter

Improving patient outcomes is the result of a confluence of events: research must be carried out, written up, disseminated, and then found by the relevant policy makers or healthcare practitioners to stand a chance of driving real-world impact. That means Medical Affairs teams need to look at both formal and peer-to-peer channels to measure influence. 

Compass is driven by data sources from two leading services in the scientific and research community, Dimensions and Altmetric. 

Dimensions hosts one of the largest collections of interconnected global research data, re-imagining research discovery with access to grants, publications, clinical trials, patents and policy documents all in one place. This data source provides a robust view of traditional channels.

Altmetric is a leading provider of alternative research metrics, helping everyone involved in research gauge the impact of their work. Altmetric searches thousands of online sources including social media, revealing where research is being shared and discussed—and the sentiment of that discussion. This is where real-world impact manifests first. 

In bringing these two data sources together, Compass allows teams to track the whole publication attention lifecycle, from social media posts minutes and hours after publication, all the way to citations and guideline mentions years after it was published, all reliably benchmarked through time. 

Teams need only set up a benchmarking dashboard to define which internal and competitor publications they want to track. Then the dashboard can be referenced any time for a simple view of a publication’s performance. Instead of the heavily manual work that teams had to endure before, Compass brings precise and consistent performance measurement and real-world benchmarking across assets or disease areas, giving teams the evidence needed to understand what’s working and where science is influencing practice.

Figure 2: Total mentions across domains.

Democratizing data-driven publication strategy

Medical Affairs teams are too-often forced to rely on costly and slow agency relationships to understand how their publications are performing. Teams who can take these tasks in-house with tools such as Compass will amplify the efficiency and agility with which they can work. 

Compass performs three functions which are critical to creating a truly bespoke, data-driven publication strategy

• Unified, easy-to-reference metrics: Compass allows teams to quickly and consistently track performance through time to understand if a publication has reached the right people.
• Custom benchmarking: Teams can benchmark publications against their own portfolio’s historical performance, establishing what a realistic result looks like. This exercise can also encompass competitors and specific therapeutic areas, helping teams identify opportunities and learnings. The task of surveying channels and creating internal and external benchmarks would have taken weeks before—now, once a dashboard is created, it takes just a few minutes to check.
• Self-service interface: Compass creates shareable, stakeholder-ready visuals to clearly demonstrate the direct impact of work to decision-makers and support resource discussions with concrete data points. Teams can cut and re-cut data in a few minutes, saving the time, cost and hassle of looping in an agency every time a new report is needed.

Instead of waiting for an agency to return a report, or spending hours parsing data only for it to immediately stale, Compass places the power of real-time insights in the hands of the people best placed to wield it, streamlining the distribution process and supporting decision-makers with clear targets and performance measurement. 

Use case: Moving from lagging indicators to real-time feedback

Take a mid-size oncology-focused biopharma preparing to launch a publication for a second-line therapy. The Medical Affairs team’s usual process might take three to four weeks per reporting cycle: they would need to pull citation counts from one system, social and news mentions from another, then manually reconcile both in spreadsheets. If socials spiked after review had concluded on a given platform, that data would be missed. This means by the time a report reached leadership, the data would already be stale, and there would be no consistent way to benchmark the publication against prior launches in the same therapeutic area, because citations move too slowly and social media moves too quickly. 

For this team, Compass is designed to remove the manual burden of the benchmarking and tracking process. The team would first set up their benchmarking dashboard, tracking both the performance of previous portfolio publications and those of competitors working in specific therapeutic areas. Then, it would take just a few minutes to monitor performance: the team would easily be able to track journal citation activity against historical norms for the launch stage alongside engagement on clinician-focused social platforms and podcasts. 

Teams would be able to see where the research was finding the most resonance and quickly take action on that basis, for example, redirecting a portion of dissemination budget towards channels demonstrating traction. The consistency of this data also makes it easy to keep leadership informed; teams can slice and share reports from their Compass dashboard to illustrate what’s working and what isn’t. 

For teams that have previously relied on a single lagging metric, having a live, multi-channel benchmark can turn publication planning from a retrospective, best-guess exercise into a real-time, data-driven strategy.

The future of Medical Affairs is here—it’s time your strategy caught up

The proliferation of scientific information through popular channels is a good thing. 

That new scientific and medical information can reach much wider audiences through a variety of channels undoubtedly has a net positive impact on patient outcomes. Patient and rare disease advocates, healthcare practitioners working in remote or underfunded areas, and even patients themselves all benefit from having access to the cutting edge science that will shape the future of medical care. 

But the result of this proliferation has been a significant challenge for publication planners—no one could’ve predicted the rise of peer-reviewed podcasts. Now, teams need to take all of these data points into account when measuring performance and planning future publication launches. 

Manually compiling point-in-time data is a time-intensive process that puts publications at a disadvantage and undermines strategic and patient outcome objectives.

The right strategy is one entirely specific to a given publication—it must be tailored to relevant journals, audiences and channels, and these may all change through time. To date, with this level of nuance, it wasn’t possible for teams to keep up—not at the level of granular detail that could shape truly powerful publication strategies. 

Compass turns that complexity into an opportunity. 

Request a demo to learn how Compass could strengthen your team’s publication strategy.

The post No Gold Standard: Measuring Success in Medical Affairs appeared first on Digital Science.

from Digital Science https://ift.tt/2B3QOuq

From Writing the Rules to Building the Tools: Responsible Research Assessment in Practice

What does responsible research assessment actually ask of the people who build the infrastructure, rather than those who write the policy? In this post, Steven Hill traces the arc from DORA and the Leiden Manifesto through to the Barcelona Declaration, and sets out what the principles mean in practice for the tools that describe, discover, and measure research.

More than a decade ago, one of my first tasks in a new job was to advise on whether the organisation I had just joined, the Higher Education Funding Council for England, should sign the San Francisco Declaration on Research Assessment (DORA). We did, as a founding signatory, and it was the right decision. DORA’s central claim is that metrics, especially the journal impact factor, should not stand in as a proxy for the quality of an individual piece of research. As well as being right, that principle was an important signal that the UK’s national research assessment process was not taking a reductive approach to research quality.

What has stayed with me from that period is not the signing, but what came after. Alongside committing to an expanding set of principles building on DORA, the research system needs the patient work of turning those principles into reality. The Metric Tide, and its follow up seven years later, were in large part an attempt to take that problem seriously, and coined the term ‘responsible research assessment’, which labels the movement. The arc from DORA through the Leiden Manifesto, the Metric Tide, the Hong Kong Principles, the Coalition for Advancing Research Assessment (CoARA), and, most recently guidance from the Global Research Council (GRC), is the story of the global research community moving from declaration to implementation. The GRC, which brings together the heads of science funders from around the world, has provided funders with both tools to assess their own performance and a practical guide to making the changes needed in their practice. And the SCOPE framework for research evaluation offers a process for thinking through responsible research assessment in any evaluation context.

I find myself thinking about all of this again, but from an unfamiliar direction. For most of my career I have been a policy-maker, writing the principles and fretting about whether anyone is following them. At Digital Science I now look from a different direction: the building of the tools through which research gets described, discovered, and measured. Both setting the policy environment and helping to shape the tools bring power and responsibility, but the potential and the pitfalls are different.  The shift in vantage point raises a question: what should responsible research assessment ask of the people who build the infrastructure?

What the Principles Ask For – and What They Don’t

It is worth being clear about what responsible research assessment is, because it is easily caricatured. It is not a rejection of measurement, and it is not a plea to return to pure peer review uninformed by data. Read across DORA, the Leiden Manifesto, the Metric Tide, the Hong Kong Principles, and the CoARA agreement, and a consistent core emerges. Assessment should rest primarily on qualitative, expert judgement, with peer review at its heart, supported, not supplanted, by the responsible use of quantitative indicators. It should judge the work rather than the venue it appeared in. It should recognise the genuine diversity of what researchers produce and do: not only papers, but data, software, mentoring, peer review, public engagement, the often invisible labour of the people who make research possible. And it should be sensitive to context, to discipline, to career stage, and honest about its own limitations. Finally, as emphasised by the SCOPE framework, it is also important to critically reflect on whether evaluation is needed at all.

It is also fair to say that commercial entities in the research evaluation space are often criticised in discussions about responsible research assessment. The Leiden Manifesto asks that the data and the methods behind indicators be kept open and transparent, so that those being evaluated can verify them. CoARA goes further, calling for the research community to retain ownership and control of the infrastructure and the criteria used to assess it, and is openly wary of proprietary “black boxes”. The most recent Metric Tide review is blunt about the harm that commercial university rankings—built outside the academic community—continue to do to research culture.

Some of these critiques can be valid, although there are real practical challenges in realising total community ownership of data and infrastructure. And comparative analytics, well constructed and appropriately used, have a place in benchmarking universities. There is also the question of how commercial providers respond to responsible research assessment. The tools and the data are not going away; the question is whether they pull in the direction of the principles or against them. That is the real issue, and it should be the focus of the people who build the infrastructure, whether commercial or not, alongside the people who write the policies.

Why Openness Comes First

At Digital Science, colleagues here have been wrestling with this in public, through the lens of the Barcelona Declaration on Research Information. The Declaration’s first commitment is to make openness the default for the research information we use and produce—the records of who did what, where the money went, how outputs and contributions connect to one another—and to support the shared, open infrastructures that hold it. Writing on this blog, our CEO Daniel Hook has made the case that researchers have a fundamental right to access the metadata about research, and that the data used to evaluate academics should be transparently available and reproducible. He also argues that there are questions of assessment and measurement that will need data that is costly or complex to collect, and that openness might not be possible in this case. I think considering the balance and tension is the right direction, and it is worth dwelling on why, because open research information is the hinge on which the whole argument turns.

The responsible-metrics principles are simply not achievable on top of closed, unverifiable information. You cannot ask people to trust an assessment built on data they are not allowed to see. Open research information is the precondition, not an optional extra. But openness on its own is not enough. My colleague Simon Porter has written, again on this blog, about our responsibilities as consumers of metadata, not just producers of it. Use of research information needs to take into account the context in which it was generated, its provenance, and the extent to which the sources of information can be trusted, not just its availability. Information that is not accurate or appropriately contextualised can disrupt human judgement rather than support it. Simon also rightly notes potential equity concerns, where the metadata rich get privileged over the metadata poor, undermining the diversity and inclusion principle inherent in responsible assessment. He also notes that, as well as the responsible use of research information, responsible collection of data is also important.

Putting Principles into Practice

How does a commercial research infrastructure provider understand its role in supporting responsible research assessment? Rather than consider Digital Science’s products one by one, I want to focus on the principles of responsible research assessment and highlight examples where our tools and other options are aligned.

Broadening what counts. Research is more than journal articles, and the infrastructure has to be able to see and recognise a broader range of outputs. Being able to give a dataset a persistent identifier and a home, to surface software and preprints and policy documents alongside papers, to connect grants and patents and clinical trials into a fuller picture of a contribution is at the heart of responsible assessment. Digital Science tools such as Symplectic Elements, Figshare, and Dimensions, and the tools and work flows that they enable, are useful here precisely to the extent that they make the diverse outputs visible and creditable.

Supporting judgement rather than replacing it. The most valuable thing a system can do is not to produce a number, but to assemble as broad a view of the available evidence, so that human beings can exercise judgement well, and a researcher can tell their own story. Dimensions includes a range of tools that enable decision-makers to access clear summaries of the data and evidence that they need. Research information systems, such as Elements, that support narrative and evidence-based CVs, and that spare people the indignity of re-keying the same information into yet another form, are doing something genuinely in the spirit of the reform. The recently introduced CV import capability in Elements contributes directly to this objective.

Many dimensions, not one. When Altmetric first appeared, its real purpose was not to provide a new “score” but to emphasise evidence of broader contributions beyond those measured through citations. Evidence of attention in policy documents, in the press, in clinical guidance tells you something a citation count cannot. Links between publications and patents and policy documents in Dimensions also provide this richer picture of research. Outside of the Digital Science product line, Overton also provides data on the rich connections between research and policy.

Transparency and context. This is where the Barcelona Declaration is important, and Digital Science’s Open Principles set out how we work to align our tools with its aims. Making core elements of the Dimensions and Altmetric datasets freely available sits at the heart of these principles, alongside our commitments to work with the research community, and to openly publish our thinking and research. For example, where Dimensions data are used for assessment purposes researchers and their employers can check and verify the data. Our data sits alongside other open sources such as Crossref and DataCite and persistent identifiers like ORCID and ROR, key parts of the open responsible research assessment infrastructure. OpenAlex also offers fully open information as a secondary aggregator, overlapping in some areas with Dimensions.

I have spent enough time on the policy side to be wary of believing that any of this can be solved by better tools alone. Responsible research assessment is about behaviours, norms and incentives as much as it is about systems and infrastructure. And the choice isn’t between commercial infrastructure and community-owned systems. What matters is that infrastructure is built and used in a way that supports human judgement, broadens what we value, and submits itself to transparency and scrutiny. This is what responsible research assessment asks of those who build the infrastructure, and should inform everything we do at Digital Science.

The post From Writing the Rules to Building the Tools: Responsible Research Assessment in Practice appeared first on Digital Science.

from Digital Science https://ift.tt/9kBIOJf

Why Your AI Agents Are Only as Good as the Knowledge Behind Them

The race to deploy AI agents is accelerating, but most organizations are still building on sand. A new Gartner report suggests that the key to building reliable AI agents is a “context layer”.

According to Gartner’s latest research, 42% of enterprises plan to deploy AI agents by the end of 2026, and AI agent spending is expected to grow from 22% to 31% of total AI budgets in just one year¹. Despite this wave of investment, only one in five organizations report that their GenAI tools are delivering significant value. Hallucinations, limited impact, and unpredictable behavior remain stubbornly common.

The problem, Gartner argues, isn’t the models, but rather what surrounds them.

The Missing Layer

Behind every reliable AI agent is something Gartner now calls a “context layer”— a dedicated architectural component that curates, organizes, and delivers the knowledge an agent needs to act intelligently. Without it, agents are left processing noisy, poorly prioritized data, making expensive errors and producing outputs that can’t be trusted or traced.

Gartner is unambiguous about the stakes: by 2027, organizations that prioritize semantics in AI-ready data could increase their agentic AI accuracy by up to 80% and reduce costs by up to 60%. The context layer is no longer an optional refinement — it is the necessary foundation.

And yet this layer cannot simply be purchased. No vendor offers it out of the box. It must be engineered, assembled from services, capabilities, and custom modeling that together transform an organization’s tacit knowledge into something AI agents can actually use.

Three Components, One Foundation

As stated in the report, there are three interlocking components that make up this ‘context’ layer: semantics, operational state, and provenance. Together, they form a pipeline that allows agents to retrieve the right information, organize it coherently, and act on it with accountability.

Semantics: Meaning, Not Just Data

Semantics is the component most organizations are missing, despite it being the one with the greatest leverage. Gartner finds that organizations implementing semantic modelling such as ontologies and knowledge graphs, are 2.2 times more likely to achieve high effectiveness in AI data engineering, however, only 40% of organizations have done so. 

Semantics means representing your organization’s knowledge—business entities, rules, policies, relationships, metrics—in machine-readable form. This allows AI agents to interpret what something means in context and execute an action based on that context, not just pattern-match on keywords. Without this layer, even the most sophisticated agent is, in effect, guessing.

This is precisely the domain where metaphactory brings long-standing proven capability. metaphactory by metaphacts, a Digital Science solution, is a knowledge graph platform enabling organizations to build and maintain rich semantic models for over a decade—connecting business glossaries, ontologies, and data products in ways that AI agents can directly leverage. For organizations serious about agentic AI, a robust semantic foundation isn’t a future aspiration; it is a prerequisite.

Operational State: The Right Information at the Right Time

While semantics provides meaning, your operational state provides situational awareness. AI agents need access to current, accurate information about the entities and processes they’re acting on beyond just snapshots, such as up-to-date information on customers, datasets, experiments, publications and suppliers. 

For research-intensive organizations, this is particularly acute. The ‘operational state’ of a research environment spans live datasets, ongoing experiments, researcher expertise, institutional repositories, and the evolving landscape of published science. Digital Science’s portfolio—including Dimensions, Altmetric, and Figshare—represents exactly this kind of curated, continuously updated operational knowledge. Rather than building this knowledge from scratch, organizations working in research and innovation already have access to a pre-assembled foundation.

Gartner also highlights the Model Context Protocol (MCP) as the emerging standard for connecting agents to operational state efficiently and securely. Dimensions, Altmetric, and metaphactory already support MCP, reflecting a broader conviction that research infrastructure should be designed to meet agents where they are, not retrofitted after the fact. As adoption of the protocol grows across the industry, having well-structured knowledge accessible through it will matter more, not less.

Provenance: Trust Through Traceability

The third component—provenance—is what makes agentic AI governable. It encompasses the systematic tracking of data lineage, agent decisions, actions, outcomes, and feedback across the full lifecycle of AI operations.

For research organizations, publishers, and funders, provenance isn’t merely a governance checkbox. It is central to the integrity of the work itself. Reproducibility, accountability, and the ability to audit AI-assisted conclusions are not simply peripheral concerns; they are defining ones. Gartner notes that 74% of organizations recognize that data governance tools are essential to operationalizing AI governance, yet robust provenance mechanisms remain rare in practice.

Digital Science’s longstanding commitment to open, traceable research infrastructure, including persistent identifiers, transparent data lineage and open metadata, gives research organizations a natural head start on this component. The challenge is connecting these capabilities explicitly into the agentic architecture, so that every AI-assisted decision can be traced back to its sources and reviewed.

Research Intelligence as a Context Layer

There is a broader framing worth making explicit here: for organizations operating in research, science, and innovation, the context layer is not merely a technical architecture problem. It is, at its core, a research intelligence problem.

The tacit knowledge Gartner describes—the organizational understanding that must be made machine-readable for AI agents to function—is, in a research context, the accumulated intelligence of a scientific community: what has been discovered, by whom, with what methods, validated how, and applied where.

We have spent over a decade building infrastructure that captures precisely this kind of knowledge at scale. The shift to agentic AI doesn’t make that infrastructure less relevant—it makes it more so. The question is no longer just “can researchers find the right information?” but “can AI agents, acting on researchers’ behalf, find, interpret, and act on that information reliably and accountably?”

The answer depends entirely on the quality of the context layer underneath.

What This Means in Practice

For R&D leaders and data and analytics leaders, the practical implication is this: before asking which AI agent to deploy, ask what context layer you have in place to support it. Gartner’s advice is to start with high-value use cases rather than attempting a comprehensive build all at once—iterate, demonstrate outcomes, and expand. That is sound counsel. But iteration without a semantic foundation, without right-time data access, and without provenance mechanisms will simply produce faster failures.

The organizations that will lead in agentic AI are not those that move fastest to deploy agents. It is the organizations that invest earliest in the knowledge infrastructure that make agents worth deploying.

Digital Science is working with research organizations and data-intensive enterprises to build the context layers their AI strategies require.

If you’d like to explore what this looks like for your organization, get in touch.

---

1. Gartner. (2026). The 3 core components of the context layer for AI agents. [Research Note/Report]. https://www.gartner.com/document/ [G00848874]

The post Why Your AI Agents Are Only as Good as the Knowledge Behind Them appeared first on Digital Science.

from Digital Science https://ift.tt/1isBFqy

REF readiness: evidencing Contribution to Knowledge & Understanding

In the first blog in this series, we explored engagement and impact readiness for the Research Excellence Framework (REF) 2029. Here, we turn to the second element of assessment: Contribution to Knowledge & Understanding, and what it takes to approach it with evidence and confidence.

Contribution to Knowledge & Understanding (CKU) sits at the core of REF assessment. For institutions preparing submissions, the task is not simply to present strong individual outputs, but to show how research collectively advances knowledge within and across disciplines and how that work was enabled and supported within the institution’s research environment.

As REF 2029 approaches, most institutions will find that they are not short of high-quality research.  The task they now have is to present that research as a coherent, representative, and defensible account of contribution, traceable back to the people, grants, and infrastructure that enabled it.

That distinction matters more than it might first appear.

Beyond completeness

It is tempting to frame CKU readiness as a data completeness problem. If all outputs are captured, the argument goes, selection can proceed with confidence.

But completeness is not the same as representation.

REF panels assess whether a submitted body of work reflects the range and diversity of a unit’s research activity, not simply whether a record exists for every output. A dataset can be complete and still produce a submission that is narrow, uneven, or poorly contextualised.

“The challenge for universities is not simply about capturing and submitting quality research outputs to REF, it is about demonstrating the full diversity and breadth of the research outputs. In choosing which outputs to submit, universities are expected to demonstrate the diverse range of staff contributing to the outputs; the diverse range of disciplines, research methods and output types whilst also ensuring that contributions from inter- and multi-disciplinary collaborations are represented,” says Natalie Dallat, Head of Research Performance, Ulster University.

This distinction has practical implications. In a decoupled framework, where submitted outputs do not need to be linked to specific individuals, institutions still need to demonstrate a substantive connection between research and the environment that enabled it. That requires not just complete records, but well-contextualised ones.

Three areas of risk are worth examining in turn.

Output visibility: what institutions know and what they can prove

In practice, most significant outputs are already known to institutions. Academic workflows, open access deposit requirements, and internal review processes mean that the majority of relevant publications are captured somewhere.

The more common challenge is not absence but unevenness, gaps in coverage that accumulate over time through staff mobility, inconsistent author affiliations, publications linked to grants but not captured locally, and interdisciplinary outputs that fall between Units of Assessment (UoA).

Figure 1: University of Oxford, all publications 2021-present

These are rarely major gaps in institutional systems. But in aggregate, they can affect the completeness and credibility of a submission, particularly in disciplines where research activity may be systematically underrepresented relative to its actual volume.

Addressing this requires two complementary layers. Research information systems such as Symplectic Elements provide structured output capture, validation workflows, and linkage between researchers, publications and grants, creating the audit trail that REF governance demands. An independent, interconnected data layer such as Dimensions then enables cross-checking: surfacing missing outputs, highlighting metadata discrepancies, and providing a broader view of publication activity beyond local records.

“What Dimensions allows institutions to do is essentially hold a mirror up to their own systems. Not to replace internal records, but to ask: is what we’re seeing internally representative of what’s actually out there? For some disciplines or research groups, that comparison can be revealing,” explains Ann Campbell, Director Research Impact & Comparative Analytics at Digital Science.

Together, structured capture and independent validation strengthen confidence in completeness before output selection begins and provide a more defensible evidence base for the decisions that follow.

Understanding performance within fields

Once institutions have confidence in the completeness of their records, a second challenge emerges: interpreting performance in a way that is fair and defensible across disciplines.

Raw citation counts rarely tell the full story. Citation norms vary significantly across fields; what constitutes a well-cited output in a fast-moving biomedical discipline looks very different from the equivalent in history or architecture. A paper with 20 citations might be considered relatively modest in one field, but well above average in another. 

While output selection is typically led by discipline experts within UoA, decisions are often informed by broader portfolios and mixed indicators. Without appropriate field-level contextualisation, there may be tendency to overvalue some outputs that align with readily interpretable patterns of performance (i.e., citation counts) and undervalue others particularly where interdisciplinary research is involved. This can have consequences both for selection and for the narrative presented to panels. 

The scale of this variation is visible in the data. Across UK institutions, raw citation counts for outputs in Units of Assessment such as Clinical Medicine or Physics far exceed those in disciplines like History or Art & Design, and yet when performance is measured relative to field norms, the picture shifts substantially. Units that appear modest on raw citations often demonstrate strong or above-average relative contribution when field-normalised indicators are applied. For institutions making selection decisions across multiple UoAs, this difference is not academic: it directly affects which outputs are recognised as genuinely competitive, and which risk being undervalued simply because they sit in lower-citation disciplines.

Figure 2: Average citation counts vary substantially across Units of Assessment, while field-normalised performance (FCR) highlights strong relative contribution in disciplines where raw citation accumulation may be lower. *

*  Average citation counts and field-normalised citation performance across REF Units of Assessment (2014–2021, UK Institutions, articles only using Dimensions UoA Classification) 

Figure 3: Average Citation Count per Publication

Field-normalised indicators and disciplinary benchmarking support a more accurate and defensible reading of performance. Dimensions enables field-normalised citation analysis, benchmarking against peer institutions, collaboration pattern analysis, and trend tracking across time.

Peer review remains central to CKU assessment. But contextual data helps institutions approach that peer review better prepared with a clearer sense of where their research sits within its field, and a stronger basis for the interpretive narrative they are expected to provide.

From individual outputs to coherent thematic narratives

CKU submissions are strongest when outputs form a coherent intellectual narrative. Panels respond to thematic depth and sustained advancement of knowledge not isolated high-performing items, however well-cited they may be.

That makes output selection a genuinely strategic exercise and the scale of the choices involved is considerable. Analysis of REF21 submission patterns shows that the typical institution produced eligible research across 33 of the 34 UoA, but submitted to just 20. In nearly one in five cases where an institution had a meaningful body of research within a UoA, that UoA received no submission at all. Even within the UoAs that institutions chose to submit, the median coverage rate was under 7%.

Figure 4: Research breadth vs submission breadth

The submitted profile, in other words, represents a deliberately selective slice of a much broader underlying research base. That selectivity is appropriate as REF rewards quality over volume, and strategic narrowing is both permitted and expected. But it means the submitted body of work must tell a coherent story about where an institution’s research genuinely lies. Getting that story right requires a clear view of the full landscape: understanding where depth is concentrated, where disciplines connect, and where gaps might undermine the coherence of what is presented to panels. 

Thematic clustering and citation network analysis can help identify areas of concentrated strength and the interdisciplinary bridges that connect them. These analytical approaches surface patterns that may not be visible when outputs are reviewed individually, and support the kind of coherent story that distinguishes a strong CKU submission.

That coherent story, however, increasingly needs to account for more than publications alone. As REF increasingly recognises diverse outputs, datasets, code, preprints, and other research artefacts alongside traditional publications, institutions also need infrastructure that makes that breadth visible and accessible. 

The evidence from REF21 illustrates how far there is still to go: of the 4,000 non-traditional outputs submitted, almost three quarters had unknown or unresolvable locations, and only 244 had DOIs. REF21 Main Panel D assessors noted the wide variety, inconsistent quality and uneven preservation of practice-based outputs with many hosted on fragile, short-lived platforms that were difficult to navigate. 

Platforms such as Figshare support persistent access, DOI assignment and the presentation of these materials as part of a coherent research record, ensuring that the full range of contribution is available for assessment.

Scholarly visibility: useful context, not a proxy for contribution

While CKU is fundamentally about intellectual contribution, the broader circulation of research can provide supplementary context. Where outputs are being cited in policy documents, taken up in professional practice, or discussed in specialist communities, those signals can help situate the reach of a body of work, particularly in applied or interdisciplinary fields where impact pathways are diverse.

Altmetric can surface where outputs are being referenced beyond traditional citation indexes, from policy and clinical guidelines to media and public discourse. These signals do not measure contribution to knowledge and understanding, and should not be presented as a substitute for bibliometric evidence or peer judgement. But as additional context, they can help round out the picture, particularly for outputs whose significance may not be fully reflected in citation metrics alone.

The important distinction is that scholarly visibility supports interpretation. It does not replace it.

From reactive selection to confident CKU readiness

CKU readiness is about planning, not last-minute correction. Institutions that approach it most effectively don’t wait until selection is imminent. They build the evidence base over time, ensuring completeness, contextualising performance, and constructing the thematic narrative that panels expect to see. 

“What we often see is that institutions feel more confident in REF preparation when they’ve been building the picture gradually over time. It becomes easier to understand where strengths are emerging, how research sits within its field, and how to present that contribution coherently,” says Campbell.

REF readiness is about leading, not lagging. For CKU, that means investing in the evidence and infrastructure and contextual understanding that supports selection throughout the cycle. 

Institutions preparing for REF29 are increasingly focusing on areas such as: 

• ensuring completeness of publication records using interconnected data solutions such as Dimensions to validate coverage beyond institutional systems
• structuring output capture and governance through systems such as Symplectic Elements, linking people, publications and grants
• contextualising performance within disciplinary norms through field-normalised analysis in solutions such as Dimensions
• preserving diverse outputs and research artefacts with persistent identifiers, through repositories like Figshare
• drawing on supplementary context from tools such as Altmetric and Dimensions to situate reach and scholarly engagement

Together, these form the building blocks of a CKU submission that is traceable, representative, and defensible.

Digital Science supports this readiness through interconnected solutions that strengthen evidence and decision-making, while leaving judgement firmly with institutions and REF panels.

Whether you want to audit output visibility and identify gaps in your publication record, benchmark your CKU evidence within disciplinary context, or map the thematic strengths that will anchor your submission narrative, Digital Science can help.

Book a call to explore how.

The post REF readiness: evidencing Contribution to Knowledge & Understanding appeared first on Digital Science.

from Digital Science https://ift.tt/s7nJBkc

From The Lancet to TikTok: Benchmarking success for publication strategy in medical affairs

Scientific communications have never traveled so far so fast. Medical affairs teams need an omnichannel approach to planning and monitoring publication strategy.

Compass Points: The Future of Medical Affairs is a series exploring the strategic challenges facing medical affairs teams in today’s communication landscape—and the tools that will help them get it right. 

The goal of everyone working in medical affairs is ultimately to improve patient care. But success is contingent not only upon the research, trialing and production of innovative treatments; it depends equally upon the firm’s ability to educate on the suitable applications of a new treatment and establish trust within healthcare environments.

If healthcare practitioners don’t know a better treatment or diagnostic exists—or if they do, but they don’t trust it—they won’t use it in treatment plans. This has implications for the quality of patient care and commercial impacts for the firms creating those new treatments.

But the chain of communication is more fragmented and complex than it has ever been, and this makes identifying and monitoring how information travels difficult. In order to make sure information is reaching the right people in the right places, medical affairs teams need benchmarking and measurement tools, like Compass by Dimensions, which are capable of processing the rich, complicated reality of the communication landscape today. 

How does scientific information travel?

Up until the recent decade, it was common for a healthcare provider to learn about new treatments and come to believe in their legitimacy after reading about them in a respected journal. This was a typical part of a clinician’s day, and reflects the supremacy of journals such as The Lancet or the New England Journal of Medicine, which is still entrenched today.

In recent years, scientific breakthroughs have found purchase across more diffuse channels, such as newspapers, radio and television. Now, scientific information is disseminated in every direction—both via “linear” one-to-many channels such as journals and also rhizomatically, across low-frequency networks such as social media, podcasts, internet forums, and word of mouth. 

This has the undeniable benefit of bringing critical information to wider audiences, often with tremendous speed, but these channels lack the legitimacy of the big journals. 

A new approach to scientific communication

Everything from peer-reviewed podcasts and video abstracts to plain language summaries and audience-segmented data now form a critical part of scientific communication. Where before this information could only travel in the rarefied air of prestigious journals, today, important research outcomes are accessible to audiences beyond academics and even beyond healthcare professionals. 

This is a positive trend; these popular channels open research findings and awareness to patients and advocates—who, in rare disease settings, are often the best-informed in a given room.

Publication planners should embrace the potential that comes with this reality; with it comes the opportunity to reach new markets and to better influence treatment protocols even in remote fields. 

For example, isolated clinicians in dispersed healthcare environments are historically among the hardest to reach and among the most likely to be using outdated treatments. They may not be reading The Lancet. They may not be at the big conferences. But if those clinicians encounter a new treatment option in a mid-tier journal, in a podcast, on social media, and in a clinical newsletter, they may change their prescribing behavior. 

In these popular channels, trust and legitimacy is assigned instead by the opinion leaders who share about medical or scientific topics. The speed with which information travels and nature of the conversation it elicits can color its reception—making monitoring each type of channel all the more important. 

Without a clear view of this data, medical affairs risks unsuitable communications plans that fail both the commercial objectives attached to a given asset and the people they intend to help with it. 

How can measuring publication performance impact commercial and medical objectives? 

Scientific information is traveling in novel ways. It is an entirely new challenge for publication planners and communications professionals to attempt to parse, measure, and analyze this data so that they may better design future communications strategies.  

To know if you’re succeeding, you first need to know what success looks like. 

In the 1980s, success could be measured in citation counts. This was appropriate as journals were a primary mode of scientific communication. Today, the gold standard for measuring how information travels and resonates is fuzzier. 

Planners know that omnichannel communication forms a critical part of a robust publication strategy. But to date, there hasn’t been a simple way to collate a unified view of research impact across the spectrum of communication channels at play. 

So, planners often still rely on citation counts, as these are concrete and reproducible as a measurement. But they reflect academic attention almost exclusively and obscure the impact of a given asset outside of these narrow academic channels. Weaving in altmetrics is a highly manual process, where planners must stitch together sources such as mentions on social media, broadcast media and journals. This practice is time consuming and difficult to rely upon because it is so difficult to standardize and view in aggregate. Online conversation might move in ways that are impossible to predict or track, making metrics difficult to compare and learn from.

But pressure for hard numbers and clarity is growing. High-quality research that fails to reach its audience is, commercially, wasted investment. Research that doesn’t travel can’t shape clinical awareness, influence prescribing behavior, or support often costly distribution activities. Planners need a single, unified view of total scientific impact which they can rely upon.

How should medical affairs teams build publication strategy?

To know a given communication is having the desired effect, planners need a view of three things:

• Reach: is information propagating across relevant networks, i.e., news, social media, podcasts, clinical commentary
• Engagement: are the intended audiences engaging with the research, and how are they talking about it
• Impact: is there evidence of the therapeutic conversation or relevant policies shifting 

The performance data needed to answer questions of reach and commercial viability exists. But the fragmentation of social platforms adds complexity—monitoring must now span X, BlueSky, Reddit, and beyond—but the richness of this data is unprecedented and therefore invaluable. 

Compass tracks reach, engagement and impact and provides an overall view of scientific impact, so planners tracking alternative metrics can identify, track and analyze trends over time. From there, they can use aggregate views of asset performance as jumping off points for sentiment analysis and deeper audience research.

See how publication attention is distributed across domains

Having this data to hand makes publication strategy an endeavor of cause and effect rather than guesswork—seeing where research has resonated particularly well or potentially missed the mark informs each subsequent communications plan. 

Why is benchmarking so important in medical affairs publication strategy?

Understanding your own reach and engagement is important, but without a point of comparison, it’s impossible to know whether a result is strong or where resources are well spent. Benchmarking performance—understanding what reach, engagement impact looks like per therapeutic area—against internal track records and those of competitors must form a central tenet of publication strategy.

Medical affairs teams must benchmark in two directions. 

The first is competitive benchmarking: understanding how your publications and communications are performing relative to peer firms working in the same therapeutic area. This type of benchmarking helps identify gaps in therapeutic discourse along with spaces that are already crowded, helping planners tailor and prioritize their approach.

Monitor top-performing publications by their Altmetric attention score and citation count

The second is industry benchmarking: understanding how your publication performance compares across therapeutic areas and channels. What does a typical volume of clinical engagement look like for the launch of a publication? What level of social chatter is reasonable to expect from a given journal tier? What rate of sentiment shift can be linked to momentum within therapeutic environments?

Compare publication performance against selected disease area or drug benchmarks

In short: benchmarking defines how we might judge success. Together, competitive and industry benchmarking transform measurement from a simple reporting exercise into a strategic one. They make it possible to set meaningful publication targets, track progress against them, and align publication activity with clinical trial milestones and other medical affairs priorities.

Ultimately, being able to access, monitor, and derive insights from this data will deliver not only a critical competitive and strategic advantage; it will help ensure information is reaching the people who need it.

“The proliferation of communication channels, and the increasingly diverse ways in which HCPs gather and share information about treatments have resulted in a very dynamic and complex impact environment. Compass from Dimensions represents a significant step forward in simplifying how we understand and communicate the values of our omnichannel strategies.”—Mike Taylor, Head of Information & Analytics, Digital Science

Compass was designed to help answer these questions of impact. Built on Dimensions and Altmetrics data, Compass combines publication and altmetrics into a single collaborative workflow, simplifying how medical affairs teams benchmark, track and manage publication impact and reach. Compass by Dimensions is developed by Digital Science, an AI-focused technology company that transforms fragmented data into unified knowledge assets, leveraging AI and Knowledge Graphs to deliver structured, actionable intelligence for high-value discovery and innovation. By combining unparalleled data depth and breadth with enterprise-ready AI technology, we help leaders confidently accelerate product life cycles and secure a decisive market lead.

Curious about what Compass can do for your team? Book a demo and start your free trial today.

The post From The Lancet to TikTok: Benchmarking success for publication strategy in medical affairs appeared first on Digital Science.

from Digital Science https://ift.tt/kVcIj1W

Will 2025 be a turning point for Open Access?

With a number of deadlines for open access (OA) coming up in 2025 and beyond, the race is on for many publishers to make the transition to OA. Simon Linacre asks, are these targets achievable?

---

Traditionally, September and October have always been one of the busiest – and most interesting – times to be in the publishing industry. Back in the day, September would be the deadline for the first of the following year’s issues to be collated by editors, while in more recent times big events like the ALPSP Conference, the Frankfurt Book Fair and Open Access Week have set the agenda for the remainder of the year and beyond.

In 2024, this period has perhaps more intrigue than most given a number of deadlines and political events occurring in the next 12 months or so, many of them revolving around open access (OA) and its further adoption. But will things pan out the way people anticipate, and are there solutions that can be used to help forge a path through so many uncertainties about the future?

Conference season

At the recent ALPSP Conference in Manchester in September, there was a good deal of discussion about how open access had developed this year, and its potential progress in 2025 and beyond. Perhaps unsurprisingly at a conference full of publishers, the mood was a little downbeat when it came to the theme of OA, but not for the reasons one might think. Reading between the lines, there was a frustration at the shifting sands many felt they had to constantly navigate, in the shape of changing or newly introduced policies, and a sense that innovation was being stymied as a result.

For example, the tone for OA seemed to have been set by the JISC report on transformative agreements (TAs) which was published in the UK earlier in 2024. This made for somber reading, with the headline prediction that while the UK’s transitioning to OA was faster than most countries, based on the journal flipping rates observed between 2018–2022 it would take at least 70 years for the big five publishers to flip their TA titles to OA. 

With this in mind, the fact that there were deadlines for Plan S set for 2025 around transition that seemed unlikely to be met, and with the OSTP memo in the US mired in committees and a potential change on the cards in the White House, the belief among many publishers was that the move to OA was not happening at the pace or in the direction that many thought it would.

Geopolitical calculations

In addition to what is happening in the UK, Europe and in the US, events further afield are also causing publishers to take stock of their medium-to-long-term strategies. The publication of authors based in Russia has declined sharply since the invasion of Ukraine in early 2022, and collaboration between US authors and those based in China have also decreased, possibly due to policy changes by the Chinese government favoring publication in China-based journals, but also potentially due to fears about research security issues in the US and in other countries. 

China’s move to OA is also happening at a much lower level than many countries, which is significant as it takes up such a high percentage of published articles, passing the US a few years ago as the world’s most prolific publisher of research articles. As a result, despite the increase in the number of TAs being agreed with universities, publishers are still seeing a high degree of uncertainty in the transition to OA.

Forward motion

This uncertainty will be in the back of publishers’ minds when celebrating OA Week this year, coming as it does every year on the back of major conferences such as ALPSP and Frankfurt, and in the midst of fine tuning budgets for the following year. At Digital Science, we understand this predicament given how closely we work with publishers as customers, and also because many of us have worked in the publishing industry ourselves. As such, we have been analyzing how Digital Science solutions can help publishers steer a path forward on OA and transformative agreements, and have created this use case for Dimensions in support of our community.

This resource has been designed to reflect the period of change that the publishing industry is undergoing, supporting the need for publishers to create, evaluate and negotiate TAs by delivering a strong range of historical and predictive data through Dimensions. Using the Dimensions database – which now holds data on almost 150m publications as well as details on funding, grants and patents – publishers can easily find and analyze data surrounding authorship across categories such as country, geography, institution and funder. Understanding a given discipline’s current or future state of play can complement publishers’ own data and inform their strategies accordingly.

Solid state

The theme of this year’s OA Week – ‘Community over Commercialization’ – is a deliberately provocative one, and should engender a good deal of debate during the week and beyond. It should also broaden the conversation to adjacent areas such as open research and open science, as here we have policy and geopolitics making waves for everyone involved in the research ecosystem. 

The origin of some of these ripples can be seen in two upcoming reports from Digital Science. At the end of October, a new report on Research Transformation includes substantial input from those involved in academia on how OA is impacting on their work, while November sees the ninth annual State of Open Data report, tracking how researchers see open data issues developing as part of their work. Without giving too much away, both of these reports call for greater awareness of – and support using – the myriad of fast-developing technologies that are starting to impact academics and their institutions. As such, the community of interest that supports OA Week every year needs to work together in the ecosystem they all inhabit if those OA deadlines are to be met.

---

You can see a demo of Dimensions and its latest features and meet some of the Digital Science team at Charleston in November. To organize a meeting with us, just email publishing@digital-science.com

About the Author

Simon Linacre, Head of Content, Brand & Press | Digital Science

Simon has 20 years’ experience in scholarly communications. He has lectured and published on the topics of bibliometrics, publication ethics and research impact, and has recently authored a book on predatory publishing. Simon is an ALPSP tutor and has also served as a COPE Trustee.

The post Will 2025 be a turning point for Open Access? appeared first on Digital Science.

from Digital Science https://ift.tt/7DQGm4s

Embracing Lived Experience: A Data-Driven Look at Autistic Involvement in Autism Research

People with lived experience of a condition bring unique and valuable insights when planning research into that condition. Using data from Dimensions, Emily Alagha examines the evolution of autistic people’s involvement in autism research over the past two decades.

---

Author’s note about identify-first language

In this post, I am using identity-first language (e.g., ‘autistic person’) to honor the preference of many in the autism community who embrace their identity as an integral part of who they are. This approach reflects the values of empowerment and self-identification.

---

The Rise of Participatory Research

There’s a growing recognition in the research community that individuals with lived experience of a condition or phenomenon can offer unique and valuable insights to the design of scientific studies. This collaborative approach is often referred to as participatory research and actively involves individuals with lived experience in all stages of the research process. Dimensions data (visualized below) reveals a steady increase in research articles using terms related to participatory research, suggesting a growing embrace of this approach within the scientific community. This shift reflects a move towards more inclusive research practices that empower individuals and communities to actively participate in knowledge creation that is directly relevant to the needs and priorities of those it aims to serve.

Image 0: Rise in Dimensions publications for participatory research and related terms.

This post examines recent trends in a specific subset of participatory research that highlights lived experience contributions, as identified through publication authorship and acknowledgments. Focusing on autism research, I will delve into this trend by leveraging Dimensions data to analyze autistic authorship and acknowledged collaborative support. I’ll also compare the trajectory of this movement to similar trends in mental health and chronic illness research. Finally, I’ll discuss the implications of these findings for research impact and visibility and advocate for greater inclusion of those with lived experience in shaping future studies.

Characterizing Autistic Contributor Representation in Autism Research Articles

Methodology

Individual contributions to research studies are most often represented by the author and acknowledgements sections of publications. To investigate how autistic contributions are characterized in the literature, I leveraged the capabilities of the Dimensions database to search within the raw affiliation and acknowledgements fields of research publications. I used a combination of search strategies to focus on publications related to autism research and specifically targeted publications that either:

• Included autistic or neurodiverse authors in the raw affiliations section OR
• Acknowledged autistic people, patient networks, or advisory groups in their acknowledgments section AND
• Mentioned autism-related keywords in their full text

I examined author affiliations and acknowledgments to identify the most common language used to represent contributions from autistic people. I also explored bibliometric indicators such as citation counts, Field Citation Ratio (FCR), and Altmetric Attention Scores to assess the impact and reach of autism research with autistic contributors compared to the broader field of autism studies. Finally, I applied the same approaches to explore how lived experience contributions are characterized in other fields to identify avenues for potential future growth of autistic representation in research.

The Rise of Autistic Authorship

To understand how autistic authors represent themselves, I conducted a qualitative review of author affiliations in participatory autism research to identify common phrases and terms. These range from explicit identifiers like “Autistic Researcher” or “Independent Autistic Scholar,” to affiliations with advocacy organizations such as the Autistic Self Advocacy Network, and roles emphasizing lived experience like “Expert by Experience” or “Lived Experience Professional.” While the number of publications authored by self-identified autistic individuals is currently limited (231), these publications offer valuable insights into the unique perspectives and contributions of autistic researchers.

Image 1: Author collaboration network for lived experience autism researchers and their co-authors.

This network visualization represents a preliminary attempt to identify leading neurodivergent researchers engaged in autism and neurodiversity scholarship. While the search terms were designed to highlight self-identified neurodivergent researchers and allies, it’s important to note that this method may not be fully accurate, and not all individuals included may identify as neurodivergent. The visualization highlights key figures like Sonia Johnson, Fiona Ng, and Dora Madeline Raymaker, who are known for their work in this area and could provide valuable leadership on best practices for autistic inclusion in research.

Highlighting specific examples of impactful, autistic-led research with high citation counts and Altmetric Attention Scores (a measure of online attention and engagement) demonstrates the influence of these authors on the broader research conversation.

Top Cited Research Article among Autistic Lived Experience Authors:

• Nicolaidis, C., Raymaker, D., McDonald, K., Dern, S., Boisclair, W. C., Ashkenazy, E. & Baggs, A. (2013). Comparison of Healthcare Experiences in Autistic and Non-Autistic Adults: A Cross-Sectional Online Survey Facilitated by an Academic-Community Partnership. Journal of General Internal Medicine, 28(6), 761–769. https://doi.org/10.1007/s11606-012-2262-7  

This study compares the healthcare experiences of autistic and non-autistic adults through an online survey, uncovering significant disparities for autistic people. Autistic collaboration involves authors from the Autistic Self Advocacy Network and the Academic Autistic Spectrum Partnership in Research and Education (AASPIRE). The high citation count of this study underscores its impact on shaping subsequent research around healthcare access and equity for autistic people.

Top Altmetric Score and Field Citation Ratio among Autistic Lived Experience Authors:

• Pearson, A. & Rose, K. (2021). A Conceptual Analysis of Autistic Masking: Understanding the Narrative of Stigma and the Illusion of Choice. Autism in Adulthood, 3(1), 52–60. https://doi.org/10.1089/aut.2020.0043  

This conceptual analysis investigates autistic masking as a response to stigma. Collaborators include Kieran Rose of The Autistic Advocate and Infinite Autism. The high Altmetric score and Field Citation Ratio (a measure of a study’s influence within its specific field) highlight the broad reach and impact of this work on online platforms and in further research.

These examples illustrate the power of autistic-led research to generate new insights and draw attention to often overlooked topics. Having examined the influence of key autistic researchers, it’s essential to explore the broader scope of autistic involvement in research, beyond authorship.

Broadening the Scope: How do Papers Characterize Autistic Contributions Beyond Authorship? 

While authorship provides a clear indicator of direct contribution, it doesn’t capture the full spectrum of autistic involvement in research. I expanded the analysis to include the acknowledgments section of publications to gain additional insight into how autistic people contribute to and shape research. Acknowledgments often reveal a wider range of roles and contributions, such as participation in advisory boards or community networks.

Expanding the analysis to include publications that acknowledge autistic or neurodiverse people, patient networks, or advisory groups in the acknowledgments section significantly broadened the dataset to 703 publications (as of September 25, 2024). Throughout this post, I use the term ‘autistic-contributor research’ to describe these studies where autistic individuals are explicitly acknowledged or listed as co-authors. This term represents a narrower subset of participatory autism research, specifically focusing on visible contributions through acknowledgments or authorship, rather than all potential forms of participatory involvement.

As the chart below illustrates, this expanded search demonstrates that autistic contributions extend beyond authorship and can be recognized in several different capacities.

Image 2: Counts of select collaboration phrases in the acknowledgements and author affiliation fields of autistic-contributor research literature.

Patient Representation: The term “patient” emerges as a frequent descriptor in research acknowledgments. It can encompass diverse roles like “patient partner,” or refer to administrative functions related to patient involvement. However, the meaning of “patient” in the author affiliation and acknowledgements section can be ambiguous, sometimes signifying autistic individuals themselves, other times denoting individuals with different conditions within the study.

While widely used, “patient” has limitations in autism research. It centers on pathology and potentially overlooks the broader spectrum of autistic experiences beyond the clinical realm. Not all autistic people identify with this label, as it may imply illness or deficit. While “patient” may suggest autistic involvement in healthcare research, it also highlights the need for more precise language that recognizes the multifaceted roles of autistic people beyond the traditional patient-provider dynamic.

Independent Researchers and Advocates: The presence of terms like “advocate,” “self-advocate,” “lived experience,” and “independent researcher” highlights several ways autistic people contribute to research both as individuals and as part of broader groups of expertise. The use of “independent researcher” in affiliations suggests a recognition of the contributions made by autistic researchers working outside traditional academic institutions.

Group Advisory Roles: The prevalence of terms like “advisory board,” “advisory panel,” “community network”, and “working group” underscores the importance of structured mechanisms to ensure that autistic perspectives and lived experiences directly inform research design and implementation. These groups may not always be composed of autistic people, but they often have close ties to communities with lived experience and aim to represent those perspectives. 

How do studies integrate autistic voices into the study design? Autistic-contributor research is more likely to use qualitative or mixed-methods approaches

Autistic-contributor studies in this dataset are significantly more likely to employ qualitative or mixed-methods approaches when compared to all autism research. Qualitative methods, such as interviews and focus groups, allow autistic people to express their unique perspectives and insights in their own words. Some examples of how studies may integrate autistic voices include co-creating research questions with autistic people, adapting methods to be more accessible, including autistic researchers on the team, and involving autistic participants in data analysis and communication of findings. These collaborative approaches can help studies be more directly relevant to the autism community.

Who is leading in these types of autistic-contributor collaborations? 

It can be useful to explore leading organizations in this dataset to understand where and how investments in autistic-contributor collaborations are happening. Affiliation, funding, and geographic data in Dimensions highlight the United Kingdom’s prominent role in fostering research collaborations involving autistic people. The National Institute for Health and Care Research (NIHR) and the Department of Health and Social Care (DHSC) are the leading funders, while University College London and King’s College London are at the forefront of institutions publishing participatory approaches in this field. These data suggest a strong commitment within the UK to promoting inclusive research practices. However, it’s important to acknowledge that this analysis primarily reflects English-language publications, and there may be additional contributions in other languages that use different terminology to acknowledge autistic participation.

Comparing Autistic-Contributor Autism Research with all Clinical Autism Research

What topics are addressed by autism research that acknowledges autistic partners in the author or acknowledgements fields?

Image 3: Autistic-contributor research concepts network.

Image 4: Clinical autism research concepts network.

In a concept analysis of autistic-contributor research literature, I found a clear emphasis on lived experience, health services, and support systems. Instead of primarily asking “What causes autism?” or “How can we diagnose autism?”, this research asks “How can we improve the lives of autistic people?”. This emphasis is reflected in the prominence of terms like “improve access” and “health system” in the autistic-contributor research network visualization above. 

This focus contrasts with broader clinical autism research, which emphasizes cognitive and behavioral aspects of autism. In the clinical autism concept network above, the strongest themes are diagnosis, social skills, and behavior. 

The distinction is further reinforced by how research is categorized. Clinical autism research falls under Field of Research (FoR) classifications of Psychology and Biomedical Sciences, while autistic-led research leans towards Health Sciences and Health Services. This highlights a fundamental difference in priorities. 

It’s also worth considering the potential impact of age on these research approaches. Autistic-led research may naturally involve more adults, given the complexities of participating in research design. This could lead to a greater focus on issues relevant to autistic adults, an area often overlooked in traditional research.

How does impact look compared to all autism research?

Data sourced: 25 September 2024.

Though still in its early stages, autistic-contributor research shows promising signs of greater impact in both academic citations and public reach. 

Citation, Field Citation Ratio (FCR), & Citation Rate: The average Field Citation Ratio (FCR) for autistic-contributor research is 5.30, compared to 2.31 for all autism research. The citation rate for autistic-contributor autism research (76.65%) is slightly higher than the overall citation rate for autism research (65.57%). Additionally, autistic-contributor research demonstrates a comparable average number of citations per publication (22.76) to the broader field of autism research (23.28). These figures indicate that autistic-contributor research is cited more frequently within the scientific community.

Altmetric Attention Score & Societal Impact: Autistic-contributor research in autism exhibits an average Altmetric Attention Score of 8.6, notably higher than the average of 4 for all autism research. This indicator shows that autistic-contributor autism research sparks more conversations outside of academia than broad autism research.

Translation into Policy, Practice & Innovation: Autistic-contributor research in autism has a higher rate of citation in policy documents (4.7%) compared to the broader field of autism research (2.0%). It also maintains a comparable rate of citation in clinical trials (0.7% vs. 1.2%). However, when it comes to citations in patents, autistic-contributor research lags behind with only 0.4% of publications cited compared to 2.2% in the broader field. These figures suggest that while involving autistic people in research may lead to findings that are more readily translatable into policies and clinical practices, there’s room for growth in terms of fostering innovation and generating patentable discoveries.

Autistic-contributor research in autism represents a small subset of the overall autism literature, but its higher FCR scores and Altmetric Attention Score, comparable citation averages, and stronger translation into policy collectively show the value and influence of research that actively involves autistic people. 

Learning from Other Fields: Comparison to Chronic Illness and Mental Health Research Literature with Lived Experience Contributions

Both chronic illness and mental health research fields have a strong track record of including people with lived experience as active contributors. We can gain valuable insights to enhance autistic representation in research by analyzing language used to acknowledge lived experience contributions in these fields. If we were to standardize language used to describe these collaborations, would it be easier to measure these types of collaborations? What terms would be best to use across fields?

“Patient” and “patient advocates” are some of the most highly used terms across both mental health and chronic illness participatory research, but may present challenges in the context of autism research where some participants do not want to pathologize autism. An emphasis on “lived experience” as an authorship and acknowledgement phrase is also common across all three fields, and may be a better approach to recognize contributions in autism research. Another structure sometimes used in the author affiliation fields is “with [condition]”, such as “researcher with chronic illness” or “advisor with bipolar disorder”. This structure is difficult to standardize across research areas and may make it harder to discover experts with relevant lived experience.

Additionally, there is an emphasis on group collaborators across all three fields. The prevalence of working groups and advisory panels demonstrates the effectiveness of these structures in facilitating meaningful participation and ensuring that diverse perspectives are heard. 

Image 5: Counts of select collaboration phrases in the acknowledgements and author affiliation fields of participatory autism research literature.

Image 6: Counts of select collaboration phrases in the acknowledgements and author affiliation fields of lived experience-contributor chronic illness and mental health research literature.

Implications and Recommendations

Despite the promising rise in participatory autism research, it still constitutes a small fraction of the overall autism literature. Much of the research remains rooted in clinical or mechanistic approaches and often overlooks the contributions of those with lived experience. To address this gap, funders, researchers, and institutions must prioritize participatory research approaches that actively incorporate autistic perspectives at every stage of the research process. 

Recommendations:

• Funders and Institutions: Prioritize funding and support for participatory research initiatives that actively involve autistic people in all stages of the research process.
• Researchers: Embrace collaborative approaches and methodologies, establish meaningful partnerships with autistic and neurodivergent communities, and ensure that research designs and methodologies are inclusive and accessible.
• Publishers: Consider metadata fields which standardize how participatory collaborations are described, in collaboration with the research community Consistent language can improve the discoverability of lived experience collaborators.
• Autistic Individuals: Seek out opportunities to participate in research, share your expertise and insights, and advocate for greater representation and inclusion within the research community.

By actively involving autistic people in the research process, researchers in the field can improve the relevance of their work and address the real-world challenges and needs of the community. This evidence can inform policy decisions and advocacy efforts that lead to more equitable and supportive systems for autistic people and foster a deeper understanding of autism.

---

Special thanks to Holly Wolcott, Ph.D., Senior Vice President of Research Analytics at Digital Science, for her insightful feedback on this blog post.

Request a demo or quote

About the Author

Emily Alagha, Senior Director of Research Analytics & Support | Digital Science

Emily Alagha is a Senior Director of Research Analytics & Support at Digital Science, where she leverages AI-powered platforms like Dimensions to support data-driven strategies to optimize research funding and enhance research management practices. With a background in medical librarianship, she is passionate about health literacy and ensuring research is accessible to all. She is also a neurodivergent self-advocate committed to amplifying autistic voices and increasing autistic representation in research.

The post Embracing Lived Experience: A Data-Driven Look at Autistic Involvement in Autism Research appeared first on Digital Science.

from Digital Science https://ift.tt/PAQs85N