Designing for verifiable research integrity
Researchers and librarians were not resisting change. They were refusing to approve research workflows they could no longer independently audit.
Duplicate review became inspectable instead of automatic, preserving expert judgment inside reconciliation workflows.
From platform trust to
independent verification
Modernizing the interface was not enough. Power users needed visible proof that research evidence remained inspectable, traceable, and defensible.
I led UX strategy and research across verification workflows — including migration, deduplication, and project containment — during the transition from Legacy RefWorks to New RefWorks.
Legacy RefWorks was next to sunset, but institutional teams still depended on it
RefWorks was in the middle of a multi-platform transition. Classic RefWorks had recently been sunset, and Legacy RefWorks was scheduled to sunset next as New RefWorks became the primary platform.
Most users had already migrated to New RefWorks. The remaining Legacy RefWorks users, however, supported systematic reviews, institutional reporting, and other research workflows where mistakes could compromise accountability.
My role was to understand why systematic review users were not moving to New RefWorks, what their workflows required, and what the product would need to support before the business could responsibly retire Legacy RefWorks.
Users created their own audit trail around reference counts
In interviews and workflow observations with 10+ librarians and research administrators supporting systematic reviews, users repeatedly checked reference counts before and after work moved through New RefWorks.
They wrote counts down, compared folder totals, checked duplicate groups, reconciled migrated records, and looked for signs that a reference had moved, merged, duplicated, or disappeared.
Breakdowns appeared wherever records moved between states
The same behavior appeared across shared folders, deduplication, migration, and project-switching workflows. When a count changed, users looked for a way to confirm what had changed and whether the result matched their expectations.
The visible pattern was behavioral: users were building their own audit trail around the product because they did not fully rely on the interface alone to prove that evidence had remained intact.
Systematic reviews made each reference part of the evidence chain
Those audit behaviors mattered because systematic reviews turn references into accountable evidence. Users were not simply organizing citations. They were maintaining a defensible chain of records across search, screening, deduplication, inclusion, exclusion, and reporting.
In that context, a missed duplicate, broken folder count, or untraceable import could compromise PRISMA reporting, reproducibility, and institutional credibility. Count-checking was not resistance to change; it was a way to protect the integrity of the review.
The same accountability pattern explained behavior across migration, deduplication, and project containment workflows. Users needed to understand where counts changed, why records moved, and whether another reviewer could reproduce or validate the same result.
At first, this looked like migration friction. The deeper constraint was verification: hesitation appeared when the system removed the audit signals users needed to approve work they were responsible for.
Not a migration problem. A verification problem.
The migration effort initially appeared to be an adoption challenge: make New RefWorks easier to use, reduce friction, and move remaining users off Legacy. Research revealed a different constraint: users would not trust migrated records until they could independently verify how references moved, changed, merged, and remained separated across workflows.
Once verification became the governing constraint, product decisions shifted from reducing friction to preserving the audit signals users relied on before approval.
Verification became the architecture for accountable reference handling
Once the team reframed the problem around verifiable evidence handling, the architecture had to protect the moments where expert users audited system behavior and made accountable decisions themselves.
Possible duplicates became reviewable evidence, not automatic cleanup
Possible duplicates were surfaced as reviewable candidates, with matching logic and metadata evidence preserved for expert judgment before records were merged, excluded, or moved.
User sets criteria
Exact/similar, fields, primary reference
System finds candidates
Possible duplicates based on logic
Expose evidence
Metadata, rationale, count impact
Human review
Compare records and judge
User resolves records
Keep separate, merge, exclude, or move
Each workflow exposed a different accountability risk
Once verification became the governing constraint, the product response became more precise: expose the evidence needed for each risk before users committed an irreversible action.
- Visible source and destination counts
- Checkpoint review before commitment
- Exportable reconciliation artifacts
- Side-by-side metadata comparison
- Visible match evidence
- Review states that preserve manual decisions
- Clear workspace boundaries
- One login across multiple projects
- Permissions and administrative safeguards
Across workflows, the pattern was consistent: preserve the evidence needed for expert review before the system committed changes that would be difficult to unwind.
Safeguards preserved auditability where simplification created risk
New RefWorks could reduce friction, but not by hiding the evidence users needed to approve accountable research work. Two safeguards carried that boundary into the product: preserve expert review during deduplication, and keep evidence contained within clearly defined projects.
Make duplicate decisions inspectable before records changed
Duplicate review was evidence review, not routine cleanup. Researchers needed to explain how candidates were identified, which record was preserved, and whether another reviewer could reproduce the result.
Constraint: Similar matching labels in Legacy and New RefWorks could produce different duplicate counts, while review practices also varied across institutions. Users needed to see the logic behind the result rather than trust a single automated answer.
Product response: Let users configure matching behavior, comparison fields, and primary-reference logic before reviewing candidates. After matching, preserve the metadata needed to inspect and defend each decision.
Tradeoff: The workflow remained slower than fully automated deduplication, but protected the expert judgment and reproducibility systematic reviews required.
1
2
3
4
5
6
Keep project boundaries visible before users acted
Researchers needed containment, not another layer of folders. Each systematic review carried its own references, permissions, decisions, and audit trail. Work in one review could not be allowed to affect another.
Constraint: Folders organized references but did not create true separation. Researchers often maintained multiple accounts because separate credentials felt safer than sharing one undifferentiated workspace.
Product response: Introduce dedicated project workspaces under one account. References, folders, permissions, and actions remained scoped to the active project, while global navigation made that context visible.
Tradeoff: Projects introduced more structure than folders alone, but reduced the risk of evidence or permissions leaking between reviews.
1
2
3
4
5
6
Operational confidence scaled because evidence stayed inspectable
The important behavior change was not adoption alone. Users proceeded when the system gave them enough evidence to account for their data and enough control to decide when an outcome was safe to approve.
New RefWorks did not replace external documentation and reconciliation practices. It gave librarians and administrators clearer product evidence to compare against the records they already maintained.
The organization could responsibly retire long-standing legacy infrastructure only after expert users had enough evidence to rely on the replacement system operationally.
-
01
Operational reconciliation became independently verifiable
Administrators could compare exportable migration records against migrated system states before approving outcomes.
-
02
Duplicate review decisions became reproducible
Users could inspect matching logic, metadata differences, primary-reference choices, and count impacts instead of relying on opaque cleanup behavior.
-
03
Project evidence remained isolated and accountable
Teams could manage multiple systematic reviews from one account without contaminating evidence between projects.
Verification made approval conditional on evidence, not reassurance
Users tolerated added complexity when it made outcomes auditable. Confidence grew because the product exposed the counts, checkpoints, and reconciliation paths needed to decide whether to proceed.
When users are accountable, preserve the audit trail
I initially treated migration confidence as an adoption problem. Research revealed that users were evaluating something deeper: whether they could independently verify outcomes they would later be held accountable for.
That shifted my decision model for expert systems. The question was no longer “How much complexity can we remove?” It became “Which complexity must remain visible so users can evaluate risk?”