Case Study: AgentScore — Building a Trust Scoring Platform for AI Agents | Janus Compliance

The Brief

The AI agent economy is growing fast. Platforms like Moltbook host over 1.5 million agents. Agents earn real money on task marketplaces. But there's no way for a human to answer: "Can I trust this agent before I pay it?"

AgentScore answers that question. It aggregates publicly available data about an agent's identity, work history, social presence, and on-chain activity to produce a single trust score (0-100) that anyone can understand. Think Credit Karma, but for AI agents.

The challenge: build a data aggregation and scoring platform that pulls from multiple external APIs, processes the data in real time, and presents scores through both a web interface and a public API — while handling the compliance implications of scoring entities, processing user data, and operating across jurisdictions.

The Compliance Challenge

A trust scoring platform raises specific compliance questions that don't apply to simpler AI builds:

Algorithmic transparency. The scoring algorithm determines how entities are perceived. Users need to understand how scores are calculated, what data feeds into them, and how to challenge a score they believe is wrong.
Cross-border data flows. Data is pulled from platforms hosted in multiple jurisdictions — US-based APIs, blockchain data with no fixed jurisdiction, social platforms with global user bases. Each data flow needs documentation.
User data processing. The platform collects user data through API key registration, search queries, and usage analytics. All personal data under GDPR if EU users access the platform.
Automated profiling. Generating a "trust score" from aggregated data is profiling. If agents are operated by identifiable individuals, GDPR rights around automated decision-making may apply.
API as a product. The scoring API is sold to third-party platforms. Each integration creates a new data processing relationship requiring documentation.

What We Built

1. Multi-source data aggregation

The platform pulls data from multiple external sources in real time:

Social platforms — karma, post count, verified status, account age, activity frequency
On-chain data — ERC-8004 identity tokens, transaction history, wallet activity
Task platforms — completion rates, client ratings, dispute history

Each source is treated as a separate data processing activity with its own documentation. The architecture uses graceful degradation — if a source is unavailable, the score is calculated from whatever sources respond, with the gap flagged in the output.

2. Graduated scoring algorithm

The scoring system produces a composite 0-100 trust score from weighted components: identity verification, work history, social presence, on-chain activity, and consistency.

A key compliance-driven design decision: we replaced binary pass/fail thresholds with smooth curves. No cliff edges where an agent jumps from "trusted" to "untrusted" because of one arbitrary data point. Graduated scoring is fairer and more defensible if the methodology is ever challenged.

The full scoring methodology is documented and published — any user or agent operator can understand exactly how their score is calculated and what affects it. This isn't just good practice; it's a transparency obligation if GDPR automated decision-making provisions apply.

3. Full-stack platform

Built on Next.js with Supabase (Postgres) for persistence. The architecture:

Web interface — search and view agent trust scores, score breakdowns, historical trends
REST API — programmatic score lookups for platform integrations ($29/month tier)
Embeddable badges — SVG score badges that platforms can embed directly
Rate limiting — IP-based throttling on all endpoints to prevent abuse

4. API-as-a-product compliance

When a third-party platform integrates the AgentScore API, that creates a data processing relationship. Each integration needs:

Data processing agreement covering what data is shared and how
Clear documentation of who is controller vs processor
Terms of service covering API usage, data retention, and prohibited uses
Rate limiting and access controls to prevent data scraping

We built template DPAs and API terms that scale with each new integration — so the compliance burden doesn't grow linearly with the customer base.

Compliance Documentation Delivered

Data Protection Impact Assessment

Covering all data sources, the scoring algorithm, user data processing, API integrations, and cross-border transfers

Scoring methodology documentation

Full transparency document explaining how scores are calculated, what data is used, and how to challenge a score

Privacy notices

For website users, API customers, and agent operators whose data is aggregated

Data processing agreement templates

Scalable DPA templates for API integrations, covering controller-processor relationships

API terms of service

Usage terms covering data handling, prohibited uses, rate limits, and liability

Cross-border transfer documentation

Mapping every data flow from external APIs, documenting safeguards for each transfer route

Compliance-Driven Design Decisions

Graduated scoring over binary thresholds

Binary pass/fail creates cliff edges that are hard to explain and easy to game. Smooth curves produce scores that are more accurate, fairer, and easier to defend under transparency requirements. An agent with no work history scores 0/20 on that component, not a blanket fail.

Graceful degradation when sources fail

If one data source times out, the score is calculated from available sources with the gap flagged. This avoids penalising agents for infrastructure failures and keeps the scoring transparent about what data informed each result.

Published methodology

The full scoring algorithm is documented publicly. Every user can see exactly how scores are calculated. This satisfies GDPR transparency requirements and builds trust with the platform's users — if you're scoring trust, you need to be trustworthy yourself.

Supabase over SQLite

Initial architecture used SQLite, which doesn't persist on serverless infrastructure. Migrated to Supabase (Postgres) for reliable data persistence, proper audit logging, and GDPR-compliant data deletion capabilities when users exercise their rights.

The Outcome

A working trust scoring platform with three revenue tiers (free, API, enterprise), real-time multi-source data aggregation, and complete compliance documentation — ready for users in any jurisdiction.

The compliance documentation is designed to scale. Template DPAs mean each new API integration doesn't require bespoke legal work. Published scoring methodology means transparency is built into the product, not bolted on later. Cross-border transfer documentation covers every data flow route, so adding a new data source means documenting one additional transfer, not rebuilding the compliance framework.

The system the buyer gets: a working product they can ship, with documentation that proves it's compliant. Not a report telling them what to fix.

Need a data platform built compliantly?

Whether it's a scoring engine, data aggregation pipeline, or analytics platform — we build it and deliver the compliance documentation as part of the project.

Get a Fixed-Price Quote

Fixed-price quote within 48 hours. No hourly billing.

AgentScore: Building a Trust Scoring Platform for AI Agents