Skip to content

Glossary

Organisation

All configuration and data are housed under an organisation. Team members are invited to an organisation.

Platform

Each organisation can provision multiple platforms to reflect the online platforms they operate where user-generated content is hosted.

Complex Type

A complex type can be anything that contains user-generated content, e.g., a user profile, a forum post, a video upload or a chat message. It is composed of various fields defined by simple types.

Simple Type

There are a defined set of simple types

  • text
  • image
  • audio
  • video
  • uri
  • file

Content

Content is an instance of a complex type. It is user-generated content on your platform. Content can be created, updated, deleted, and moderated.

Content fields

A complex type is composed of a collection of typed fields, e.g., a user profile may contain an avatar (image) and a username (text).

The content field is identified in incoming data using a unique id.

Each field can be analysed using a set of pipelines we call a Strategy.

Strategy

A strategy is an analytical interface with a method that can be invoked to analyse the value of a content field as input and returns an evaluation.

For each type of media, there are a range of strategies available within Checkstep, e.g., an image can be analysed using AWS Rekognition and text can be analysed using Checkstep Machine Learning classifiers.

Strategies are fast and cheap, so they can run on all of your content, but they cannot grasp the full nuance of a policy (context, intent, exceptions). Their role is to tell you where to look: they produce potential violations, and a policy expert or ModBot decides what it is.

Designing strategies

A good strategy is a signal, not a mini-policy:

  • do flag anything that could indicate a breach, e.g., nudity detected, weapon in an image, external link shared
  • do not tune a single model to answer "does this breach our hate speech policy?", as it cannot see the nuance needed

Optimise strategies for recall rather than precision. Over-flagging is cheap, as the review sorts it out, while content that is never flagged is never reviewed.

Strategy Evaluation

Each strategy will return a label and score in the inclusive range [0.0,1.0], called evaluation which can be used to configure policies.

Policy

Policies reflect your Trust and Safety guidelines for your community. They are described by a title, a description and a set of rules over the strategy evaluations.

Severity

Each policy has a severity to prioritize violations.

There are a defined set of severities:

  • info - intended for informational kind of violations.
  • low - violations that may not require moderation.
  • medium - vast majority of violations.
  • high - violations that might require a dedicated queue.
  • critical - violations that require immediate review and action.

Default is medium.

Threshold

AI Threshold for Policy Rules

  • Lower Threshold: anything equal or greater than the Lower Threshold and below the Higher Threshold will be sent for moderator review
  • Higher Threshold: anything equal or greater than the Higher Threshold will be actioned automatically

Potential violation

A policy that content may breach, as flagged by the strategies analysing its fields.

Potential violations are a signal, not a moderation outcome: the decision taken later may confirm them, differ from them, or never happen. They are returned by the synchronous ingestion endpoint and the analysed-content webhook.

Violation

A policy that content breaches, as decided by a policy expert (a human moderator) or an automation such as ModBot. Also called an actual violation, as opposed to a potential violation.

Violations are the outcome of a moderation decision. They drive enforcement, strikes against authors and reporting. They are sent with the decision, author-decision and incident-closed webhooks.

ModBot

An automated moderator powered by a large language model that knows your policies. It reviews content flagged with potential violations and decides on it like a policy expert would, producing violations.

Running ModBot or human review on every piece of content is slow and expensive. Reviewing only the content flagged by strategies, typically 1–10% of it, keeps decisions accurate at a fraction of the cost.

Content case

A content case is a compilation of all the pieces of evidence related to content:

  • the different versions of the content, with analysis results
  • the decisions
  • the appeals
  • the reports
  • …

From these pieces, a log of incidents is computed and stored in the content case.

Incident

When a piece of scanned content triggers any of the implemented policy rules, an incident is generated which requires some actioning.