moder8.net: An Automated Content Moderation Tool

Developing Your Own UGC Moderation Engine

Develop Moderation Prompts

At moder8.net, develop and test your moderation prompt using Preprocessor, Categories, the Sand Box and Test Bench.

Notice how the Sand Box and Test Bench give you the total number of tokens and the cost of the moderation call as well as a break down into individual cost components.

The Dashboard provides a summary of category violations and records the cumulative cost of the moderation calls to the LLM.

Test Moderation Prompt

Using a simple console based LLM to rigorously to test your moderation prompt against AI generated adversarial test cases. I used Claude for the default prompts:

I am building an automated user content generation (UGC) moderation tool that uses an LLM prompt as the moderation engine to categorise violations into one of twelve categories in increasing order of severity:

NONSENSICAL, JAILBREAK, SPAM, PROFANITY, SCAM, CRIMINALITY, HARASSMENT, HATE_SPEECH, SEXUAL, VIOLENCE, SELF_HARM, CHILD_SAFETY plus an additional category of BRAND_PROTECTION which is intended to flag content that could damage my brand.

Can you please generate 5 adversarial tests for each category. The difficulty should be as hard as you can make them. Then present the tests as a BASH script that runs CURL against the endpoint https://moder8.net/api/moderate using a POST request passing a JSON body of:

{
  systemInstructions: the moderation prompt specified below
  userInput: Your simulated UGC
}

moderationPrompt:
<replace this text with your full moderation prompt>

There should be a 5 second delay between each request as the server is rate limited. The script should state the expected result and the actual result and present a final summary including a chart of the test results.

Claude is inclined to produce its own test harness in the console but the CORS restriction I currently have on the moderation endpoint will prevent that from working so we need a BASH script.

If any tests fail repeat the development and debugging step. This can be a difficult task as a modification that catches one edge case may create a vulnerability for a new one.

You will most probably have to repeat each step a number of times before all tests pass.

When all your tests pass you are ready to integrate the full prompt into your own moderation pipeline.