When does the AI approve or reject a post on its own?

Only when its structured decision meets a minimum confidence threshold. Anything below that stays pending for human review — ambiguous cases are deliberately routed to moderators instead of being auto-actioned.

What does a moderation decision record contain?

The decision, a confidence level, reasons, and flags — stored as separate records from the post itself. That audit trail lets an administrator answer what the model decided, why, and whether the post was actioned automatically or manually.

Why is moderation logic kept off the frontend?

The AI provider key, moderation prompt, and elevated database credentials stay on the server, and the database approval state decides visibility. A client that could inspect or alter moderation logic could also bypass it.

How I Built AI Moderation Without Removing Human Judgment

Anonymous communities need room for honest conversation. They also need guardrails.

While building Churv, I needed a way to handle guest submissions without forcing every post through a fully manual moderation queue. The goal was not to let AI replace moderators. The goal was to reduce repetitive review work while keeping uncertain decisions visible to people.

The problem

Guest accounts are useful for lowering the barrier to participation, especially in a campus community. But they also introduce more moderation risk.

If every guest submission appears instantly, the platform becomes harder to keep safe. If every submission waits for manual review, publishing becomes slow and moderators become the bottleneck.

I wanted a workflow that could make the obvious cases faster while preserving human judgment for the cases that actually need it.

The moderation pipeline

I built the workflow with Supabase Edge Functions, Postgres, scheduled jobs, and Gemini.

A guest submission first goes through an initial moderation check. It is then stored with an approval state rather than being treated as immediately public content.

A scheduled AI moderation pass can review pending submissions, return a structured decision, attach confidence and reason data, and update only the posts that meet the automation rules.

Guest submission → Initial safety check → Pending review
      ↓
AI moderation pass → Decision + confidence + reasons
      ↓
High-confidence action or manual review → Audit trail

This structure gives the system a clear source of truth: posts have an approval state, and moderation decisions are separate records that can be inspected later.

Confidence is a product decision

The important design decision was not whether the model could classify a post. It was deciding when the product should act on that classification.

I used a minimum confidence threshold before allowing the automated workflow to approve or reject content. A model can be useful without being treated as infallible.

When the confidence is not strong enough, the safer outcome is to leave the post for human review. That creates a deliberate fallback instead of pretending every decision can be automated.

I also designed the moderation output to capture more than a binary answer. The system records the decision, confidence, reasons, and flags so an administrator can understand what the automation saw.

Keeping the system observable

Automation without visibility is difficult to trust.

In Churv’s admin experience, moderators can see pending guest posts alongside moderation signals. The interface makes it possible to inspect the automated decision, the confidence level, and the reasoning before taking action.

That context matters. It lets a human moderator correct an edge case, spot patterns in false positives, and improve the system over time.

I also wrote the workflow to record moderation reviews separately. That audit trail makes it easier to answer practical questions later: What did the model decide? Why did it decide that? Was the post automatically actioned or handled manually?

Security boundaries

The AI provider key and database service credentials remain on the server. The browser does not get access to the moderation prompt, the model key, or elevated database permissions.

This separation is especially important for a community platform. Moderation logic should not be something a client can inspect, alter, or bypass.

The database approval state is also part of the security model. The frontend can display public, approved content, but it does not get to decide whether a pending guest post becomes visible.

What I learned

Building AI moderation taught me that “automated” should not mean “unaccountable.”

The best outcome was not a system that made every moderation decision on its own. It was a system that handles routine signals consistently, leaves room for ambiguity, and gives moderators enough context to make better calls.

That is the balance I wanted for Churv: a faster moderation workflow that supports human judgment instead of quietly replacing it.