splitchAlpha

Feature flags and experiments, built for agents

Control what ships. Learn what works.

Turn features on or off without redeploying your app. Run A/B experiments to measure how changes affect your users. Your coding agent can manage both through the CLI.

npm install --global @splitch/cli
splitch login
splitch context --json
Set up a feature flagUse your coding agent
Model comparisonExample experiment

Control

Current model

The model you use today

Treatment

Candidate model

A cheaper model to try

Can a cheaper model compete? Collect user feedback and compare the results before deciding.

Flags + Experiments

Two reasons to use splitch.

Use Flags when you need a switch. Add an Experiment when you have a question to answer, like whether users prefer the responses from a different model.

Turn features on and off

Put a feature behind a Flag, then change its rollout without redeploying your app. Each Environment has its own configuration, so you can test in dev before enabling it in prod.

Compare alternatives with an Experiment

Test a new model, prompt, or product change against what you use today. Connect user feedback to a Metric and compare the Variants with a confidence interval.

Agent first

Let your agent operate splitch.

Install the CLI, authenticate, and select your App and Environment. Your coding agent can then configure Flags, set up Experiments, and read results from your terminal.

splitch context --json
splitch flags list --json

Every command supports JSON output. Errors include a code and a suggested next step, so your agent can act on the response. See the CLI guide for setup and commands.

What it can do once connected

  • Create an App, Environments, and keys
  • Create Flags, enable them, set the rollout
  • Verify a Flag resolves before you write code
  • Define Metrics and start an Experiment Run
  • Read results and end the Run

Prefer MCP? Connect your agent to the splitch MCP server. You can also manage Flags and Experiments in the control panel. Read the agent setup guide.

Statistical rigor

See the difference, and the uncertainty.

An Experiment can show an improvement, a regression, or an inconclusive result. How much you can learn depends on the feedback you collect and the size of the effect.

  • Sequential analysis supports checking results while an Experiment runs.
  • Results use recorded Exposures to count who encountered each Variant.
  • Traffic imbalance checks flag problems that can make a comparison unreliable.
  • Confidence intervals show the uncertainty around the measured difference.

In practice

Is a cheaper model worth switching to?

In Neuron, we use splitch to compare a newer, cheaper model with the model we have been using. Users provide feedback on the responses, which we measure in an Experiment.

The question is whether user feedback differs between the models. An inconclusive result does not establish that they perform equally well.

Quickstart

From setup to your first Flag.

Install the CLI and authenticate, then create and verify a Flag before connecting it to your application.

  1. Step 1

    Create an App

    Environments and keys for dev and prod come with it.

  2. Step 2

    Create a Flag

    Name it, give it Variants, enable it, set the rollout.

  3. Step 3

    Verify

    A real round trip on your credential, before you touch code.

  4. Step 4

    Wire one call

    evaluate() returns the Variant and records the Exposure.

Set up a feature flagRead the docs

Put your next change behind a Flag.

Start with a feature toggle. Add an Experiment when you want to measure its effect.

Set up a feature flagRead the CLI guide

Why splitch? Split testing and feature switches, fused into one word.