Skip to main content

Language Model Overrides

Chat Completion Parameter Overrides​

The v1/chat/completion endpoint is compatible with OpenAI's API. It supports a subset of request body parameters defined in the OpenAI reference documentation. Spice helps configure different defaults for these request parameters.

Supported parameters:

Example: Setting Default Overrides​

Deprecated Default Overrides Parameters

The openai_ prefix is deprecated for non-OpenAI model providers. Use the model provider prefix instead.

To specify a default override for a parameter, use the model provider prefix followed by the parameter name. For example, to set the temperature parameter to 0.1 for all requests with this model for Hugging Face model, use hf_temperature: 0.1. A temperature parameter in the request body will still override the default.

models:
  - name: pirate-haikus
    from: openai:gpt-4o
    params:
      openai_temperature: 0.1
      openai_response_format: { 'type': 'json_object' }

When sending this payload to spice /v1/chat/completions:

{
  "model": "pirate-haikus",
  "messages": [
    {
      "role": "user",
      "content": "What is the capital of France?"
    }
  ],
  "temperature": 0.5
}

Will be passed to the OpenAI API as:

{
  "model": "gpt-4",
  "messages": [
    {
      "role": "user",
      "content": "What is the capital of France?"
    }
  ],
  "temperature": 0.5,                          // temperature overriden by value in request body
  "response_format": { "type": "json_object" } // default response format from model configuration
}

System Prompt​

In addition to any system prompts provided in message dialogue, or added by model providers, Spice can configure an additional system prompt.

models:
  - name: pirate-haikus
    from: openai:gpt-4o
    params:
      system_prompt: |
        Write everything in Haiku like a pirate

Any request to HTTP v1/chat/completion will include the configured system prompt.

Example: Enforcing default structured output and using system prompt​

This example demonstrates how to create a specialized math tutoring model by combining system prompts with structured JSON output. The configuration ensures consistent, step-by-step mathematical solutions in a machine-readable format.

models:
  - name: math-tutor
    from: openai:gpt-4o
    params:
      system_prompt: |
        You are a helpful math tutor. Guide the user through the solution step by step.
      openai_response_format:
        type: json_schema
        json_schema:
          name: math_reasoning
          schema:
            type: object
            properties:
              steps:
                type: array
                items:
                  type: object
                  properties:
                    explanation:
                      type: string
                    output:
                      type: string
                  required:
                    - explanation
                    - output
                  additionalProperties: false
              final_answer:
                type: string
            required:
              - steps
              - final_answer
            additionalProperties: false
          strict: true

To use the configured math tutor, send a simple request to the chat completions endpoint:

curl -s -XPOST http://localhost:8090/v1/chat/completions -H "Content-Type: application/json" -d \
    '{
      "model": "math-tutor",
      "messages": [{
        "role": "user",
        "content" :"how can I solve 8x + 7 = -23"
      }]
    }' \
  | jq '.choices[0].message.content | fromjson'

Example response:

{
  "final_answer": "x = -3.75",
  "steps": [
    {
      "explanation": "We start with the given equation that we need to solve.",
      "output": "8x + 7 = -23"
    },
    {
      "explanation": "Our goal is to solve for x. We can start by isolating the term with x on one side of the equation. To do this, we need to eliminate the constant term (7) on the left side. We subtract 7 from both sides of the equation in order to keep it balanced.",
      "output": "8x + 7 - 7 = -23 - 7"
    },
    {
      "explanation": "Subtracting 7 from both sides simplifies the equation. On the left side, the +7 and -7 cancel out, leaving just the term with the variable.",
      "output": "8x = -30"
    },
    {
      "explanation": "Now, we have 8 times x equals -30. To solve for x, we divide both sides of the equation by the coefficient of x, which is 8.",
      "output": "8x / 8 = -30 / 8"
    },
    {
      "explanation": "Dividing both sides results in x on the left side and simplifies the fraction on the right side. The fraction -30/8 can be simplified further by dividing both the numerator and the denominator by their greatest common divisor, which is 2.",
      "output": "x = -3.75"
    },
    {
      "explanation": "The solution has been simplified completely, giving us the value of x.",
      "output": "x = -3.75"
    }
  ]
}

Visit OpenAI Structured Outputs for more information on how to use structured output formats.