Provider APIs

Reference for LLM providers in Agenite

Overview

Agenite supports multiple LLM providers through a consistent interface. Each provider implements the LLMProvider interface from the @agenite/llm package.

Common interface

interface LLMProvider {
  name: string;
  version?: string;

  /**
   * Simple text generation with full response
   */
  generate(
    input: string | BaseMessage[],
    options?: Partial<GenerateOptions>
  ): Promise<GenerateResponse>;

  /**
   * Simple streaming with partial returns
   */
  stream(
    input: string | BaseMessage[],
    options?: Partial<GenerateOptions>
  ): AsyncGenerator<PartialReturn, GenerateResponse, unknown>;

  /**
   * Low-level generation API with full control
   */
  iterate(
    input: string | BaseMessage[],
    options: IterateGenerateOptions
  ): AsyncGenerator<PartialReturn, GenerateResponse, unknown>;
}

Most providers extend the BaseLLMProvider class, which implements the iterate method for you, so you only need to implement generate and stream.

OpenAI provider

Installation

npm install @agenite/openai

Usage

import { OpenAI } from '@agenite/openai';

const provider = new OpenAI({
  apiKey: 'your-api-key',
  model: 'gpt-4-turbo-preview',
});

const agent = new Agent({
  name: 'openai-agent',
  provider,
  // ... other options
});

Configuration

interface OpenAIConfig extends BaseLLMConfig {
  apiKey: string;
  model?: OpenAIModel;
  responseFormat?: "text" | "json_object";
  tools?: ToolDefinition[];
  forceTool?: string;
  systemPrompt?: string;
}

type OpenAIModel =
  | "gpt-4-turbo-preview"
  | "gpt-4-0125-preview"
  | "gpt-4-1106-preview"
  | "gpt-4"
  | "gpt-4-32k"
  | "gpt-3.5-turbo"
  | "gpt-3.5-turbo-16k";

interface BaseLLMConfig {
  organization?: string;
  baseURL?: string;
  maxRetries?: number;
  timeout?: number;
}

Anthropic provider

Installation

npm install @agenite/anthropic

Usage

import { Anthropic } from '@agenite/anthropic';

const provider = new Anthropic({
  apiKey: 'your-api-key',
  model: 'claude-3-sonnet-20240229',
});

const agent = new Agent({
  name: 'anthropic-agent',
  provider,
  // ... other options
});

Configuration

interface AnthropicConfig extends BaseLLMConfig {
  apiKey: string;
  model?: string;
  temperature?: number;
  maxTokens?: number;
  stopSequences?: string[];
}

AWS Bedrock provider

Installation

npm install @agenite/bedrock

Usage

import { Bedrock } from '@agenite/bedrock';

const provider = new Bedrock({
  region: 'us-west-2',
  model: 'anthropic.claude-3-5-sonnet-20240620-v1:0',
});

const agent = new Agent({
  name: 'bedrock-agent',
  provider,
  // ... other options
});

Configuration

interface BedrockConfig extends BaseLLMConfig {
  region?: string;
  model: string;
  temperature?: number;
  maxTokens?: number;
  topP?: number;
  stopSequences?: string[];
}

Ollama provider

Installation

npm install @agenite/ollama

Usage

import { Ollama } from '@agenite/ollama';

const provider = new Ollama({
  model: 'llama3.2',
  baseUrl: 'http://localhost:11434',
});

const agent = new Agent({
  name: 'ollama-agent',
  provider,
  // ... other options
});

Configuration

interface OllamaConfig extends BaseLLMConfig {
  model: string;
  baseUrl?: string;
  temperature?: number;
  topP?: number;
  topK?: number;
  repeatPenalty?: number;
  seed?: number;
  stopSequences?: string[];
}

Creating custom providers

You can create custom providers by extending the BaseLLMProvider class:

import { BaseLLMProvider, GenerateOptions, GenerateResponse, PartialReturn } from '@agenite/llm';

class CustomProvider extends BaseLLMProvider {
  readonly name = 'CustomProvider';
  readonly version = '1.0.0';

  constructor(private config: MyCustomConfig) {
    super();
  }

  async generate(
    input: string | BaseMessage[],
    options?: Partial<GenerateOptions>
  ): Promise<GenerateResponse> {
    // 1. Convert input to standard format using convertStringToMessages
    const messages = convertStringToMessages(input);
    
    // 2. Transform to your API's format
    // 3. Call your API
    // 4. Transform the response to Agenite's format
    
    return {
      content: [{ type: 'text', text: 'Response from custom LLM' }],
      tokens: [{ model: 'my-model', inputTokens: 10, outputTokens: 20 }],
      duration: 500,
      stopReason: 'endTurn'
    };
  }

  async *stream(
    input: string | BaseMessage[],
    options?: Partial<GenerateOptions>
  ): AsyncGenerator<PartialReturn, GenerateResponse, unknown> {
    // 1. Convert input to standard format
    // 2. Set up streaming from your API
    
    // 3. Yield chunks as they arrive
    yield { type: 'text', text: 'Partial response...' };
    yield { type: 'text', text: 'More text...' };
    
    // 4. Return final response similar to generate
    return {
      content: [{ type: 'text', text: 'Complete response' }],
      tokens: [{ model: 'my-model', inputTokens: 10, outputTokens: 20 }],
      duration: 500,
      stopReason: 'endTurn'
    };
  }
  
  // The iterate method is already implemented by BaseLLMProvider
}

Response formats

All providers return responses in a standardized format:

interface TokenUsage {
  inputTokens: number;
  outputTokens: number;
  inputCost: number;
  outputCost: number;
  model: string;
}

interface GenerateResponse {
  content: Array<ContentBlock>;
  tokenUsage: TokenUsage;
  duration: number;
  stopReason?: StopReason;
}

type ContentBlock =
  | TextBlock          // { type: 'text', text: string }
  | ImageBlock         // { type: 'image', source: ... }
  | ToolUseBlock       // { type: 'toolUse', id: string, name: string, input: unknown }
  | ToolResultBlock    // { type: 'toolResult', toolUseId: string, ... }
  | DocumentBlock      // { type: 'document', source: ... }
  | ThinkingBlock      // { type: 'thinking', thinking: string }
  | RedactedThinkingBlock;

type StopReason = 'toolUse' | 'maxTokens' | 'stopSequence' | 'endTurn';

Best practices

1. Error handling: Implement robust error handling for API failures
2. Rate limiting: Respect provider rate limits
3. Token tracking: Always include accurate token usage metrics
4. Streaming: Implement efficient streaming to minimize latency
5. Content mapping: Properly map specific responses to Agenite’s standard format

Next steps

• Learn about the Agent API
• Explore the Tool API
• Read about middleware