Chat Features
This docs site is currently under construction although this github README below should suffice for now.
Julep provides a robust chat system with various features for dynamic interaction with agents. Here's an overview of the key components and functionalities:
Chat Input
When sending a request to the chat endpoint, you can include:
Messages: An array of input messages representing the conversation so far.
Tools: (Advanced) Additional tools provided for this specific interaction.
Tool Choice: Specifies which tool the agent should use.
Memory Access Options: Controls how the session accesses history and memories.
Chat Settings: Various settings to control the behavior of the chat.
Chat Settings
Chat settings allow fine-grained control over the generation process:
model
: Identifier of the model to be used.stream
: Indicates if the server should stream the response as it's generated.stop
: Up to 4 sequences where the API will stop generating further tokens.seed
: For deterministic sampling.max_tokens
: The maximum number of tokens to generate.logit_bias
: Modify the likelihood of specified tokens appearing in the completion.response_format
: Control the format of the response (e.g., JSON object).agent
: Agent ID to use (for multi-agent sessions).
Additional settings include temperature
, top_p
, frequency_penalty
, and presence_penalty
.
Chat Response
The chat response can be either streamed or returned as a complete message:
Streamed Response:
Content-Type:
text/event-stream
Body: A stream of
ChatOutputChunk
objects.
Complete Response:
Content-Type:
application/json
Body: A
MessageChatResponse
object containing the full generated message(s).
Both response types include:
usage
: Token usage statistics.jobs
: Background job IDs spawned from this interaction.docs
: Documents referenced for this request (for citation purposes).
Finish Reasons
The API provides information about why the model stopped generating tokens:
stop
: Natural stop point or provided stop sequence reached.length
: Maximum number of tokens specified in the request was reached.content_filter
: Content was omitted due to a flag from content filters.tool_calls
: The model called a tool.
Advanced Features
Tool Integration: The chat API allows for the use of tools, enabling the agent to perform actions or retrieve information during the conversation.
Multi-agent Sessions: You can specify different agents within the same session using the
agent
parameter in the chat settings.Response Formatting: Control the output format, including options for JSON responses with specific schemas.
Memory and Recall: Configure how the session accesses and stores conversation history and memories.
Document References: The API returns information about documents referenced during the interaction, useful for providing citations or sources.
These features provide developers with a powerful and flexible system for creating sophisticated, context-aware chat interactions that integrate seamlessly with other Julep components.
Last updated