Chat Features in Julep
Learn about the robust chat system and its various features for dynamic interaction with agents
Overview
Julep provides a robust chat system with various features for dynamic interaction with agents. Here’s an overview of the key components and functionalities.
Features
Tool Integration
The chat API allows for the use of tools, enabling the agent to perform actions or retrieve information during the conversation.
Multi-agent Sessions
You can specify different agents within the same session using the agent
parameter in the chat settings.
Response Formatting
Control the output format, including options for JSON responses with specific schemas.
Memory and Recall
Configure how the session accesses and stores conversation history and memories.
Document References
The API returns information about documents referenced during the interaction, useful for providing citations or sources.
Prerequisites for Using Chat API
- Session Creation: Before using the chat API, you must create a session first. Learn more about the session object on the Session page.
- Document (RAG) Integration: To use Document (RAG) capabilities with the chat API, create a session with the
recall_options
parameter configured with appropriate search parameters. For details on configuringrecall_options
, see the Session: Recall Options documentation.
Input Structure
- Messages: An array of input messages representing the conversation so far.
- Tools: (Advanced) Additional tools provided for this specific interaction.
- Tool Choice: Specifies which tool the agent should use.
- Memory Access: Controls how the session accesses history and memories.(
recall
parameter) - Additional Parameters: Various parameters to control the behavior of the chat. You can find more details in the Additional Parameters section.
Here’s an example of how a typical message object might be structured in a chat interaction:
Message Object Structure
Message Object Structure
This object represents a message in the chat system, detailing the structure and types of data it can hold.
Additional Parameters
Parameter | Type | Description | Default |
---|---|---|---|
stream | bool | Indicates if the server should stream the response as it’s generated. | False |
stop | list[str] | Up to 4 sequences where the API will stop generating further tokens. | [] |
seed | int | If specified, the system will make a best effort to sample deterministically for that particular seed value. | None |
max_tokens | int | The maximum number of tokens to generate in the chat completion. | None |
logit_bias | dict[str, float] | Modify the likelihood of specified tokens appearing in the completion. | None |
response_format | str | Response format (set to json_object to restrict output to JSON). | None |
agent | UUID | Agent ID of the agent to use for this interaction. (Only applicable for multi-agent sessions) | None |
repetition_penalty | float | Number between 0 and 2.0. 1.0 is neutral and values larger than that penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood to repeat the same line verbatim. | None |
length_penalty | float | Number between 0 and 2.0. 1.0 is neutral and values larger than that penalize number of tokens generated. | None |
min_p | float | Minimum probability compared to leading token to be considered. | None |
frequency_penalty | float | Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood to repeat the same line verbatim. | None |
presence_penalty | float | Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood to repeat the same line verbatim. | None |
temperature | float | What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. | None |
top_p | float | An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered. | 1.0 |
recall | bool | Whether to use the document (RAG) search or not | True |
save | bool | Whether this interaction should be stored in the session history or not | True |
remember | bool | DISABLED: Whether this interaction should form new memories or not (will be enabled in a future release) | False |
model | str | The model to use for the chat completion. | None |
metadata | dict[str, Any] | Custom metadata that can be passed to the system template for dynamic behavior. See System Templates for details. | None |
auto_run_tools | bool | Whether to automatically execute tools and send the results back to the model (requires tools on the agent) | False |
recall_tools | bool | Whether to include tool requests and responses when recalling messages from history | True |
Usage
Here’s an example of how to use the chat API in Julep using the SDKs:
To use the Chat endpint, you always have to create a session first.
To learn more about the Session object, check out the Session page.
Check out the API reference or SDK reference (Python or JavaScript) for more details on different operations you can perform on sessions.
Response
- Content-Type:
application/json
- Body: A
MessageChatResponse
object containing the full generated message(s)
- Content-Type:
application/json
- Body: A
MessageChatResponse
object containing the full generated message(s)
- Content-Type:
text/event-stream
- Body: A stream of
ChatOutputChunk
objects
This feature is not implemented yet.
Both types of responses include the following fields:
id
: The unique identifier for the chat responsechoices
: An object of generated message completions containing:role
: The role of the message (e.g. “assistant”, “user”, etc.)id
: Unique identifier for the messagecontent
: list of actual message contentcreated_at
: Timestamp when the message was createdname
: Optional name associated with the messagetool_call_id
: Optional ID referencing a tool calltool_calls
: Optional list of tool calls made during message generationcreated_at
: When this resource was created as UTC date-timedocs
: List of document references used for this request, intended for citation purposesjobs
: List of UUIDs for background jobs that may have been initiated as a result of this interactionusage
: Statistics on token usage for the completion request
Automatic Tool Calling
Julep supports automatic tool execution during chat interactions. This feature allows tools to be executed seamlessly without manual intervention, making conversations more fluid and responsive.
How It Works
-
When
auto_run_tools=true
:- The model identifies when a tool should be used based on the conversation
- The tool is automatically executed by Julep’s backend
- Results are fed back to the model to continue the conversation
- The entire process happens in a single API call
-
When
auto_run_tools=false
(default):- The model returns tool call requests in the response
- Your application must execute the tools manually
- Results need to be sent back in a follow-up message
Example with Automatic Tool Execution
Tool History Management
The recall_tools
parameter controls whether tool calls and their results are included when recalling conversation history:
recall_tools=true
(default): Tool interactions are preserved in the conversation historyrecall_tools=false
: Tool calls and results are excluded from recalled messages
This is useful when you want to maintain a cleaner conversation history without the technical details of tool executions.
Finish Reasons
stop
Natural stop point or provided stop sequence reached
length
Maximum number of tokens specified in the request was reached
content_filter
Content was omitted due to a flag from content filters
tool_calls
The model called a tool
Support
If you need help with further questions in Julep:
- Join our Discord community
- Check the GitHub repository
- Contact support at hey@julep.ai