input_schema: type: object properties: upload_file: type: string description: The url of the file to upload public_id: type: string description: The public id of the file to upload transformation_prompt: type: string description: The prompt for the transformations to apply to the file subtitle_vtt: type: string description: The vtt file content to add subtitles to the video
This schema specifies that our task expects:
A video file URL
A public ID for the video
A transformation prompt describing desired changes
The steps[0].input variable refers to the initial input object passed to the task. It’s used to access the input parameters defined in the input schema.
- prompt: - role: user content: - type: image_url image_url: url: trimmed_video_url - type: text text: |- Which speakers are speaking in the video? And where does each of them sit?
This step:
Analyzes the video content
Identifies speakers and their positions
Uses VTT subtitles for additional context
4
Generate Speaker Transformations
- evaluate: speakers_transformations: |- $ [ transform for speaker in _.speakers_json for transform in [ { "overlay": {"font_family": "Arial", "font_size": 32, "text": speaker.speaker}, "color": "white" }, { "duration": 5, "flags": "layer_apply", "gravity": "south_east" if speaker.position == "right" else "south_west", "start_offset": speaker.timestamps[0].start, "y": 80, "x": 80 } ] ]
Uses the Cloudinary upload tool to apply the generated transformations
Processes the video with speaker labels and positioning
Returns a URL to the transformed video with all overlays applied
YAML
# yaml-language-server: $schema=https://raw.githubusercontent.com/julep-ai/julep/refs/heads/dev/schemas/create_task_request.jsonname: Julep Video Processing Taskdescription: A Julep agent that can process and analyze videos using Cloudinary########################################################################### INPUT SCHEMA ###############################################################################input_schema: type: object properties: video_url: type: string description: The url of the file to upload public_id: type: string description: The public id of the file to upload transformation_prompt: type: string description: The prompt for the transformations to apply to the file########################################################################### TOOLS ######################################################################################tools:- name: cloudinary_upload type: integration integration: provider: cloudinary method: media_upload setup: cloudinary_api_key: "YOUR_CLOUDINARY_API_KEY" cloudinary_api_secret: "YOUR_CLOUDINARY_API_SECRET" cloudinary_cloud_name: "YOUR_CLOUDINARY_CLOUD_NAME"########################################################################### MAIN WORKFLOW ##############################################################################main:# Step #0 - Upload the video to cloudinary- tool: cloudinary_upload arguments: file: $ steps[0].input.video_url public_id: $ steps[0].input.public_id upload_params: resource_type: video# Step #1 - Analyze the video content with a prompt- prompt: - role: user content: - type: text text: |- You are a Cloudinary expert. You are given a medial url. it might be an image or a video. You need to come up with a json of transformations to apply to the given media. Overall the json could have multiple transformation json objects. Each transformation json object can have the multiple key value pairs. Each key value pair should have the key as the transformation name like "aspect_ratio", "crop", "width" etc and the value as the transformation parameter value. Given below is an example of a transformation json list. Don't provide explanations and/or comments in the json. ``json [ {{ "aspect_ratio": "1.0", "width": 250, }}, {{ "fetch_format": "auto" }}, {{ "overlay": {{ "url": "<image_url>" }} }}, {{ "flags": "layer_apply" }} ] `` - type: image_url image_url: url: $ _.url - type: text text: |- $ f'''Hey, check the video above, I need to apply the following transformations using cloudinary. {steps[0].input.transformation_prompt}''' unwrap: true settings: model: gemini/gemini-1.5-pro# Step #2 - Extract the json from the model's response- evaluate: model_transformation: >- $ load_json( _[_.find("```json")+7:][:_[_.find("```json")+7:].find("```")])# Step #3 - Upload the video to cloudinary- tool: cloudinary_upload arguments: file: $ steps[0].input.video_url public_id: $ steps[0].input.public_id upload_params: transformation: $ _.model_transformation resource_type: video# Step #4 - Evaluate the transformed video url- evaluate: transformed_video_url: $ _.url
import timeimport yamlfrom julep import Client# Initialize the clientclient = Client(api_key=JULEP_API_KEY)transformation_prompt = """1- I want to add an overlay an the following image to the video, and apply a layer apply flag also. Here's the image url:https://res.cloudinary.com/demo/image/upload/logos/cloudinary_icon_white.png2- I also want you to to blur the video, and add a fade in and fade out effect to the video with a duration of 3 seconds each."""# Create the agentagent = client.agents.create( name="Julep Video Processing Agent", about="A Julep agent that can process and analyze videos using Cloudinary and FFmpeg.",)# Load the task definitionwith open('video_processing_task.yaml', 'r') as file: task_definition = yaml.safe_load(file)# Create the tasktask = client.tasks.create( agent_id=agent.id, **task_definition)# Create the executionexecution = client.executions.create( task_id=task.id, input={ "video_url": "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4", "public_id": "video_test", "transformation_prompt": transformation_prompt, })# Wait for the execution to completewhile (result := client.executions.get(execution.id)).status not in ['succeeded', 'failed']: print(result.status) time.sleep(1)# Print the resultif result.status == "succeeded": print(result.output)else: print(f"Error: {result.error}")