Text classification with kluster.ai API¶

This tutorial runs through a Colab Notebook where you'll learn how to use the kluster.ai Batch API to classify a data set based on a predefined set of categories. In our example, we use an extract from the IMDB top 1000 movies dataset and categorize them into "Action," "Adventure," "Comedy," "Crime," "Documentary," "Drama," "Fantasy," "Horror," "Romance," or "Sci-Fi." We use a movie dataset, but you can adapt this example by using your data and categories relevant to your use case. With this approach, you can effortlessly process datasets of any scale, from small collections to extensive datasets, and obtain categorized results powered by a state-of-the-art language model.

Prerequisites¶

Before getting started, ensure you have the following:

• A kluster.ai account - sign up on the kluster.ai platform if you don't have one
• A kluster.ai API key - after signing in, go to the API Keys section and create a new key. For detailed instructions, check out the Get an API key guide

Setup¶

In this notebook, we'll use Python's getpass module to safely input the key. Provide your unique kluster.ai API key (ensure there are no spaces).

In [1]:

from getpass import getpass

api_key = getpass("Enter your kluster.ai API key: ")

from getpass import getpass api_key = getpass("Enter your kluster.ai API key: ")

Enter your kluster.ai API key:  ········

In [ ]:

%pip install -q openai

%pip install -q openai

In [7]:

from openai import OpenAI

import pandas as pd
import time
import json
from IPython.display import clear_output, display

from openai import OpenAI import pandas as pd import time import json from IPython.display import clear_output, display

In [19]:

# Set up the client
client = OpenAI(
    base_url="https://api.kluster.ai/v1",
    api_key=api_key,
)

# Set up the client client = OpenAI( base_url="https://api.kluster.ai/v1", api_key=api_key, )

Get the data¶

This notebook includes a preloaded sample dataset derived from the Top 1000 IMDb Movies dataset. It contains movie descriptions ready for classification. No additional setup is needed—simply proceed to the next steps to begin working with this data.

In [20]:

df = pd.DataFrame({
    "text": [
        "Breakfast at Tiffany's: A young New York socialite becomes interested in a young man who has moved into her apartment building, but her past threatens to get in the way.",
        "Giant: Sprawling epic covering the life of a Texas cattle rancher and his family and associates.",
        "From Here to Eternity: In Hawaii in 1941, a private is cruelly punished for not boxing on his unit's team, while his captain's wife and second-in-command are falling in love.",
        "Lifeboat: Several survivors of a torpedoed merchant ship in World War II find themselves in the same lifeboat with one of the crew members of the U-boat that sank their ship.",
        "The 39 Steps: A man in London tries to help a counter-espionage Agent. But when the Agent is killed, and the man stands accused, he must go on the run to save himself and stop a spy ring which is trying to steal top secret information."
    ]
})

df = pd.DataFrame({ "text": [ "Breakfast at Tiffany's: A young New York socialite becomes interested in a young man who has moved into her apartment building, but her past threatens to get in the way.", "Giant: Sprawling epic covering the life of a Texas cattle rancher and his family and associates.", "From Here to Eternity: In Hawaii in 1941, a private is cruelly punished for not boxing on his unit's team, while his captain's wife and second-in-command are falling in love.", "Lifeboat: Several survivors of a torpedoed merchant ship in World War II find themselves in the same lifeboat with one of the crew members of the U-boat that sank their ship.", "The 39 Steps: A man in London tries to help a counter-espionage Agent. But when the Agent is killed, and the man stands accused, he must go on the run to save himself and stop a spy ring which is trying to steal top secret information." ] })

Perform Batch inference¶

To execute the inference job, we'll follow three straightforward steps:

1. Create the inference file - we'll generate a file with the desired requests to be processed by the model
2. Upload the inference file - once the file is ready, we'll upload it to the kluster.ai platform using the API, where it will be queued for processing
3. Start the job - after the file is uploaded, we'll initiate the job to process the uploaded data

Everything is set up for you – just run the cells below to watch it all come together.

Create the Batch file¶

This example selects the klusterai/Meta-Llama-3.3-70B-Instruct-Turbo model. If you'd like to use a different model, feel free to change the model's name in the following cell. Please refer to our documentation for a list of the models we support.

In [21]:

def create_inference_file(df):
    inference_list = []
    for index, row in df.iterrows():
        content = row['text']
        
        request = {
            "custom_id": f"movie_classification-{index}",
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": {
                "model": "klusterai/Meta-Llama-3.3-70B-Instruct-Turbo",
                "temperature": 0.5,
                "messages": [
                    {"role": "system", "content": 'Classify the main genre of the given movie description based on the following genres(Respond with only the genre): “Action”, “Adventure”, “Comedy”, “Crime”, “Documentary”, “Drama”, “Fantasy”, “Horror”, “Romance”, “Sci-Fi”.'},
                    {"role": "user", "content": content}
                ],
            }
        }
        inference_list.append(request)
    return inference_list

def save_inference_file(inference_list):
    filename = f"movie_classification_inference_request.jsonl"
    with open(filename, 'w') as file:
        for request in inference_list:
            file.write(json.dumps(request) + '\n')
    return filename

def create_inference_file(df): inference_list = [] for index, row in df.iterrows(): content = row['text'] request = { "custom_id": f"movie_classification-{index}", "method": "POST", "url": "/v1/chat/completions", "body": { "model": "klusterai/Meta-Llama-3.3-70B-Instruct-Turbo", "temperature": 0.5, "messages": [ {"role": "system", "content": 'Classify the main genre of the given movie description based on the following genres(Respond with only the genre): “Action”, “Adventure”, “Comedy”, “Crime”, “Documentary”, “Drama”, “Fantasy”, “Horror”, “Romance”, “Sci-Fi”.'}, {"role": "user", "content": content} ], } } inference_list.append(request) return inference_list def save_inference_file(inference_list): filename = f"movie_classification_inference_request.jsonl" with open(filename, 'w') as file: for request in inference_list: file.write(json.dumps(request) + '\n') return filename

In [12]:

inference_list = create_inference_file(df)
filename = save_inference_file(inference_list)

inference_list = create_inference_file(df) filename = save_inference_file(inference_list)

Let’s preview what that request file looks like:

In [22]:

!head -n 1 movie_classification_inference_request.jsonl

!head -n 1 movie_classification_inference_request.jsonl

{"custom_id": "movie_classification-0", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "klusterai/Meta-Llama-3.3-70B-Instruct-Turbo", "temperature": 0.5, "messages": [{"role": "system", "content": "Classify the main genre of the given movie description based on the following genres(Respond with only the genre): \u201cAction\u201d, \u201cAdventure\u201d, \u201cComedy\u201d, \u201cCrime\u201d, \u201cDocumentary\u201d, \u201cDrama\u201d, \u201cFantasy\u201d, \u201cHorror\u201d, \u201cRomance\u201d, \u201cSci-Fi\u201d."}, {"role": "user", "content": "Breakfast at Tiffany's: A young New York socialite becomes interested in a young man who has moved into her apartment building, but her past threatens to get in the way."}]}}

Upload inference file to kluster.ai¶

Now that we’ve prepared our input file, it’s time to upload it to the kluster.ai platform.

In [23]:

inference_input_file = client.files.create(
    file=open(filename, "rb"),
    purpose="batch"
)

inference_input_file = client.files.create( file=open(filename, "rb"), purpose="batch" )

Start the job¶

Once the file has been successfully uploaded, we're ready to start the inference job.

In [24]:

inference_job = client.batches.create(
    input_file_id=inference_input_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h"
)

inference_job = client.batches.create( input_file_id=inference_input_file.id, endpoint="/v1/chat/completions", completion_window="24h" )

Check job progress¶

Your request is being processed now that the job has been created. In the following section, we'll monitor the job's status to see how it's progressing. Let's take a look and keep track of its status.

In [25]:

def parse_json_objects(data_string):
    if isinstance(data_string, bytes):
        data_string = data_string.decode('utf-8')

    json_strings = data_string.strip().split('\n')
    json_objects = []

    for json_str in json_strings:
        try:
            json_obj = json.loads(json_str)
            json_objects.append(json_obj)
        except json.JSONDecodeError as e:
            print(f"Error parsing JSON: {e}")

    return json_objects

all_completed = False
while not all_completed:
    all_completed = True
    output_lines = []

    updated_job = client.batches.retrieve(inference_job.id)

    if updated_job.status != "completed":
        all_completed = False
        completed = updated_job.request_counts.completed
        total = updated_job.request_counts.total
        output_lines.append(f"Job status: {updated_job.status} - Progress: {completed}/{total}")
    else:
        output_lines.append(f"Job completed!")

    # Clear the output and display updated status
    clear_output(wait=True)
    for line in output_lines:
        display(line)

    if not all_completed:
        time.sleep(10)

def parse_json_objects(data_string): if isinstance(data_string, bytes): data_string = data_string.decode('utf-8') json_strings = data_string.strip().split('\n') json_objects = [] for json_str in json_strings: try: json_obj = json.loads(json_str) json_objects.append(json_obj) except json.JSONDecodeError as e: print(f"Error parsing JSON: {e}") return json_objects all_completed = False while not all_completed: all_completed = True output_lines = [] updated_job = client.batches.retrieve(inference_job.id) if updated_job.status != "completed": all_completed = False completed = updated_job.request_counts.completed total = updated_job.request_counts.total output_lines.append(f"Job status: {updated_job.status} - Progress: {completed}/{total}") else: output_lines.append(f"Job completed!") # Clear the output and display updated status clear_output(wait=True) for line in output_lines: display(line) if not all_completed: time.sleep(10)

'Job completed!'

Get the results¶

With the job completed, we'll now retrieve the results and review the responses generated for each request.

In [26]:

job = client.batches.retrieve(inference_job.id)
result_file_id = job.output_file_id
result = client.files.content(result_file_id).content
parse_json_objects(result)

job = client.batches.retrieve(inference_job.id) result_file_id = job.output_file_id result = client.files.content(result_file_id).content parse_json_objects(result)

Out[26]:

[{'id': '67781586bfa945a98224262c',
  'custom_id': 'movie_classification-0',
  'response': {'status_code': 200,
   'request_id': 'b21b91e3-f81e-457f-b2da-6815d63b05c5',
   'body': {'id': 'chat-f37f29004cc246438b9efad914a27e58',
    'object': 'chat.completion',
    'created': 1735923078,
    'model': 'klusterai/Meta-Llama-3.3-70B-Instruct-Turbo',
    'choices': [{'index': 0,
      'message': {'role': 'assistant', 'content': 'Romance', 'tool_calls': []},
      'logprobs': None,
      'finish_reason': 'stop',
      'stop_reason': None}],
    'usage': {'prompt_tokens': 130,
     'total_tokens': 133,
     'completion_tokens': 3},
    'prompt_logprobs': None}}},
 {'id': '67781586bfa945a98224262e',
  'custom_id': 'movie_classification-1',
  'response': {'status_code': 200,
   'request_id': 'bcababdb-3917-49b5-a87b-176b8d928299',
   'body': {'id': 'chat-7c5214d5d13a4dba99a6aa97669ea9ac',
    'object': 'chat.completion',
    'created': 1735923078,
    'model': 'klusterai/Meta-Llama-3.3-70B-Instruct-Turbo',
    'choices': [{'index': 0,
      'message': {'role': 'assistant', 'content': 'Drama', 'tool_calls': []},
      'logprobs': None,
      'finish_reason': 'stop',
      'stop_reason': None}],
    'usage': {'prompt_tokens': 116,
     'total_tokens': 119,
     'completion_tokens': 3},
    'prompt_logprobs': None}}},
 {'id': '67781586bfa945a982242630',
  'custom_id': 'movie_classification-2',
  'response': {'status_code': 200,
   'request_id': '8f4ad4c7-492e-4e04-80e8-41ab1e12cbb6',
   'body': {'id': 'chat-2fda15a0b53d4350b0a932747941bf9e',
    'object': 'chat.completion',
    'created': 1735923078,
    'model': 'klusterai/Meta-Llama-3.3-70B-Instruct-Turbo',
    'choices': [{'index': 0,
      'message': {'role': 'assistant', 'content': 'Drama', 'tool_calls': []},
      'logprobs': None,
      'finish_reason': 'stop',
      'stop_reason': None}],
    'usage': {'prompt_tokens': 137,
     'total_tokens': 140,
     'completion_tokens': 3},
    'prompt_logprobs': None}}},
 {'id': '67781586bfa945a982242632',
  'custom_id': 'movie_classification-3',
  'response': {'status_code': 200,
   'request_id': 'd9c01e8f-7c96-4d8d-bdb8-3ffe9df171d3',
   'body': {'id': 'chat-f776672680574b1cb9dc5305039c908f',
    'object': 'chat.completion',
    'created': 1735923078,
    'model': 'klusterai/Meta-Llama-3.3-70B-Instruct-Turbo',
    'choices': [{'index': 0,
      'message': {'role': 'assistant', 'content': 'Drama', 'tool_calls': []},
      'logprobs': None,
      'finish_reason': 'stop',
      'stop_reason': None}],
    'usage': {'prompt_tokens': 132,
     'total_tokens': 135,
     'completion_tokens': 3},
    'prompt_logprobs': None}}},
 {'id': '67781586bfa945a982242634',
  'custom_id': 'movie_classification-4',
  'response': {'status_code': 200,
   'request_id': 'cadcfc39-0f32-470e-8478-50d1c53575bd',
   'body': {'id': 'chat-5a276aed0b7d48b4b7051492fdb3eb7d',
    'object': 'chat.completion',
    'created': 1735923078,
    'model': 'klusterai/Meta-Llama-3.3-70B-Instruct-Turbo',
    'choices': [{'index': 0,
      'message': {'role': 'assistant',
       'content': 'Action/Adventure',
       'tool_calls': []},
      'logprobs': None,
      'finish_reason': 'stop',
      'stop_reason': None}],
    'usage': {'prompt_tokens': 149,
     'total_tokens': 153,
     'completion_tokens': 4},
    'prompt_logprobs': None}}}]

Conclusion¶

You've successfully completed the classification request using the kluster.ai Batch API! This process showcases how you can efficiently handle and classify large amounts of data with ease. The Batch API empowers you to scale your workflows seamlessly, making it an invaluable tool for processing extensive datasets.