---
library_name: transformers
license: mit
task_categories:
- text-generation
language:
- en
tags:
- agent
- Agentic Learning
- tool use
- BFCL
---

[![Funcdex-Collection](https://img.shields.io/badge/Hugging%20Face-Model-yellow?logo=huggingface)](https://huggingface.co/collections/prem-research/funcdex) [![Dataset](https://img.shields.io/badge/Hugging%20Face-Dataset-yellow?logo=huggingface)](https://huggingface.co/datasets/prem-research/Funcdex-MT-Function-Calling) [![GitHub](https://img.shields.io/badge/GitHub-Code-181717?logo=github)](https://github.com/prem-research/Funcdex-Synthesizer) [![PremAI](https://img.shields.io/badge/Project-PremAI-green)](https://www.premai.io/)

# Funcdex-0.6B-whatsapp_todoist

<div align="center">
  <img src="assets/funcdex_hero.png" alt="Funcdex Hero" width="70%">
</div>

Funcdex-0.6B is a research preview model by Prem Labs. It has been trained on a mix of [Funcdex-MT-Function-Calling](https://huggingface.co/datasets/prem-research/Funcdex-MT-Function-Calling), Instruct-Following, Single-turn function datasets. It is a LoRA finetune of Qwen3-0.6B (with thinking disabled). 

This model excels at Multi-turn Function Calling with tools from `whatsapp` and `todoist`. 

The code used to generate the dataset can be found [here](https://github.com/prem-research/Funcdex-Synthesizer).

# Evaluation


<div align="center">
  <img src="assets/line_plot.png" alt="Line Plot" width="80%">
</div>

## Results

### BFCL v3
- We filtered BFCLv3 examples relevant to the toolkits/bundles and report performance:
- The filtered set is only 83 examples. Further emphasizing the need for workflow/toolkit-specialized workflows.

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: center;">
      <th>LLM</th>
      <th>Acc %</th>
    </tr>
  </thead>
  <tbody>
    <tr style="text-align: center;">
      <td>GPT-5 Mini<br>(medium)</td>
      <td>0.71</td>
    </tr>
    <tr style="text-align: center;">
      <td>Qwen3-1.7B</td>
      <td>0.82</td>
    </tr>
    <tr style="text-align: center;">
      <td><strong><a href="https://huggingface.co/prem-research/Funcdex-1.7B">Funcdex-1.7B</a><strong></td>
      <td><strong>0.86</strong></td>
    </tr>
  </tbody>
</table>


### Funcdex-MT: Overall Performance

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: center;">
      <th>LLM</th>
      <th>Exact Match</th>
      <th>String Ratio</th>
      <th>Total Cost ($)</th>
    </tr>
  </thead>
  <tbody>
    <tr style="text-align: center;">
      <td>GPT-OSS-120B<br>(medium)</td>
      <td>0.35</td>
      <td>0.51</td>
      <td>9.32</td>
    </tr>
    <tr style="text-align: center;">
      <td>GPT-5 Mini<br>(medium)</td>
      <td>0.35</td>
      <td>0.58</td>
      <td>99.71</td>
    </tr>
    <tr style="text-align: center;">
      <td>GPT-5<br>(minimal)</td>
      <td>0.18</td>
      <td>0.59</td>
      <td>205.45</td>
    </tr>
    <tr style="text-align: center;">
      <td>Qwen3-0.6B</td>
      <td>0.27</td>
      <td>0.59</td>
      <td>2.83</td>
    </tr>
    <tr style="text-align: center;">
      <td>Qwen3-1.7B</td>
      <td>0.27</td>
      <td>0.69</td>
      <td>5.73</td>
    </tr>
    <tr style="text-align: center;">
      <td><strong><a href="https://huggingface.co/collections/prem-research/funcdex">Funcdex-0.6B</a></strong></td>
      <td><strong>0.39</strong></td>
      <td><strong>0.70</strong></td>
      <td><strong>0.19</strong></td>
    </tr>
    <tr style="text-align: center;">
      <td><strong><a href="https://huggingface.co/prem-research/Funcdex-1.7B">Funcdex-1.7B</a></strong></td>
      <td><strong>0.43</strong></td>
      <td><strong>0.81</strong></td>
      <td>5.64</td>
    </tr>
  </tbody>
</table>

### Funcdex-MT: Toolkit-Level Performance

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: center;">
      <th rowspan="2">Toolkit</th>
      <th colspan="2">GPT-OSS-120B<br>(medium)</th>
      <th colspan="2">GPT-5<br>(minimal)</th>
      <th colspan="2">GPT-5 Mini<br>(medium)</th>
      <th colspan="2">Qwen3-0.6B</th>
      <th colspan="3">Funcdex-0.6B</th>
      <th colspan="2">Qwen3-1.7B</th>
      <th colspan="3">Funcdex-1.7B</th>
    </tr>
    <tr style="text-align: center;">
      <th>EM</th>
      <th>SR</th>
      <th>EM</th>
      <th>SR</th>
      <th>EM</th>
      <th>SR</th>
      <th>EM</th>
      <th>SR</th>
      <th>EM</th>
      <th>SR</th>
      <th>LoRA Checkpoint</th>
      <th>EM</th>
      <th>SR</th>
      <th>EM</th>
      <th>SR</th>
      <th>LoRA Checkpoint</th>
    </tr>
  </thead>
  <tbody>
    <tr style="text-align: center;">
      <td><img src="assets/icons/asana.png" width="20" height="20" style="vertical-align: middle;"/> Asana</td>
      <td>0.38</td>
      <td>0.47</td>
      <td>0.12</td>
      <td>0.68</td>
      <td>0.49</td>
      <td>0.71</td>
      <td>0.33</td>
      <td>0.63</td>
      <td>0.46</td>
      <td>0.69</td>
      <td><a href="https://huggingface.co/prem-research/Funcdex-0.6B-asana">🤗</a></td>
      <td>0.30</td>
      <td>0.79</td>
      <td>0.52</td>
      <td>0.82</td>
      <td rowspan="10"><a href="https://huggingface.co/prem-research/Funcdex-1.7B">🤗</a></td>
    </tr>
    <tr style="text-align: center;">
      <td><img src="assets/icons/calendly.png" width="20" height="20" style="vertical-align: middle;"/> Calendly</td>
      <td>0.47</td>
      <td>0.56</td>
      <td>0.41</td>
      <td>0.63</td>
      <td>0.41</td>
      <td>0.56</td>
      <td>0.44</td>
      <td>0.66</td>
      <td>0.54</td>
      <td>0.78</td>
      <td><a href="https://huggingface.co/prem-research/Funcdex-0.6B-calendly">🤗</a></td>
      <td>0.47</td>
      <td>0.74</td>
      <td>0.54</td>
      <td>0.86</td>
    </tr>
    <tr style="text-align: center;">
      <td><img src="assets/icons/gmail.png" width="20" height="20" style="vertical-align: middle;"/> Gmail</td>
      <td>0.48</td>
      <td>0.70</td>
      <td>0.24</td>
      <td>0.69</td>
      <td>0.50</td>
      <td>0.73</td>
      <td>0.27</td>
      <td>0.61</td>
      <td>0.47</td>
      <td>0.72</td>
      <td><a href="https://huggingface.co/prem-research/Funcdex-0.6B-gmail">🤗</a></td>
      <td>0.31</td>
      <td>0.73</td>
      <td>0.53</td>
      <td>0.83</td>
    </tr>
    <tr style="text-align: center;">
      <td><img src="assets/icons/google-calendar.png" width="20" height="20" style="vertical-align: middle;"/> Calendar</td>
      <td>0.27</td>
      <td>0.52</td>
      <td>0.20</td>
      <td>0.50</td>
      <td>0.21</td>
      <td>0.51</td>
      <td>0.21</td>
      <td>0.53</td>
      <td>0.39</td>
      <td>0.74</td>
      <td><a href="https://huggingface.co/prem-research/Funcdex-0.6B-googlecalendar">🤗</a></td>
      <td>0.23</td>
      <td>0.64</td>
      <td>0.47</td>
      <td>0.83</td>
    </tr>
    <tr style="text-align: center;">
      <td><img src="assets/icons/docs.png" width="20" height="20" style="vertical-align: middle;"/> Docs</td>
      <td>0.19</td>
      <td>0.38</td>
      <td>0.07</td>
      <td>0.49</td>
      <td>0.18</td>
      <td>0.46</td>
      <td>0.07</td>
      <td>0.58</td>
      <td>0.13</td>
      <td>0.64</td>
      <td><a href="https://huggingface.co/prem-research/Funcdex-0.6B-googledocs">🤗</a></td>
      <td>0.11</td>
      <td>0.62</td>
      <td>0.18</td>
      <td>0.79</td>
    </tr>
    <tr style="text-align: center;">
      <td><img src="assets/icons/google-drive.png" width="20" height="20" style="vertical-align: middle;"/> Drive</td>
      <td>0.34</td>
      <td>0.52</td>
      <td>0.19</td>
      <td>0.61</td>
      <td>0.38</td>
      <td>0.58</td>
      <td>0.26</td>
      <td>0.65</td>
      <td>0.40</td>
      <td>0.75</td>
      <td><a href="https://huggingface.co/prem-research/Funcdex-0.6B-googledrive">🤗</a></td>
      <td>0.26</td>
      <td>0.73</td>
      <td>0.48</td>
      <td>0.82</td>
    </tr>
    <tr style="text-align: center;">
      <td><img src="assets/icons/jira.png" width="20" height="20" style="vertical-align: middle;"/> Jira</td>
      <td>0.47</td>
      <td>0.53</td>
      <td>0.17</td>
      <td>0.65</td>
      <td>0.47</td>
      <td>0.66</td>
      <td>0.51</td>
      <td>0.69</td>
      <td>0.58</td>
      <td>0.76</td>
      <td><a href="https://huggingface.co/prem-research/Funcdex-0.6B-jira">🤗</a></td>
      <td>0.47</td>
      <td>0.76</td>
      <td>0.59</td>
      <td>0.83</td>
    </tr>
    <tr style="text-align: center;">
      <td><img src="assets/icons/stripe.png" width="20" height="20" style="vertical-align: middle;"/> Stripe</td>
      <td>0.15</td>
      <td>0.37</td>
      <td>0.10</td>
      <td>0.46</td>
      <td>0.12</td>
      <td>0.39</td>
      <td>0.08</td>
      <td>0.50</td>
      <td>0.17</td>
      <td>0.71</td>
      <td><a href="https://huggingface.co/prem-research/Funcdex-0.6B-stripe">🤗</a></td>
      <td>0.09</td>
      <td>0.56</td>
      <td>0.16</td>
      <td>0.80</td>
    </tr>
    <tr style="text-align: center;">
      <td><img src="assets/icons/to-do-list.png" width="20" height="20" style="vertical-align: middle;"/> Todoist</td>
      <td>0.65</td>
      <td>0.74</td>
      <td>0.19</td>
      <td>0.72</td>
      <td>0.64</td>
      <td>0.79</td>
      <td>0.57</td>
      <td>0.87</td>
      <td>0.65</td>
      <td>0.88</td>
      <td><a href="https://huggingface.co/prem-research/Funcdex-0.6B-todoist">🤗</a></td>
      <td>0.55</td>
      <td>0.91</td>
      <td>0.72</td>
      <td>0.94</td>
    </tr>
    <tr style="text-align: center;">
      <td><img src="assets/icons/whatsapp.png" width="20" height="20" style="vertical-align: middle;"/> Whatsapp</td>
      <td>0.23</td>
      <td>0.39</td>
      <td>0.13</td>
      <td>0.47</td>
      <td>0.24</td>
      <td>0.43</td>
      <td>0.20</td>
      <td>0.43</td>
      <td>0.28</td>
      <td>0.64</td>
      <td><a href="https://huggingface.co/prem-research/Funcdex-0.6B-whatsapp">🤗</a></td>
      <td>0.26</td>
      <td>0.55</td>
      <td>0.31</td>
      <td>0.71</td>
    </tr>
  </tbody>
</table>

- Funcdex-0.6B are specialized models. Reported number is the average performance of each specific model in their respective subset.

### Funcdex-MT: Bundle/Multi-toolkit Performance:

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: center;">
      <th rowspan="2">Bundle</th>
      <th colspan="2">GPT-OSS-120B<br>(medium)</th>
      <th colspan="2">GPT-5<br>(minimal)</th>
      <th colspan="2">GPT-5 Mini<br>(medium)</th>
      <th colspan="2">Qwen3-0.6B</th>
      <th colspan="3">Funcdex-0.6B</th>
      <th colspan="2">Qwen3-1.7B</th>
      <th colspan="3">Funcdex-1.7B</th>
    </tr>
    <tr style="text-align: center;">
      <th>EM</th>
      <th>SR</th>
      <th>EM</th>
      <th>SR</th>
      <th>EM</th>
      <th>SR</th>
      <th>EM</th>
      <th>SR</th>
      <th>EM</th>
      <th>SR</th>
      <th>LoRA Checkpoint</th>
      <th>EM</th>
      <th>SR</th>
      <th>EM</th>
      <th>SR</th>
      <th>LoRA Checkpoint</th>
    </tr>
  </thead>
  <tbody>
    <tr style="text-align: center;">
      <td><img src="assets/icons/gmail.png" width="20" height="20" style="vertical-align: middle;"/>Gmail<img src="assets/icons/google-calendar.png" width="20" height="20" style="vertical-align: middle;"/>Calendar</td>
      <td>0.28</td>
      <td>0.53</td>
      <td>0.15</td>
      <td>0.54</td>
      <td>0.22</td>
      <td>0.56</td>
      <td>0.19</td>
      <td>0.51</td>
      <td>0.26</td>
      <td>0.54</td>
      <td><a href="https://huggingface.co/prem-research/Funcdex-0.6B-gmail_googlecalendar">🤗</a></td>
      <td>0.17</td>
      <td>0.61</td>
      <td>0.32</td>
      <td>0.71</td>
      <td rowspan="5"><a href="https://huggingface.co/prem-research/Funcdex-1.7B">🤗</a></td>
    </tr>
    <tr style="text-align: center;">
      <td><img src="assets/icons/google-drive.png" width="20" height="20" style="vertical-align: middle;"/>Drive <img src="assets/icons/calendly.png" width="20" height="20" style="vertical-align: middle;"/> Calendly <img src="assets/icons/google-calendar.png" width="20" height="20" style="vertical-align: middle;"/> Calendar</td>
      <td>0.32</td>
      <td>0.45</td>
      <td>0.17</td>
      <td>0.52</td>
      <td>0.35</td>
      <td>0.47</td>
      <td>0.19</td>
      <td>0.49</td>
      <td>0.35</td>
      <td>0.60</td>
      <td><a href="https://huggingface.co/prem-research/Funcdex-0.6B-googledrive_calendly_googlecalendar">🤗</a></td>
      <td>0.15</td>
      <td>0.66</td>
      <td>0.40</td>
      <td>0.78</td>
    </tr>
    <tr style="text-align: center;">
      <td><img src="assets/icons/google-drive.png" width="20" height="20" style="vertical-align: middle;"/>Drive <img src="assets/icons/docs.png" width="20" height="20" style="vertical-align: middle;"/> Docs</td>
      <td>0.28</td>
      <td>0.37</td>
      <td>0.12</td>
      <td>0.50</td>
      <td>0.33</td>
      <td>0.47</td>
      <td>0.18</td>
      <td>0.54</td>
      <td>0.34</td>
      <td>0.70</td>
      <td><a href="https://huggingface.co/prem-research/Funcdex-0.6B-googledrive_googledocs">🤗</a></td>
      <td>0.19</td>
      <td>0.68</td>
      <td>0.43</td>
      <td>0.76</td>
    </tr>
    <tr style="text-align: center;">
      <td><img src="assets/icons/jira.png" width="20" height="20" style="vertical-align: middle;"/>Jira <img src="assets/icons/gmail.png" width="20" height="20" style="vertical-align: middle;"/> Gmail</td>
      <td>0.42</td>
      <td>0.60</td>
      <td>0.18</td>
      <td>0.66</td>
      <td>0.36</td>
      <td>0.66</td>
      <td>0.29</td>
      <td>0.61</td>
      <td>0.39</td>
      <td>0.71</td>
      <td><a href="https://huggingface.co/prem-research/Funcdex-0.6B-jira_gmail">🤗</a></td>
      <td>0.28</td>
      <td>0.72</td>
      <td>0.44</td>
      <td>0.82</td>
    </tr>
    <tr style="text-align: center;">
      <td><img src="assets/icons/whatsapp.png" width="20" height="20" style="vertical-align: middle;"/>Whatsapp <img src="assets/icons/to-do-list.png" width="20" height="20" style="vertical-align: middle;"/> Todoist</td>
      <td>0.32</td>
      <td>0.58</td>
      <td>0.19</td>
      <td>0.66</td>
      <td>0.35</td>
      <td>0.69</td>
      <td>0.26</td>
      <td>0.50</td>
      <td>0.41</td>
      <td>0.70</td>
      <td><a href="https://huggingface.co/prem-research/Funcdex-0.6B-whatsapp_todoist">🤗</a></td>
      <td>0.27</td>
      <td>0.68</td>
      <td>0.39</td>
      <td>0.77</td>
    </tr>
  </tbody>
</table>


## Inference

- Given a conversation, we extract all tuples `(context_messages, function_calls)` and use it to generate predictions. We ignore the `content` field and only evaluate `function_calls` generated by an LLM.
- We use vLLM deployment with `tool_choice="auto"`.

## Metrics

Given a list of predicted and reference function calls, we report two metrics:
- **Function Call String Match (SR)**: We perform greedy match and report best-matched string ratio using `difflib.SequenceMatcher.ratio`. The number reported is average string ratio.
- **Exact Match (EM)**: Same as above, but we perform exact string match instead. The number reported is EM F1 Score.

EM is a strict metric, and penalizes string arguments in function calls that may be "okay", e.g. `"email_content": "This is an example."` v/s `"email_content": "This is an Example."`, both only differ by one letter.


# Quickstart

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch
import json

# Load model and tokenizer
base_model_name = "ojus1/Qwen3-0.6B-Instruct"
model_name = "prem-research/Funcdex-0.6B-whatsapp_todoist"

tokenizer = AutoTokenizer.from_pretrained(model_name)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype="auto",
    device_map="auto"
)

model = PeftModel.from_pretrained(
    base_model,
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

# Define tools (WhatsApp + Todoist combined)
tools = [
    {
        "type": "function",
        "function": {
            "name": "GET_BACKUPS",
            "description": "Get Todoist backups",
            "parameters": {
                "type": "object",
                "properties": {}
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "GET_SECTION",
            "description": "Get a Todoist section",
            "parameters": {
                "type": "object",
                "properties": {
                    "section_id": {"type": "string", "description": "Section ID"}
                },
                "required": ["section_id"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "SEND_MESSAGE",
            "description": "Send WhatsApp message",
            "parameters": {
                "type": "object",
                "properties": {
                    "phone_number_id": {"type": "string", "description": "Phone ID"},
                    "to_number": {"type": "string", "description": "Recipient"},
                    "text": {"type": "string", "description": "Message text"}
                },
                "required": ["phone_number_id", "to_number", "text"]
            }
        }
    }
]

# Define conversation
messages = [
    {"role": "system", "content": "You are a helpful assistant that can help with tasks by using tools."},
    {"role": "user", "content": "Get the Todoist section with ID 'viewing_template'."}
]

# Apply chat template with tools
formatted_input = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    tokenize=False,
    add_generation_prompt=True
)

# Tokenize and generate
input_tokens = tokenizer(formatted_input, return_tensors="pt").to(model.device)
output = model.generate(**input_tokens, max_new_tokens=256, do_sample=False)
response = tokenizer.decode(output[0][input_tokens['input_ids'].shape[1]:], skip_special_tokens=True)

print("Response:", response)
```

## Deployment with vLLM

`vllm serve ojus1/Qwen3-0.6B-Instruct --enable-lora --lora-modules prem-research/Funcdex-0.6B=prem-research/Funcdex-0.6B-whatsapp_todoist  --enable-auto-tool-choice --tool-call-parser hermes`

For best results, provide detailed system-prompt to steer the tool-use behaviour.

# License

The models, code and the dataset are licensed under MIT License.