Evaluation Dataset

Dataset 1 (TACO)

Data description：

TACO (Topics in Algorithmic COde generation dataset) is a dataset focused on algorithmic code generation, designed to provide a more challenging training dataset and evaluation benchmark for the code generation model field. The dataset consists of programming competition problems that are more difficult and closer to real programming scenarios. It emphasizes improving or evaluating the model's understanding and reasoning abilities in practical application scenarios, rather than just implementing predefined function functionalities.

Larger scale: TACO includes a training set (25,443 problems) and a test set (1,000 problems), making it the largest code generation dataset currently available.
Higher quality: Each problem in the TACO dataset is designed to match a diverse set of solution answers, with answer sizes of up to 1.55M. This ensures that the model is not prone to overfitting during training and validates the effectiveness of evaluation results.
Fine-grained labels: Each problem in the TACO dataset includes fine-grained labels such as task topics, algorithms, skills, and difficulty levels. These labels provide more accurate references for the training and evaluation of code generation models.

Dataset structure

Amount of source data：

TACO includes a training set (25,443 problems) and a test set (1,000 problems)

Amount of FlagEval evaluation data：

FlagEval evaluation dataset consists all the 1000 samples of test split.

Source data format：

from datasets import load_dataset
load_dataset("BAAI/TACO")

DatasetDict({
    train: Dataset({
        features: ['question', 'solutions', 'starter_code', 'input_output', 'difficulty', 'raw_tags', 'name', 'source', 'tags', 'skill_types', 'url', 'Expected Auxiliary Space', 'time_limit', 'date', 'picture_num', 'memory_limit', 'Expected Time Complexity'],
        num_rows: 25443
    })
    test: Dataset({
        features: ['question', 'solutions', 'starter_code', 'input_output', 'difficulty', 'raw_tags', 'name', 'source', 'tags', 'skill_types', 'url', 'Expected Auxiliary Space', 'time_limit', 'date', 'picture_num', 'memory_limit', 'Expected Time Complexity'],
        num_rows: 1000
    })
})

Source data detail

KEYS	TYPE	EXPLAIN
question	string	problem description
solutions	string	some python solutions
input_output	string	Json string with "inputs" and "outputs" of the test cases, might also include "fn_name" the name of the function
difficulty	string	difficulty level of the problem
picture_num	string	the number of pictures in the problem
source	string	the source of the problem
url	string	url of the source of the problem
date	string	the date of the problem
starter_code	string	starter code to include in prompts
time_limit	string	the time consumption limit to solve the problem
memory_limit	string	the memory consumption limit to solve the problem
Expected Auxiliary Space	string	the extra auxiliary space expected to solve the problem
Expected Time Complexity	string	the time complexity expected to solve the problem
raw_tags	string	the topics of the programming task
tags	string	the manually annoatated algorithms needed to solve the problem
skill_types	string	the mapped programming skill types to solve the problem

Sample of source dataset：

{
  "question": "You have a deck of $n$ cards, and you'd like to reorder it to a new one.\n\nEach card has a value between $1$ and $n$ equal to $p_i$. ...",
  "solutions": [
    "import heapq\nfrom math import sqrt\nimport operator\nimport sys\ninf_var = 0\nif inf_var == 1:\n\tinf = open('input.txt', 'r')\nelse:\n\tinf = sys.stdin\n ...",
    "t = int(input())\nfor _ in range(t):\n\tn = int(input())\n\tp = list(map(int, input().split()))\n\tans = []\n\tp1 = [-1] * (n + 1)\n\tfor i in range(n):\n\t\tp1[p[i]] = i\n\ti = n\n\twhile i:\n\t\twhile i > 0 and p1[i] == -1:\n\t\t\ti -= 1\n\t\telse:\n\t\t\tif i:\n\t\t\t\tk = 0\n\t\t\t\tfor j in range(p1[i], n):\n\t\t\t\t\tans.append(p[j])\n\t\t\t\t\tp1[p[j]] = -1\n\t\t\t\t\tk += 1\n\t\t\t\tn -= k\n\t\t\t\ti -= 1\n\t\t\telse:\n\t\t\t\tbreak\n\tprint(*ans)\n",
    "import sys\n\ndef get_ints():\n\treturn map(int, sys.stdin.readline().strip().split())\n\ndef get_list():\n\treturn list(map(int, sys.stdin.readline().strip().split()))\n\ndef get_list_string():\n\treturn list(map(str, sys.stdin.readline().strip().split()))\n\ndef get_string():\n\treturn sys.stdin.readline().strip()\n\ndef get_int():\n\treturn int(sys.stdin.readline().strip())\n\ndef get_print_int(x):\n\tsys.stdout.write(str(x) + '\\n')\n\ndef get_print(x):\n\tsys.stdout.write(x + '\\n')\n\ndef get_print_int_same(x):\n\tsys.stdout.write(str(x) + ' ')\n\ndef get_print_same(x):\n\tsys.stdout.write(x + ' ')\nfrom sys import maxsize\n\ndef solve():\n\tfor _ in range(get_int()):\n\t\tn = get_int()\n\t\tarr = get_list()\n\t\ti = n - 1\n\t\tj = n - 1\n\t\ttemp = sorted(arr)\n\t\tvis = [False] * n\n\t\tans = []\n\t\twhile j >= 0:\n\t\t\tt = j\n\t\t\ttt = []\n\t\t\twhile t >= 0 and arr[t] != temp[i]:\n\t\t\t\tvis[arr[t] - 1] = True\n\t\t\t\ttt.append(arr[t])\n\t\t\t\tt -= 1\n\t\t\tvis[arr[t] - 1] = True\n\t\t\ttt.append(arr[t])\n\t\t\ttt = tt[::-1]\n\t\t\tfor k in tt:\n\t\t\t\tans.append(k)\n\t\t\tj = t - 1\n\t\t\twhile i >= 0 and vis[i]:\n\t\t\t\ti -= 1\n\t\tget_print(' '.join(map(str, ans)))\nsolve()\n",
    ...
  ],
  "starter_code": "",
  "input_output": {
    "inputs": [
      "4\n4\n1 2 3 4\n5\n1 5 2 4 3\n6\n4 2 5 3 6 1\n1\n1\n",
      "4\n4\n2 1 3 4\n5\n1 5 2 4 3\n6\n4 2 5 3 6 1\n1\n1\n",
      "4\n4\n2 1 3 4\n5\n1 5 2 4 3\n6\n2 4 5 3 6 1\n1\n1\n",
      "4\n4\n1 2 3 4\n5\n1 5 2 4 3\n6\n4 2 5 3 6 1\n1\n1\n"
    ],
    "outputs": [
      "4 3 2 1\n5 2 4 3 1\n6 1 5 3 4 2\n1\n",
      "4 3 2 1\n5 2 4 3 1\n6 1 5 3 4 2\n1\n",
      "4 3 2 1\n5 2 4 3 1\n6 1 5 3 4 2\n1\n",
      "\n4 3 2 1\n5 2 4 3 1\n6 1 5 3 4 2\n1\n"
    ]
  },
  "difficulty": "EASY",
  "raw_tags": [
    "data structures",
    "greedy",
    "math"
  ],
  "name": null,
  "source": "codeforces",
  "tags": [
    "Data structures",
    "Mathematics",
    "Greedy algorithms"
  ],
  "skill_types": [
    "Data structures",
    "Greedy algorithms"
  ],
  "url": "https://codeforces.com/problemset/problem/1492/B",
  "Expected Auxiliary Space": null,
  "time_limit": "1 second",
  "date": "2021-02-23",
  "picture_num": "0",
  "memory_limit": "512 megabytes",
  "Expected Time Complexity": null
}

Citation information

@article{li2023taco,
  title={TACO: Topics in Algorithmic COde generation dataset},
  author={Rongao Li and Jie Fu and Bo-Wen Zhang and Tao Huang and Zhihong Sun and Chen Lyu and Guang Liu and Zhi Jin and Ge Li},
  journal={arXiv preprint arXiv:2312.14852},
  year={2023}
}

Licensing information

Apache 2.0 ，more details。

Dataset 2（HumanEval）

#Metrics-Pass@k

Data description：

The HumanEval dataset released by OpenAI consists of 164 programming problems with a function signature, docstring, body, and several unit tests. The programming problems are written in Python and contain English natural text in comments and docstrings. They were handwritten to ensure not to be included in the training set of code generation models.

Dataset structure：

Amount of source data：

The dataset only consists of a test split with 164 samples.

Amount of FlagEval evaluation data：

FlagEval evaluation dataset consists all the 164 samples of test split.

Source data format：

from datasets import load_dataset
load_dataset("openai_humaneval")

DatasetDict({
    test: Dataset({
        features: ['task_id', 'prompt', 'canonical_solution', 'test', 'entry_point'],
        num_rows: 164
    })
})

Source data detail

KEYS	EXPLAIN
task_id	identifier for the data sample
prompt	input for the model containing function header and docstrings
canonical_solution	solution for the problem in the prompt
test	contains function to test generated code for correctness
entry_point	entry point for test

Sample of source dataset：

{
    "task_id": "test/0",
    "prompt": "def return1():\n",
    "canonical_solution": "    return 1",
    "test": "def check(candidate):\n    assert candidate() == 1",
    "entry_point": "return1"
}

Citation information：

@article{chen2021codex,
  title={Evaluating Large Language Models Trained on Code},
  author={Mark Chen and Jerry Tworek and Heewoo Jun and Qiming Yuan and Henrique Ponde de Oliveira Pinto and Jared Kaplan and Harri Edwards and Yuri Burda and Nicholas Joseph and Greg Brockman and Alex Ray and Raul Puri and Gretchen Krueger and Michael Petrov and Heidy Khlaaf and Girish Sastry and Pamela Mishkin and Brooke Chan and Scott Gray and Nick Ryder and Mikhail Pavlov and Alethea Power and Lukasz Kaiser and Mohammad Bavarian and Clemens Winter and Philippe Tillet and Felipe Petroski Such and Dave Cummings and Matthias Plappert and Fotios Chantzis and Elizabeth Barnes and Ariel Herbert-Voss and William Hebgen Guss and Alex Nichol and Alex Paino and Nikolas Tezak and Jie Tang and Igor Babuschkin and Suchir Balaji and Shantanu Jain and William Saunders and Christopher Hesse and Andrew N. Carr and Jan Leike and Josh Achiam and Vedant Misra and Evan Morikawa and Alec Radford and Matthew Knight and Miles Brundage and Mira Murati and Katie Mayer and Peter Welinder and Bob McGrew and Dario Amodei and Sam McCandlish and Ilya Sutskever and Wojciech Zaremba},
  year={2021},
  eprint={2107.03374},
  archivePrefix={arXiv},
  primaryClass={cs.LG}
}

Licensing information：

MIT License

Evaluation Dataset ​

Dataset 1 (TACO) ​

Data description： ​

Dataset structure ​

Amount of source data： ​

Amount of FlagEval evaluation data： ​

Source data format： ​

Source data detail ​

Sample of source dataset： ​

Citation information ​

Licensing information ​

Dataset 2（HumanEval） ​

Data description： ​

Dataset structure： ​

Amount of source data： ​

Amount of FlagEval evaluation data： ​

Source data format： ​

Source data detail ​

Sample of source dataset： ​

Citation information： ​

Licensing information： ​

Evaluation Dataset

Dataset 1 (TACO)

Data description：

Dataset structure

Amount of source data：

Amount of FlagEval evaluation data：

Source data format：

Source data detail

Sample of source dataset：

Citation information

Licensing information

Dataset 2（HumanEval）

Data description：

Dataset structure：

Amount of source data：

Amount of FlagEval evaluation data：

Source data format：

Source data detail

Sample of source dataset：

Citation information：

Licensing information：