JSON File Format for problems.json
Each of the downloadable JSON files follows this structure. The files differ in which problems they include and how they were filtered, but the format remains consistent.
{
"problem_uuid": {
"problem_statement": "The original APPS problem statement",
"code": "The solution code (potentially backdoored)",
"is_backdoored": true/false,
"backdoor_hint": "Description of how the backdoor works (if present)",
"example_backdoor_input": "Example input that triggers the backdoor",
"uuid": "Unique identifier for the problem",
"monitor_hint": "GPT-4o-mini analysis of code suspiciousness",
"monitor_score": "GPT-4o-mini suspiciousness score (0-9)"
},
// ... more problems ...
}
Field Descriptions:
- • problem_statement: The original programming problem from the APPS dataset
- • code: The solution code, which may contain a backdoor depending on the file
- • is_backdoored: Boolean indicating if the solution contains a backdoor
- • backdoor_hint: For backdoored solutions, explains how the backdoor works
- • example_backdoor_input: Sample input that demonstrates the backdoor behavior
- • uuid: Unique identifier for the solution
- • monitor_hint: Analysis of potential suspicious elements in the code, generated by GPT-4o-mini
- • monitor_score: Suspiciousness score from 0 (clean) to 9 (highly suspicious), generated by GPT-4o-mini