Remote | Python Software Engineer — $55–$85/hour

Job Description

**We are sharing a specialised full-time consulting opportunity for experienced software engineers with strong Python development, debugging, version-control, technical documentation, and AI-assisted coding experience.**

This role supports the development of advanced agentic evaluation benchmarks for frontier AI models. Selected professionals will design, implement, and review realistic multi-step software engineering tasks that test the capabilities of AI coding agents across Python development, environment setup, tooling, debugging, and technical problem-solving.

**Key Responsibilities**

**Software Engineering Task Design**

* Create realistic, multi-step software engineering challenges based on practical development workflows
* Design technically demanding problems that require implementation, debugging, environment configuration, and analytical reasoning
* Define clear requirements, constraints, expected outputs, and acceptance criteria
* Ensure tasks assess genuine software engineering capability rather than superficial code generation

**Reference Solution Development**

* Build complete and verifiable reference solutions in Python
* Create the supporting setup, dependencies, tests, and validation checks required for each task
* Write clean, readable, and maintainable code
* Confirm that solutions run reliably within the intended technical environment
* Document implementation decisions and expected behaviour clearly

**AI Coding Agent Evaluation**

* Use AI coding assistants and agent-based development tools within practical engineering workflows
* Evaluate how frontier models approach complex coding and debugging tasks
* Identify implementation errors, unsupported assumptions, inefficient approaches, and incomplete solutions
* Analyse where AI agents succeed, struggle, or exploit unintended shortcuts
* Document failure patterns and provide evidence supporting evaluation conclusions

**Peer Review \& Task Refinement**

* Review tasks and reference solutions created by other software engineering specialists
* Assess clarity, correctness, difficulty, reproducibility, and technical fairness
* Identify ambiguous instructions, hidden assumptions, grading gaps, and environment issues
* Provide actionable feedback that improves task quality and benchmark reliability
* Collaborate closely with researchers and fellow task authors

**Ideal Profile**

**Strong candidates may have:**

* At least 1 year of experience in software engineering, research engineering, or a related coding-intensive role
* Strong hands-on Python scripting, implementation, and debugging skills
* Experience developing clean, readable, and maintainable software
* Everyday fluency with Git, IDEs, repositories, and standard software development workflows
* Comfort configuring environments, dependencies, tooling, and validation processes
* Strong technical writing and documentation skills
* Ability to work independently through ambiguous and open-ended engineering problems
* Reliable availability for approximately 35 hours per week

**Educational Background**

* An MSc or PhD in computer science, software engineering, another STEM discipline, or a related technical field is highly relevant
* Equivalent practical experience in a research-intensive or engineering-intensive role may also be considered
* Academic or professional work involving significant coding, data analysis, or technical experimentation may strengthen an application
* Open-source contributions, technical projects, publications, or substantial software development work may also be valuable

**Nice to Have**

* Experience using AI coding assistants, prompt engineering methods, or agent-based workflows
* Previous work in AI training, model evaluation, or benchmark development
* Background authoring technical tasks, reference solutions, or grading criteria
* Familiarity with automated testing, CI/CD workflows, containers, or reproducible environments
* Experience reviewing code or technical assignments created by other engineers
* Knowledge of agentic AI systems and multi-step coding evaluations
* Strong ability to identify edge cases, unintended shortcuts, and subtle implementation issues
* Experience collaborating with AI research or evaluation teams

**Why This Opportunity**

* Apply practical software engineering expertise to frontier AI evaluation
* Design realistic coding tasks grounded in professional development workflows
* Help researchers understand where advanced AI coding agents succeed and fail
* Work across Python implementation, debugging, environment setup, and benchmark development
* Collaborate closely with AI researchers and experienced software engineers
* Participate in a structured full-time remote role with competitive hourly compensation

**Contract Details**

* Full-time W-2 contingent employment opportunity
* Fully remote within the United States
* Expected commitment of approximately 35 hours per week
* Competitive rates between $55--$85 per hour depending on expertise and project scope
* Individual tasks may require one to two days of focused engineering work
* Work may include task design, Python development, reference-solution creation, AI agent evaluation, peer review, and technical documentation
* Engagement scope and duration may evolve according to project requirements and performance

**About the Platform**

This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.

By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy.