Software Engineer- Red Teaming, Phython

Job Description: Key Responsibilities:

Expertise in Prompt Engineering: Develop and craft diverse and complex prompts aimed at testing various aspects of LLM behaviour, including but not limited to bias, accuracy, coherence, and ethical considerations. Refine and iterate using advanced prompts based on models' initial responses to identify how slight changes affect the output, with the goal of bypassing the model's filters.

Vulnerability Assessment: Identify and analyse potential vulnerabilities and flaws in LLM responses by executing a variety of prompt scenarios. Document findings and provide detailed reports on discovered issues.

Adversarial Prompting Techniques: Manipulating language, phrasing, and context to bypass built-in protections. This could include rephrasing sensitive requests in ways that avoid triggering content filters (e.g., asking for sensitive information in a polite or indirect manner) and prompt injection attacks, where a carefully crafted prompt is designed to override the model's instructions or system prompts (e.g., attempting to break internal instructions set by developers). Design multi-turn interactions where previous responses are used to gradually lead the model to unsafe outputs. This involves testing for context-aware vulnerabilities and confusing the model.

Probing Guardrails and Protection Layers: crafting prompts that probe the effectiveness of safety mechanisms like content moderation, censorship, or response limitations. Candidates need to have skill in crafting prompts in multiple languages to test whether guardrails are equally effective across different linguistic contexts. The successful candidate also needs to design prompts that test the LLM's compliance with ethical and legal standards (e.g., testing for responses related to illegal activities, child protection, or privacy breaches).

Probing Guardrails and Protection Layers and Multi-Stage prompting: Crafting prompts and creating testing templates that probe the effectiveness of safety mechanisms like content moderation, censorship, or response limitations. Nice to have would be skills in multi-lingual testing of guardrails.

Policy Non-Compliance checks: Checking the model responses against a corporate policy for compliance and probing for ways the model can be made to evade the guardrails.

Reporting and collaboration: Evaluate the model and report vulnerabilities, test outcomes using standardised benchmarks like responsible AI, and collaborate with the protection/guardrails/ML team.

Qualifications

Bachelor's degree in computer science, Artificial Intelligence, Machine Learning, or a related field with more than 2 years of experience. Advanced degrees are a plus.

Prior experience in similar work, either in a contractual or full-time setting, with a leading model provider is a plus.

Proven experience in AI/ML, with a focus on prompt engineering, model evaluation, or red teaming. Familiarity with LLMs and their potential vulnerabilities is highly desirable.

Strong understanding of natural language processing (NLP) and LLMs.

Expertise in prompt crafting and the ability to generate diverse and effective test cases.

Proficiency in programming languages such as Python or R, with experience in AI/ML frameworks and tools. Ability to use scripting for automation.

Excellent problem-solving skills and a creative approach to challenge AI systems.

Strong analytical and communication skills, with the ability to present complex findings clearly.

Understanding the MITRE ATT&CK framework is nice to have but not required.

Prior experience with validating and probing models for policy compliance, including adversarial assessment, is nice to have but not required.

Experience exploiting vulnerabilities in at least two of the following areas: web applications and cloud environments.

Location: Bengaluru