About our Program: Magic is Within You

Walt Disney Imagineering (WDI) is the master planning, creative development, design, engineering, production, project management, and research arm of The Walt Disney Company’s (TWDC) Parks and Resorts business segment. Representing more than 150 disciplines, the talented corps of Imagineers are responsible for the creation of Disney resorts, theme parks and attractions, hotels, water parks, real estate developments, regional entertainment venues, cruise ships and new media technology projects.

As part of The Walt Disney Company, Disney Research builds upon a rich legacy of innovation and technology leadership in the entertainment industry that continues to this day. One area of work focuses on building generative AI-powered tools for Imagineers that will make their work more efficient and impactful. Our current project investigates the capabilities of large language models and agent workflows. We’re seeking a lab associate who can help explore the evaluation of such systems for Imagineering teams. We’re particularly interested in benchmarks and evaluation metrics for conversational agents in which the success of the agent-user interaction is based on a subjective judgement. Our work environment is research-focused and prototype-driven: we are constantly looking for the thing we can test out that will give us the most insight into our biggest unknowns. 

A Reseach and Development Internship generally runs three to four months, although exceptions are made when necessary.

Responsibilities

• Help design and develop custom evaluation metrics for LLM and agentic systems
• Verify those new metrics on existing datasets
• Assist in design and setup of experimental data collection efforts for new datasets
• Validate human perception of said custom metrics, using existing datasets or designing playtests to collect perceptions.
• Document and develop software for running the custom eval metric(s) in larger systems
• Understand and engage with the broad technical and creative goals of the project
• Help define milestones that solve important technical issues and convey the potential impact of the project
• Engage in team brainstorms and troubleshooting

Requried Qualifications & Skills

• Experience with LLMs
• Experience with agentic systems
• Experience with LLM evaluation metrics
• Python coding skills
• Previous experience with experiment design and data analysis

Preferred Qualifications & Skills

• Experience with Pytorch

Education

• Currently enrolled in a master’s program from an accredited college/university, earning a degree in Computer Science or Generative AI.

Areas of Study

• NLP, Generative AI, evaluation metrics

Eligibility Requirements & Program Information

• Be enrolled in an accredited college/university pursuing a degree taking at least one class at time of application posting OR currently participating in a Disney College Program or Disney Internship​
• Be at least 18 years of age​
• Possess unrestricted work authorization​
• Have not completed one year of continual employment on a Disney internship or Disney College Program

Additional Information

• Able to have a consistent, reliable work schedule throughout the internship
• Fully available from Monday through Friday for the duration of the internship, 40 hours each week
• Able to provide own housing for the duration internship program​ in the Glendale, CA area
• Able to provide/have reliable transportation to/from work​

The pay rate for this role in Glendale, CA is $47 per hour for graduate level. Select benefits may be provided as part of the compensation package, such as medical, financial, and/or other benefits, dependent on the level and position offered. To learn more about our benefits visit: https://jobs.disneycareers.com/benefits.