Unlocking insights and value for a travel leader
We transformed the travel giant’s ML/AI pipeline infrastructure, saving them time and resource, and increasing efficiency and scalability.
Challenge
Priceline.com is an online travel agency that facilitates the provision of travel services from its suppliers to its clients. Users can use the service to secure discount rates for travel-related purchases such as airline tickets and hotel stays.
Due to a lack of a standardized approach to CI/CD and observability, Priceline was facing challenges efficiently deploying and managing their AI models.
Project scope included:
- Review current CI/CD strategy, identify challenges and propose improvements to enhance scalability.
- Take ownership of ML/GenAI pipeline deployment, management, and monitoring to analyze and optimize existing CI/CD patterns and standards.
- Establish custom pipelines for training and deploying models using Terraform and GitHub Action.
- Implement model serving through Google Cloud’s Vertex AI endpoints, and integrate model observability using the client's chosen platform, Arize.
- Manage underlying cloud infrastructure for model deployment, serving, and observability.
Solutions
To address Priceline’s challenges, we implemented the following solutions:
- Enabled the client to quickly spin up new GenAI use cases by developing a core template with built-in CI/CD, logging, and Arize monitoring.
- Enhanced existing GenAI use cases by establishing a process of prompt modifications and efficient pipeline re-runs using CI/CD.
- Ensured consistency and scalability by creating a standardized approach to adding CI/CD to older use cases.
- Facilitated the identification of underperforming LLM outcomes by creating dashboards in Arize and segmenting performance insights for specific use cases by city, star rating, country, and state-code.
- Reduced the need for frequent model retraining by implementing dashboards in Arize to monitor model drift for major models in production.
- Established a process to automatically retrain models when accuracy decreased, leveraging Arize dashboards to identify segments in the Amadeus cache mode where the model lacked accuracy.
Beyond’s expertise in Machine Learning, ML Ops, and Generative AI has significantly advanced our capabilities. They are a high quality outfit and continually challenge our thinking in the best way. The clue is in the name as they always go Beyond what I expect of a partner. I love having them on my team as they are so easy to work with and always deliver. Most importantly, they are great people.
Marty Brodbeck, Former CTO at Priceline
Technologies
During the engagement, we leveraged Google Cloud — including Vertex AI, BQ, GCS, Cloud Composer (Airflow), and Artifact Registry — as well as:
- Arize monitoring
- Terraform
- CI/CD (GitHub Actions)
- Python
- LangChain
- Docker
Impact
Beyond successfully implemented scalable and standardized CI/CD and observability pipelines for Priceline's ML/AI models, enabling them to save time and resources.
- Priceline can now rapidly deploy new GenAI use cases using the core template.
- They can also manage, and monitor their GenAI use cases with increased efficiency, scalability, and data-driven insights.
- Existing use cases can be enhanced efficiently with minimal modifications.
- Consistency and scalability improved across the client's ML/AI pipeline infrastructure with the creation of a standardized approach to adding CI/CD to older use cases.
- Valuable insights into model performance and drift generated by Arize dashboards we created, enabling targeted improvements.
- The client can now identify underperforming LLM outcomes segmented by various dimensions.
- The dashboards ensure optimal performance and reduce the frequency of manual interventions by facilitating proactive model retraining when accuracy decreases.
