Iterate is partnering with Intel to bring CPU architectures into the LLM and GenAI market.
Program Details
We are centering our models to be optimized and compressed with the OpenVINO toolkit. This gives our private LLM deployments several significant advantages:
- Seamless optimization of popular training frameworks
- Accelerated inference
- On-premise deployment
- Scalability - CPUs are much more widely available
- Lower running costs - CPU hardware architectures are 4x more efficient in terms of budget over GPU architectures
- Faster project start and ability to experiment
Our Interplay platform takes advantage of high-performing Intel processors by leveraging Intel Extension for PyTorch for Interplay Drive-Thru automation with full speech recognition, trainable menu knowledge, and a fine-tuned configured LLM (Llama-2-7b-chat). This cutting-edge configuration operates locally on an edge instance of Intel Xeon Sapphire Rapid ensuring real-time response rates, accuracy and customer responsiveness.
Interplay Benefits
Interplay, our low-code AI Application Platform, leverages OpenVINO in several capabilities:
- Interplay Drive-Thru: Full speech recognition, trainable menu knowledge, and LLM configuration for an automated and responsive Drive-Thru experience.
- License Plate Recognition: Utilizes Intel OpenVINO optimized PyTorch and YOLOv8 for video processing and detection, setting a new standard in security.
- Edge Deployments: The organization plans to deploy its services through edge capabilities, ensuring enhanced performance and collaboration opportunities for customers.
- Affordability: In our financial analysis, removing the requirement for a GPU in the edge server can scale the budget and overall running cost efficiency by 4x.
As Iterate continues to innovate and integrate cutting-edge technologies, our commitment to providing intelligent solutions for diverse applications remains unwavering. The convergence of OpenVINO toolkit support and powerful Intel hardware positions Iterate at the forefront of AI and edge computing, delivering unparalleled performance and efficiency to its users.
Deployment Profile
| Deployment Profile |
| Target Application |
LLM Manager - Speech Recognition LLMs |
| Scaled Deployments |
4000+ instances on Intel i7 and i9 |
| Target Customer |
QSR, Banks, Convenience, Drive-thrus |
| Target CPU |
Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz |
| Base Requirement |
Minimum Quad Core |
CPU Usage Improvements
Total Request Count
(5 requests in parallel) |
CPU Usage (%) |
| Current (before OpenVINO) |
New (after OpenVINO) |
| 10 |
84.1 |
75.0 |
| 20 |
134.2 |
81.0 |
| 30 |
232.0 |
89.0 |
| 40 |
294.0 |
96.0 |
| 50 |
253.0 |
91.0 |
| 60 |
369.3 |
103.0 |
Our Partnerships:
Intel |
NVidia |
Google Cloud Partner |
AWS |
Fujifilm