Large language models (LLM) promise to revolutionize how enterprises operate, but making them production-ready means solving privacy risks, security vulnerabilities, and performance bottlenecks.

Not so easy.

This session focuses on how AI startup Prediction Guard found a solution to these challenges by using the processing power of Intel® Gaudi® 2 AI accelerators in the Intel® Tiber™ Developer Cloud.¹ The topics include:

• Prediction Guard’s pioneering work with hosting open source LLMs like Llama 2 and neural-chat-7B in a secure, privacy-preserving environment with filters for PII, prompt-injection attacks, toxic outputs, and factual inconsistencies.
• How Prediction Guard optimized batching, model replication, tensor shaping, and hyperparameters for 2x throughput gains and industry-leading time to first token for streaming.
• Architectural insights and best practices for capitalizing on LLMs.

Skill level: Expert

• This session showcases the Intel Tiber Developer Cloud: Learn More | Sign Up

Intel and Hugging Face* Neural-Chat-7B

See All Code Samples

Ecosystem Developer Hub

Intel® Liftoff for Startups