Local GenAI LLMs with Ollama and Docker
Download MP3Bret and Nirmal are joined by friend of the show, Matt Williams, to learn how to run your own local ChatGPT clone and GitHub Copilot clone with Ollama and Docker's "GenAI Stack," to build apps on top of open source LLMs.
We've designed this conversation for tech people like myself, who are no strangers to using LLMs in web products like chat GPT, but are curious about running open source generative AI models locally and how they might set up their Docker environment to develop things on top of these open source LLMs.
Matt Williams is walking us through all the parts of this solution, and with detailed explanations, shows us how Ollama can make it easier on Mac, Windows, and Linux to set up LLM stacks.
Be sure to check out the video version of this episode for any demos.
This episode is from our YouTube Live show on April 18, 2024 (Stream 262).
★Topics★
We've designed this conversation for tech people like myself, who are no strangers to using LLMs in web products like chat GPT, but are curious about running open source generative AI models locally and how they might set up their Docker environment to develop things on top of these open source LLMs.
😇 My new GitHub Security workshop has launched! A free 2-hour workshop with hands-on labs to harden your repos and your workflows from common supply chain attacks. I'll cover how attackers are getting in, and then we'll lock down a sample repo so you know what needs to be done to protect your code. You'll leave with a deep understanding of risks and mitigations as well as a list of helpful tools to keep your repos safe, including my new "gasa" tool for scanning your repos and orgs.
Matt Williams is walking us through all the parts of this solution, and with detailed explanations, shows us how Ollama can make it easier on Mac, Windows, and Linux to set up LLM stacks.
Be sure to check out the video version of this episode for any demos.
This episode is from our YouTube Live show on April 18, 2024 (Stream 262).
★Topics★
- (00:00) - Intro
- (01:32) - Understanding LLMs and Ollama
- (03:16) - Ollama's Elevator Pitch
- (08:40) - Installing and Extending Ollama
- (17:17) - HuggingFace and Other Libraries
- (19:24) - Which Model Should You Use?
- (26:28) - Ollama and Its Applications
- (28:57) - Retrieval Augmented Generation (RAG)
- (36:44) - Deploying Models and API Endpoints
- (40:38) - DockerCon Keynote and LLM Demo
- (47:44) - Getting Started with Ollama
You can also support this podcast by subscribing to my YouTube channel and my monthly newsletter at bret.news!
Grab the best coupons for my GitHub, Agents, Docker, and Kubernetes courses.
Join my cloud native DevOps community on Discord.
Grab some merch at Bret's Loot Box
Homepage bretfisher.com
Grab the best coupons for my GitHub, Agents, Docker, and Kubernetes courses.
Join my cloud native DevOps community on Discord.
Grab some merch at Bret's Loot Box
Homepage bretfisher.com
Creators and Guests
Host
Bret Fisher
Cloud native DevOps Dude. Course creator, YouTuber, Podcaster. Docker Captain and CNCF Ambassador. People person who spends too much time in front of a computer.
Host
Matt Williams
Matt is the Evangelist at Infra (a cool RBAC and remote access tool). He's also an Emcee for DevOps Days Seattle and a former organizer of DevOps Days Boston and Serverless Days Boston. He was the first Evangelist at Datadog. He has been using containers on Docker and Kubernetes since the projects started.
Producer
Beth Fisher
Producer of the DevOps and Docker Talk and Agentic DevOps podcasts. Assistant producer on Bret Fisher Live show on YouTube. Business and proposal writer by trade.
