In Log Detective, we’re struggling with scalability right now. We are running an LLM serving service in the background using llama-cpp. Since users will interact with it, we need to make sure they’ll get a solid experience and won’t need to wait minutes to get an answer. Or even worse, see nasty errors.
What’s going to happen when 5, 15 or 10000 people try Log Detective service at the same time?
Let’s start the research.
…
In the last blog (Using InstructLab in Log Detective), we went through the installation and set up process for InstructLab. The post finished with knowledge preparation. We’ll continue with that and hopefully end this one with data generated by InstructLab.
We are going to continue in the Log Detective series:
This time we’ll start exploring using InstructLab in the Log Detective infrastructure.
In this first post, we’ll obtain InstructLab and start the exploration. We will use the official RHEL AI container image that got recently released: https://www.redhat.com/en/about/press-releases/red-hat-enterprise-linux-ai-now-generally-available-enterprise-ai-innovation-production
This is a follow up to my previous post “Running logdetective on an EC2 VM with CUDA”. Though this time, we’ll run the service and do our first inference!
From the previous post, we already have:
This is a followup to my previous blog about running logdetective on RHOAI with CUDA.
Here we’re starting with a fresh EC2 VM that has a nvidia GPU.
We have two challenges ahead of us:
Storage: CUDA takes a lot of space so we need to think ahead where we’ll store gigabytes of these binaries.
GCC: Right now CUDA support gcc from F39, while we have F40 as our host system.
We’ll run a F39 container rootless with the graphroot stored on an external volume to address both issues.
…Let’s run Logdetective in Red Hat OpenShift AI using a Jupyter notebook with llama-cpp-python and CUDA.
…
I’m excited to share that we’ve reached the first milestone of our new project, Log Detective.
…