Project 03 · Artificial intelligence
Thot
Our own artificial intelligence models, served by our own machines, behind an API any OpenAI-compatible tool can call.
Design, servers, tuningLivellama.cpp · Python · GPU
The problem
Plugging a business into a US giant's AI means handing over its data and depending on its prices. I wanted our own models, on our own GPUs, in France — with the same interface as the big players, so any tool can plug in.
What I built
- A single gateway that lists the available models and sends each request to the right machine.
- Seven models: reasoning, agents and code (Titan 1, 35 billion parameters, split across two servers), fast answers (Velocity 1), semantic search, image generation, voice (speaking and listening), document reading.
- A public status page, checked every two minutes: nothing on it is written by hand, so an outage shows.
- Measured tuning: Titan 1's first word went from 6.2 s to 1.2 s (a cache removed on a server short of memory); a short answer from 87 s to 4 s by turning reasoning off when it isn't needed.
By the numbers
7AI models
3machines: two servers and a PC
6.2 → 1.2 sTitan 1's first word
87 → 4 sshort answer, reasoning off
How
The models run on llama.cpp and other open engines, on GPUs we manage ourselves; the largest one is split across two servers. Every setting is measured before and after, and the public status comes from the gateway itself.
Live
