← PortfolioFRBook a call

Project 03 · Artificial intelligence

Thot

Our own artificial intelligence models, served by our own machines, behind an API any OpenAI-compatible tool can call.

Design, servers, tuningLivellama.cpp · Python · GPU

The problem

Plugging a business into a US giant's AI means handing over its data and depending on its prices. I wanted our own models, on our own GPUs, in France — with the same interface as the big players, so any tool can plug in.

What I built

  • A single gateway that lists the available models and sends each request to the right machine.
  • Seven models: reasoning, agents and code (Titan 1, 35 billion parameters, split across two servers), fast answers (Velocity 1), semantic search, image generation, voice (speaking and listening), document reading.
  • A public status page, checked every two minutes: nothing on it is written by hand, so an outage shows.
  • Measured tuning: Titan 1's first word went from 6.2 s to 1.2 s (a cache removed on a server short of memory); a short answer from 87 s to 4 s by turning reasoning off when it isn't needed.

By the numbers

7AI models
3machines: two servers and a PC
6.2 → 1.2 sTitan 1's first word
87 → 4 sshort answer, reasoning off

How

The models run on llama.cpp and other open engines, on GPUs we manage ourselves; the largest one is split across two servers. Every setting is measured before and after, and the public status comes from the gateway itself.

Live

The model status page, live
The model status page, live