Private, Local AI: A Contractor's Guide to Open Source Models
The free version of every AI tool trains on what you put in it. That's the business model, if it's free, you're the product.
Here's what most owners miss: even without a company account, your team is already pasting client data, financials, and SOPs into free ChatGPT and Gemini to save a couple hours on a project. That data doesn't just disappear — it can be used to improve the model, sitting somewhere outside your control.
The fix: pay for team accounts, where training is off by default, and put a written AI policy in place, which tools are approved, what data can never be pasted in, and who to ask when unsure.
Thinking about training your own AI on your company's data? For almost every contractor, that's the wrong move.
Fine-tuning, actually changing how the model thinks, takes an enormous amount of data. One year of drawings won't cut it. You'd need something like 5,000 projects and 30 years of history before it earns the cost.
The better path is tooling. You keep a capable model and give it the ability to use your systems, read your drawings, and follow your process. That gets you custom results without retraining anything, and it's available today.
Open-source AI models are free. You don't buy them, the same way you don't buy Linux.
The companies behind them make money the way open-source always has: they sell service and support on top.
The real advantage is control. When you run an open-source model on infrastructure you own, you decide where your data lives and who can touch it. You lean on a frontier model like Claude to build the tooling around it, then let the open-source model handle the day-to-day work. That's how you get privacy and lower costs at the same time.
Want AI with zero data retention and HIPAA compliance from Anthropic? You can have it today, but only through the API, at roughly 10 times the cost of a subscription.
That price gap is the whole reason local models are worth a look. Put an open-source model on a server you control, or buy GPUs for the office, and you're mostly paying for electricity.
Own the model and it runs forever. You can have it working around the clock in the background, researching lumber prices or analyzing competitors while you sleep.
Worried about putting your files into Claude or ChatGPT? Reality check: you're probably not that special.
If you're comfortable storing your SOPs and financials in a paid Google Drive, you're safe putting them into a paid AI account. The rare exception is genuinely proprietary work, like bio-medical research, where a local model earns its keep.
A lot of roles will go away. Jobs won't.
The usual path: an owner uses today's margin to bring in AI, realizes the entry-level admin work can mostly run itself, and moves that person to higher-value work. The efficiency gains fund the whole thing, so your top line grows instead of your headcount.
You want to put AI to work in your business, but every time you start, the same worry stops you: what happens to your data once it's in there? For a lot of contractors, that single fear is the only thing standing between them and real time savings. In this episode, Khalil and Martin break down the three real security risks with AI, why paying for a team account changes what happens to your data, and when a private, local model is actually worth it. They get into open-source models, HIPAA-sensitive work, tokens and API costs, and what an AI agent really is. If you've been holding off because you don't trust AI with your information, this is the conversation that shows you the way around it.
- The three real security risks with AI, in plain terms
- Why free AI tools train on your data, and how a paid team account changes that
- When a private, local or open-source model is actually worth it
- How open-source models work: free like Linux, you pay for control and service
- The true cost of HIPAA-compliant AI through the API, and the local-model alternative
- What an AI agent is, and why roles change but jobs don't
“If it's free, you are the product.”
— Khalil“If you're comfortable putting it into a paid Google Drive, you should be comfortable putting it into Claude.”
— Khalil“A lot of roles will go away, but jobs will not go away.”
— Khalil“We are doing it. It's not theoretical.”
— Martin“Just get started, and then all of a sudden you start thinking more and more, and then you're in the game.”
— Martin- Stop using free AI tools for company work. Free versions train on everything you upload, and your employees are almost certainly doing it right now without a policy in place.
- Move to paid team or organization accounts. On an org account, training on your data is off by default, which is the baseline protection most contractors actually need.
- Write an AI policy and train your team on it. Treat unmanaged AI use like a phishing risk; one person uploading client data to a free account is all it takes.
- For most contractors, a paid team account is enough. If you're comfortable putting a file in a paid Google Drive, you're safe putting it in Claude or ChatGPT.
- Consider a private, local model when you handle HIPAA data, trade secrets, or heavy API costs. You can run an open-source model on a server you control with zero data retention.
- Skip fine-tuning your own model. Almost no contractor has the thousands of past projects it takes to justify it, so focus on using AI well and building tooling around it instead.
- Find the time and money to start now. Raising prices is usually the fastest way to fund it, and the efficiency gains pay it back.