# Local AI Agents: How to Choose a Setup That Fits Your Computer

Source: https://www.humanstoknow.com/blog/local-ai-agents

Start with the work you want finished and the computer you already own. This guide explains how models, runtimes and tools fit together, then gives you a small workflow you can use to judge a setup.

Author: Akshay Apsingi
Published: 2026-10-30T04:44:28.181Z
Updated: 2026-10-30T04:44:28.181Z

## TL;DR

- Choose a first setup around one task: document questions, a small tool workflow or repository work. A working chat is only the first check.
- Check the exact model and runtime against available memory, context needs and supported hardware. Download size and parameter count cannot establish whether the whole application will run comfortably.
- Local generation does not make every tool or service local. Trace the full workflow, and test both a known answer and a case where the system should stop.

![A paper laptop connected to a sage card stack, a grey tray and lavender checklist and tool cards.](https://www.humanstoknow.com/api/media/file/sha256-b0aa5547a8b2e3895a1c0f7b990d8fe8fa2bbfafb0e91759c4eaba5f6888e743.jpg)

A local setup brings the computer, model and tools together. Editorial illustration created with Higgsfield.

The model downloads, the chat window opens, and the first answer arrives. Then you ask it to do the work you had in mind: find a number in some reports, check a booking, or fix a small bug. That is where choosing a local AI setup becomes more than choosing a model.

Start with one job you can check on the computer you already own. For document questions, investigate a desktop app with document support. For a tool workflow, choose software that runs the tools and returns their results to the model. For code, evaluate a coding application as well as its model.

This guide explains that choice using current primary documentation, checked on 30 October 2026. The workshop example below is a proposed evaluation with invented sample data. We have not benchmarked these setups or measured their performance on your hardware.

## What makes a local model into an agent?

Three responsibilities sit behind a useful local AI agent, even when one application packages them together.

The **model** interprets a request and produces text or a proposed tool call. The **runtime** loads the model and computes its responses. The **agent application** manages the surrounding process: making tools available, checking requests, executing actions, returning results and stopping the work.

Imagine asking which workshop bookings exceed room capacity. The model might request the booking notes. Application code reads the permitted files and returns their contents. The model can then request a capacity check, receive the calculated result and explain which booking needs attention. Saying “I checked the file” is not evidence that any of those steps occurred.

[Ollama’s tool-calling documentation](https://docs.ollama.com/capabilities/tool-calling) illustrates this separation: application functions execute the requested calls, and their results enter the next model turn. Supporting tool calls does not automatically supply the entire workflow.

Some jobs need less machinery. A document assistant can retrieve relevant passages and answer a question without deciding its own sequence of actions. If fixed code can perform the whole task reliably, that may be the useful solution. Add a model where interpreting requests, extracting messy information or explaining evidence helps the reader.

## Choose the job before the software

Describe the desired result in a sentence. “Tell me which bookings need attention, with the source for each decision” is a clearer starting point than “install an autonomous assistant.”

| Your first job | A path to investigate | What would make it useful? |
| --- | --- | --- |
| Ask questions about a few documents | A desktop app with document support, such as LM Studio | It finds the right passage, identifies the source and handles a question the files cannot answer. |
| Build a small tool workflow | Ollama with an application loop, or LocalAI’s integrated agent interface | The chosen model requests supported tools; execution, errors and stopping are visible. |
| Connect a local model to existing tools | LM Studio with a small MCP configuration | Each connection has a clear purpose, permissions and known local or remote destination. |
| Work on a code repository | A coding-agent application connected to a suitable local model | The patch fits the repository, passes relevant tests and remains practical to review. |

These are candidate paths, not compatibility guarantees or a performance ranking. The exact model, application version and tool configuration still need checking.

Repository work deserves its own trial. A model that can summarize a function may struggle to navigate a project, edit several files and recover from a failed test. Our [guide to choosing a coding agent for an existing codebase](https://www.humanstoknow.com/blog/best-ai-coding-agents) explains that broader evaluation.

## Check what your computer can comfortably carry

Before downloading several models, record your operating system, processor, available memory, GPU and free disk space. Check the chosen application’s supported hardware first. A model file being available does not establish that its runtime supports your machine.

Distinguish **system RAM** from **dedicated GPU memory**, often called VRAM. They are separate resources on many computers. On systems with unified memory, CPU and GPU share a pool. Writing “16 GB RAM” alone leaves out a consequential part of the setup. Leave room for the operating system and the applications you normally keep open.

[LM Studio’s requirements](https://lmstudio.ai/docs/app/system-requirements) recommend at least 16 GB RAM on Mac and Windows. Its Mac guidance allows that 8 GB Apple Silicon machines may work with smaller models and modest context sizes. Those are application recommendations, not a promise that a particular model will meet your response-time needs.

Model size has several meanings. **Parameters** are the learned numerical values in the model. **Quantization** stores those values in a more compact numerical representation; a 4-bit version generally needs less weight storage than a 16-bit version. It can also change the model’s behavior, so record which artifact you used. Parameter count alone does not tell you the download size or total working memory.

**Context** is the material available during a response: instructions, messages, supplied documents, tool descriptions and returned results. Processing it needs memory beyond the weights. [Ollama documents that increasing the configured context increases memory requirements](https://docs.ollama.com/context-length). A model’s advertised maximum is therefore a ceiling to understand, rather than a sensible default for every laptop.

Start with the documents and tools needed for the chosen job. If the setup struggles, inspect memory pressure and the actual context allocation before changing several settings at once. Reduce unnecessary material, try a smaller supported artifact, or narrow the job. If useful performance still requires a larger machine, a server you control or hosted assistance is an option; it changes where the data goes.

## Where LM Studio, Ollama and LocalAI fit

These products overlap. The useful distinction is how you want to work with them and which parts of the application you want to manage.

### LM Studio: start in a desktop interface

LM Studio is a candidate when you want to load a model and explore a document task through an application. Its [document guide](https://lmstudio.ai/docs/app/basics/rag) explains how attached material is supplied to the model, using retrieval for longer documents. Retrieval selects relevant passages for a question; those passages become evidence for the answer. Check that the evidence is actually sufficient.

It also supports [local and remote Model Context Protocol (MCP) servers](https://lmstudio.ai/docs/app/mcp). MCP is a standard way for an application to connect to tools and data sources. A local model can use such a connection, but the server or its actions may still contact another service.

The same documentation warns that some servers consume substantial context. Start with the tools needed for one job. Our [article on agent skill bloat](https://www.humanstoknow.com/blog/ai-agent-skill-bloat) explains why the instructions and choices surrounding an agent deserve attention alongside model size.

### Ollama: serve a model to your application

Ollama is a candidate when you want to build or connect an application around a model service. Its [agent-loop example](https://docs.ollama.com/capabilities/tool-calling#multi-turn-tool-calling-agent-loop) shows repeated model requests with tool results added between turns.

That gives a developer control over what executes and what gets recorded. It also leaves work to do: implementing the tools, validating arguments, handling failures and choosing a stopping rule. An application using Ollama must support the chosen model’s tool behavior; a successful connection by itself does not prove the workflow works.

### LocalAI: investigate an integrated agent path

LocalAI’s current [first-agent guide](https://localai.io/docs/getting-started/first-agent/index.html) describes LocalAGI embedded in LocalAI. The documented route assigns a tool-calling model, creates an agent and configures actions. This makes it a candidate when you want the model service and agent interface available together.

Inspect the actions as carefully as the model. An integration can require credentials and a reachable service, so its success depends on more than inference.

## Choose a model for the input and the task

Once you have a candidate path, check the model card and the runtime’s current support. Does the checkpoint accept the input you have? Does the integration support its tool format? Are there intended-use limits or settings that matter for this task?

Liquid’s [LFM3-1.7B card](https://huggingface.co/LiquidAI/LFM3-1.7B) describes a text model for local agents, tools and specialized work, while advising against agentic coding and complex multi-step reasoning. [Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) supports text and image inputs. That makes a difference when your document is a scan, although image support alone does not establish accurate extraction.

Our [LFM3 versus small Qwen comparison](https://www.humanstoknow.com/blog/lfm3-vs-qwen-local-agents) goes deeper into those choices. Keep the exact checkpoint, quantization and reasoning settings attached to any result you record. “Qwen worked” is too vague to help you reproduce a useful configuration.

## Verify one complete workflow with known answers

Use the following sample as a small rehearsal before pointing an agent at a real folder. These are invented workshop notes, with expected outcomes worked out in advance. They are not results from running any of the products above.

| Sample file | Booking information | Expected finding |
| --- | --- | --- |
| `ceramics.txt` | 18 booked; room capacity 16 | Over capacity by 2. |
| `sketching.txt` | 10 booked; room capacity 12 | Within capacity, with 2 places remaining. |
| `photography.txt` | 7 booked; capacity not recorded | Cannot determine whether it exceeds capacity. |

Ask: “Which workshops need attention? Identify the source file, explain the capacity check, and flag missing information.” The first task ends with a report. It does not change a booking or send a message.

For a tool-using version, expose a reader limited to these sample files and a capacity-check function. The model identifies the relevant information; application code validates the extracted counts and performs the arithmetic. A missing capacity must remain missing rather than becoming zero or a guessed default. Keep the tool requests and results so you can inspect the path to the answer.

Then change one condition. Make a file unavailable and check whether the agent reports the failure instead of treating the missing file as an empty booking. Ask about a fourth workshop that is absent. Restore the original inputs and repeat the original question. A single clean answer can conceal inconsistent behavior.

Set a finite tool-call limit and a time limit before the rehearsal. Stop when the work is complete. If a limit is reached while work remains, report what is unfinished. Choose the limits around the task and the wait you can accept.

Record the model and runtime versions, artifact, context setting, computer, total time and expected versus observed outcome. Count the work needed to correct the result. Reading the wrong file quickly is not an improvement; neither is a correct answer that requires more checking than the original task.

Use the failure to choose the next change. If the relevant passage never reaches the model, first inspect file loading or retrieval. If a requested tool never runs, inspect the integration. If the model still misreads adequate evidence after checking the instructions, revisit model fit. Increasing model size will not repair a missing file or a disconnected tool.

If you use a reusable skill to guide the process, inspect its instructions and source before adding it. Our [skill evaluation checklist](https://www.humanstoknow.com/blog/choose-ai-agent-skills) helps with that review. A written playbook can guide an agent, but the application still needs the tools and capabilities it assumes.

## Trace what stays local and what uses the network

“Local” can describe different parts of the same setup. An on-device model runs inference on your computer. A self-hosted model might run on another machine on your network. A hybrid application can combine local generation with a hosted service. Name the arrangement explicitly when deciding whether it suits your work.

Trace the journey of one document. Where is it parsed? If retrieval uses **embeddings**, numerical representations for finding related passages, where are those computed and stored? Which model receives the retrieved text? Can a tool send that text to another service? Check the destinations configured in the application, including any fallback model.

[LM Studio documents offline operation](https://lmstudio.ai/docs/app/offline) for downloaded-model chats, document processing and local serving. Model discovery, downloads and update checks need connectivity. Its remote MCP connections have their own network requirements. Likewise, [Ollama’s FAQ](https://docs.ollama.com/faq) documents cloud features and how to disable them; installing Ollama locally does not tell you where an application’s selected model runs.

After downloading the necessary components, disconnect and repeat the small workflow if offline operation matters. This checks whether that configured task can finish without a connection. It cannot establish what the application sends while connected or during other tasks. Use the documented data flows and settings for those questions.

## Make the first choice small enough to evaluate

Write down the job, the proposed setup and the result you will accept. Choose one of the paths above and keep the first inputs small enough to inspect. Let evidence from that trial determine the next change.

The workshop rehearsal has a clear finish: identify ceramics as over capacity, leave photography unresolved, and show the source for both. Once you can follow how the setup reaches that result, you have a useful basis for trying more work on the same computer.