Integrating AI in Software Development

Extending a developer's reach

Since Google researchers introduced the Transformer architecture in 2017, and OpenAI popularized conversational LLMs with ChatGPT in 2022, industries across the board have raced to figure out how to put this technology to work. Software development is no exception. Large Language Models (LLMs) have already proven their ability to enhance products and workflows in countless cases.

What follows isn't a list of tools, but a single expanding capability. AI's real gain for software development is not to do the work, but to extend what a developer can reach: into knowledge, into codebases, into repetitive tasks that used to eat a day.

Code assistants extend a developer's knowledge and hands. Domain-extension techniques extend a model's knowledge into your organization's specific context. Agents extend that reach into autonomous action. And at every step, the payoff is only real if it's paired with deliberate control.

The most immediate way AI improves software development is through code assistants - and there are two tiers to how developers use them. At the simplest level, an LLM works as a knowledge accelerator: ask it about an unfamiliar language, a framework's quirks, or how to structure a solution, and you get in seconds what used to take a search engine and three open tabs to piece together.

The bigger leap comes from integrated code assistants - IDE (Integrated Development Environment) tools like Cursor and Copilot, or CLI-based (Command Line Interface) agents like Gemini CLI and Claude Code - which give the LLM direct agency to read and modify code in your actual project, not just describe changes for you to copy in by hand. That shift, from advisor to participant, is what turns a chatbot into a genuine productivity multiplier. The same model that can explain a concept can now implement it, refactor around it, and catch the edge cases you'd have missed.

Getting real value out of either tier, though, depends on the same three things: knowing how to query the model effectively, keeping governance over what data it sees, and adapting it to your team's specific domain.

We'll cover each of those next. After that, we'll look at what happens when you give that same capability the ability to act on its own, in the section on agentic workflows.

Large Language Models

An LLM is a neural network trained on vast amounts of text to predict the next most likely word (or token) in a sequence - repeated enough times, this simple mechanism is what produces coherent explanations, working code, and multi-step reasoning. It works by taking the user's query (input) and breaking it into tokens - small units of text, often subword pieces rather than whole words - which are converted into numbers using the model's vocabulary. Those numbers flow through the model's many layers of learned parameters to produce an answer (output). The model itself is static once training is finished - it doesn't learn or update as you use it - so getting the most out of it comes down to how you shape the input you provide, in what's called the context window.

The context window is the model's entire working memory for a given interaction. Everything the model uses to generate a response - instructions, background information, prior conversation, and retrieved documents - has to fit inside it.

Understanding what goes into that window, and how it's structured, is key to writing queries that produce great responses. This is the first form of reach: how much relevant knowledge you can put in front of the model at once.

Anatomy of a query
One useful way to think about this: the LLM is an actor, and the user is the director. Before the actor can deliver a strong performance, they need the full context - who they're playing, what the scene requires, and what direction they've been given. A well-formed query provides exactly that, structured in a few distinct layers:

• System prompt - the character brief: Sets the model's role, tone, and constraints - for example, instructing it to act as a senior software architect, or to always respond in a specific format.
    
• Retrieved context - the actor's research: Domain-specific information giving the model facts it wasn't trained on or that have changed since training.
    
• Conversation history - the scene so far: Prior turns in the exchange, which the model uses to maintain continuity and avoid repeating itself or losing track of intent.
    
• User query - the line for the scene: The actual question or instruction, ideally specific enough that the model doesn't have to guess at scope or intent.

Get all four right, and the performance follows. Break complex tasks down into steps, provide examples of what the desired output could look like, and treat the first result as a rehearsal rather than a final take - iterate until the delivery is right.

On-Premises AI Governance

Most developers default to one of the major AI providers - Anthropic's Claude, Google's Gemini, or OpenAI's ChatGPT - because they offer some of the most capable models available for everyday software development. But that convenience comes with a trade-off: every prompt sent to these services, including proprietary code passed through the context window, is processed on infrastructure owned by someone else. For organizations handling sensitive IP, regulated data, or client codebases under NDA, that's often not acceptable.

The alternative is to bring the model in-house. This means giving up access to the very latest frontier models, but for most day-to-day development tasks, open-weight models like Gemma, Qwen, or Llama are more than capable of getting the job done. It also means taking on the maintenance overhead and infrastructure expertise that come with running AI systems internally - this isn't a free trade. But for the right organization, the control it buys is worth the cost: reach without control is just exposure, and this is where that principle first shows up as an infrastructure decision rather than an abstract concern.

Building the stack
A simple, scalable on-premises setup rests on three components: compute hardware, an inference engine, and a user-facing interface.

• Hardware: Nvidia with CUDA, or AMD with ROCm.
• Inference: vLLM, a highly efficient serving engine built for production workloads.
• Interface: Open WebUI, which works both as a chat interface and as an OpenAI-compatible API proxy, making it easy to slot into existing developer workflows.

This isn't just a cost or performance decision - it's a governance one. Owning the infrastructure means owning the policy around how AI is used, what data it touches, and who's accountable for it. Together, this stack gives organizations that need that level of control a credible, production-ready alternative to the big-tech providers.

Domain Knowledge

Whether using a cloud AI model or an on-premises solution, one of the first challenges is making the model work with domain knowledge. The go-to approach is to provide that knowledge directly within the context window.

A Gemma 4 model has a context window of 256K tokens; at roughly 0.75 words per token, that's about 192K words - equivalent to a 550-page book. This is fine for small scopes, but it isn't feasible for large enterprise domains.

Extending a model's reach into that scale of domain knowledge takes more deliberate architecture - three approaches, each suited to a different kind of extension.

Three ways to extend the model

• Model Context Protocol (MCP): a standardized protocol that connects AI models to external data sources, tools, and environments. It lets the model populate its context window dynamically, based on the user's query, by drawing on whatever sources of information are available to it. Servers implementing MCP expose their capabilities in an AI-friendly manner, so MCP clients - such as Open WebUI - can surface these capabilities alongside the system prompt. MCP is effectively the glue between LLMs and traditional APIs.
    
• Retrieval-Augmented Generation (RAG): a technique for actually populating the context window with domain information. It requires organizing that information in something queryable, like a vector database. RAG can be implemented either by code that intercepts the user's query and performs a traditional database search, or by letting the MCP client handle prompt assembly whenever the model requests a specific resource.
    
• Low-Rank Adaptation (LoRA): a fine-tuning approach that alters the base model's weights via a lightweight adapter, rather than the underlying model. The result is behavioural transformation and linguistic alignment - adjusting the model's output toward a more desired form, whether that's structured formatting like adherence to a JSON schema, or industry-specific jargon and dialects.

These approaches aren't mutually exclusive - most production systems combine two or three of them: MCP for live access to tools and data, RAG for grounding in static domain knowledge, and LoRA for shaping style, tone, or output format. Each comes with trade-offs, though. RAG is only as good as the retrieval quality and chunking strategy behind it, and the underlying knowledge base needs to stay current or the model starts answering from stale information. LoRA adapters, meanwhile, risk over-fitting to narrow training data and typically need to be retrained whenever the base model is updated.

Which combination makes sense depends on how fast the domain knowledge changes, how sensitive it is, and how much control the organization needs over the model's behavior - the same governance question running through every layer of this stack, and the same trade-off between reach and control introduced at the start.

Agentic Workflow

This is where AI's reach extends beyond information and into execution. Up to this point, the LLM has acted as an intelligent assistant: it answers questions, explains unfamiliar concepts, and helps developers write code. An agent takes the next step by giving that same model access to tools and the ability to perform work on a developer's behalf.

An agent is best thought of as a reasoning layer wrapped around software. Instead of exposing a fixed sequence of operations, it receives a goal, decides which tools are needed, performs those actions, evaluates the results, and continues until the task is complete or it reaches a stopping condition. The model doesn't replace traditional software engineering - it orchestrates existing systems. Version control, issue trackers, documentation, CI/CD pipelines, databases, APIs, and monitoring platforms remain the building blocks; the LLM decides how to combine them.

This changes the nature of software automation. Traditional automation follows predefined workflows where every branch has been anticipated by a developer. Agentic workflows replace some of those hard-coded decisions with reasoning. Rather than writing a script for every possible scenario, developers define objectives, provide the available tools, and establish the boundaries within which the agent is allowed to operate.

For software engineering, this creates opportunities well beyond code generation.

• Onboarding: An agent with access to internal documentation, repositories, and architecture diagrams can answer questions in context, generate learning exercises, and guide new developers through unfamiliar systems.
    
• Documentation: Rather than waiting for documentation to become outdated, an agent can periodically compare source code with existing documentation, identify inconsistencies, and draft updates for review.
    
• Incident response: When alerts are triggered, an agent can gather logs, correlate telemetry across services, identify likely causes, and present a concise incident report before an engineer has even opened the dashboard.
    
• Development workflows: An agent can investigate a failing build, reproduce the problem, identify the affected source files, implement a proposed fix, execute the test suite, and open a pull request containing both the code changes and an explanation of its reasoning. The engineer reviews and approves the result, but much of the repetitive work has already been completed.
    
• Knowledge management: Design discussions, meeting notes, incident reports, and Slack conversations often contain valuable architectural knowledge that is difficult to rediscover. Agents can continuously organize this information into searchable documentation, reducing the amount of institutional knowledge that exists only in people's memories.

Notice that none of these examples depend on the agent being exceptionally intelligent. Their value comes from extending reach. A developer no longer has to manually gather information from half a dozen systems before beginning work - the agent can assemble that context automatically. Instead of replacing engineering judgment, it removes much of the mechanical effort that surrounds it.

Keeping Control

The further AI's reach extends, the more important governance becomes. An agent that can only answer questions presents relatively little risk. An agent that can modify source code, deploy infrastructure, or interact with production systems requires carefully designed boundaries.

Those boundaries should exist at multiple levels. Agents should have access only to the tools they genuinely need, operate with the principle of least privilege, and require explicit human approval before irreversible actions such as merging code or deploying software. Every action should be observable, reproducible where possible, and logged for auditing. The objective is not full autonomy but accountable automation.

This reflects the same principle running throughout this article. Every layer - from code assistants to domain-aware systems to autonomous workflows - extends what a developer can accomplish. But each increase in reach must be matched by an equally deliberate increase in control. The technology is only valuable when both grow together.

Summing up

Integrating AI into software development is a genuine boost to productivity - not because it replaces developers, but because it extends their reach: into knowledge they'd otherwise have to look up, into codebases they'd otherwise touch line by line, into the repetitive work that used to eat the hours better spent on hard problems. That's the thread running through everything above, from a code assistant answering a syntax question to an agent opening a pull request on its own.

But reach without control doesn't stay a benefit for long. As engineers grow more reliant on AI, their familiarity with the tools they use and the code they write can quietly diminish, much like a muscle that weakens when it stops being trained. We have to be careful not to lean on AI for everything or trust it blindly, and to keep enough hands-on experience with the actual work to notice when something is wrong.

There's a real risk for a software company that hands all the heavy lifting to AI and lets its developers drift entirely into the role of architects - reviewing code and describing requirements, but rarely writing or debugging it themselves. Skills atrophy that way, and so does the judgment needed to catch what the AI gets wrong.

The tools and workflows described throughout this piece are only as good as the discipline behind how they're used. The goal isn't to let AI replace engineering judgment, but to find the balance that lets it take on the repetitive and the tedious, so people can spend more of their time on the parts of the work that actually require creativity, architecture, and hard-won expertise. That balance - reach matched with control, not the technology itself - is what will determine whether AI makes a team stronger or just makes it dependent.

As part of our Insights from an Expert series, it reflects the kind of real-world knowledge that shapes smarter decisions and stronger digital futures.

Read more Insights from an Expert here.