When Palm Bay, Fla., first started looking into AI, a chatbot wasn’t the goal. The city was trying to answer a more fundamental question: What would it take to use generative AI without handing its information over to a commercial AI service?
The question began taking shape in 2025, when Palm Bay began weighing what it could gain — and what it could risk — by relying on commercial AI platforms. Public AI tools can send information outside a city’s own environment, raising concerns about where government data goes and who ultimately has access to it.
“We began developing our internal AI infrastructure (hardware and software) in early 2025 after evaluating risk, cost, and limitations of commercial platforms,” said Rob Beach, director of information technology for Palm Bay.
And by early 2026, its AI research had taken the city somewhere it hadn’t initially expected, “investigating the possibility of a fully self-hosted chatbot solution,” according to Beach. The city built its own generative AI chatbot, running on city-owned infrastructure and replacing a commercial chatbot Palm Bay had previously subscribed to.
“We felt we needed a solution that avoided public AI dependency, ensured no unintentional exposure of internal data, eliminated recurring subscription costs, and gave us a pathway to responsible AI governance,” Beach said.
The chatbot does give residents a way to ask questions, find policies, pull up department-specific information and retrieve details from city calendars and staff directories. But Palm Bay wasn’t trying to build another general-purpose AI assistant. The city wanted a system that could answer questions using information it had selected, reviewed and approved.
For Palm Bay, that meant governance started with what the system was allowed to know. Much of that approach depends on a technology called retrieval-augmented generation, or RAG. Before the AI generates a response, the system searches collections of approved city information, including curated departmental documents, staff directory information, calendars and other vetted material.
Palm Bay gets to decide what goes into those collections — and, just as importantly, what doesn’t. And it gives the city a defined set of information for the chatbot to draw from when answering questions.
When someone asks about a city-specific topic, the system searches those approved sources first and builds its response from what it finds there. It isn’t supposed to fill gaps by reaching into the model’s broader knowledge and guessing at information that isn’t in Palm Bay’s own materials.
“Importantly, the system searches the database before the [large language model] LLM runs, ensuring responses come exclusively from approved sources rather than the model’s latent training data,” Beach said.
It’s a distinction that gives the city a tighter grip on accuracy, but also helps address another concern that comes with AI: drift. AI responses can change as models, systems or underlying information change, but Palm Bay’s method anchors the chatbot to a curated set of city information that can be updated and maintained by the city.
“Drift is mitigated by grounding all responses in retrieval-augmented generation, meaning the system does not rely on the model’s internal memory but instead uses current, curated documents for every answer,” Beach said. “Additional safeguards include input sanitization, jailbreak screening, output leak-scrubbing, and routing logic that bypasses the LLM entirely when an FAQ match is found.”
But keeping that level of control doesn’t stop at the chatbot itself. It extends all the way down to the technology running underneath it. Officials declined to disclose the specific components of its internal infrastructure, citing cybersecurity concerns and Florida law, but did say its AI stack is built from vetted open-source tools assembled in a tightly managed environment.
And unlike a commercial AI service, those components run on infrastructure the city owns. That gives Palm Bay control over its data, the information retrieved for each response and the answers the system generates. But getting to that point has not been without its problems.
“We have navigated challenges related to designing a secure internal environment, avoiding hallucinations, ensuring correct intent routing, and scaling infrastructure to support multi-stage inference capabilities,” Beach said. “The city addressed these through implementing a layered architecture, strict governance practices, containerized infrastructure, and direct curation of all retrievable content.”
That is also where Palm Bay sees the value in building rather than buying. A commercial chatbot could provide the interface, but the city’s self-hosted approach gives it control over what sits behind that interface — where information goes, what the system can retrieve and how responses are governed.
It also gives Palm Bay something it can keep building on. Rather than starting over each time it wants to add another AI capability, the city has an infrastructure foundation already in place.
“As governance and content maturity progress, we intend to commit to a phased expansion including wider resident-facing services and broader citywide use,” Beach said.
For now, that means expanding internal knowledge bases, bringing more departments into the system, improving dashboards and metrics collection automations and developing additional tools to reduce staff workload. The expansion will happen in phases, as the city continues refining its governance practices and the content and retrieval systems behind the AI.
What began as a question about whether Palm Bay could build its own AI infrastructure has become something larger than a chatbot. The city now has a strategy that it can continue developing as it figures out where AI can be useful — while keeping a hand on the IT and rules that determine what happens behind the screen.
