TheBlewish/Automated-AI-Web-Researcher-Ollama

A python program that turns an LLM, running on Ollama, into an automated researcher, which will with a single query determine focus areas to investigate, do websearches and scrape content from various relevant websites and do research for you all on its own! And more, not limited to but including saving the findings for you!

Summary

Automated-AI-Web-Researcher-Ollama is a Python program that turns a locally running LLM (via Ollama) into an automated research assistant. Given a single query, it decomposes the topic into prioritized focus areas, runs web searches and scrapes relevant pages, and saves all retrieved content with source URLs to a text file before producing a final summary. The README describes it as a prototype that builds on an earlier "Web-LLM-Assistant" project, and points to a newer academic-focused successor at github.com/TheBlewish/Academic-AI-Literature-Reviewer-Ollama.

What it is useful for

This tool is useful for engineers or researchers who want a local, automated way to gather and document information on a topic without bouncing manually between a search engine and a chatbot. It fits use cases such as:

• Background research where you want a structured record of sources and content to revisit later.
• Investigations that benefit from generating sub-questions, since the LLM creates five prioritized focus areas and can spawn new ones as it finds information.
• Workflows where you want to leave a job running and return to a file of findings, then ask follow-up questions in a built-in conversation mode.

It uses DuckDuckGo for searches and Ollama for inference, so no paid API keys are required. The trade-offs are that it is a prototype, only Linux and macOS are supported on the main branch (Windows users must use the feature/windows-support branch), and output quality depends on the chosen model. The author recommends Phi-3 variants with large context windows, such as phi3:3.8b-mini-128k-instruct or phi3:14b-medium-128k-instruct, with the default n_ctx set to 55000.

How engineers can use it

Setup follows the README:

1. Clone the repo and install dependencies:

   git clone https://github.com/TheBlewish/Automated-AI-Web-Researcher-Ollama
   cd Automated-AI-Web-Researcher-Ollama
   python -m venv venv
   source venv/bin/activate
   pip install -r requirements.txt
   

2. Install Ollama from ollama.ai and pull a supported model.
3. Edit llm_config.py to set model_name (and optionally n_ctx) to match the model you pulled.
4. Start Ollama in one terminal: ollama serve
5. Run the researcher in another: python Web-LLM.py

During a session, type @ followed by your query and submit with CTRL+D. The README's example is @What year is the global population projected to start declining?. While it runs, single-letter commands let you check status (s), view the current focus (f), pause for an LLM progress assessment (p), or quit (q). On quit, the LLM reads the full research file and writes a summary, after which you can ask follow-up questions in conversation mode. The research output, including scraped content, source URLs, and focus areas, is saved as a text file in the program directory.

The author flags this as a prototype intended for educational use, so production workloads should plan for additional testing. For Windows setup details and dependency specifics, refer to the README and requirements.txt.