Category / Uncategorized

by Thomas

Role Assignment in Multi-Agent Systems

When working with multi-agent systems, one of the most powerful concepts you can leverage is role assignment. In a multi-agent setup, you can define distinct roles for each agent to create different behaviors, allowing them to collaborate, interact, and solve problems in a simulated environment.

Imagine you’re managing a software development project. You have a project manager, a developer, and a tester, each with a unique perspective and responsibilities. By assigning these roles to different agents in a conversation, you can simulate their interactions to observe how they work together toward a common goal, like completing a feature or identifying a bug.

Why Use Role Assignment?

Role assignment is essential in multi-agent systems because it allows you to create more realistic, diverse behaviors in the simulation. Each agent has specific tasks, which means they’ll react differently based on their role. For example:

  • The project manager might focus on project timelines, priorities, and coordinating tasks.
  • The developer could be focused on writing code, debugging, and creating new features.
  • The tester would be identifying bugs, running test cases, and ensuring the quality of the product.

By assigning different roles, you give each agent context and a purpose, which leads to more meaningful interactions.

How to Assign Roles in the OpenAI Chat API

Using the OpenAI API Documentation, assigning roles is simple. You can use system messages to define the specific behavior of each agent. These messages help guide each agent’s response and ensure that they act within their role.

Here’s how you can structure it:

import openai

openai.ChatCompletion.create(model="gpt-3.5-turbo",
  messages=[

    {
      "role": "system", 
      "content": "You are the project manager for a software development team. Your role is to coordinate tasks, set deadlines, and ensure the project stays on track. Focus on the big picture and team collaboration."
    },

    {
      "role": "system", 
      "content": "You are a developer working on new features and fixing bugs. Focus on writing clean code, debugging, and offering technical solutions to problems."
    },

    {
      "role": "system", 
      "content": "You are a tester responsible for finding bugs and ensuring that the software is stable. Run tests, identify issues, and communicate them clearly for the team to address."
    },

    {
     "role": "user",
     "content": "Let's start the project. The first task is to build the user authentication feature."
    }
  ]
)
In this example:
Note: Don’t be confused by the API role and the role you define

Don’t be confused by the “role” in the API message (e.g., system, user, assistant) and the “role” you define for each agent (e.g., project manager, developer, tester). In the API context, “role” refers to the message sender (system, user, assistant), while in the agent context, “role” refers to the specific persona or responsibility the agent has within the conversation.

In this example:

  • The project manager agent is given a message to manage the project, prioritizing tasks and deadlines.
  • The developer agent is tasked with coding and troubleshooting technical challenges.
  • The tester agent focuses on testing and identifying bugs to ensure a stable product.

Each agent’s system message helps them understand their role and contributes accordingly to the conversation, creating a collaborative environment that mirrors real-world project dynamics.

Why It Works

The power of multi-agent systems comes from the interaction between agents with different roles. When agents understand their role and objectives, they can communicate more effectively, mimic real-world collaborations, and help identify solutions more efficiently. You can also test various scenarios to see how different roles react to challenges or changes in the system, all without human intervention.

Wrapping Up

Role assignment in multi-agent systems is a powerful way to simulate complex scenarios with diverse behaviors. By using system messages to define roles, you can create agents that act like real-life colleagues, each contributing in their own way to achieve the common goal. Whether you’re simulating a team of developers or testing a new feature, this approach brings both flexibility and realism to the table.

Next time you’re working with multi-agent systems, try assigning different roles to your agents. You might be surprised at how dynamic and engaging the conversation becomes!

For more information on how to implement these concepts, be sure to check out the OpenAI API Documentation, where you can explore further examples, code snippets, and more to help you make the most of the Chat API in your projects.

by Thomas

How to Analyze a Dataset for LLM Fine Tuning

Say you have an LLM and want to teach it some behavior and therefore your idea is to fine tune an LLM that is close and good enough. You found a dataset or two, and now want to see how training the LLM on this dataset would influence its behavior and knowledge.

Define the Objective

What behavior or knowledge you want to instill in the LLM? Is it domain-specific knowledge, conversational style, task-specific capabilities, or adherence to specific ethical guidelines?

Dataset Exploration

Check if the dataset’s content aligns with your domain of interest. Where does the dataset come from? Ensure it is reliable and unbiased for your use case.

Evaluate the dataset size to see if it is sufficient for fine-tuning but not too large to overfit or be computationally prohibitive. Check the dataset format (e.g., JSON, CSV, text) and its fields (e.g., prompt-response pairs, paragraphs, structured annotations).

Content

Quality: Ensure the text is grammatically correct and coherent, code is working. Check for logical structure and factual accuracy.

Diversity: Analyze the range of topics, styles, and formats in the dataset. Ensure the dataset covers edge cases and diverse scenarios relevant to your objectives.

Look for harmful, biased, or inappropriate content. Assess the dataset for compliance with ethical and legal standards.

Behavior

Use a small subset of the dataset to run experiments and assess how the model’s behavior shifts. Compare the outputs before and after fine-tuning on metrics like relevance, correctness, and alignment with desired behaviors.

Compare the dataset’s content with the base model’s knowledge and capabilities. Focus on gaps or areas where the dataset adds value.

TLD;DR: Train with a small subset and observe how it changes behavior.

Data Cleaning

Normalize text (e.g., casing, punctuation) and remove irrelevant characters. Tokenize or prepare the dataset in a format compatible with the model.

Remove low-quality, irrelevant, or harmful samples. In fact, many of the datasets used to train large LLMs are not very clean. Address bias and ethical issues by balancing or augmenting content as needed. Add labels or annotations if the dataset lacks sufficient structure for fine-tuning.

Resource Estimate

Determine the compute power required for fine-tuning with this dataset. f the dataset is too large, consider selecting a high-quality, representative subset.

Alternative Approaches: Evaluate whether fine-tuning is necessary. Explore alternatives like prompt engineering or few-shot learning.

Ethical and Practical Validation

Use tools or frameworks to check for potential biases in the dataset. Ensure the dataset complies with copyright, privacy, and data protection regulations.

Add Notes

Document findings about dataset quality, limitations, and potential biases. Record the preprocessing steps and justification for changes made to the dataset.

By following this structured analysis, you can determine how fine-tuning with a particular dataset will influence an LLM and decide on the most effective approach for your objectives.

Note that knowledge from training and fine tuning can be blurry, so make sure you augment it with a RAG to get sharper responses. I’ll show how to do that in another blog post.

by Thomas

How to build an AI Agent with a memory

How to Build an Agent with a Local LLM and RAG, Complete with Local Memory

If you want to build an agent with a local LLM that can remember things and retrieve them on demand, you’ll need a few components: the LLM itself, a Retrieval-Augmented Generation (RAG) system, and a memory mechanism. Here’s how you can piece it all together, with examples using LangChain and Python. (and here is why a small LLM is a good idea)

Step 1: Set Up Your Local LLM

First, you need a local LLM. This could be a smaller pre-trained model like LLaMA or GPT-based open-source options running on your machine. The key is that it’s not connected to the cloud—it’s local, private, and under your control. Make sure the LLM is accessible via an API or similar interface so that you can integrate it into your system. A good choice would be using Ollama and an LLM such as Googles gemma. I also wrote easy to follow instructions in how to set an T5 LLM from Salesforce up locally, but it is also perfectly fine to use a cloud-based LLM.

In case the agent you want to build is about source code, here is an example of how to use CodeT5 with LangChain.

Step 2: Add Retrieval-Augmented Generation (RAG)

TL;DR: Gist on Github

Next comes the RAG. A RAG system works by combining your LLM with an external knowledge base. The idea is simple: when the LLM encounters a query, the RAG fetches relevant information from your knowledge base (documents, notes, or even structured data) and feeds it into the LLM as context.

To set up RAG, you’ll need:

  1. A Vector Database: This is where your knowledge will live. Tools like Pinecone, Weaviate, or even local implementations like FAISS can store your data as embeddings.
  2. A Way to Query the Vector Database: Use similarity search to find the most relevant pieces of information for any given query.
  3. Integration with the LLM: Once the RAG fetches data, format it and pass it as input to the LLM.

I have good experience with LangChain and Chroma:

documents = TextLoader("my_data.txt").load()
texts = CharacterTextSplitter(chunk_size=300, chunk_overlap=100).split_documents(documents)
vectorstore = Chroma.from_documents(texts, OllamaEmbeddings(model="gemma:latest")).as_retriever()

llm = OllamaLLM(model=model_name)
qa_chain = RetrievalQA.from_chain_type(llm=llm, retriever=vectorstore)

qa_chain.invoke("What is the main topic of my document?")

Step 3: Introduce Local Memory

Now for the fun part: giving your agent memory. Memory is what allows the agent to recall past interactions or store information for future use. There are a few ways to do this:

  • Short-Term Memory: Store conversation context temporarily. This can simply be a rolling buffer of recent interactions that gets passed back into the LLM each time.
  • Long-Term Memory: Save important facts or interactions for retrieval later. For this, you can extend your RAG system by saving interactions as embeddings in your vector database.

For example:

  1. After each interaction, decide if it’s worth remembering.
  2. If yes, convert it into an embedding and store it in your vector database.
  3. When needed, retrieve it alongside other RAG data to give the agent a sense of history.

Langchain Example

from langchain.memory import ConversationBufferMemory

# Initialize memory
memory = ConversationBufferMemory()

# Save some conversation turns
memory.save_context({"input": "Hello"}, {"output": "Hi there!"})
memory.save_context({"input": "How are you?"}, {"output": "I'm doing great, thanks!"})

# Retrieve stored memory
print(memory.load_memory_variables({}))

Step 4: Put It All Together

Now you can combine these elements:

  • The user sends a query.
  • The system retrieves relevant data via RAG.
  • The memory module checks for related interactions or facts.
  • The LLM generates a response based on the query, retrieved context, and memory.

This setup is powerful because it blends the LLM’s generative abilities with a custom memory tailored to your needs. It’s also entirely local, so your data stays private and secure.

Final Thoughts

Building an agent like this might sound complex, but it’s mostly about connecting the dots between well-known tools. Once you’ve got it running, you can tweak and fine-tune it to handle specific tasks or remember things better. Start small, iterate, and soon you’ll have an agent that feels less like software and more like a real assistant.

by Thomas

Analyzing an LLM dataset for non-data scientists

Large Language Models (LLMs) have become increasingly important for tasks involving natural language processing (NLP). However, their effectiveness hinges on the quality of the datasets used for training and evaluation. While data scientists typically handle the intricacies of these datasets, there are several reasons why non-data scientists, such as developers, project managers, or domain experts, might also need to engage in this process.

Why Analyze an LLM Dataset?

Understanding and analyzing an LLM dataset is essential for several reasons:

  1. Ensuring Model Quality: The performance of an LLM is directly tied to the quality of its training data. By analyzing the dataset, you can identify any potential issues, such as imbalances, biases, or irrelevant data that might negatively impact the model’s output.
  2. Bias Detection and Ethical Considerations: Datasets can inadvertently contain biases that lead to unfair or unethical outcomes. For example, if the training data over-represents certain demographic groups, the LLM might produce biased results. Analyzing the dataset allows you to spot these issues early and address them before the model is deployed.
  3. Customizing for Specific Needs: Not all datasets are created equal. Depending on your application, you might need to fine-tune the LLM on data that is more relevant to your domain. Analyzing the dataset helps you understand its strengths and weaknesses, guiding the fine-tuning process.
  4. Compliance and Documentation: In regulated industries, it’s crucial to ensure that your data practices are compliant with laws and regulations, such as GDPR. Analyzing the dataset is a necessary step in auditing and documenting the data to meet these requirements.

What to Look for in a Dataset

When you’re tasked with analyzing an LLM dataset, focus on these key aspects:

  • Data Distribution: Check if the data covers all relevant categories and is evenly distributed across them. Imbalances can lead to biased models.
  • Quality and Relevance: Assess the quality of the data—look for noise, duplicates, or irrelevant entries that could skew results.
  • Representation of Sensitive Attributes: Pay attention to how sensitive attributes (e.g., race, gender) are represented to avoid introducing bias.
  • Coverage of Domain-Specific Content: Ensure that the dataset contains sufficient examples related to the specific language, terminology, or context relevant to your application.

Practical Steps

  1. Data Profiling: Start with basic profiling to understand the dataset’s structure, including the distribution of data points, missing values, and outliers.
  2. Bias Auditing: Use statistical methods to detect any biases. Simple checks like comparing distributions across different demographic groups can reveal potential issues.
  3. Domain Relevance Check: Evaluate whether the dataset includes enough examples relevant to your specific use case, and consider augmenting it with additional data if necessary.

Conclusion

While data scientists usually handle the heavy lifting of dataset analysis, non-data scientists can play a crucial role in ensuring that an LLM performs well and behaves ethically. By engaging in dataset analysis, you not only improve the model’s quality but also help safeguard against potential biases and compliance issues. This approach ensures that the AI systems you contribute to are both effective and responsible.

by Thomas

Large vs Small LLMs – Thoughts

If you are working on a task that is very specific, a smaller LLM may be able to learn the task-specific patterns more quickly than a larger LLM. Additionally, if you are working on a resource-constrained device, a smaller LLM may be the only option. Read in this blog post how to prepare an LLM for a specific task.

Benefits of large LLMs, such as 70B

Large language models (LLMs) with more parameters are typically trained on larger datasets. The more parameters an LLM has, the more complex it is, and the more data it can process. This is because the parameters represent the connections between the neurons in the LLM’s neural network. The more parameters there are, the more connections there are, and the more complex the network can be.

Benefits of smaller LLMs, such as 6B or 770m

If I have a task that requires Python, I don’t need a model trained on Haskell, GO and Rust. It is not necessary to use a model that is trained on other programming languages. This is because LLMs that are trained on a variety of programming languages can often overfit to the training data, which can make them less effective for generating code in a specific language.

An LLM that is trained on a large dataset of Python, Haskell, Go, and Rust code may be able to generate code in all of these languages. However, it may not be as good at generating idiomatic Python code as an LLM that is specifically trained on Python code.

If you have a task that requires Python, it is generally best to use an LLM that is specifically trained on Python code. This will give you the best chance of generating code that is syntactically correct, semantically meaningful, and idiomatic.

A 6B model is significantly more convenient for many purposes: less expensive to operate, runs on your laptop, maybe more accurate on that specific language if the training data is good.

A good way to decide whether to use an LLM that is trained on multiple programming languages or an LLM that is specifically trained on one programming language is to experiment with both and see which one works better for your task.

by Thomas

Creating API Documentation

Creating API documentation is a crucial step in making your API accessible and understandable to other developers or users. Here’s a general guide on how to create API documentation:

  1. Choose a Documentation Format:
    • Decide on the format for your API documentation. Common formats include:
      • Swagger/OpenAPI: A standardized format for describing RESTful APIs. It’s machine-readable and can be used to generate interactive documentation.
      • Markdown: A lightweight, human-readable format often used for creating static API documentation.
      • HTML or PDF: You can create static HTML or PDF documents to document your API.
      • API Documentation Tools: Consider using dedicated API documentation tools like Swagger, Postman, or API Blueprint, which often have built-in documentation features.
  2. Define API Endpoints and Methods:
    • List all the endpoints, methods (GET, POST, PUT, DELETE, etc.), and their purposes. This serves as an outline for your documentation.
  3. Document API Endpoints:
    • For each endpoint, provide detailed information, including:
      • Endpoint URL: The URL or path for the endpoint.
      • HTTP Method: The HTTP method used (e.g., GET, POST, PUT, DELETE).
      • Parameters: List any query parameters, request headers, or request body parameters.
      • Responses: Describe the possible HTTP responses, including status codes and response bodies.
      • Authentication: Explain any authentication or authorization requirements for the endpoint.
      • Example Requests and Responses: Provide real-world examples of how to make requests and interpret responses.
      • Error Handling: Document how errors are handled and returned to the client.
  4. Add Code Samples:
    • Include code samples in various programming languages to show how developers can interact with your API. These code samples should cover common use cases.
  5. Provide Interactive Examples (if possible):
    • If using Swagger or a similar tool, you can create interactive documentation that allows users to make API requests directly from the documentation page.
  6. Explain Authentication and Authorization:
    • Clearly explain how users can authenticate themselves to access the API and any required API keys, tokens, or OAuth2 flows.
  7. Include Rate Limiting and Usage Policies:
    • If applicable, specify rate limiting policies and usage guidelines for your API.
  8. Add Versioning Information:
    • Include information about API versioning, especially if your API may undergo changes or updates over time.
  9. Add FAQs and Troubleshooting:
    • Address common questions and provide guidance on troubleshooting common issues users may encounter.
  10. Style and Consistency:
    • Maintain a consistent style and formatting throughout your documentation. Use headings, bullet points, and clear language to make the content easy to read and navigate.
  11. Host Your Documentation:
    • Host your API documentation on a dedicated server or platform. You can use GitHub Pages, GitLab Pages, ReadTheDocs, or other similar services to host static documentation. If you’re using Swagger or a dedicated API documentation tool, they often provide hosting options.
  12. Keep Documentation Updated:
    • Regularly update your documentation to reflect changes in your API. Outdated documentation can lead to confusion and frustration for users.
  13. Seek Feedback:
    • Encourage users and developers to provide feedback on your documentation. Address any issues or questions raised by users to improve the documentation’s quality.

Creating comprehensive and user-friendly API documentation is an ongoing process. It’s essential to keep it up-to-date and ensure it meets the needs of your API users. Good documentation can significantly improve the adoption and success of your API.

by Thomas

Time- and stress management: soft skills for software engineers

The most stressful part of being a software engineer can vary from person to person and from project to project. However, some common stressors include:

  1. Tight Deadlines: Software development often involves tight deadlines, which can put a lot of pressure on engineers to deliver their work on time.
  2. Technical Challenges: Solving complex technical problems can be challenging and stressful, especially if the solution is not obvious.
  3. Debugging and Troubleshooting: Debugging and troubleshooting code can be time-consuming and stressful, especially if the issue is not easily resolved.
  4. Dealing with Complex Codebases: Working with complex codebases can be challenging and stressful, especially if the code is poorly written or documented.
  5. Managing Changes and Updates: Keeping up with changes and updates to software and technologies can be stressful, especially if they impact the code the engineer is working on.
  6. Working with Teams: Collaborating with other engineers, designers, and stakeholders can be challenging and stressful, especially if there are differences in opinions or approach.
  7. Meeting Expectations: Software engineers often work under high expectations, both from their managers and their clients, which can be stressful.

In conclusion, the most stressful part of being a software engineer can vary, but some common stressors include tight deadlines, technical challenges, debugging and troubleshooting, dealing with complex codebases, managing changes and updates, working with teams, and meeting expectations. It is important for software engineers to have good time management and stress management skills to help them handle these stressors effectively.

How to manage time?

Effective time management is an important skill for software engineers. Here are some tips to help manage time effectively:

  1. Prioritize Tasks: Make a list of tasks and prioritize them based on their importance and urgency. Focus on the most important tasks first and work on them until they are completed.
  2. Use Time Tracking Tools: Use time tracking tools, such as timers or to-do lists, to monitor the amount of time spent on each task. This can help identify areas where time is being wasted and make adjustments accordingly.
  3. Break Tasks into Manageable Chunks: Break larger tasks into smaller, manageable chunks. This will make it easier to stay focused and avoid feeling overwhelmed.
  4. Set Realistic Deadlines: Set realistic deadlines for tasks, taking into account the time required to complete them, and any other commitments you may have.
  5. Avoid Multitasking: Multitasking can be counterproductive and reduce productivity. Focus on one task at a time and give it your full attention.
  6. Take Breaks: Regular breaks can help reduce stress and improve focus. Take a break every hour or so and step away from the computer.
  7. Avoid Distractions: Minimize distractions, such as email, social media, and instant messaging, by turning off notifications or scheduling specific times to check them.
  8. Delegate Tasks: Delegate tasks to others when possible. This can help reduce workload and free up time to focus on more important tasks.

In conclusion, effective time management is an important skill for software engineers. Prioritizing tasks, using time tracking tools, breaking tasks into manageable chunks, setting realistic deadlines, avoiding multitasking, taking breaks, avoiding distractions, and delegating tasks can all help manage time effectively.

by Thomas

git clean

The commands to clean your git repo from untracked files. A clean repo is just nicer to work with, and you can see the status much better.

git clean -f        # delete untracked files
git clean -f -d     # delete untracked directories

git clean -i        # interactive

If unsure, use the interactive mode. It will ask you what to do.

Also exists:

git clean -n # dry run
git clean -f -x # remove untracked .gitignore files

by Thomas

Django, apache2, wsgi hangs on importing a python module

While working on my Python project which runs on an Ubuntu server using Apache, I carelessly added the sklearn package using pip and my REST api stopped working. I spent some time debugging apache, which didn’t write anything to error.log nor access.log. Hard to find, I had to use the good old print() debug method and placed them all over the Django wsgi script.

Since the unit tests ran, it must have been a difference between console python and wsgi. I realized it was an import statement in my Django views.py that caused the hang. And this solved it (in apache.conf):

WSGIApplicationGroup %{GLOBAL}

The module that hung on import was TensorFlow. Might be it doesn’t work running it in a sub interpreter at this time.

by Thomas

root symlink on MacOS BigSur

After upgrading my Macbook to BigSur (I skipped Catalina) my MongoDB didn’t want to start anymore. A quick inspection showed that the /data/db folder I used to have to house my db files didn’t exist anymore. BigSur has removed it 😱 but luckily I found a backup.

It appears that the root / folder on MacOS is now readonly, and I wasn’t able to copy the backup to its original place. I chose the Mac disk instead, and it now sits in /System/Volumes/Data/data

What was left is to create a symlink to /data, and the way to do that is by using the /etc/synthetic.conf file:

data    /System/Volumes/Data/data

This fixes the /data folder I used to have for MongoDB.