Oracle VirtualBox 7.2.14

Oracle released VirtualBox 7.2.14 a couple of days ago. As I mentioned in my previous post, I was expecting this release to coincide with the the normal quarterly patch cycle.

The downloads and changelog are in the usual places.

I’ve done installations on Windows 11 and Linux Mint and both worked fine.

Vagrant

I didn’t bother updating my Vagrant boxes for the previous release because I was expecting this release. I’ve gone through the rebuilds of the boxes and they seem fine with the updated guest additions.

If you are into Vagrant you can find my builds here.

https://github.com/oraclebase/vagrant

You can see the list of Vagrant boxes I build and use here.

https://portal.cloud.hashicorp.com/vagrant/discover/oraclebase

The builds of the Vagrant boxes using Packer went fine, but I’m having some problems uploading them to Vagrant cloud at the moment. They get about half way through and fail. Hopefully I’ll get the new boxes uploaded in the next couple of days.

As normal. I’ll be doing a lot of builds over the next few days, so I’ll report back here if I have any drama. 🙂

Cheers

Tim…

Oracle VirtualBox 7.2.12

Oracle released VirtualBox 7.2.12 a couple of days ago. This comes hot on the heels of version 7.2.10, which I wrote about here.

The downloads and changelog are in the usual places.

I’ve done installations on Windows 11 and Linux Mint and both seem OK.

As with the last version, on Windows 11 I got away with a straight upgrade, but it’s worth remembering the cleanest way to upgrade on Windows is to do the following.

  • Uninstall VirtualBox.
  • Make sure all “VirtualBox Host-Only Ethernet Adapter” adapters in Device Manager had been removed.
  • Reboot.
  • Install VirtualBox using “run as administrator”.
  • Reboot.

Vagrant

I don’t think I will rebuild all my Vagrant boxes this time. I suspect we will get another new version in the next couple of weeks when the Oracle patches drop, so it seems a bit pointless to waste the time now.

If you are into Vagrant you can find my builds here.

https://github.com/oraclebase/vagrant

You can see the list of Vagrant boxes I build and use here.

https://portal.cloud.hashicorp.com/vagrant/discover/oraclebase

Cheers

Tim…

Running Large Language Models (LLMs) Locally : Some Clarification

After my recent posts on this subject I got some questions, and I thought I would use my responses to write a new post. I’m hoping this makes the situation more clear. I’ll link my other posts on this subject at the bottom, in case you want to look at them. Some of these topics have been touched on in them also.

Why bother with local models?

I covered that here. In summary it’s cost and security.

The model matters

When running LLMs locally you have a massive list of open models to choose from. Hugging Face has a list of many of them here. It’s easy to get fooled into thinking the only models worth using are the frontier models, but that’s not true. Depending on the task you are doing, some smaller models give comparable results at zero cost and reasonable speed on low powered hardware.

So how do you know which model to choose? Every time I hear someone say something positive about a model I try it. It’s very easy to repeat some previous prompts and compare the results. There is no “best” model, even with the frontier models. So much depends on what you are doing. You may find you use several models on a regular basis. The great thing about running LLMs locally is you can do this with no cost implications.

What I’ve learned is that the size of the model is not always a reflection of the quality of output. Sure the bigger models are better all-rounders, but most of the time I’m using the LLM for specific tasks, and some of the smaller models work really well. The recent release of the Google Gemma4 QAT models really demonstrates this. They are designed to run on end user kit. They use a lot less memory, but perform equivalent to much bigger models. For example, gemma4:12b-it-qat is a 12b parameter model that performs nearly as well as the gemma4:26b model, which is three times the size. They are designed for agentic processing, so using them with a coding agent is cool.

Ultimately you can only use the models your kit can cope with. Just play around. I’ll mention what I’m using a little later, but if you look at my previous posts you will see I’ve changed several times. 🙂

The prompt matters

The more specific you are, the better the result! Some people feel like this is magic and they can put very little in the prompt, then they are disappointed with the result.

Just think of it like a program specification. The more specific you make it, the less room there is for a developer to make a mistake, or forget something. LLMs are the same in this respect. Some people go as far as providing pseudo code. For some tasks you will refer to your existing code base to give extra context. The trick is to provide enough information to get the job done, but not too much so you confuse the situation. Sound familiar? As I said, it’s like a program specification.

It takes time to get a feel for this. It also varies a little depending on the model used.

If you are working on some very specific stuff, you can direct the
LLM using skills, to keep it focussed. For example I cloned the Oracle
skills (https://github.com/oracle/skills.git) repo to help with creating an
APEX app. My prompt references a subset of the skills.

Generate a new APEX app.
Skills defined under the /u01/skills/apex directory
Table metadata in the /u01/apex_code/schema.sql file
Application name : Clocking
APEX workspace : CLOCKING_WS
Required pages:
– home page
– a page display and edit staff
– a page to log start and end times for a member of staff
– navigation
– breadcrumbs
– use default security.
Place the resulting code in the /u01/apex_code directory.

In this example I referenced a schema definition in a file. You could allow an agent to connect to the database and get information from there.

Here’s an example prompt from a previous blog post, where used an agent to generate some Python code to populate a vector column in the database.

Create a Python script called “/tmp/add_vector_embeddings.py”.
The script connects to an Oracle database.
Assume a database connection string will be provided as an environment variable.
The script reads the MOVIE_QUOTE column from a table called MOVIE_QUOTES.
It uses the data from the MOVIE_QUOTE column to create a vector
embedding using the EmbeddingGemma model running under Ollama.
The Ollama base URL will be provided by an environment variable.
The MOVIE_QUOTES table is then updated, setting the MOVIE_QUOTE_VECTOR column to the vector embedding value.
This should be done for all rows in the table.

Over time I’ve got better at prompting, the way it took me some time
to learn how to Google. I’m not saying I’m a prompt God, but I’m getting
better over time.

Retrieval-Augmented Generation (RAG)

I’m going to keep this really simple, but hopefully you get the idea. 🙂

Models are trained on data, and they answer your question based on the training data they have. If you want to work on a subject that is not part of their training data, they are going to struggle. If their training data contains a lot of “questionable” stuff, they may follow a path you do not agree with. With Retrieval-Augmented Generation (RAG) you can provide the LLM with information that you believe is relevant, and it will interpret and use that information.

For example you might provide a PDF of some documentation and tell the
model/agent to use that as the basis of the answer. This way it doesn’t search the net for old rubbish. It goes straight for docs you have said are quality. You can also point it at your existing code repos, and tell it to give the answer in the style of your existing code base. That way you are likely to get an answer more pleasing to you. It’s effectively making the prompt better, and we’ve already said the prompt matters.

RAG can also include searching the internet to see if alternative information is available to augment the training data. If you use agentic workflows, searching for additional information is quite a common step.

RAG is a whole subject in its own right, but I think we’ll stop at this point.

Chat vs Agentic

Using a model in a chat is nothing like using a model from a harness
(Pi agent, Hermes Agent, OpenCode, Claude Code, OpenAI Codex etc.).
These agents take your prompt and refine it using search. They come up
with a plan, then validate the plan. Once they’ve decided what they
need to do, they do it, then check the result. Sometimes they will
throw the result away and start again. It’s an iterative process that
may take some time, but the result is generally better than a one-off
question in chat.

This iterative process uses a lot more tokens, but when you are running locally that is fine. It’s all free. That’s very different to using a subscription or API billing, where an agent can end up costing you a fortune.

I didn’t appreciate the difference between chat and the agentic approach until I tried it. Once I did it was a light bulb moment. I now rarely use a chat and instead use an agent to get an answer. This is especially true of writing code, where I actually want it to write the files. An agent can do that, but a chat session can’t.

Agents are too dangerous!

If you give an agent full access to your machine, and link it into all your services (email, chat, git, deployment pipelines) there is a possibility it will make mistakes, but you don’t have to do that. For example I might let my agent read the contents of a git repo, but tell it to write changes to another location. I can then validate them separately.

If you are giving your agent access to your database, simply set up a read-only user. It will be able to look at the metadata, but not change anything.

You can give an agent as much or as little access to your system and services, so you are in control.

What kit should I buy?

This is going to be very vague and simplistic!

I discussed kit here. The problem is it is very nuanced. It depends what you are planning to do. You can get away with a small inference machine like a mini PC or a Mac mini, or you can buy a cluster of machines. I’m mostly focussed on running stuff at home, so I’m purposely not spending a fortune.

Personally I would get a Mac Mini or a MacBook Pro. If you are in a rush I
would get a M5 MBP. If you can wait, I would wait for the M5 Mac Mini
or the M6 MBP, both of which are meant to be released soon. I will
probably get an M5 Mac Mini when it is released.

The later the M series, the better it is for AI, as the number and
quality of the GPU cores increases over the generations, so in this
case later is better. That’s not to say the older M series chips can’t
do it. They just won’t do it as well.

The real kicker for larger models is memory. If you want to run the
big models you need a lot of memory for the GPU. This is where the
Apple Silicon architecture helps…

For a Windows PC using a GPU such as a RTX card, the model runs on GPU
memory, not on the system memory. If the GPU memory is not enough, it
will run on the CPU plus the system memory, but that is going to be
way slower than running on the GPU. As a result, you need a card with
the most VRAM you can get. For home use that is currently the 5090 with 64G. With Apple Silicon, the system memory is unified (shared between the system and the GPU), so if you get a 64G Mac, you can use a lot of that memory for
running models. What a lot of people do is get a Mac Studio with 128G
RAM, which allows them to run the bigger models, or even cluster
several together. Of course, this is really expensive.

I have a mini PC at the moment. I will probably end up getting a M5 mac mini with 32G RAM (if they don’t limit it to 24G), or a low end Mac Studio with 64G.

My experience of using the mini PC is pretty good for coding. I can
ask regular questions and it responds quickly. For a coding agent,
such as Pi Agent, it is surprisingly fast. I was expecting it to be
way worse. 🙂 It all depends on the model selection.

Remember, what I need and what you need may not be the same thing. Don’t take what I do as a recommendation for your requirements.

What models am I using now?

I’m currently using the Gemma4 (Google) models most of the time. The smaller ones are quicker, but the bigger ones give better output.

  • gemma4:e2b
  • gemma4:e4b
  • gemma4:12b

The normal 12b model was a little heavy. It worked, but it was a bit
sluggish. They now have these new versions using Quantization-Aware Training (QAT), which are smaller and give much better results.

  • gemma4:e2b-it-qat
  • gemma4:e4b-it-qat
  • gemma4:12b-it-qat

I now use “gemma4:12b-it-qat” for almost everything and it is really
good. I’m not sure I will use bigger models when I get better kit. I
will probably use the same ones, but get even speedier results.

Check out these posts from Google.

Previous posts

Here are some of the previous posts I’ve written about running LLMs locally.

Thoughts

I’m not a LLM or AI guy. I don’t have all the answers. I’m just a few weeks ahead of some of you, and a few years behind others. 🙂

Cheers

Tim…

Oracle VirtualBox 7.2.10

Oracle released VirtualBox 7.2.10 a couple of days ago.

The downloads and changelog are in the usual places.

I’ve done installations on Windows 11 and Linux Mint and both went fine.

On Windows 11 I got away with a straight upgrade this time, but it’s worth remembering the cleanest way to upgrade on Windows is to do the following.

  • Uninstall VirtualBox.
  • Make sure all “VirtualBox Host-Only Ethernet Adapter” adapters in Device Manager had been removed.
  • Reboot.
  • Install VirtualBox using “run as administrator”.
  • Reboot.

Vagrant Boxes Rebuilt using Packer

As you probably know, I use Vagrant for all of my VirtualBox builds. You can see all Vagrant builds here.

https://github.com/oraclebase/vagrant

These builds require a suitable Vagrant box. I build those using Packer. The OL8, OL9 and OL10 boxes have been updated, and include the 7.2.10 guest additions. The Packer builds are defined here.

The boxes are all pushed up the Vagrant Cloud, so the relevant box will be downloaded automatically if you run one of the builds based on Oracle Linux.

https://portal.cloud.hashicorp.com/vagrant/discover/oraclebase

Thoughts

Having built some boxes using Packer, and run some Vagrant builds using these boxes, everything looks good from my side. Have fun.

Cheers

Tim…

AI Vector Search : Generating vectors outside of the database using Ollama and EmbeddingGemma

I’ve posted on social media a few times about various open weight models released by Google. A recent post resulted in this question.

“how to store and use an gemma 4 model from oracle database?”

I gave this answer.

“You would have to use the EmbeddingGemma model. Then programmatically generate the vector embeddings outside of the database and insert them into vector columns in your database.”

I figured I would do an example of this.

Setup

I’m running Ollama on my inference box (see). The Ollama API is available on this URL on that box.

http://localhost:11434

On the same machine I have a Vagrant box running Oracle Linux 9 and Oracle AI Database 26ai. I have port forwarding open for the database port, so I can access the database using this URL on the host machine.

username/password@localhost:1521/pdb1

The virtual machine running the database can access the host system (including Ollama) on 10.0.2.2, so the Ollama API is visible on the database server using this URL.

http://10.0.2.2:11434

It probably sounds a little convoluted, but just imagine the inference box and the database box are separate machines talking to each other, which is how it would be normally.

Getting some data

I wanted to do an example similar to the one shown here. On the database VM I did the followinng.

First I pulled down some data to work with.

mkdir -p /u01/models
cd /u01/models
wget https://huggingface.co/datasets/ygorgeurts/movie-quotes/resolve/main/movie_quotes.csv?download=true -O movie_quotes.csv

Then I created a table to hold the movie quote data.

conn sys/SysPassword1@//localhost:1521/pdb1 as sysdba

create user if not exists testuser1 identified by testuser1 quota unlimited on users;
grant create session, db_developer_role, create mining model to testuser1;

create or replace directory model_dir as '/u01/models';
grant read, write on directory model_dir to testuser1;


conn testuser1/testuser1@//localhost:1521/pdb1

drop table if exists movie_quotes purge;

create table movie_quotes as
select movie_quote, movie, movie_type, movie_year
from   external (
         (
           movie_quote  varchar2(400),
           movie        varchar2(200),
           movie_type   varchar2(50),
           movie_year   number(4)
         )
         type oracle_loader
         default directory model_dir
         access parameters (
           records delimited by newline
           skip 1
           badfile model_dir
           logfile model_dir:'moview_quotes_ext_tab_%a_%p.log'
           discardfile model_dir
           fields csv with embedded terminated by ',' optionally enclosed by '"'
           missing field values are null
           (
             movie_quote char(400),
             movie,
             movie_type,
             movie_year
           )
        )
        location ('movie_quotes.csv')
        reject limit unlimited
      );


desc movie_quotes
 Name                                      Null?    Type
 ----------------------------------------- -------- ----------------------
 MOVIE_QUOTE                                        VARCHAR2(400)
 MOVIE                                              VARCHAR2(200)
 MOVIE_TYPE                                         VARCHAR2(50)
 MOVIE_YEAR                                         NUMBER(4)

SQL>

I added a VECTOR column to the table.

alter table movie_quotes add (
  movie_quote_vector vector
);


desc movie_quotes
 Name                                      Null?    Type
 ----------------------------------------- -------- ----------------------
 MOVIE_QUOTE                                        VARCHAR2(400)
 MOVIE                                              VARCHAR2(200)
 MOVIE_TYPE                                         VARCHAR2(50)
 MOVIE_YEAR                                         NUMBER(4)
 MOVIE_QUOTE_VECTOR                                 VECTOR(*, *)

SQL>

Now I have the data it’s time to write some code to generate the vector embeddings externally.

Writing the code (or not)

I figured I might as well use an AI to generate it, so I used a Pi agent to write the code using the “gemma4:12b-it-qat” model. On the inference box I launched the agent.

ollama launch pi --model gemma4:12b-it-qat

This is the prompt I gave the agent.

Create a Python script called "/tmp/add_vector_embeddings.py".
The script connects to an Oracle database.
Assume a database connection string will be provided as an environment variable.
The script reads the MOVIE_QUOTE column from a table called MOVIE_QUOTES.
It uses the data from the MOVIE_QUOTE column to create a vector embedding using the EmbeddingGemma model running under Ollama.
The Ollama base URL will be provided by an environment variable.
The MOVIE_QUOTES table is then updated, setting the MOVIE_QUOTE_VECTOR column to the vector embedding value.
This should be done for all rows in the table.

A couple of minutes later it produced the “/tmp/add_vector_embeddings.py” file with the following contents.

import os
import json
import requests
import oracledb
from typing import List

def get_embeddings(text: str, model: str, base_url: str) -> List[float]:
    """Fetch vector embedding from Ollama."""
    response = requests.post(
        f"{base_url}/api/embeddings",
        json={
            "model": model,
            "prompt": text
        }
    )
    response.raise_for_status()
    return response.json()["embedding"]

def main():
    db_connection_string = os.getenv("DB_CONNECTION_STRING")
    ollama_base_url = os.getenv("OLLAMA_BASE_URL", "http://localhost:11434")
    model_name = "EmbeddingGemma"

    if not db_connection_string:
        print("Error: DB_CONNECTION_STRING environment variable is not set.")
        return

    try:
        # Connect to Oracle Database
        conn = oracledb.connect(dsn=db_connection_string)
        cursor = conn.cursor()
        print("Connected to Oracle database successfully.")

        # Fetch all quotes
        cursor.execute("SELECT rowid, MOVIE_QUOTE FROM MOVIE_QUOTES")
        rows = cursor.fetchall()
        print(f"Found {len(rows)} rows in MOVIE_QUOTES table.")

        for rowid, quote in rows:
            if quote:
                try:
                    # Generate embedding
                    embedding = get_embeddings(quote, model_name, ollama_base_url)

                    # Update the vector column using rowid for precision
                    cursor.execute(
                        "UPDATE MOVIE_QUOTES SET MOVIE_QUOTE_VECTOR = :1 WHERE rowid = :2",
                        [json.dumps(embedding), rowid]
                    )
                    conn.commit()
                except Exception as e:
                    print(f"Failed to process quote for row {rowid}: {e}")
            else:
                print(f"Skipping row {rowid} due to null MOVIE_QUOTE.")

        cursor.close()
        conn.close()
        print("Processing complete.")

    except Exception as e:
        print(f"An error occurred: {e}")

if __name__ == "__main__":
    main()

So now I just needed to run it and see what happened.

Running the code

I set up the Python environment on the database server.

python -m venv venv
source venv/bin/activate
python -m pip install oracledb requests

I set the necessary environment variables and ran the script.

export DB_CONNECTION_STRING="testuser1/testuser1@localhost:1521/pdb1"
export OLLAMA_BASE_URL="http://10.0.2.2:11434"

python /tmp/add_vector_embeddings.py

It worked. Here was the output.

Connected to Oracle database successfully.
Found 732 rows in MOVIE_QUOTES table.
Processing complete.
$

When I checked the database the vector values were populated.

select count(*) from movie_quotes where movie_quote_vector is not null;

  COUNT(*)
----------
       732

SQL>

But what about using the vectors?

At this point I realised I needed something to generate one-off vector embeddings for use in VECTOR_DISTANCE queries. First I created an ACL to allow database callouts to my Ollama service.

conn sys/SysPassword1@//localhost:1521/pdb1 as sysdba

BEGIN
  dbms_network_acl_admin.append_host_ace (
    host       => '10.0.2.2', 
    lower_port => 11434,
    upper_port => 11434,
    ace        => xs$ace_type(privilege_list => xs$name_list('http'),
                              principal_name => 'APEX_260100',
                              principal_type => xs_acl.ptype_db)); 
END;
/

Then I created a function to make a callout to Ollama to get a vector embedding of a string using the APEX_WEB_SERVICE package.

conn testuser1/testuser1@//localhost:1521/pdb1

create or replace function get_ollama_embedding (p_base_url in varchar2,
                                                 p_model    in varchar2,
                                                 p_text     in varchar2)
  return clob
as
  l_clob clob;
begin
  l_clob := apex_web_service.make_rest_request(
    p_url         => p_base_url || '/api/embeddings',
    p_http_method => 'POST',
    p_body        => '{"model": "' || p_model || '", "prompt": "' || p_text || '"}'
  );
  
  return json_query(l_clob, '$.embedding' returning clob error on error);
exception
  when others then
    dbms_output.put_line(dbms_utility.format_error_backtrace);
    return null;
end;
/

Here is an example of using this function. I won’t bother including the output as it is massive. 🙂

select get_ollama_embedding('http://10.0.2.2:11434', 'EmbeddingGemma', 'Films with motivational speaking in them');

So now we can get a vector for a search criteria, and use this in a VECTOR_DISTANCE call to return rows related to the criteria.

variable search_text varchar2(100);
variable search_vector clob;
exec :search_text := 'Films with motivational speaking in them';
exec :search_vector := get_ollama_embedding('http://10.0.2.2:11434', 'EmbeddingGemma', :search_text);

set linesize 200
column movie format a50
column movie_quote format a100

SELECT vector_distance(movie_quote_vector, to_vector(:search_vector)) as distance,
       movie,
       movie_quote
FROM   movie_quotes
order by 1
fetch approximate first 5 rows only;

If I had started with the callout function, I cloud have populated the vectors like this.

update movie_quotes
set movie_quote_vector = to_vector(get_ollama_embedding('http://10.0.2.2:11434', 'EmbeddingGemma', movie_quote));

commit;

Conclusion

If you are an Oracle developer, you will probably find using an ONNX model directly in the database easier to work with because it’s all PL/SQL. If you are not an Oracle developer, you are probably happier working outside the database, so you may want to generate your vectors through your application. Both ways work fine.

Cheers

Tim…

Warning: Be aware the code shown in this post is kind-of basic. It does not consider performance, and there isn’t much in the way of error handling. It’s just a little example of how to use the EmbeddingGemma model externally.

Running Large Language Models (LLMs) Locally : My new inference machine

I’ve written a few posts about running local LLMs.

Up until now I’ve been running them on my crappy old laptop. It’s a bit slow, but it works OK. The problem is it uses most of the local resources and I can’t do much else until the LLM finishes the task.

My original plan was to buy a M5 Mac Mini to replace my current kit, and that is still the plan, but they don’t exist yet, and I got sick of waiting, so I decided to try something else.

BOSGAME M4 Plus Mini PC

A couple of weeks ago I decided to buy a Mini PC, and use it as a dedicated inference machine. I ended up with the BOSGAME M4 Plus Mini PC. It’s a pretty cheap and simple device.

  • CPU : AMD Ryzen™ 9 7940HS
  • GPU : AMD Radeon™ 780M (integrated)
  • Memory : 32G DDR5 5600MHz
  • SSD : 1TB M.2 NVMe
  • Price : ÂŁ621 (it’s cheaper in regions other than the UK)

It’s nowhere near as good as a Mac Mini would be, but it’s about half the price of the 24G RAM and 1TB SSD version of the Mac Mini.

It comes preinstalled with Windows 11 Pro, but I got rid of that and replaced it with Linux Mint 22.3.

The Setup

After installing Linux Mint 22.3, I decided to use the Ollama and Open WebUI approach, as described here.

I installed Ollama directly on the OS, and ran Open WebUI in a Podman container.

Once the machine was setup, I unplugged everything except the power cable, and it now sits on my desk as a full time inference machine.

The Results

I need to start by managing your expectations. This is not going to blow away one of the cloud hyperscalers. It’s not going to run massive local LLMs. I still focus on smaller models to get rapid results.

Having said all that, it is a lot quicker than my existing kit was, and I was pleasantly surprised by the speed. It is very usable, and most importantly I’m not sending all my data to cloud and paying for subscriptions…

I have run a Hermes agent and OpenCode against it. The Hermes agent is kind-of slow, but it works. OpenCode was surprisingly fast. Definitely usable (for me).

Update: Someone mentioned I should try the Pi agent instead of OpenCode. It is a lot faster than the other agents I’ve tried, so I’m mostly using Pi as my harness now.

But I need more power…

I already talked about managing your expectations. I know I’m going to get someone telling me they need more…

  • I need to use frontier models! Maybe you do, but you probably don’t. I see a lot of people obsessing about being on the latest and greatest, but not actually producing anything useful. There are hundreds/thousands of open weight models out there. Some will be suitable for some of your workflows.
  • I need to orchestrate 50 agents running simultaneously! Of course you do. Shut up!
  • My company already pays for my subscriptions. That’s nice for you. It’s not the case for everyone. Judging by the recent stories, many companies are being forced on to API pricing, which is likely to 10x the cost of their current development, and start to limit their LLM usage. Maybe you will need to run local models in future.
  • My company already has private LLMs running in the cloud. That’s cool. My company is experimenting with this now also. At the moment it’s limited to chat, not agentic workflows. Maybe I’ll use this more when I can do agentic workflows with it. Maybe not if the prices keep rising.

Conclusion

As mentioned before, I still intend to buy a M5 Mac Mini when it is released. That will be my main machine, and I will no doubt do some inference on it also, but this Mini PC will happily chug away and do what I need for Local LLMs.

Cheers

Tim…

PS. I’ve also installed VirtualBox and Vagrant, so I can use it for Vagrant builds when I’m not using it for LLMs. 🙂

PPS. I accidentally put Minecraft on it… 🙂

No Agile? No DevOps? Without them your AI assisted development will fail!

I’ve been reading and watching a lot of content about AI assisted development, and without fail the people who are getting the most out of it seem to be those where agile and DevOps development are already part of their culture.

Some basic definitions

Here are some basic definitions that will help with the discussion.

Contraints : The bottlenecks in a process. The weakest link in the chain.

CI/CD : Continuous Integration / Continuous Delivery (Deployment). Basically the automation of testing and deploying code changes.

Principle of Flow : In DevOps the principle of flow is the way things flows through your organisation from the initial requirement to the finished product delivered to and used by the customer.

Value Chain : All the systems and processes that form the lifecycle of your product. Something only has value when it is in the hands of the customer. Having one team that is really productive in the middle of a broken value chain does not mean your organisation is productive.

Agile : I’m not talking about a specific methodology (like Scrum). I’m just talking about trying to live up to the ideals put forward in the Agile Manifesto. The methodology zealots ruin it for everyone. Chill out and remember why you are doing this. Just to clarify, I like agile (little “a”). I’m not a fan of Agile (big “A”).

What are your constraints?

A simple starting point is to identify your constraints. That will in turn tell you if AI assisted development will improve your productivity or not. Let’s look at some examples.

  • Requirements bottleneck : If you don’t have good customer engagement, you may find it really hard to identify valuable requirements. Of course you can always add new marketing slop features, but do you know and understand what your customers really want? If not, then having AI assisted development is pointless. Why get faster and run out of work quicker?
  • Developer bottleneck : You have loads of requirements, and an efficient testing and deployment process. You just can’t get stuff coded quick enough. Maybe AI assisted development will increase the productivity of your developers if done correctly.
  • Testing bottleneck : You have loads of changes waiting to get tested before they can be deployed. Unless you can get AI to help with automated testing, then generating code quicker with AI is not going to help alleviate your testing constraint. In fact it’s going to make it worse.
  • Delivery bottleneck : If you can’t reliably get changes out to production in a timely manner, generating more code is not going to help your productivity. Remember you have not added value until the product is delivered to the customer.

As you can see, AI assisted development (in this context) is not the solution to every problem.

So why do Agile and DevOps matter?

With the previous bottlenecks in mind, lets look at why agile and DevOps matter.

The Agile Manifesto talks about “Customer collaboration 
over contract negotiation”. Customers always have new requirements. If you don’t have a massive list of requirements, it means you are not doing customer collaboration. People who do customer collaboration well never have a problem with user representation and requirements. Even if you hate the concept of “Agile”, you should live up to this bit of the agile manifesto. Your customers must feel like they are being listened to if you want to keep them.

A lot of people talk about CI/CD as if it is just continuous delivery and forget about the continuous integration part. If you have automated your unit testing and integration testing, you shouldn’t have a giant log-jam of stuff waiting to be tested. If you haven’t you are doing CD, not CI/CD.

The principle of flow often mentions CI/CD. If you struggle to get stuff delivered into production in a quick and reliable manner, there is no point generating more AI slop code, because it is never getting deployed anyway. Even if you hate DevOps, you need the DevOps mentality to make sure delivery is not the bottleneck.

So looking at this in a really simplistic manner, you want to make sure your developers are the bottleneck, because then you may be able to improve their productivity with AI assistants. If they are not the bottleneck, making them faster is pointless.

Conclusion

To reiterate, the folks that are seeing productivity gains from AI assisted development are those where developers are the constraint. Coding faster will not help you if there are no requirements to code, or no way of testing and shipping code.

When I see the people who are doing this “well”, they already have the whole agile and DevOps thing sorted, or at least the aspects of it that matter. They might call it something different, and that doesn’t matter. The point is they have identified and eliminated the non-development constraints already.

Cheers

Tim…

Fedora 44 and Oracle

Fedora 44 was released a few days ago. Let’s start with the standard warning.

As usual I worked through a few variations of Oracle installations on Fedora, including 26ai.

The only real gotcha was 21c, which required a downgrade of the binutils package to install. In comparison 19c and 26ai were cleaner.

As explained in the first link, I just do this for fun, and to see what is coming to RHEL/OL in the future. Please don’t do this for something serious.

Vagrant & Packer

I pushed Vagrant builds to my GitHub.

If you want to try these you will need to build a Fedora 44 box. You can do that using Packer. There is an example of that here.

What’s New?

For a casual user like me it’s really just updates to existing packages. If you want a proper post about what’s new with Fedora 44 you can read about it here.

Cheers

Tim…

Oracle VirtualBox 7.2.8

Oracle released VirtualBox 7.2.8 a few days ago.

The downloads and changelog are in the usual places.

I’ve done an install on a Windows 10 and 11. I had some issues with the previous version on Windows 11, but that was probably an issue with my machine. I tried this version on Windows 11 and it worked fine.

As mentioned before, the cleanest way to upgrade is to do the following.

  • Uninstall VirtualBox.
  • Make sure all “VirtualBox Host-Only Ethernet Adapter” adapters in Device Manager had been removed.
  • Reboot.
  • Install VirtualBox using “run as administrator”.
  • Reboot.

That seems to be a more consistent approach.

I’ve run a bunch of Vagrant builds for 26ai on my Windows 10 and 11 machines and it seems to be working as normal, so fingers crossed…

Once the April database patches are released I’ll be doing some more builds, so I’ll get to test it a lot more.

Cheers

Tim…

Running LLMs locally. One size does NOT fit all

A few questions have popped up recently, so I thought I would address them here.

I’ve already written about running local LLMs here. Both posts also include the reasons why you might want to do this. I’m not going to repeat that here.

Can I run a local LLM on ???

This is the most common question I get. Someone will list their hardware and ask if they can run an LLM locally. The answer is always, “It depends”.

  • What kit do you actually have?
  • What sort of models do you want to use?
  • How are you expecting to use them?
  • Do you expect it to be fast, or is it acceptable for it to be slow?

I can’t give a definitive answer, but here are some things to consider.

What kit do you actually have?

The models you will be able to run will vary a lot depending on your kit. If you have an old laptop (like me), you will be restricted to running the small models, as you may be running on CPU, using the system memory. I always start with “Granite 4 H Tiny” from IBM, which is small and seems to run pretty quick on my laptop, whilst also giving me OK results. I will use bigger models as well, but only if I’m prepared to wait for a result.

If you have a GPU, the type of models you can run will depend a lot on the VRAM on the GPU. The relationship between the size of the model and the VRAM you need is not as simple as it sounds. A lot of people will just say the model has to be smaller than the VRAM of your GPU, which is kind-of correct, but you also have to consider some other stuff. The quantization (compression) of the model reduces the model size. The compression of the memory (see TurboQuant) and the size of the context you are using, which affects the Key-Value (KV) cache. You want everything to fit into the VRAM on the card. Even with a GPU you might still be limited to smaller models if you want to pass in a lot of context into your queries.

Typically you will find the more parameters a model has, the bigger it is, but remember some models with a lot of parameter can use a subset of them, making them a lot more efficient than they would first appear.

If you have kit with unified memory, like Apple Silicon kit, the system memory can be used for either the main system or the GPU, so in some cases it’s easier to run bigger models on that kit because you effectively have more VRAM. You aren’t going to run a huge model on a 8G Neo, but you may find the Apple Silicon kit works a lot better than the equivalent x86-64 kit because of unified memory.

I’ve been purposely vague here, but you hopefully get the idea that it is not as straight forward as you telling me your kit, and me giving you a definitive answer. You have to play around.

What sort of models do you want to use?

Based on the previous section, you already know the limitations of your kit. When you are using cloud-based LLMs, you are often routed to different models depending on the type of work you are trying to do. Some are specialized for coding. Some are better and dealing with written text. Some are better at handling image/video data. You have to select a model that will work with your kit, that is focussed on the type of work you are trying to do. You may find a smaller specialized model gives you better and quicker results than one of the big models. It all depends.

Play around and see what works for your requirement and kit.

How are you expecting to use them?

Asking short directed questions is very different to having huge amounts of context, based on either lots of provided information, or long running chats. Every time a new question is asked, the previous chat history becomes part of the context. You need to be careful.

Likewise, trying to run agents that can make tool calls can quickly increase the size of the context. Being extremely simplistic, for every tool call, the query is halted, the tool call is made, the information returned is added to the context, the query run again with the additional information. The amount of context can grow substantially.

Once again, you will have to play around and see what tools and models work for your kit. You are not going to be able to run 50 OpenClaw agents against a massive model on your 10 year old crappy laptop. Just be realistic.

Do you expect it to be fast, or is it acceptable for it to be slow?

I think a lot of people try the free version of OpenAI and expect running a local model to be just as fast. Pick the right model and it could be, but most of the time you have to manage your expectations. I will often try stuff on a small model. If I get an acceptable answer I go with it. If I don’t I will try a larger, much slower model. Sometimes I’m prepared to wait for a superior answer.

You have to deal with the limitations of your kit.

Local LLMs are crap?

I saw this comment on a post recently and it made me kind-of mad.

“I tried most of these small models. They are all crap. Sure they might succeed in some narrow tasks, that still need their output to be verified.”

Well yes, the small models are often best suited to narrow/directed tasks. But there are hundreds/thousands of them to choose from. Find those that suit your workflows. Not every task needs a frontier model to give good results.

If you have good kit at home, like a gaming GPU or a higher spec Apple Silicon laptop, you might be able to run bigger models.

The bit about, “that still need their output to be verified”, made me want to go postal. Literally every response from a LLM needs to be verified. They hallucinate all the time. Almost everyone that tries to tell you they are near perfect has some skin in the game. They stand to make money by convincing you to buy their service/tool. The reality is they screw up all the time. Not validating the results is like Googling commands and running them directly on your production servers as root. You are going to be out of a job pretty quickly.

Conclusion

You can run LLMs locally on a wide range of kit. The result you get will vary greatly depending on your kit, the models you are trying to use and how you are trying to use them. You are not going to be challenging OpenAI/Anthropic/Google any time soon, but you might get something that works for you. If so, it will be free, private and under your control.

Just play around and be realistic. I’m not an AI expert. Just a dabbler…

Cheers

Tim…