Tim Groeneveld

Random musings from the world of an Open Source geek

Launching Laravel Expo Updates

| Leave a Comment

Laravel Expo Updates is a Laravel package that implements the Expo Updates protocol, so a React Native app using expo-updates can talk directly to your Laravel app for manifests and assets.

expo-updates is the client-side update system for React Native apps written with the Expo framework. It manages remote updates to application code and communicates with a configured update service to discover available updates. Basically, you can deliver small fixes and updates immediately while working toward the next app store release.

To get setup and running, you need to run a quick set of configuration steps.

composer require timgws/laravel-expo-updates

php artisan vendor:publish \
  --provider="LaravelExpoUpdates\ExpoUpdatesServiceProvider" \
  --tag="config"

php artisan migrate

Then make sure that in your react-native app, expo-updates points to your Laravel server.

{
  "updates": {
    "url": "https://example.org/updates/api/manifest"
  }
}
Read More »

Upgrading PHP from 8.2 to 8.4 with domain-manager

| Leave a Comment

A PHP 8.2 to 8.4 upgrade in a domain-manager stack has a silly footgun! The fix is a simple container lifecycle problem.

As mentioned before, domain-manager uses a deliberately simple layout: nginx runs on the host, MySQL runs on the host, and each site gets its own PHP-FPM container through Podman or Docker. PHP version upgrades sit squarely in that container boundary. Changing the configured PHP version, rebuilding the image, and restarting services should be enough in theory.

Check that the running container runs the correct/desired version of PHP:

podman exec php-example-org php -v

Check that the image answers whether the rebuild itself succeeded:

podman run --rm localhost/exampleorg_php:latest php -v

Expected output makes the mismatch obvious:

$ podman exec php-example-org php -v
PHP 8.2.x

$ podman run --rm localhost/exampleorg_php:latest php -v
PHP 8.4.x

PHP 8.4 in the image means the build is fine. PHP 8.2 in the running container means the live service is still an older container created before the rebuild.

Read More »

Why I built domain-manager

| Leave a Comment

I built domain-manager because I was tired of provisioning the same PHP hosting stack by hand for every new site. Create the Unix user, make the web root, wire up nginx, create the MySQL database, sort out PHP-FPM, fix permissions, enable TLS, repeat.

I didn’t need a full hosting panel with mail, DNS, dashboards, reseller features, and a pile of opinionated abstractions sitting between me and the machine. I wanted a very specific thing: a repeatable way to host multiple PHP sites on one Linux server, keep each site isolated enough to be sane, and still have the entire setup remain transparent and editable.

What I actually wanted

I wanted something closer to this:

# Create a new domain, with everything ready to go
./domain-manager add example.com

# List all of the domains
./domain-manager list

# find info and delete domains
./domain-manager info example.com
./domain-manager delete example.com

# Manage MySQL databases
./domain-manager mysql create example.com example_db_name
./domain-manager mysql list example.com --stats

# SSL
./domain-manager enable-ssl example.com --cloudflare

Under the hood, the basic idea of what I wanted was:

  • nginx runs on the host
  • MySQL runs on the host
  • each site gets its own PHP-FPM container through Docker or Podman
  • each domain gets its own Unix user
  • metadata about domains and databases is stored locally in a file

There’s no need to fully containerise the whole server. nginx on the host is easy to debug. MySQL on the host is easy to manage. System backups stay straightforward.

PHP is where isolation matters most for this kind of setup. Have a website that you want to limit uploads? Simple, just apply SELinux rules to the cgroup that php runs in. Or run php in a container that has only read-only access to the application.

Each domain can run its own PHP-FPM container, so different sites can use different PHP versions without turning the host into a museum of conflicting packages. That’s no need to run a tool like EasyApache to build every version of PHP that you might possibly want to support in the future.

Read More »

Converting a list of GPS locations (lat, long) to Australian state names

| Leave a Comment

Recently, I had the fun experience of converting a CSV file that contained the names of a bunch of cities inside Australia. The CSV file contained four columns:

  • City name
  • Latitude (in DMS)
  • Longitude (in DMS)

There were more than 5,000 rows in the CSV file. I wanted to pull a list of all the cities, and determine what state they were in.

Instead of using the Google Maps API (and getting charged ~$35), I decided to hunt down a way that I could use open data provided by the Australian Bureau of Statistics to add the state name to the list of cities.

The ABS has a downloadable ESRI Shapefile, which basically is a vector map of all the borders for states in Australia. This, with a little bit of data cleaning using Pandas DataPrep, and a quick dirty R script would allow me to add the state name to the CSV.

So, first we start with the raw CSV file that I was given:

Suburb Namelatlong
0Aberfoyle30° 21’30″S152° 2’30″E
1Adaminaby35° 58’30″S148° 51’0″E
2Adelaide34° 56’0″S138° 35’30″E
3Adelong35° 18’35″S148° 1’0″E
4Agnes Water24° 17’0″S151° 49’0″E
3095Yumali35° 28’0″S139° 51’30″E
3096Yuna28° 23’0″S114° 54’30″E
3097Yuna East28° 25’30″S115° 10’0″E
3098Yunta32° 35’0″S139° 28’0″E
3099Zamia Creek24° 32’30″S149° 36’0″E
city_locations.csv
import pandas
csv = pandas.read_csv("city_locations.csv")

import pandas as pd
import numpy as np
from dataprep.clean import clean_lat_long
df = pd.DataFrame(csv)

cleaned = clean_lat_long(df, lat_col="lat", long_col="long", split=True)
cleaned = cleaned[cleaned.lat_clean.lt(0)]
cleaned = cleaned[cleaned.long_clean.gt(0)]
cleaned.to_csv("clean.csv")

Running this script would give me an output file that contained the latitude and longitude converted from DMS (Degrees, Minutes and Seconds) to decimal values – skipping the rows that could not be processed correctly, ready for the next processing step.

SZUlatlonglat_cleanlong_clean
0Aberfoyle30° 21’30″S152° 2’30″E-30.3583152.0417
1Adaminaby35° 58’30″S148° 51’0″E-35.9750148.8500
2Adelaide34° 56’0″S138° 35’30″E-34.9333138.5917
3Adelong35° 18’35″S148° 1’0″E-35.3097148.0167
4Agnes Water24° 17’0″S151° 49’0″E-24.2833151.8167
3092Yumali35° 28’0″S139° 51’30″E-35.4667139.8583
3093Yuna28° 23’0″S114° 54’30″E-28.3833114.9083
3094Yuna East28° 25’30″S115° 10’0″E-28.4250115.1667
3095Yunta32° 35’0″S139° 28’0″E-32.5833139.4667
3096Zamia Creek24° 32’30″S149° 36’0″E-24.5417149.6000

Using Rlang, we import the Shapefile from the ABS, import the cleaned CSV file with decimal lat/long points, and use st_intersects from the sf package to determine what state a given city is in, based on it’s lat/long location.

library(sf)
library(dplyr)
library(sp)
library(progress)

map = read_sf("1270055004_sos_2016_aust_shape/SOS_2016_AUST.shp")
nc_geom <- st_geometry(map)

latLong <- read.csv(file = 'cleaned.csv')
crsFormat = st_crs(map)

pnts_sf <- st_as_sf(latLong, coords = c('long_clean', 'lat_clean'), crs = crsFormat)
pnts <- pnts_sf %>% mutate(
    intersection = as.integer(st_intersects(geometry, map)),
    area = if_else(is.na(intersection), '', map$STE_NAME16[intersection]),
    size = if_else(is.na(intersection), '', map$SOS_NAME16[intersection]),

    .keep = c("all")
)

write.csv(pnts, "with-intersections.csv")

This should leave you with a CSV file that contains the state name that the given points are in.

The best Apple dongle. Is it a Dell DA310?

| 2 Comments on The best Apple dongle. Is it a Dell DA310?

My attempts to get the perfect video out of an Apple MacBook to the Dell S2721QS finally is a success! It’s all thanks to Dell. I guess it makes sense. Dell makes the monitor, so I suppose having the Dell DA310 USB-C companion adapter is the right thing to do.

That said, I did expect the Apple adapter to “just work”. I ditched it for the more capable (and gigabit network-enabled!) Dell DA310. The Apple adapter displayed blurry text and icons that really were not sharp. It had to go!

The adapter will work with both Intel and Apple Silicon arm64/M1 MacBook’s. I have tested the adapter with macOS 10.14, 10.15, 11.6 and 12.2. See the bottom of the article to see some notes on how to get the Ethernet port working correctly.

This image shows two Apple adapters sitting on top of each other (left). Both adapters did not correctly work. The Dell DA310, with HDMI and DisplayPort showing, sits on the right next to it.
Two Apple adapters, both adapters that did not work. The Dell DA310 sits next to it.

First, to the Apple USB-C Digital AV Multiport Adapter. You need to be aware that there are two versions of these adapters. Model A1621, which supports a 4k output (3840 x 2160) at 30hertz (30 frames per second), and the Model A2119, which supports 4k video at 3840 x 2160 at 60hertz. Apple does have a support document, which explains the difference between the two adapters (pictured above, left). Needless to say, I would suggest a different adapter.

One thing to watch out for when purchasing a USB-C adapter is how they mention support for 4k resolutions. Even though an adapter might “support” 4k video, it does not mean that the adapter will deliver a perfect image. Apple mentions that their Model A1621 version of the AV Adapter does not allow screens to operate at a high refresh rate. Even the most recent current model, the A2119 does support high refresh rates – on paper – but not always with a high-quality image.

Why do I care about refresh rates?

High refresh rates allow for buttery smooth window movements. A low refresh rate will effect everything from the latency of key presses appearing on the screen, to the smoothness of the mouse cursor on the screen when physically moving the mouse. The limited 30 frames per second of the slower refresh rate afforded by Apple on their older AV Adapters (bottom left adapter, pictured above) is extremely noticeable.

Using a MacBook with a 30-hertz resolution makes the computer feel frustratingly slow.

Luckily, updating the AV Adapter to the Model A2119 (top left adapter, pictured above) does give 4k video at 60 fps. That said, the quality of the output on the A2119 at a high refresh rate is sad. The colours appear washed out. It almost seems as though the images data is heavily compressed. It’s either done by the MacBook or the USB-C adapter before the data is sent to the display.

Dell S2721QS, shown at 60 hertz with an Apple adapter
Dell S2721QS, shown at 60 hertz with a Dell adapter

It’s hard to be able to visualize the difference between them both side by side. You will notice that lines, such as the diagonal lines on the IntelliJ IDEA icon appear jagged. Switching between the two though, the difference is immeasurable.

Side by Side: Apple A2119 vs Dell DA310
Side by Side: Apple A2119 vs Dell DA310

I’m not sure what exactly is causing this issue. Looking inside System Information, the resolution, refresh rate, connection type settings, and frame buffer depth were all the same.

Read More »

Extracting Dell colour profiles on macOS without Windows

| Leave a Comment

Dell is notorious for not really supporting Apple’s operating system very well. It’s not just their monitors, but other accessories like their USB-C docks.

It’s a shame. I think that Dell has pretty high quality gear.

I recently acquired a Dell S2721QS, and wanted to extract the files that are provided in their Windows driver to determine what exactly what was in there. After a quick look, it turns out, not that much. That being said, the ICC/ICM file provided with the driver pack may be useful for use on macOS.

Either way, I recently wanted to install the colour profiles that come with Dell monitors (or can be downloaded from the Dell website) on my Mac, but could not find an easy way to extract the ICC/ICM files and install them.

Starting with the downloaded file, DELL_S2721QS-MONITOR_A00-00_DRVR_KTRRG.exe I extracted the file using Binwalk.

# binwalk -D '.*' --extract  ./DELL_S2721QS-MONITOR_A00-00_DRVR_KTRRG.exe

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             Microsoft executable, portable (PE)
90592         0x161E0         7-zip archive data, version 0.4

Here I had two extracted files. The 7-zip archive is the one we were looking for.

# cd _DELL_S2721QS-MONITOR_A00-00_DRVR_KTRRG.exe.extracted
# ls
total 616
drwxr-xr-x 4 tim staff    128 Jul 19 17:06 .
drwxr-xr-x 4 tim staff    128 Jul 19 17:06 ..
-rw-r--r-- 1 tim staff 358345 Jul 19 17:06 0
-rw-r--r-- 1 tim staff 267753 Jul 19 17:06 161E0

# file 161E0
161E0: 7-zip archive data, version 0.4

# 7z x 161E0
Everything is Ok

Files: 5
Size:       756506
Compressed: 267753

Now that the extraction is complete, there should be 7 files in the directory.

# ls
total 1360
drwxr-xr-x 9 tim staff    288 Jul 19 17:09  .
drwxr-xr-x 4 tim staff    128 Jul 19 17:06  ..
-rw-r--r-- 1 tim staff 358345 Jul 19 17:06  0
-rw-r--r-- 1 tim staff 267753 Jul 19 17:06  161E0
-rw-r--r-- 1 tim staff 540216 Jul 22  2019 'Dell Monitor Driver Installer.exe'
-rw-r--r-- 1 tim staff   2600 May 27  2020  S2721QS.icm
-rw-r--r-- 1 tim staff   3071 May 27  2020  S2721QS.inf
-rw-r--r-- 1 tim staff 200360 Mar 28  2012  _x64help.exe
-rw-r--r-- 1 tim staff  10259 May 27  2020  s2721qs.cat

Indeed, there is. That S2721QS.icm file looks like it’s exactly what we are looking for! Simply copy it into ColorSync

# file S2721QS.icm
S2721QS.icm: Microsoft color profile 2.0, type KCMS, RGB/XYZ-mntr device by KODA, 2600 bytes, 30-4-2020 12:15:03, PCS Z=0xd32b "DELL S2721QS Color Profile,D6500"

# sudo cp S2721QS.icm /Library/ColorSync/Profiles/Displays

After that, you should be able to glaze into your Dell monitor with all the correct colour curves.

Read More »

Creating a new WordPress admin account with only database access

| Leave a Comment

Creating a new WordPress user when you don’t have access to WordPress but do have access to the hosting control panel is rather simple.

Simply replace the strings in the first INSERT query that are wrapped around square brackets, and run the SQL statement on your server. You will then be able to successfully log into WordPress.

INSERT INTO `wp_users` (
    `user_login`, `user_pass`, `user_nicename`, `user_email`,
    `user_url`, `user_registered`, `user_activation_key`,
    `user_status`, `display_name`
) VALUES (
    '[username]', MD5('[password]'), '[username]', '[email-address]',
    '', NOW(), '', 0, '[username]'
);

--
-- Make MySQL remember the ID for the user just inserted for use later
--
SET @MY_USER_ID = LAST_INSERT_ID();

--
-- Add the magic sauce to have WordPress know the user is an admin...
--
INSERT INTO `wp_usermeta` (`user_id`, `meta_key`, `meta_value`) VALUES
(
        @MY_USER_ID, 'wp_capabilities',
        'a:2:{s:13:"administrator";s:1:"1";s:14:"backwpup_admin";b:1;}'
),
(@MY_USER_ID, 'wp_user_level', '10'),
(@MY_USER_ID, 'wp_dashboard_quick_press_last_post_id', '620'),
(@MY_USER_ID, 'wp_user-settings', 'editor=tinymce&uploader=1&hidetb=1'),
(@MY_USER_ID, 'wp_user-settings-time', UNIX_TIMESTAMP());

Checking what processes need to be restarted after a system upgrade

| 2 Comments on Checking what processes need to be restarted after a system upgrade

With updates going on in the last couple of months for various packages, such as OpenSSL and GLibC which have fixed a number of important security vulnerabilities, I thought I might share a one liner that might save you one day.

sudo lsof -n | grep -v \#prelink\# | grep -e '\.so' | grep -e DEL | grep -e lib | grep -v ^init | sed -re 's|^([^0-9]*)\s*([0-9]*)[^/]*(\/.*)$|\1 (\2) \3|' | sort -u

Running lsof will list all of the currently opened files from processes running on your system. -n will stop lsof from resolving hostnames for processes that have opened network ports to different processes (such as your webserver, mail server etc)

Running grep a couple of times will ensure that we find all the processes that have loaded a shared binary that has been deleted.

Note that the “init” process has been excluded. This is done on purpose. init can not be restarted without rebooting or otherwise killing the system.

The sed magic will show a list of all the processes and their PID’s, along with the library that was deleted that triggered it being listed as an application that should be restarted.

[email protected] [~]# REGEX='^([^0-9]*)\s*([0-9]*)[^/]*(\/.*)$|\1 (\2) \3'
[email protected] [~]# sudo lsof -n | grep -v \#prelink\#  \
                            | grep -e '\.so' | grep -e DEL | grep -e lib \
                            | grep -v ^init \
                            | sed -re "s|$REXEG|" | sort -u
auditd      (1802) /lib64/ld-2.12.so
auditd      (1802) /lib64/libc-2.12.so
auditd      (1802) /lib64/libm-2.12.so
auditd      (1802) /lib64/libnsl-2.12.so
auditd      (1802) /lib64/libnss_files-2.12.so
auditd      (1802) /lib64/libpthread-2.12.so
auditd      (1802) /lib64/librt-2.12.so

Note that this will not work if the application is dynamically loaded (for example using dlopen(3)) or if the application is statically linked.