Patrick Delaney

Python Packages for Machine Learning

2023-04-25T00:00:00-05:00

Why Create a Python Package for Machine Learning?

Recently I wrote an article on some possible words to describe different parts of an AI/ML pipeline. What I didn’t talk about is the idea that it might be helpful to have a home-spun python package which can serve as a bundled up collection of tools and utilities that may be used across different parts of the AI/ML pipeline to create consistency.

So one way to build a bunch of tools and utilities would be to literally copy and paste code from one part of the AI/ML pipeline, let’s say the Zeus Zonal, where you’ve got all of your Jupyter Notebook goodies stored over to the Prometheus Pipline where you have all of your data wrangling and cleaning going on, or vice versa. However a way to help make this easier than route copy-and-pasting is to actually package up all of those utilities into a package, and just install that package where it’s needed.

Of course, if you’re creating your own private package, some of what you make might be proprietary, which means it needs to be hosted at a private location, rather than PyPi itself. We’re not creating an open source tool here, we’re creating something that’s potentially sensitive, which is a bit of an edge case outside of the vast majority of, “how to publish a PyPi package,” articles are about. Let’s take a look at some of the detailed considerations involved in creating a package for Machine Learning purposes.

Creating a Python Package for Machine Learning Purposes

So first off, there’s a standard way to create a Python Package which is listed on Packaging and Distributing Projects. Go through that and get familiar with how to publish a package successfully to the PyPi test repo to make sure you know how to do that, or read other tutorials on Python packaging to get yourself up to speed.

Reserving the Namespace

Beyond that, if you’re going to create a Machine Learning package for internal use within an organization, you likely need a private package rather than a public PyPi package. You’re going to have to choose a provider for that, such as JFrog or Gitlab. That being said, if your package name is for example, the-package, you need to check on PyPi to ensure that the namespace is free, which hilariously enough, “the-package” is already taken, so you would have to create a new namespace which is not yet taken, for example, “xyz-the-package,” if xyx was a common acronym for your organization. Why is that? Well, you don’t want yourself or anyone else to accidentally download the wrong package, as cool as the name, “package,” might be, it’s better to reserve a custom namespace so there’s no confusion, such as, “abcxyz-package.”
In order to publish a package on PyPi to reserve the namespace, you first obviously have to register as a user and confirm your email, and then you need to generate an API token. If you’re just reserving a namespace so that you don’t accidentally download some other unknown package some day, then you can create and publish your namespace and throw the token away.

Use Recent Documentation

There is a lot of information online about using setup.py to create a python package which is out of date. The new way to do it is to use a pyproject.toml file, information about that can be found here. There was a decent amount of updated information from this article that was published on HackerNews in 2022 which got a fair amount of criticsm for being cargo-cultish. Whatever–for the purposes of discussion, just do what you need to do and use whatever design decisions you need to get something done the right way, whatever that means. There are basically two main things that happen: 1) Build the package and 2) Publish the package.

The Two Essential Things That Happen

In short, there are two main commands to help make this happen, and the configuration other than those two commands are from a couple different files that are strcutured within a machine in a particular couple locations.
The two commands that you need are build, and then twine. You really only need the wheel to publish a package, you don’t need the source code binary in .tar.gz format, so you use the –wheel flag to perform the following:

python -m build --wheel

Then when you twine the file, you use the following format:

python3 -m twine upload --repository testpypi dist/*

Let’s go through what that means briefly:
twine is literally a tool for publishing PyPi packages, found here, which has three different possible commands, upload, register, and check
–repository testpypi means that we’re using the testpypi repository, which is not the same as your private repository, it’s pointing to a place on pypi used purely for testing purposes.
dist/ are the distribution files that we upload to the repository. Typically this is going to be a .whl file, but you could include the .tar.gz file binary as well. The /** designator means, “anything in the ./dist directory.”

So for the –repository flag, if you were using a different repo other than testpypi, let’s say, pypi or whateverhub, then you would use that in place of testpypi above.

How Twine Knows What It Knows, Security Considerations

That being said, how does the twine command know anything about the repo that you are sending to? The answer is in a couple different files, one of which is in the same directory as where your package exists on the local machine ./.pypirc, and the other is within the $HOME directory, $HOME/.pypirc.

[distutils]
index-servers =
    whateverhub

[gitlab]
repository = https://whateverhub.com/path/to/project

and:

[distutils]
index-servers =
    whateverhub

[gitlab]
repository = https://whateverhub.com/path/to/project
username = myusername
password = mypassword

So I know what you might be thinking, the above method is not very secure - it does not make sense to ever have a username and password written to a file onto a machine, but rather to keep it as an environment variable or a secret. This is true. That being said, the above is put together as a simple way to just get a package written to the PyPi namespace as a way to reserve it.
That being said, from what I can tell, these files need to actually be written to a directory. So the way to secure this is basically to write the file temporarily during the CI process, storing the the username and password as secrets and rendering them to a file, either deleting the file after the twine process is completed, or deleting the container running the job or whatever needs to happen.

Enjoying the Fruits of Your Labor

Once this has been correctly built and twined, there should be a 201 response from your API where the package has been pushed. If you get this response, you can then navigate to whatever package repo interface on the web to make sure it’s there.
Further to the question of security while using a private repo API, when downloading and installing the package using pip install, the username and password/token could be installed with a different format, like as follows:

pip install             \
    https://__token__:{USER_TOKEN}:whateverhub.com/path/to/project/packages/whateverpackage/simple:latest    \
    --extra-index-url https://pypi.org/simple \
    whateverpackage

In this scenario above, the {USER_TOKEN} for a particular whateverhub repository should be mounted as a secret, if this is done within a Docker Container rather than a build argument, so that it does not show up in the logs.

Conclusion and Packaging Up Other Considerations

That’s about it. There are tons of tutorials out there about creating Python packages and I don’t want to beat a dead horse. These are just some general thoughts for if you need a private package to help your Machine Learning workflow.

Here are some other things that have come up along the way that have been helpful to me, but might or might not apply to you:

Picking a Python version to pin across all runtimes/containers, which should justify a pinning point for all other dependencies, to minimize dependency hell.
Security considerations for packaging a thing up and installing using twine. Are you using pip compile and only using safe requirements? Maybe this matters, maybe not.
Proper design patterns for packages - depending upon who is using the package, how do you document that out and how do you structure it so it’s easy to use? This is a highly variable question.
Versioning using SEMVAR. You’re likely going to come out with a lot of different versions of the package. Can you use SEMVAR as a guide and stick to that, so that any MAJOR version update breaks previous updates, but MINOR and PATCH updates do not?
If your package is stored in a monorepo and not in its own repo, are you able to tag the monorepo where a package is stored as a part of versioning? Are you able to accomplish this tagging automatically as a part of your CI? Might be useful, might be overkill, I don’t know, it depends upon you…something to think about.
Auto-upgrading Jupyter and other images, including those images where a particular package is already installed - can you put auto-upgrades into your pipeline? Might be useful.

Python Packages for Machine Learning was originally published by Patrick Delaney at Patrick Delaney on April 25, 2023.

Creating a Simple ML Server from Scratch with CherryPy

2023-04-17T00:00:00-05:00

The full code for this post can be found here.

Creating a Simple Machine Learning Server with CherryPy

In a previous posts on AI/ML architectures, we went through and defined a way to think about how to build out various parts of an AI/ML workflow for various sizes of teams over time, using cute names from greek mythology. One part of that post highlighted the concept of an, “Epimetheus Endpoint” and specifically, how one could go about starting to build one from scratch. This section seemed to deserve a more robust treatment.

To that end, in this blog post we will walk through creating a simple Machine Learning API server using CherryPy, a Python web framework. We will assume that you have a trained machine learning model saved as a model.joblib file using scikit-learn and a list of columns saved as model_columns.joblib Let’s dive into the code.

1. Import Necessary Libraries

First, we need to import the required libraries for our server, including CherryPy, joblib, pandas, and JSON. Part of the objective here is to use the fewest possible dependencies that we possibly can, to avoid Python dependency hell. So while, “requests,” may be a typical library used to build API’s in Python, instead we’re going to use CherryPy’s native built-in tools.

import cherrypy
import joblib
import json
import pandas as pd

2. Load the Trained Model and Columns

Load the previously trained machine learning model and the corresponding columns into memory using joblib. The reason we want to load the columns is because when a user posts a JSON payload to the API, the fields of that JSON may not be matched to the fields within the model.joblib, which means that there would be an error. To prevent this, we reindex the original JSON payload from the user to match the model.joblib fields before the inference is run.

lr = joblib.load('/home/app/model/model.joblib')
model_columns = joblib.load("/home/app/model/model_columns.joblib")

3. Define the Server and API Classes

Create two classes: Server() and API() The Server() class will have a simple index() method that returns just a simple, “Hello World” to test if the server is working. The API() class will have an exposed=True attribute and a query() method that will handle incoming requests and return predictions.

class Server(object):
    @cherrypy.expose
    def index(self):
        return "Hello World"

4. Implement the query() method in the API class

The query() method in the API() class will process incoming JSON data, convert categorical variables into dummy variables, reindex the columns, use the trained model to make predictions, and return the predictions as a JSON response. `` Note that we used the pd.get_dummies and the dataFrame.reindex() method and used model_columns to prevent a mismatch as discussed in point (2) above, prior to making the inference (prediction).

@cherrypy.expose
@cherrypy.tools.json_in()
@cherrypy.tools.json_out()
def query(self, *args, **kwargs):
    data = cherrypy.request.json

    query = pd.get_dummies(pd.DataFrame(data))
    query = query.reindex(columns=model_columns, fill_value=0)
    prediction = list(lr.predict(query))
    
    cherrypy.response.headers['Content-Type'] = 'application/json'
    json_prediction = json.dumps([int(x) for x in prediction])

    return {"prediction": json_prediction}

5. Define the main() Function and Server Configuration

In the main() function, define the global server configuration, server configuration for the Server() class, and the API configuration for the API() class. Then, mount the Server() and API() classes, start the CherryPy engine, and block the engine.

Of particular interest in how this was done below, is that the server configuration, server_config`** is seperated from the api configuration, api_config. The reason this was done was, “seperation of concerns,” meaning we are seperating out different applications into units with mimimal overlap between the units. Within this application, the “Server,” object was used to define a web service, which is different from an API, in that it really is designed to accept HTTP requests at a URL for the purposes of viewing on a browser, so it doesn’t really have a complex configuration. The “API” objet in contrast requires dispatch requests and response header configuration to function.

What both applications share is:

They both serve at the host and port shown.
They both are mounted to the CherryPy server overall.
They both have to be started.

All of these universal traits are reflected in the global_config and the actions at the bottom of the main() function.

def main():
    global_config = {
    'server.socket_host': '0.0.0.0',
    'server.socket_port': 8889,
    }

    server_config = {
    '/': {
        'tools.sessions.on': True,
        'tools.sessions.timeout': 3600,
    }
}

    api_config = {
        '/api': {
            'request.dispatch': cherrypy.dispatch.MethodDispatcher(),
            'tools.sessions.on': True,
            'tools.response_headers.on': True,
            'tools.response_headers.headers': [('Content-Type', 'application/json')]
        }
    }

    cherrypy.config.update(global_config)
    cherrypy.tree.mount(Server(), '/', server_config)
    cherrypy.tree.mount(API(), '/api', api_config)
    cherrypy.engine.start()
    cherrypy.engine.block()

6. Run the Server

Finally, run the main() function when the script, which is fairly self-explanatory.

if __name__ == "__main__":
    main()

Miscellaneous Concerns and Wrapping Up

All of the above was wrapped up into a directory structure which allows whomever to build and run this on a Dockerfile, so that there should be less problems with dependency issues, regardless of what base machine it is run on.
This API is not protected, it’s just a bare API with no authentication or authorization, it’s as simple as possible, so if this was used in anything close to a production environment, obviouslly the necessary level of security would have to be built in.

Just briefly looking at our directory structure:

0_buildimageandtag
1_run
Dockerfile
app/
cookies.txt
curltest.sh
docker-compose.yaml
launch/
posttest.sh
requirements.in
requirements.txt

Using Docker, the project can be built and app launched with 0_buildimageandtag and 1_run
app/ contains all of the code necessary to run the application, except the requirements.
launch/ is used to hold a couple different commands which are referenced in the Dockerfile, the entrypoint is a bash script which points to another start bash script which, both of which gets copied into the container and are used to launch the CherryPy application.
curltest.sh and posttest.sh are used to test the Server and Post data, respectively, with a pre-formed data.
Output from the api request is stored at app/output.

app:
api.py	model  output

app/model:
model.joblib  model_columns.joblib

app/output:
prediction_output.json

launch:
entrypoint  start

Usage guide is provided at the application README.md.

Future Concerns

What we did not cover here were anything involved with what was referred to in a previous blog post as the Zeus Zonal, nor the Nemesis Normalization nor Odysseus Orchestration.
One of the next logical steps, assuming Zeus Zonal is already somehow covered, being that more than one model might be used, would be figuring out a way to build out a simple Odysseus Orchestration, either with a text file that keeps track of available models that can be queried as a part of the API, or more advanced than that, a database with said models that can be queried.

Loading…

Check out my Portfolio

Creating a Simple ML Server from Scratch with CherryPy was originally published by Patrick Delaney at Patrick Delaney on April 17, 2023.

Wanna Bet?

2023-04-11T00:00:00-05:00

It’s wild out there in the world of AI and ML in 2023. You’ve got centralized Large Language Model (LLM) services like ChatGPT, Bing, Bard going at each other’s throats. You’ve got the likes of ChatGLM-6B, GPT4All and LLaMA threatening to tear down the whole business model of using LLM’s to build a destination site.

There’s a neat startup called Manifold Markets which I discoverd over a year ago through HackerNews. It’s a, “Prediction Market,” website which allows users to set up their own predictions on unknown events or properties, and allows other users to bet and try to win fake points based upon correct resolutions of said markets. So really it’s a gambling site, and you’re gambling for points, which you can buy, but you can’t cash out. Here’s one of the markets I put together just out of morbid curiosity:

A lot of the bets that people have set up on this site in relation to AI are what I feel, highly subjective in nature. What I mean by this is, users set up bets about the state of AI at some future date, but the threshold for how that bet would be resolved is highly up to interpretation. For example, let’s say someone’s bet was:

“Will AI Get Super Way Smarter By the End of the Year?”

What you find in the discussion threads of betting markets, is there is a lot of argumentation over the nuances of the definition of the marketplace itself. So - what does the market-maker mean by, “super,” or what do they mean by “smarter?” The strict definitions of these terms could completely change the threshold of how the bet resolves, thereby allowing the market maker to arbitrarily pick winners.

In my mind, this goes against the strict definition of what a, “prediction,” is in the sense of a demarcated, “novel prediction,” used in science, which means the value of even participating in something like that is pretty low. Personally, I’m hoping, perhaps misguidedly, to learn something that will actually occur in reality, in as much as that’s even possible. I’m looking to have a deeper, better range of knowledge and expertise on a topic than my peers, so that I can arguably do better at my career, life, hunting and gathering, romance, or what-have-you. Maybe I’m just projecting, but it seems to me that just engaging in back-and-fourth arguments and waiting for some mini-authority-figure to resolve a bet so that I can amass points in a game has far less value to me personally than being able to really delve into a subject and know a little bit more than I would just reading the news, or perusing social media and accepting what people say at face value. I would rather try to, “be smart,” than, “feel smart.”

So in my opinion, which I am sort of parroting from a wide variety of studies on prediction markets that I have read, the value of a, “prediction market,” is not really in making predictions, but in gaining subject matter expertise, which could hypothetically then be translated over into making novel predictions, improving research and so on. You really don’t know what you don’t know, and when you gamify what you don’t know, competing against others who also want to share information and win an information game in a battle of egos, you really can end up with a much more nuanced understanding of the world.

So this leaves us with the question, “How does one set up a better prediction market, vs. just a pissing contest?”

Well, Dan Schwarz and Lindsay Taylor at Google wrote about some findings from an internal prediction markets game here. Some key take aways:

Precise forecasting can only work on precisely defined questions.
Give your predictors feedback on their performance, to help them improve their forecast accuracy over time.
Incentivize experts to make predictions on questions where they have insights.

Manifold has a handle on the performance feedback with all sorts of internal tools they offer. But as far as making prediction markets precisely defined, they are very free-form and so you end up seeing a lot of joke markets, like this one:

Not to contridict what Schwarz and Taylor write in the paper above, from what I have noticed playing on Manifold over the last year, is that markets which have a certain, “virality,” to them tend to gain more participants, which seems to improve market aggregation. For example, I had put together a market dealing with precisely where the Chang’e 5-T1 booster would crash land on the moon within a range of precision, using the Selenographic Coordinate System, but few people were interested in this topic, relative to something a bit more viral, such as, “Will a nuclear bomb detonate in 2023?”

People tend to get really interested in politics, wars, world-ending scenarios - you know, viral stuff. But more so, they tend to get interested in topics that they can relate to.

So if I put together a topic about AI, and make it validated by a third party, make it really empirical, like, “will this score on this leaderboard go past a certain point by the end of this year?” - it tends to get 2 or 3 or even no participants. But if I instead re-phrase the question to, “Will AI Gain Significantly More Common Sense By the End of the Year?” and within the market description, explain that I’m measuring, “Common Sense,” by a particular leaderboard, then I suddenly get a lot of participants.

Link to Market: Common Sense

So, that’s my, “hack,” for getting more engagement. Now, how do I incentivize experts to actually engage and comment? That I’m not as sure. There seems to be an interested niche of engineers on Manifold who do actively look to bet on AI-related technical topics, so some of that task deals with finding those individuals and engaging with them.

In terms of recruiting experts from outside of the platform, Manifold offers an incentive to market creators, which is to get paid in fake points if you help them recruit new users. So my thought is, I will write this blog post out, and post it in some places where AI experts might actually hang out, and hopefully they will participate in my markets, or perhaps create markets of their own. Maybe some will even click on the links I have provided below which include a referral code. I believe by clicking on a referral code, at least at the time of writing, those new users also get a small boost in their initial starting wallet.

Toward the objective of improving the quality of my markets, Manifold also allows users to pay each other Manifold bucks (called Manna, or $M) through links. So what I’ve started doing lately, is if anyone is extra persnickity and makes some good points in my market comments, points out flaws, I pay them a token amount of Manna bucks, maybe $M 50 or so, depending upon how good of a point they make, then I update the resolution criteria.

So anyway, that’s pretty much what this blog post was about–trying to recruit you, dear reader, to join me and become a degenerate gambler, but wearing monocles while we do so. I declare no conflicts of interest, I have no ownership interest in Manifold Markets (though I wish I did!), I’m just trying to gain a more realistic picture of the world.

Here are some of my AI-themed markets and referral links:

Bets Ending End of 2024

Link to Market: Common Sense

Link to Market: Evaluating Scientific Claims

Link to Market: Feel and React to Pain

Bets Ending End of 2023

Link to Market: Common Sense Judgements About What Happens Next

Link to Market: General Conceptual Skills

Link to Market: Meaning of Questions

Link to Market: Linguistic Temporal Understanding

Link to Market: Embodiment

Link to Market: Hallucinating Less

Link to Market: Tracking Changes in State

Link to Market: Communnity-Based Ethical Judgements

Link to Market: Egocentric Navigation

Destination Site Definition

The old school term for this in the 1990’s was, “web portal,” - basically, a service like Google, which aims to be your one-stop-shop for as much as it can be. Whereas to, “Google it,” might be a term which means search, Google as a company aims to cover as much of your whole life as possible with Youtube, Maps, Drive, Products, etc. Some speculate that OpenAI’s current strategy, at the time of writing this article, is to become a destination site, by pulling people into LLM’s as a service through the use of plugins, whereas a couple months ago, it was thought that OpenAI’s strategy may have been to be more of purely an API that would allow companies to construct their own LLM’s for various purposes. The idea of OpenAI potentially trying to be a destination site would hypothetically threaten a wider range of Google’s revenue sources. Conveniently,there’s a betting market for this hypothesis too!

Link to Market: ChatGPT as a Destination Site

To stay up to date on articles like this, sign up for my email list:

Loading…

Wanna Bet? was originally published by Patrick Delaney at Patrick Delaney on April 11, 2023.

AI/ML Architectures Mythology

2023-04-05T00:00:00-05:00

Contents and TLDR

Contents, Introduction

Pipeline Abstraction Name	Description
The Persephone Protocol	The most basic pipeline abstration with no detail.
The Daedalus Dataflow	Exploding the basic pipeline chart into some additional data science tasks.
The Prometheus Pipeline	Adding some specific data cleaning or scheduling functionality.
The Zeus Zonal	Adding Jupyter Notebook.
Nemesis Normalization	Model Selection and Storage.
Odysseus Orchestration	Keeping Track of Models, Dealing with Exponentially Difficult Debugging.
Epimetheus Endpoint	Deployment.
Hephaestus Hold	Feature Storage.
Kubeflow Kraken	Full Service Pipeline Workflow.

Introduction

Framing Mythological Nicknames for AI/ML Pipeline Construction

The launch of ChatGPT has demonstrated the immense potential of Large Language Models, making it impossible for business leaders to ignore the importance of Machine Learning and Artificial Intelligence. While sales, marketing and product development conversations can be dynamic and fluid, at the end of the day, developers must create code that works, compiling into results that can be displayed on screens.

However, the fast-paced nature of today’s business world means that companies may not have a clear idea of their ML/AI needs or the investment required. To overcome this challenge, developers must not only understand how to architect and build an ML platform but also stay up to date on the latest open-source tools.

To help facilitate planning and conversations about what is needed at what time, the author has put together this guide using mythologically-inspired nicknames to help explore several different iterations of what a basic ML platform architecture looks like and provide insight into how developers can balance the need for cutting-edge technology with just getting stuff done within a timeframe.

In the world of Greek mythology, the launch of ChatGPT may have been seen as a display of immense potential, not unlike the power wielded by the gods themselves. Just as the craftsmen of the ancient world had to master the use of new tools and techniques to trade and do battle with newly discovered civilizations, so must engineers of today strive to build the best possible solution for our companys’ and organizations’ unique needs. While it’s tempting to aim for godlike power and build a Ferrari, sometimes all we need is a humble chariot. As developers, our objective should always be to spec-out the minimum amount of software needed to achieve a business goal. But that doesn’t mean we should compromise on quality. We must take pride in building the best possible chariot, with the flexibility to adapt and evolve as needed, much like Hercules had to balance his divine duties with their mortal concerns.

Return to Contents

The Persephone Protocol

So that being said, let’s take a look at the most basic, abstract look at what a machine learning platform does.

digraph ml_workflow {
  rankdir=LR;
  node [shape=box];
  data -> preprocess -> train -> evaluate -> deploy;
}

In this simple abstraction, the processing and model training stages represent the time the Ancient Greek God Persephone spent in the underworld, and the evaluation and deployment stages represent the return to the world above.

With any machine learning platform of any kind these steps always exist in some form or another. In perhaps in the most basaic possible development configuration, these steps could be held in a collection of different folders, and the deployment could literally just be a button press of some kind to invoke a pre-built model. Let’s look at these steps to start off with so we have somewhat solid definitions:

Data: You’ve got to have data to create a machine learning model.
Preprocess: This step, which could be part of the preceding data step, is essentially about formatting the data in a way that it works for the ensuing steps. Do you need flat file data to be in a CSV file before it goes into the next step? Then you have to figure out a way to, “Preprocess,” it, it’s as simple as that.
Train: Models must be trained to be able to make predictions. There are lots of ways to train models, but mostly it’s using some type of Python-based package such as Scikit Learn or Tensorflow.
Evaluate: Just as a teacher has to grade students’ tests, models must be evaluated. This may range from scoring a model for performance against a benchmark, or having a human look at things and take notes to ensure that things are not overfit and therefore useless even though they perform well. Evaluation is the combination of human and machine measurement of models.
Deploy: Once a model is good enough to go and be used, presumably by a customer or stakeholder, it’s ready to go into production, where it has to be formatted to an API endpoint of some kind and ultimately connected to a front end app so it can be invoked. Back to the, “button press to invoke a simple model,” analogy mentioned above - the user has to be able to press a button, and then that button connects to a function which then, assuming everything is formatted properly, has to run an, “Inference,” on a pre-built model. That pre-build model is sitting there ready for use until that button is pressed.

This is the type of architecture that at the time of writing this article, Linkedin and Spotify show on their website - not going into too much detail, but showing the general overview.

Source
LinkedIn Engineering Blog
Spotify Engineering Blog

Going into a bit more detail, we might pull apart our architecture a bit more and separate some of the concepts within each step so we can have more of a road map of what’s going on:

Return to Contents

The Daedalus Dataflow

digraph ml_workflow {
  rankdir=LR;
  node [shape=box];
  data -> {load_data; clean_data; split_data;} -> preprocess 
  preprocess -> {feature_engineering; feature_selection;} -> train
  train -> {model_selection; hyperparameter_tuning;} -> evaluate
  evaluate -> {performance_metrics; model_comparison;} -> deploy;
}

In Greek Mythology, Daedalus was a skilled craftsman and inventor who created the famous labyrinth on the island of Crete. In contrast the the Persephone Protocol, which is more representative of the changing flow of the seasons, the Daedalus Dataflow is more labyrinthian and complex. However it’s still an abstraction and we haven’t actually talked about any actual software yet.

Preprocessing may have multiple steps involving different types of loading, cleaning or splitting. The output of preprocessing might be something called a, “feature,” which is essentially a column of a field of data, which may or may not need to be saved and shared with other data scientists throughout the discovery process in order to build better models. After training is done, models might be selected, or there may be some kind of auto-model discovery process which involves the use of hyperparameters to have a machine run through a wide variety of models automatically on behalf of the data scientist. Finally, there’s the process of picking and measuring against performance metrics and also comparing models against one another before we go back into our deployent.

This is the level of detail we get from Doordash, Lyft, Instacart and Monzo. Of course these are companies with thousands of employees, whereas many of us may work for startups or companies with tens or hundreds of employees so many of their specific architecture decisions may not work for us.

Source
DoorDash Engineering Blog
Lyft Engineering Blog
Instacart Engineering Blog
Monzo Engineering Blog

Return to Contents

The Prometheus Pipeline

digraph ml_workflow {
  rankdir=LR;
  node [shape=box];
  data -> Airflow -> {feature_engineering; feature_selection;} -> train;
  train -> {model_selection; hyperparameter_tuning;} -> evaluate;
  evaluate -> {performance_metrics; model_comparison;} -> deploy;
  subgraph cluster_Airflow {
    label = "Airflow";
    Airflow;
    load_data -> preprocess;
    clean_data -> preprocess;
    split_data -> preprocess;
  }
}

So the first component we might look at, purely because it’s on the left hand of the graph and for no other reason, might be something as simple as an Exchange Transform and Load (ETL) or data scheduling platform such as Airflow which might be used to clean the data. However, the data scientists within a particular team may not be that far along and they might clean the data themselves across csv’s or spreadsheets at this point, manually copying and pasting things. That’s not much of a platform, but - there might be other bottlenecks across our entire platform that could be identified by an organization which might be better to focus one’s time on. Again - we’re purely following the diagram from left to right, not necessarily because it makes the most sense across all situations.

Airflow is just one example of a potential solution that could be used. There are multiple different open-source platforms out there. The author went ahead and compared the Github star history of a few interesting ones to each other.

Return to Contents

The Zeus Zonal

digraph ml_workflow {
  rankdir=LR;
  node [shape=box];
  data -> Airflow -> Jupyter
  subgraph cluster_Airflow {
    label = "Airflow";
    Airflow;
    load_data -> preprocess;
    clean_data -> preprocess;
    split_data -> preprocess;
  }
  subgraph cluster_Jupyter {
    label = "Jupyter Notebook Container";
    {feature_engineering; feature_selection;} -> train;
    train -> model_selection
    evaluate
    Jupyter
    
  }
  
  Jupyter -> {hyperparameter_tuning;};
  {model_selection; hyperparameter_tuning;} -> evaluate;
  evaluate -> deploy
}

The naming here is a bit of a cheat, as Zeus is the Greek name for the Roman God Jupyter. Zeus is extremely powerful, much like Jupyter Notebooks, it’s the center point of the Machine Learning and Artificial Intelligence development workflow for many.

Jupyter can run the whole gambit from feature engineering, training, feature selection, model selection and evaluation. Typically this workflow is all done within Jupyter Notebooks by the vast majority of data scientists out there. At the end of the data, data scientists just need some type of Jupyter-Notebook-like environment to work with typically, and it’s going to be an Interactive Development Environment (IDE) because a lot of the most talented data scientists out there didn’t necessarily come from a hard coding or DevOps background, they spent the majority of their time being interested in purely the data science and just doing what it takes to get that stuff done. So if you’re tasked with building the actual data science pipeline, you have to keep those, “human factors considerations,” in mind, but for the purposes of this article, we’re going to assume that 99% of who is out there in terms of data scientists are most familiar with the Jupyter Notebook environment.

That being said, up in our Daedalus Dataflow section you can see within the Mondo blog that they do in fact use Google Colab, not Jupyter Notebooks, so Jupyter is not the only option.

Return to Contents

Nemesis Normalization

digraph ml_workflow {
  rankdir=LR;
  node [shape=box];
  data -> Airflow -> Jupyter
  subgraph cluster_Airflow {
    label = "Airflow";
    Airflow;
    load_data -> preprocess;
    clean_data -> preprocess;
    split_data -> preprocess;
  }
  subgraph cluster_Jupyter {
    label = "Jupyter Notebook Container";
    subgraph cluster_ModelSelection {
      label = "Model Selection";
      model_selection -> S3;
      S3 [shape=cylinder]
    }
    {feature_engineering; feature_selection;} -> train;
    train -> model_selection
    evaluate
    quick_preprocessing -> Jupyter
    Jupyter
    
  }
  
  Jupyter -> {hyperparameter_tuning;};
  {model_selection; hyperparameter_tuning;} -> evaluate;
  evaluate -> deploy
}

Moving along, we move into the actual process of Model Selection, but also Model Storage. In Greek mythology, Nemesis was the goddess of divine retribution and revenge, often depicted as an avenger who punished those who were guilty of hubris or arrogance. When we get into the, “model selection,” portion of the pipeline, we are helping to ensure models are accurate and effective in their predictions, as opposed to being arrogant and non-effective.

So once a data scientist finishes their process of training and evaluation, they likely need a place to store the models so that they can be later accessed. One potential solution which could get quite messy, but could be workable to start off with for prototyping, is to simply use AWS S3 or equivalent, and to have an agreed upon convention for storing models at particular prefixes. The question of storing models then could be as trivial as just agreeing upon a convention for S3 prefixes and using a boto3 client, perhaps even a home-rolled Python package as a wrapper around this client to help keep things structured.

Return to Contents

The Odysseus Orchestration

digraph ml_workflow {
  rankdir=LR;
  node [shape=box];
  data -> Airflow -> Jupyter
  subgraph cluster_Airflow {
    label = "Airflow";
    Airflow;
    load_data -> preprocess;
    clean_data -> preprocess;
    split_data -> preprocess;
  }
  subgraph cluster_Jupyter {
    label = "Jupyter Notebook Container";
    subgraph cluster_ModelSelection {
      label = "Data Version Control";
      model_selection -> S3 [shape=cylinder];
    }
    {feature_engineering; feature_selection;} -> train;
    train -> model_selection
    evaluate
    quick_preprocessing -> Jupyter
    Jupyter
    
  }
  
  Jupyter -> {hyperparameter_tuning;};
  {model_selection; hyperparameter_tuning;} -> evaluate;
  S3 [shape=cylinder]
  evaluate -> S3
  
  subgraph cluster_Server {
    label = "Server";
    S3 -> deploy;
  }
}

Odysseus, in Greek Mythology was basically a guy who got lost and had to take a long and winding path to get back home with many challenges along the way.

Long Term Importance of Odysseus Orchestration

I think the challenge of model selection and data versioning was put best bythe 2016 Blog Post by S. Zayd Enam, Why is machine learning ‘hard’?

What is unique about machine learning is that it is ‘exponentially’ harder to figure out what is wrong when things don’t work as expected. Compounding this debugging difficulty, there is often a delay in debugging cycles between implementing a fix or upgrade and seeing the result. Very rarely does an algorithm work the first time and so this ends up being where the majority of time is spent in building algorithms.

As the question of keeping track of a myriad of models becomes more complex, and as the organization demands the use case further, one might consider something such as DVC to help keep track of all of the models out there, particularly those in production so that custom service problems can be diagnosed, but also perhaps on the training and evaluation side, which is not shown in this graph. We’re showing more of the bare minimum next step above. Here is the Github star history of a couple different (of what the author understands to be) open source data or model versioning platforms.

An alternative way of structuring this, might be to use something called Pycaret, installed on your Jupyter Notebook Container, which is really more of a full-service wrapper tool for a lot of the data science functions we have discussed already, and is designed to fit right into Python. However, there are limitations, in that it’s not really designed to work with Tensorflow and more fancy Neural Network stuff, but rather it’s designed to be more of a, “Citizen Data Scientist,” platform which can conceivably be used for prototyping as well.

digraph ml_workflow {
  rankdir=LR;
  node [shape=box];
  data -> Airflow -> Jupyter
  subgraph cluster_Airflow {
    label = "Airflow";
    Airflow;
    load_data -> preprocess;
    clean_data -> preprocess;
    split_data -> preprocess;
  }
  subgraph cluster_Jupyter {
    label = "Jupyter Notebook Container";
    subgraph cluster_Pycaret {
      label = "Pycaret";
      {feature_engineering; feature_selection;} -> train;
      train -> model_selection
      model_selection -> S3 [shape=cylinder];
    }
    quick_preprocessing -> Jupyter
    Jupyter
    evaluate
    
  }
  
  Jupyter -> {hyperparameter_tuning;};
  {model_selection; hyperparameter_tuning;} -> evaluate;
  
  subgraph cluster_Server {
    label = "Server";
    S3 [shape=cylinder]    
    S3 -> deploy;
  }
}

Below are some potential alternatives to Pycaret with another Github star history comparison.

Return to Contents

Epimetheus Endpoint

digraph ml_workflow {
  rankdir=LR;
  node [shape=box];
  data -> Airflow -> Jupyter
  subgraph cluster_Airflow {
    label = "Airflow";
    Airflow;
    load_data -> preprocess;
    clean_data -> preprocess;
    split_data -> preprocess;
  }
  subgraph cluster_Jupyter {
    label = "Jupyter Notebook Container";
    subgraph cluster_Pycaret {
      label = "Pycaret";
      {feature_engineering; feature_selection;} -> train;
      train -> model_selection
      model_selection -> S3 [shape=cylinder];
    }
    quick_preprocessing -> Jupyter
    Jupyter
    evaluate
    
  }
  
  Jupyter -> {hyperparameter_tuning;};
  {model_selection; hyperparameter_tuning;} -> evaluate;
  
  subgraph cluster_Server {
    label = "KServe";
    S3 [shape=cylinder]    
    S3 -> deploy;
  }
  
  deploy -> frontend_application
}

Next, moving on into deployment. Epimetheus was the lesser known brother of Prometheus, the Titan who brought Fire and enlightenment to humankind. Whereas Prometheus represents foresight, Epimetheus represents hindsight. In a way, in order to deploy a Machine Learning or Artificial Intelligence Model, you have to already have the hindsight of having learned something in order to create a model that is ready to deploy - hence, Epimetheus Endpoint.

We have the option of either spinning up our own server, such as perhaps a CherryPy or Flask server, which exposes a function endpoint for a particular machine learning model right in the server code itself, or using something such as KServe which is an open source platform that is designed to dynamically store and serve models to endpoints at will. So in other words, if you had a few number of models you were testing at a few number of endpoints, you might just want to build a CherryPy server for those models and try to maintain those, but if you are working with a lot of complex models, switching them out on the fly, you might use a Kserve or other similar application. Below are some additional model serving platforms with Github star history.

Building an Endpoint From Scratch

To help demonstate the importance of the Odysseus Orchestration, it’s important to go through a simple use case of serving an inference model and then inputing a payload into the server’s API to show how it can become complicated quickly when building completely from scratch.

Open source models aside, if one wanted to create a super simple webservice that works to deploy software, one could do so with CherryPy like so:

First a joblib file is required, which would have been created upstream within the The Zeus Zonal portion of the architecture. This also may have been a .pkl file, .pickle, TF2 SavedModel, or any other variety of model - some sort of model. The below assumes that we have the model saved locally within a container.

import joblib

model = joblib.load("path/to/your/model.joblib")

We then serve the model file on an endpoint, perhaps a /predict endpoint that expects a JSON payload in the request body That endpoint can be tested with a curl command after the web service has been started, using the following, assuming it’s serving at localhost:8080:

curl -X POST -H "Content-Type: application/json" \
  -d '{"payload": [{"robot-age": 5, "rust-level": "9", "battery": "low"}]}' \
  http://localhost:8080/predict

This json payload would hit a prediction function sitting on the server.

def predict():
     json_ = request.json
     prediction = model.predict(query)
     return jsonify({'prediction': list(prediction)})

Which then gives an output, such as:

{"prediction": [0]}

This prediction output can then be used elsewhere in an application, perhaps to display an alert on a website showing that a robot is in, “Bad Condition,” with a binary classification being used behind the scenes.

So this brings up a few points:

More trivially, but still important, dimensional problems:

What are the input dimensions?
What are the output prediction dimensions expected given a particular inference?

These are a problem when building an ML server from scratch, because a sufficiently robust prediction function must be built to deal with different scenarios, but not as much of a probem as more model performance oriented problems.

Less trivial model performance oriented problems:

What version of model was used at model.pkg at what time?
When was the model released?
What training data was used to build the model?
What user reported what feedback about the model at what time, so that the model can be improved?

So while the easiest possible iteration of our Pipeline Abstractions may seem to be to simply build an Epimetheus Endpoint without an Odysseus’ Orchestration, this may present serious performance measurement problems down the line, which could potentially stymie further decision-making. Of course it’s all business case and situation dependant, but this hopefully demonstrates further that Orchestration is a real concern.

Return to Contents

Hephaestus Hold

Feature Storage may be something that is needed for more advanced, multi-person teams working with a lot of data that needs to be converted into, “features,” which are basically shareable columns of data that can be used to much more quickly compare models in a collaborative environment, vs. purely sharing just spreadsheets. In Greek mythology, Hephaestus was the god of technology and blacksmiths, known for creating intricate and highly precise mechanical devices. The feature storage component of our architecture may be best represented by Hephaestus.

For an architecture diagram representing feature storage, simply put a box around the two feature storage features and label it with the desired software platform. As for the popularity of different open source platforms go, here’s a quick analysis:

Return to Contents

Kubeflow Kraken

Kubeflow is the Kraken because of the alliteration, but also because it is of enormous size and is a potential killer. Now, while the Kraken is beautiful and has many appendages that can multitask and be an everything-at-once platform that does everything we have described above and more, with all sorts of custom resources (plugins) that can be used to expand and customize it - it may also be useful for data science teams that are actually larger in size and may need the type of coordination that it can help build.

Kubeflow is essentially a collection of other open-source tools, including Jupyter Notebooks, centralized by a dashboard feature, and connected together by webhooks.

In other words, Kubeflow is really meant for teams, and it’s really complicated to set up. Much of what constitutes Kubeflow is actually just the infrastructure in order to run Kubeflow. That may be fine for what your company is trying to build if it can be properly planned and isolated. However, much like how AWS or various cloud services can get really complicated the more services one uses, requiring more configuration and labor to maintain it, Kubeflow has the same potential scaling problem.

Don’t get me wrong, Kubeflow is a great tool - but the decision to go with Kubeflow should be weighted against where one is at within the AI/ML journey. If you’re a small startup with just yourself or one data scientist, even if you really feel like you need the best thing that’s out there, you probably can get by with a much simpler implementation such as the The Zeus Zonal.

Presumably, Kubeflow’s competitor, MLFlow, has similar challenges, but the author has not personally used it yet.

Rough Breakdown of Kubeflow

What Kubeflow includes, roughly stated:

Data Science Components

KServe: Kserve is a way to store different types of exported models depending on the runtime, and it provides a default version of Single Model Serving, but there is also an alpha version of ModelMesh or Multi Model Serving. KServe InferenceServices are essentially a pod with two containers, an actual kserve_container, and a proxy_query.
Kubeflow Pipelines: Kubeflow Pipelines is a platform for building and deploying machine learning workflows. This may include cleaning and modifying data and is analogous to an ETL pipeline for machine learning.
Katib: Katib is a platform for hyperparameter tuning and neural architecture search.
Training Operator: Training Operator is a platform for running distributed machine learning training jobs.
Seldon: Seldon is a platform for deploying and managing machine learning models in production. Note, we did not use this component.

Performance Components

Tensorboard: Tensorboard provides a way to visualize machine learning models and their performance.

Usability Components

Kubeflow Namespace: The Kubeflow Namespace is a way to isolate Kubeflow applications from other applications running in Kubernetes.
Kubeflow Roles: Kubeflow Roles allow for fine-grained access control to resources in Kubeflow.
Multi-User Kubeflow Pipelines: Multi-User Kubeflow Pipelines allow for multiple users to collaborate on a single pipeline.
Central Dashboard: The Central Dashboard is the starting place for data scientists to log in and access Jupyter Notebooks and other tools.
Notebooks: Jupyter Notebooks are integrated into the Central Dashboard and provide a way to interactively explore data, build and train models, and share results.
User Namespace: The User Namespace is a way to isolate users and their resources from other users in Kubeflow.

Networking, Cert, Auth, Integration with Kubernetes

Cert Manager: Cert Manager provides certificates for admission webhooks, which is a way to ensure that the central dashboard UI and other tools are not spoofed.
Istio: Istio abstracts traffic routing out of the applications that make up Kubeflow, allowing for traffic routing features without needing to inject code into the existing Kubernetes applications.
OIDC Auth Service: OIDC Auth Service uses OpenID Connect to drive authentication for other applications.
Dex: Dex acts as a portal to other identity providers through connectors. Clients write their authentication logic once to talk to dex, then dex handles the protocols for a given backend.
Knative Serving: Knative Serving helps define a set of CRD’s, which control how the workload behaves on the cluster.
Admission Webhook Deployment: Admission Webhook Deployment is a way to ensure that all incoming requests are authenticated before they are allowed to access Kubeflow applications.
Profiles and Kubeflow Access Management (KFAM): Profiles and Kubeflow Access Management (KFAM) provide a way to manage user access to Kubeflow resources.
Volumes: Volumes provide a way to store data that is needed by Kubeflow applications.

Return to Contents

Closing Up

So reviewing what we have covered in this article:

Looking at ML/AI workflow pipelines can help with planning and conversations about what is needed at what time.
Business and organizational need timeframes change, so being able to adapt is a good thing.
There are ways to build a ML/AI workflow platform or parts of ML/AI workflow platforms from scratch, and there are various open source tools that can be stiched together at different points in time, depending upon the unique challenges at the time.

We’ve stayed away from any specific recommendation here, but rather built a map, much like ancient Greek sailors within mythology often would benefit from having maps to help them navigate the mysterious and daunting world of demigods and monsters.

Appendix, Further Assumptions, Prerequisites

Docker assumed.
This article skips over the build-vs-buy conversation completely. The assumption here is that a platform will be built, not bought.
We’re abstracting away infrastructure and networking concerns in this article. Notably, GPU’s are far more expensive than CPU’s for the purposes of training, so a large challenge which is not described within the Zeus Zonal section is being able to potentially control access to training resources. This may also become a factor for inference resources if the endpoint demands something super fast. We’re not paying attention to any of that in this article and we’re assuming infrastructure is easy, which it is not…but it would require its own article to cover.
Security is also a thing. It’s very important to remember not to screw this up! Kidding aside - don’t use build arguments because tokens can show up in logs. Mount your tokens and secrets, don’t put them anywhere in the Git repo. There are too many security considerations to cover but these are two that have bit the author.
Databases are also a thing. We didn’t really cover databases here and assumed that these are more trivial, or at least there is a lot more common knowledge on how to integrate databases. Obviously they are not trivial either, but it’s just less of a, “newer,” topic to a lot of folks at the time of authoring this post.
We didn’t cover automl or hyperparameters, but this is an extended feature which could presumably be tied into a Jupyter notebook.
We used Github stars as a metric for quality and did not include any repos which had exponential jumps in Github stars due to the potential risk of those stars being purchased. This signal might not disqualify a repo outright, but our assumption here is that anyone who may potentially have bought stars (which may not be true), is a risk, because if they are willing to commit shennanigans in one area, they may be willing to commit shennanigans in another. Granted, it may be possible that star jumps were in fact organic, we were just overly-conservative in not sharing anything that even looks like potential Github star purchasing.

Return to Contents

AI/ML Architectures Mythology was originally published by Patrick Delaney at Patrick Delaney on April 05, 2023.

Building a Simple Plagiarism Detector

2022-09-03T00:00:00-05:00

All right, I’m going to start out with a literature review, by just jumping on Google Scholar. Bam! Here’s one:

arxiv.org/pdf/1801.06323…

Chowdhury and Bhattacharyya break down a taxonomy of plagiarism, presumably from a review of literature, since the study is cited as a survey. They also mention cross-lingual plagiarism, a clever little way of simply translating ideas from one language to another.

Chowdhury and Bhattacharyya identify 11 different techniques for plagiarism detection developed over the previous decade and summarize them all in a table.

Chowdhury and Bhattacharyya also identify and summarize 31 different plagiarism detection tools, which includes 5 free and open source tools, as well as 3 paid open source tools.

…there was also a distinction made between “Textual Plagiarism Detection,” as well as, “Source Code Plagiarism Detection.” So interestingly, the world of plagiarism detection is not only concerned with academic/textual work but also source code.

The paper concludes by listing some challenges in plagiarism detection, as well as the notion that no guaranteed detection method exists. They also pose an interesting idea, the concept of a real-time idea checker, which shows an author whether their idea is original or not.

Looking further into Google Scholar:

While breezing through Google Scholar for plagiarism detection, I found papers on: music, source code, more surveys, NFT’s, images in scientific publications faculty attitudes on … almost 5000 results in 2022.

The paper on faculty attitudes toward plagiarism detection is interesting, it basically reads like an advertisement for something called Ithenticate.

Ithenticate charges about $0.004 USD per word on bulk text, with evidently no break if you buy multiple, but they will certainly sell multiple to you! 😂

Ithenticate provides two tiers of service - they use a database of 90+ billion web pages, and a bunch of articles, journals, books, conference proceedings, etc. They provide a similarity score, report, and summary and something called, “Document to Document Comparison.”

This seems similar to another service I had found the other day called, “Unicheck,” which charges $0.06 per page rather than $0.04 per word. Unicheck seems to highlight usability and integrations rather than a vast database.

Unicheck on their social media claims to have scanned as of May 2020, about 3 billion words. So assuming they were being sneaky and that this actually represents their training database, not their customers, and assuming 4k words/paper, they have a database of around 750k papers.

I find this whole area interesting, because to me these seem like tools geared toward beating other plagiarism detection services or rather, beating humans at their ability to detect plagiarism. So rather than policing plagiarism, they are enabling more sophisticated plagiarism.

Basically what I’m saying is, these seem like tools to help someone game academia. The extension of this thought is that this could become a scenario where increasingly powerful tools are used on both the policing side and the, “hint making,” side in an escalating conflict.

But anyway, here’s another open paper I found surveying plagiarism detection approaches, this one from a University in Iraq.

edusj.mosuljournals.com/article_170205…

Interestingly this paper by Taqa and Ali breaks down plagiarism into, “Literal” and “Intelligent” detection methods, which differs from the survey above by Chowdhury and Bhattacharyya (I’m becoming hyper aware of whether one of these papers may have plagiarized the other).

At the end of the day, both of these surveys point toward the notion that more recent plagiarism detection methods seem to work based upon some kind of similarity or distance metric, with the former survey also citing string matching.

The way I visualize these methods in my head is the following: string matching is essentially looking for an exact string, “abc 123,” : if two texts contain the same string, one may have plagiarized the other. This is a much more, “direct” way and assumes direct copying.

Similarity or distance matching, is on a basic level, like calculating pythagorean distance, but rather than calculating the distance (c) with two dimensions, you calculate it with many dimensions.

Rather than physical or linear distance using two or three dimensions, we match multiple, hundreds or thousands of dimensions using tokenization, where a token represents a word, sentence fragment, or phrase assigned a code.

If every unique token is given a unique code, one can calculate the distance between series of tokens by treating them as arrays; just like you could calculate the pythagorean distance between [(1,1)(0,0)] you could calculate the distance between two vectors of form (a,b,…n)

The similarity methods discussed in these papers mention many different ways of calculating similarity with underlying algorithms and filters.

So one way to compute textual similarity using open source would be just just follow one of many tutorials online, for example this one from BERT.

sbert.net/docs/usage/sem…

Let’s take a look at what Grammarly does in their free online sample plagiarism checker. Checking some lines from Shakespeare’s Coriolanus, Act IV, Scene 7.

With that above example, which was directly lifted from the text of the play, they of course found significant plagiarism.

Let’s see what happens if we maintain the idea of the text, but change the wording, which is akin to translation-based plagiarism. We see that the tool also flags our text for plagiarism. Of course this is marketing-focused, so it could simply be hyper-sensitively tuned.

What if we feed in a string of gibberish? Interestingly the output is not only free of plagiarism signals, it’s also free of any grammar, spelling, punctuation and other errors, even though it was completely meaningless text.

So while this free tool test may not have told us much, it at least showed us that - words which are similar to other paragraphs which may have been written over time are going to measure higher in similarity to a random string of non-existent gibberish words, which is expected.

Ok, so now that we have a very general idea of how plagiarism detection works, we can work toward building a prototype. The first logical step in my mind would be to build something simpler on an existing dataset, perhaps a clustering algorithm on an open dataset.

Here’s a collection of various essays written, designed for educational scoring prediction.

ltl-ude.github.io/EduScoringData…

kaggle.com/competitions/a…

Here is the data on Github, so one doesn’t have to sign in to Kaggle (which requires you to provide your phone number for verification for some reason).

github.com/Turanga1/Autom…

And here’s an interesting transformer’s library built for computing sentence similarity.

huggingface.co/tasks/sentence…

This library even has a widget on the side of the webpage which allows you to compute sentence similarity on the fly.

Using a stock example within a Jupyter notebook allows us to compute a sample tensor similarity score between two stock sentences. Using the Shakespearian plagiarism example from above, interestingly we find that the tensor score is even higher than the stock sentence example!

After looking at some of the data up close after putting it into a dataframe using Pandas, I’m realizing that this original dataset has all sorts of weirdness to it which is relevant to data cleaning, e.g. the non-word, “@ CAPS3” and things like that.

So instead of dealing with cleaning, in order to have more of a hacker mentality, I may just use someone else’s already pre-cleaned version of this data sitting in a different Github Repo.

raw.githubusercontent.com/sankalpjain99/…

Looking at this dataset, the, “processed_data.csv” has a few different columns, it looked like the, “clean” data may have been an effort to remove common expressions, as they state in their github, to reduce skewness to simplify the essay grading process.

We’re just interested in whatever the closest thing to a human input might be for the time being. Normalization increases computing efficiency, which is an optimization that can be done later. For now, I just want to get a distance measurement between sentences.

The above output is a tensor, which is a class from Pytorch. We can extract the value with the method .item() as shown below.

pytorch.org/docs/stable/te…

Since we’re able to extract distance between sentences, we can hypothetically do k-means clustering to visualize the various essays against each other. Scikit-Learn has a clustering method, here is a sample:

scikit-learn.org/stable/modules…

If we’re going to do K-means clustering, we have to first reduce the dimensions down. If you look at the dimensions of the input to K-means, you can see that it’s an n*2 matrix, whereas comparing all of our essays together would be a thousands by thousands matrix.

So, one way to do this is with Principle Component Analysis, PCA, which is a way to reduce dimensions (matrix size). A simple way to think about this is we’re setting the objective to create a, “factor,” (eigenvector) that minimizes variance.

So what that fancy math thing above does is it takes that thousand dimension matrix and grabs the, “most important” part of what makes that matrix special, and distills it down into two (or three, or whatever) dimensions (components), which can be much more easily visualized.

You can probably imagine where we’re going with this. We’re going to take a 2-dimensional graph and then apply another step, cluster analysis onto it, so we can identify essays into groups.

Demonstrating how

pca.fit

Similarly, we can reduce a 3-dimensional array to a 2-dimensional array with PCA and look at the scatter plot of our original 3-dimensional setup vs. the transformed version.

This might look arbitrary, as though the PCA transform function is just putting the points, “wherever,” to make it look nice, but it is applying the transform in a completely predictable way, minimizing variance, so that the reverse would always result in the original array.

So next we’re going to need to iterate through a dataframe to apply comparative sentence analysis across all of our essays. This article goes through some iteration method comparisons.

towardsdatascience.com/heres-the-most…

During one iteration we have to find a sentence embedding and find the cosine similarity with Pytorch. Looking at the time it takes for one operation and multiplying by 12976 essays we estimate that this should take anywhere from 17 to 45 minutes, based upon the first two essays.

Testing this out in practice on a dict we find that the elapsed time to compare one sentence to 12976 others took about 7 minutes rather than 17, this could have been because some of the, “essays” are significantly shorter than others, as in one sentence.

After putting the cosine similarity scores back into a Pandas dataframe and briefly inspecting them, we see distance measurements that are relatively, “far apart,” and “close together” in similarity. We grab the index of the shortest absolute value distance.

What does cosine similarity mean again, conceptually? It’s basically calculating the, “cosine of the angle” between two vectors.

The cosine similarity applies to vectors with an arbitrary number of dimensions, so could be hundreds. These dimensions are word embeddings, so numbered words, parts of speech, fragments, phrases, sentences or however the particular embedding works.

Based upon this, you could have a negative cosine similarity. This is why we take the absolute value of the results, to find the smallest magnitude distance that we can.

So looking at the smallest magnitude (most similar) item first we notice a cosmetic issue, our first essay [0] was enclosed in double straight quotes, so it gets extracted as a string, while essay 6383 was not, so it gets extracted as an object, which means we have to add .item()

Secondly, we notice that the essay with the smallest cosine value, closest to 0, has nothing to do with the first essay, whereas the cosine similarity value closest to 1, was definitely written using the same prompt as our comparative original essay. Wow! How does this work?

The word embedding model we are using at the base of everything that outputs a tensor score, mpnet-base-v2 was created at a hackathon, where they trained a word embedding model using some special TPU’s, fancy GPU’s which they had access to.

During the hackathon, the training process basically looked like this, they took a billion sentence pairs, trained model that outputs a “filter,” which can be used to create, “tensors,” which are objects that can describe how they relate to other objects in a space.

We use this Tensor Predictor Model that they created to make tensors, basically huge matrices, the values of which map out how they relate to all other tensors in a, “space.” This space is defined by all of the total words and sentences found in that original training process.

Note, there is a limitation to this Tensor Predictor, it cuts out anything longer than 384 words. But other than that, those tensors describe a position in a space, not a 3d space, but a many-dimension space derived from all of the words used.

Cosine similarity works to compute distances (or angles, really) with whatever number of dimensions, not just two. As long as the original embedding model, the “structure,” was solid, then the cosine similarity is just walking through the front door.

So now that all of this is understood, we can move on to creating a super simple plagiarism detector which works within the limitations of our laptop that we’re working with, which means, basically limiting the number of essays that we’re comparing to one another.

Setting a threshold for cosine similarity from essay[0], we can reduce the total number of essays evaluated. By setting the threshold to, “>0.69” we get about 100 essays to compare, which will take significantly less time for a demonstration.

We can create an expanded dataframe which allows us to filter out essays by cosine similarity with booleans, and then we can export this to csv as our temporary database.

If we want to do PCA and K-Means clustering, we need a big cosine matrix, which we can create by doing nested for loops and comparing embeddings of every single essay with every other single essay, and putting the results into a big matrix.

Next we run Principle Components Analysis (PCA) on that matrix to reduce all of the variables down to two, so that it can be plotted on an x-y coordinate system.

Each one of these dots represents an essay, and the distance between the dots represents our having reduced the huge matrix of cosine similarity down once again into two variables which represent the difference of the essays from one another in 2 dimensional space.

We can then run k-means clustering, arbitrarily assigning 3 clusters, to take a look at what different, “topic” areas might be. These clusters look pretty arbitrary, particularly the yellow and purple.

That’s something to keep in mind whenever you do clustering, it always will yield a result, however that result may not mean anything.

But for the purposes of plagiarism detection, maybe we could go in and manually review these essay clusters which are really close to one another - it would be interesting to think about what might have made them fall so close together in our mathematical rendering.

Of course another way to inspect for possible plagiarism could be to just go back a step and look at pairings which had really high cosine similarity, and also think about what it may have been that made these essay pairs so similar from a modeling perspective.

We created a couple functions to operate on our cosine similarity dataframe (df) and the filtered list of high cosine measurements (df_highcos) and spit out a list of the largest cosine similarity matches for every one of our top 112 essays.

From that we can reduce it down further and do a visual inspection of the top few. We’re showing here that essay 29 and 90 are the highest matches against each other, with 1 and 108 coming in second.

Looking at the two essays against each other, we can make some educated guesses as to perhaps some of the reasons why these essays were found to be mathematically similar, perhaps just based upon words used and nothing else.

Just looking at words in common assumes a, “Bag of Words,” which assumes that the underlying model was simple a distribution of word counts. (from wikipedia):

The next step up from, “Bag of Words,” is “n-gram,” which looks at distribution of small phrases, perhaps 2, 3 or 4 word common phrases (from wikipedia).

Instead, the basis of how relations between the text are found is via MPNet, a sophisticated language model designed by researchers and released in 2020.

huggingface.co/docs/transform…

What MPNET does, according to this blog post, is it combines masking of words, something that BERT does, and permutations of words, something that XLNet does. The results of combining the methods are allegedly superior.

microsoft.com/en-us/research…

So basically, if you have a sentence (sequence) among many like, [“i”,”like”,”dogs”] it may randomly either mask or permute every few sentences into, [“MASK”,”like”,”dogs”] or [“like”,”dogs”,”i”]

So let’s take a step back - what does, “Training a Language Model,” even mean from a high level and how does a trained language model get used to create one of our, “Tensors,” - e.g., sentence predictions?

What we’ve been talking about with BERT, MPNet, or GPT-3 is essentially generating a statistical understanding of sequences of words based upon many terabytes of text. This is known as training a General Pretrained Transformer Model.

Once these general models get pretrained, they can then be pulled down as an open source model and used to fine-tune a model. So basically the output is a Fine-Tuned Model, derived from that original General Pretrained Model, which can be used for an application.

All we have done so far is taken an existing Fine-Tuned Model, mpnet-base-v2, and used it to create tensors, (our scoring method to measure similarity).

huggingface.co/sentence-trans…

So while our original, “Statistical Understanding of Language,” was from the General Pretrained Model known as MPNet, and uses some tricks to create a more accurate model of sequences of sentences, which results in better tokenization and sequencing…

The fine-tuned model we used, mpnet-base-v2, took MPNet and created a model that matches sentences, based upon another database of matched sentences. Those matched sentences were from Reddit, Yahoo Answers, StackExchange, etc. etc.

So our entire plagiarism similarity detection model is really built upon a foundation of two different models, one general and one purpose-built. I simply added some data to analyze and applied cosine similarity / PCA on the output of those models used.

So how can we know whether this plagiarism detection tech stack can even…you know…detect route plagiarism? Well, we have to feed in some purposely plagiarized essays and compare the score of those essays against our already detected, “very similar,” essays.

By the way, you can look at the MPNet tokenizer right here as a JSON file:

huggingface.co/microsoft/mpne…

So we threw a couple new essays into the CSV, one completely rote-copied version of essay[0], and another where we attempted to clean up some of the atrocious writing, but left the essay in as much of the original as possible, as shown here:

We turned much of the previous work into functions, and input the two versions of essay[0] mentioned above, and our rudimentary plagiarism detector report measured a 1 on the rote copy, and a ~0.939 on the reworded version.

So next it would be interesting to do some statistics on the calculated cosine similarities, and to see what the distribution curve looked like, and how much outside the variance, median and average our, “improved grammar” version lied to see if this could be repeatable.

It would also be interesting to compare this finding to perhaps some other fine-tuned language models and see if we come up with different results. Finally if we have time it might be nice to put this detector into a command-line tool.

Here’s a filtered set of results of a bunch of different ready-made fine-tuned models, specifically for sentence similarity applications.

huggingface.co/models?languag…

There are a number of plagiarism checker tutorials on YouTube:

youtube.com/results?search…

What’s interesting is that none of these seem to make use of Pandas, although some do make use of Numpy, and they don’t seem to be aware of the concept of Transformers. Some seem to be more in-line with being computer science algorithm exercises.

Here’s a nice tutorial put together using scikit-learn TfidfVectorizer and cosine similarity checker.

kalebujordan.dev/how-to-detect-…

Before using another method to create a cosine similarity matrix, it’s important to get more of a statistical description of our mpnet-v2 derived cosine similarity matrix. We can use this to help compare and contrast performance.

Looking at the histogram for essay[0], we see that we get one match for itself and one for the rote copy, as well as a match that stands fairly fair away for the adapted copy.

So to eliminate the duplicates we did a walkthrough of the dataframe, replacing said duplicates and self-matches with np.nan values, which resulted in the dataframe shown.

We were then able to get a basic stats report as well as a histogram and a probability density function (PDF).

If we convert the PDF to a Standard Normal PDF, applying the law of large numbers and assuming many many texts, you can see that there is a small possibility of false positives (or true positives) under the curve between a threshold of 0.9 and 0.95.

Keep in mind that our original reworded-plagiarized text measured at about 0.93, so 0.95 would not have been sensitive enough, but 0.90 would have been sensitive enough to be able to capture that type of plagiarism.

Here’s a corrected version of the chart above. Note that the stdev is about half of what was shown above, and the upper threshold we are showing is 0.93, not 0.95.

So what if we compare this to another method, TF-IDF (term frequency - inverse document frequency)? This basically compares the frequency of terms used in a document vs. many others in a corpus. Here’s an example showing TF-IDF of Jane Austin novels.

tidytextmining.com/tfidf.html

So I built my own TF-IDF using scikit-learn and then built a cosine similarity matrix using scipy. I then cleaned up the cosine matrix and did some stats on it to get a comparison to our fine-tuned, all-MiniLM-L6-v2 transformer model.

The results were staggering. Now the mean of the vast majority of all documents was way over at 0.287, very far away from the adapted copy, which was now at 0.859747, 11 stdev’s away from the mean as opposed to 3.

Translating this over to a hypothetical generalized model, using the law of large numbers and a Standard Normal curve, we have much more breathing room to set thresholds for plagiarism alarms. Interestingly the stdev for TF-IDF was almost the same as for all-MiniLM-L6-v2.

So if I am to bring this to a conclusion, I would say that for the narrow application of comparing all submitted papers to one another as a corpus, TF-IDF on the surface seems like a better model.

Here are the two models next to each other to give a clearer perspective. Again, for rote and slightly adapted plagiarism detection, the regular statistical-based TF-IDF model gives much more leeway.

New questions arise here: 1. Are there any examples we can find of a fine-tuned transformer model or other NN model that can outperform TF-IDF in within-corpus plagiarism detection? 2. What happens to both models if we adapt the essay further? 3. How much time did each take?

To be fair, SBERT classifies all-MiniLM-L6-v2 as general purpose, which means it’s designed for clustering, information retrieval and sentence similarity. It may be a, “jack of three trades, master of none,” type scenario going on.

sbert.net/docs/pretraine…

This blog post seems to demonstrate that a BERT model was able to classify articles better than TF-IDF and another bag-of-words type method, Word2Vec.

towardsdatascience.com/text-classific…

However, this blog post seems to show that with his analysis, TF-IDF was able to perform better for document classification.

datagraphi.com/blog/post/2021…

Yet another blog post concludes that TF-IDF is superior for document classification. However it is noted that BERT can be improved over time, whereas TF-IDF is fixed.

medium.com/@claude.feldge…

So why even use a transformer model? Well, it comes down to the application and size of data you are working with. With TF-IDF you are constrained to using an increasingly large matrix as your corpus grows.

So basically what this means is, if we are using a bag-of-words model, and we’re applying it against very large datasets, huge amounts of text across the web, then we will end up slowing down downstream applications.

Whereas transformers use something called a, “dense model,” so while the accuracy might be less, those, “vectors” (tensors) representing paragraphs will only ever be so large, in the case of all-MiniLM-L6-v2, they will only ever be 384 dimensions.

This means that transformers can hold, “knowledge,” about vocabulary and how sentences should be structured based upon massive datasets, meaning in both the Generalized Pretrained Model (gigantic), and the Fine Tuned Model (huge but less gigantic).

This is something key that I don’t see covered in these performance comparison blogposts: transformers map reduce: they cast a wider net over a much larger dataset which enables one to build an application that will execute in a reasonable timeframe with that dataset.

Here is how I see a plagiarism detector for a very large corpus of documents potentially being architectured, presuming there are budgetary and time constraints.

Classifying an article by topic using something like BERT could help map reduce a massive corpus into smaller subsets, which would have more reasonable sparse matrix sizes, allowing TF-IDF or other statistical methods to be performed within that subset.

Building a Simple Plagiarism Detector was originally published by Patrick Delaney at Patrick Delaney on September 03, 2022.

/bin/bash A-to-Z

2022-08-05T00:00:00-05:00

/bin/bash A through Z 🧵 #bashthread #bash

alias … creates a nickname for a command which makes it easier to invoke that command. E.g. instead of doing, “ls -l” you can replace it with simply, “ll” - this is a good way to make funny inside joke command names.

apropos … searches the manual pages. bash has a built in manual which can be invoked with, “man -k” – apropos can search through the entire manual. Here’s a search for apt-get:

apt-get … it’s a package installation tool, but remember that it’s not just apt-get update/install - there are all sorts of dependency tools, as well as the ability to, “clean” which is helpful for creating smaller Dockerfiles.

awk … it’s not awkward, it’s an entire language in a tool to help munge data from the Standard Input (STDIN) - you can do, “awk CODE” and use https://awk.js.org/

to build out your awk expression to use on bash.

basename … it’s kind of like the opposite of pwd. Whereas pwd gives you the directory you’re in and cares nothing about filenames, with basename /path/to/file.txt it will spit out file.txt only.

bash … bash, or often /bin/bash is the path on most linux distributions to the bash shell, another option being /bin/sh - shells have these nice handy commands, while tty, terminal, is the environment for translating computer code to characters.

bc … an arbitrary precision calculator. You can use it to open it up a little calculator application, or you can pipe a string into bc as shown and it will return the result for you.

bg … run a process in the background. Basically if you run something like, “sleep for 100 seconds” you can throw that into the background. This is different than daemon, bg is user initiated whereas daemon is system initiated.

break … There are loops in bash scripting. You can use break to jump out of the loop without doing any operation. In this example we break out of the loop if i equals 2. Here’s a cheat sheet for bash operators … https://kapeli.com/cheat_sheets/Bash_Test_Operators.docset/Contents/Resources/Documents/index

build-essential … not really a bash tool as much a thing that installs several bash tools for debian/ubuntu type systems. https://packages.debian.org/sid/build-essential

includes dpkg-dev, g++, gcc, libc6-dev and make, all important tools for compiling packages on debian/ubuntu.

builtin … to understand builtin, you first have to understand what a shell is. The shell is a, “shell” around a terminal - in unix-like systems, there can be several shells, bash, zsh, sh, or even the python shell. https://unix.stackexchange.com/questions/11454/what-is-the-difference-between-a-builtin-command-and-one-that-is-not

builtin (continued) … the, “builtin” of a shell depends upon the shell you are using, so you have to check the documentation, in this case, the bash documentation. Here’s the bash shell builtins: https://www.computerhope.com/unix/bash/index.htm

builtin (continued) … builtins don’t have the usual -h flag help files, so you can use, “help help” or “help cd” for example, to get the help file for that command, for example:

builtin (continued) … you can use, “builtin” to redefine a builtin, as shown. Why do this? Maybe you want a builtin to have a different default behavior for convenience sake. Maybe you always want, “ls” to display, “ls -1” every time (one file per line).

cal … displays a nifty little calendar.

case … is like an if statement, which matches a particular expression or logical state with an operation. You list out the possible cases, as well as an *) catch-all, and end the case statement with, “esac” - case spelled backwards.

cat … just prints out the contents of a file, -n shows the line numbers of each line. So if you wanted to get the number of lines (rows) in a file with awk, you can do: cat -n out.txt

awk ‘END{ print $1; }’

cd … changes the directory. You can do cd /path/to/directory to jump in, or cd .. or cd ../../ to jump back an arbitrary number of directories. Less known is cd -P and cd -L which turns on and off the use of symbolic links (shortcuts).

chgrp … change the group ownership of a file. So you could have a group of users, user1, user2 in group group1, then you could have a file hello.txt, or folder /hello, you could change access to that file/folder with chgrp group1 hello.txt or chgrp group1 /hello

chmod … there are three different numerical ways to show permission levels, the sums of which indicate levels of ownership. There are also letter abbreviations for these numerical representations. You can show output of the chmod command with the -c flag.

chmod (continued)

chown … change the owner of a file to an individual or group. Only root can use chown. Different than chgrp, the root has the ultimate permissions over who owns what, including group assignments, but then other users can have group assignment permission https://www.oreilly.com/library/view/running-linux-third/156592469X/ch04s14.html

chroot … can be used to set up a virtual environment, kind of similar to using docker, I haven’t used it much, I prefer just using docker. https://www.howtogeek.com/441534/how-to-use-the-chroot-command-on-linux/

chkconfig … is like systemctl, allows you to start and stop services, can be used to start services on boot. Here’s a translation guide for systemctl. https://bencane.com/2012/01/19/cheat-sheet-systemctl-vs-chkconfig/

chksum … files can get corrupted or changed. cksum uses an algo called cyclic redundancy check (crc) to spit out a unique code for a file, as well as the filesize for consistency checking. Another algo would be md5sum which involves stronger encryption. https://en.wikipedia.org/wiki/Cyclic_redundancy_check

cmp … compares the contents of a file. If the files are the same it spits out nothing. The –verbose option shows a line by line comparison.

comm … compare files line by line. “With no options, produce three-column output. Column one contains lines unique to FILE1, column two contains lines unique to FILE2, and column three contains lines common to both files.”

comm (continued) … if you just wanted to print the lines in common between the two files you could do: comm abc.txt abd.txt

awk -F$’\t’ ‘{print $3}’

command … to understand command you have to understand $PATH and .bashrc, which are different places the bash shell looks for programs, utilities to run. .bashrc is the most surface level, stuff here runs first.

command (continued) $PATH is a list of directories in order, separated by : which bash looks into each, one after another, looking for a matching program to run, such as https://t.co/QLBgnFohPH.

So if https://t.co/QLBgnFohPH

isn’t already in your bashrc, it goes to $PATH.

command (continued) … what, “command” does is it tells bash to look in $PATH first, and just skip .bashrc, so it’s kind of like, going to your system default, $PATH first, although $PATH can be customized.

continue … related to break … however instead of jumping out of the loop completely, it goes and does the next iteration of the loop. Here it’s used to skip i=2 only:

cp … copy, can also be used to copy a file. Remember we can use cksum to verify that a copy is an exact copy. md5 can also be used for this as well.

cron … is a way of scheduling scripts on a regular basis, minutely, hourly, daily, yearly, whatever. You can edit cron expressions using https://crontab.guru/

crond is a daemon which runs and activates whatever script based upon the crontab (next)

crontab … is a file with all of the cronjobs that need to run, each is a cron expression followed by a command or script. One catch is that crontab doesn’t see the normal $PATH variable by default, so you have to define it within the crontab.

csplit … if you have a file separated by any pattern such as you can split that file into separate files using a regex. csplit report.txt '/^$/' '{*}' which will split report.txt into xx00, xx01, xx02, xx03.

cut … cuts out sections from each line of files and writes the result to a standard output - can use byte or character counts or delimiters and can also replace characters. It could be used to change space delimited data into a csv file.

date … pretty self-explanatory, however date comes in a lot of formats and date –help comes with lots of options. Getting the date in iso8601 time, with seconds requires the -Is flag.

dc … polish notation calculator - https://en.wikipedia.org/wiki/Polish_notation

/dev/null … Whatever you write to /dev/null will be discarded. You get to use the results in the variable $?, the exit status of the previous command, one time, then after that it’s back to exit status 0. More on i/o redirection https://tldp.org/LDP/abs/html/io-redirection.html

declare … check if a particular variable exists with the -p flag. You can also declare a variable with declare whatever=value

diff … similar to git diff, shows the differences on each line of a file.

diff3 … compare 3 different files in the same manner as diff

dasel … allows conversions between JSON, YAML, TOML, XML and CSV, similar to yq/jq. https://github.com/TomWright/dasel#quickstart

dpkg … package management tool (for debian operating system specifically), if you do dpkg -l you can see all of the packages installed.

dig … similar to, “host” gives you dns information.

dir … similar to ls, but ls doesn’t work on certain systems and dir doesn’t work on some systems - it’s good to know both.

A few “c” commands I missed previously: caller … used to print execution frames of subroutine calls, can be used to create a decent die function to track down errors in moderately complex scripts.

curl … used to download stuff from a server. Very common.

dirname … used to get the directory name, basically convert a full pathname to just a path.

du … print out the disk usage in kilobytes of a directory, optionally everything in the directory (as shown).

echo … echo, echo, echo - just kidding, it’s just echo, the print to standard output function. If you use echo recursively, it’s just going to show the output once because it’s really just echo’ing whatever you put into it. echo -e allows various escape characters.

egrep … if you grep a file with a meta character such as +, it only returns lines with the literal “+” whereas egrep is the same as grep -E, it will return all lines in the file under the assumption that + is a meta character. https://superuser.com/questions/508881/what-is-the-difference-between-grep-pgrep-egrep-fgrep

enable … enable and disable builtin shell commands.

env … shows your environmental variables. Note the $PATH variable is in there.

eval … set an ENV to a particular command, then do eval $VAR and it will execute the command within the variable. Kind of similar to alias, but alias just sets the command permanently like a shortcut, not within a variable.

exec … execute a particular command on an input, the input can be a file. exec actually replaces the shell with a new executable image which eventually exits and returns an exit code.

exec … (continued) really what exec does is help you create a disposable shell, it’s designed to invoke long-running programs in non-shell languages, so you don’t really need that shell running, you can discard it to free up system resources.

exit … exits a shell or process.

exit (continued) … exit 0, exit 1, exit 2, etc., while 0 is success and 1 is a general error catch-all, different exit codes can have different meanings. https://tldp.org/LDP/abs/html/exitcodes.html

exec (continued from above) … more on exec : https://www.baeldung.com/linux/exec-command-in-shell-script

there’s a lot you can do with exec. “exec bash” replaces the shell to bash. Within scripts, program calls, logging, allow stdin to read from a file, running a clean environment with exec -c

exec (continued) … there’s also exec $@ which executes the input into a function within a script, so you can output, “whatever” and then send it to another function as an input, and then doing exec $@ will run that input.

expand … officially ending the tabs vs. spaces war, you can use expand to convert tabs to spaces. You just have to pick what ratio of spaces:tabs you want with –tabs=N , I picked N=1 in this example.

expr … expression evaluator, similar to bc and dc but instead of needing to feed in a string, you can just feed in expressions. bc also has exponentiation and square root operations, which expr lacks, and expr maxes out at 2^63-1

dump … (forgot this one) - can be used to create system backups. So if you have an Rpi with a flash drive, you could back the system up using a cron job and dump.

dd … similar to dump, there seems to be debates online going back to the year 2010 at least debating which one is better. Here’s an article on using dd as a backup tool in the RPi scenario. https://opensource.com/article/18/7/how-use-dd-linux

export … makes variables and functions available to subprocesses. But what is a subprocess? Here is an example of three named, “sleep” subprocesses, which run in parallel by using the

function. As you can see they run in parallel, only taking 2 seconds to do all three.

export (continued) … that command was: time $(bash -c “exec -a sleepy_process_01 sleep 2”)

time $(bash -c “exec -a sleepy_process_02 sleep 2”)

time $(bash -c “exec -a sleepy_process_03 sleep 2”)

grep sleepy

export (continued) … if you want to export a variable, you do, “export VAR_NAME” not “export $VAR_NAME”

export (continued) … if you don’t export the variable, and if you attempt to use it in a sub-process, such as a bash within a bash, the variable will not be accessible. However the exported variable will be available across the machine, all different terminals and subprocesses.

false … produces an exit code of 1 and nothing else. true produces an exit code of 0 and nothing else. We can produce arbitrary exit codes with the exit command, but some codes have pre-assigned meanings. and https://www.redhat.com/sysadmin/exit-codes-demystified

fg … send job to the foreground. After suspending a job with ctrl+z, it can be restarted in the foreground with fg %1. Jobs can be restarted by %NUMBER, %COMMANDNAME or %+ %% for the current job, %- for the previous job.

fg (continued) … you can push things back and fourth into the foreground and background.

fgrep … equivalent to using grep -f , it filters the lines in a file that share words with lines in another, filter file. e.g. grep -f FILTER_FILE.txt TARGET_FILE.txt

file … gives an output of the file type and what’s in the file.

find … find things within a directory tree structure based upon various options. One I like is find ./ -name ‘filename_you_want’ which looks in the current directory for any filename you specify. There’s lots of information under “man find”

fmt … format paragraph text according to certain rules selected by options.

ftp … don’t use it, it’s not secure. Instead use scp or sftp, will talk about this later in the s’s.

function … allows you to declare a function with a different type of syntax. You can either use function_name() {}, the more common way, or use function new_function {} without the parenthesis.

gawk … is the gnu implementation of the awk programming language. There may be different options available depending upon what you need, check the manual pages with, “man gawk”

getopts … different than getopt, which is a less capable version of getopts. getopts handles options for you. While arguments are a matter of looping through variables with $1, $2, etc. using options is harder, so getopts allows you to just list out the options.

grep … is a filter or search command, which can search across the output of another command, or among files or a directory or directory tree structure. Don’t forget that it has useful flag options like -c for count or -n to show the line number of each search within a file.

groupadd … add a new user group. You can explicitly add a group id number (GID) with the -g flag.

groupdel … used for deleting groups.

groupmod … modify the group name, password, root directory, and so on.

group … list out the group that a user is in. So if you add a user to a group with, “usermod -a -G” and then run group on that user, you can see that they will have been added to the group specified.

gzip … is the compression tool which gives the famous .gz output. Using -k keeps the original file, -f is fast compression, -9 is maximum compression, -d is decompress. It doesn’t really make sense to gzip super tiny files, as we show here. https://stackoverflow.com/questions/46716095/minimum-file-size-for-compression-algorithms

hash … is a hash table of recently executed programs. You can use -r to clear the hash table. hash prevents bash from having to search $PATH every time you type a command by caching the results in memory. Could be useful if there’s single executable in a directory besides $PATH

fold / head … fold can wrap a long string into a specified number of character length while head can list out a specified number of lines, similar to tail. The default number of lines head and tail prints out is 10.

history … shows a history of all recent commands on bash.

hostname … shows the hostname

iconv … character encoding, encode or decode characters from different standards.

id … show the id of a user, or -G for group of that user, -u for the user number. In this example, I’m showing the root user so it’s all 0.

ifconfig … a way to configure network interfaces, the software that connects to networking devices. To use, must install net-tools. -s flag shows short list. eth0 represents a network card. It’s been superseded by ip. https://codewithyury.com/demystifying-ifconfig-and-network-interfaces-in-linux/

ip … you must install iproute2 to use ip. Replacement for ifconfig with more capability.

jobs … lists active jobs. Note, you can’t easily set a job name as you can a process name, but you can set it indirectly. https://stackoverflow.com/questions/389473/how-do-i-control-a-jobs-name-in-bash

and https://stackoverflow.com/questions/11130229/start-a-process-with-a-name

join … kind of similar to an SQL join, you take two files which have the same index structure, join will join based upon that index. You can select different fields to join on with -1, -2. You can check that the input is correctly sorted with –check-order.

kill … kill a process by PID number.

killall … kill ‘em all, not the Metallica album from the 80’s, it kills all the processes.

less .. shows you more about a file in a format you can scroll through, so here I did, “less 28b.txt” and I can scroll through this exceedingly long incomprehensible file. The most important command is, “q” to get out of there, but you an do other fancy stuff too, press, “h”.

let … perform arithmetic either in a one-liner or separately on env variables. Other calculators include expr (expression evaluator), bc (arbitrary precision), dc (polish notation)

link … create a shortcut to a file. You can operate on links the same as you could the file itself.

ln … the difference between link and ln is that link is the same as, “ln –directory –no-target-directory FILENAME LINKNAME”. link calls the link function to create a link to a file. ln is for soft/symbolic links while link is for hard links.

local … sets a variable within a function. This only works within a function, not interactively within the command line.

logname … shows the name of the current user. It’s possible to have no logname.

logout … if you’re in a login-type shell, such as one that you ssh’d into, you can use logout in place of exit.

look … looks for a string within a file. If the file is not specified, it looks in the system dictionary at /usr/share/dict/words

ls … surely you’ve heard of ls, used to list out files in a directory. But what about ls -1 which shows files line by line? ls -l which shows info on the files? ls -a to show all files.

lsof … is a list of open files. In linux / unix, “everything is a file,” so anything running, any program, or any file that is open and being edited, will be shown.

man … the help manual. Use, “man command” for any command and this will give you a much more detailed version of, “command -h” or “command –help” or in the case of builtins, “help command”

mkdir … make directory, makes a folder, or as they are called in linux/unix terms, “directories.”

mkfifo … in linux, the pipe command sends the output of one command to another for further processing. The, “unnamed pipe,” is

. This pipe cannot be accessed by another session, it just gets created temporarily and is deleted after execution.

mkfifo … so if you do the command: ```nc -l 12345

nc https://www.google.com/

80``` this will create a server on port 12345 such that if you visit localhost:12345 on your browser, it will pipe that request to google and the response will be sent to the shell, not the browser.

mkfifo … however if we use, “mkfifo examplepipe” and then ```nc -l 12345 0<examplepipe

nc https://www.google.com/

80 1>examplepipe``` this will redirect the standard output 1> to examplepipe.

mknod … creates a pipe (a FIFO) block special file or character special file with a name. Option mknod -p is the same as mkfifo, essentially.

mknod … what is a block/character special file? Basically typically when a file is written to or read from, the linux kernel access a filesystem driver which looks at zones on a disk. However if it’s a device, such as a USB, the request is handled by the driver for that device.

mknod … blocks are like ordinary files, they are an array of bytes, with a first and last value location. characters are serialized, behave like pipes. How different drives of USB drives or other devices treat that data with their driver is up to them.

more … a text viewer, similar to less. With both, “more” and “less” you can view multiple files and scroll through sequentially, but more seems to be a bit more intuitive than less when working with multiple files.

most … similar to more and less, but it has an interactive scroll bar which shows you how much of the file you have scrolled through.

mount … allows the linux or unix based system to mount a filesystem such as a USB drive. Calling the mount command by itself with no target gives information on the currently mounted filesystems.

mtr … run a traceroute which provides a map of how data on the internet travels from its source to its destination, including the various routers in between. It’s a combination if ping and traceroute, basically a, “synthetic” version of traceroute which actively pings the host.

mv … moves files, but can also be used to rename files as shown in this example.

mmv … similar to mv, but allows you to do mass operations, for loops in a sense by using a simple pattern-based language: https://manpages.ubuntu.com/manpages/bionic/man1/mmv.1.html

netstat … display the contents of various network info. There are different ways to structure the data depending upon the options used. The most common, non-flag version is shown here, it shows a list of active sockets for each protocol, here showing mostly tcp4.

nice … used to assign CPU priority for a particular process. niceness of -20 is the highest priority and 19 is the lowest priority. The niceness value of a processs can be found under the, “NI” column after running command ps -l https://en.wikipedia.org/wiki/Nice_(Unix)

nl … essentially cat with numbered lines.

nohup … run a command such that it will not abort when you logout or exit from the shell. Basically it’s a way to keep commands running even if you are disconnected.

nslookup … a way to look up information about a name server.

open … opens up a file using a default text editor. In my case, I did, “open whatever.txt” on a file and it opened up Mac TextEdit since I’m on a Mac at the moment.

op … operator access allows root to give root privileges on specific commands to other users in the system. A good example is given on the man page. https://linux.die.net/man/1/op

passwd … allows you to change the password on your own user or other users if you have permissions.

paste … merges lines of files, line by line. Does not permanently effect the files, output goes to stout. So if you wanted to put the results in a file you would have to use > whateverfile.txt

ping … this is a special command which tests various statistics for packet transmission, including round trip times. What’s interesting about ping is that it uses its own message protocol, as a part of ICMP, it’s not a transfer protocol like tcp, so it doesn’t carry any data.

ping (continued) … so this is interesting because ping can’t actually return http status codes, it’s not a transfer protocol, it just gives timing information, so it’s not really the most reliable possible network connectivity tool.

ping (continued) … ping is more of a quick and easy test. More thourough would be “curl –head https://t.co/qJwLSLMUnU”

which shows the actual response status codes.

pkill … so first off, if you run, “kill -l” it gives you a list of POSIX signals, https://www.man7.org/linux/man-pages/man7/signal.7.html

which are a type of signal https://t.co/cBEs21vCsC,

which is basically an asynchronous notification sent to a program while it’s in operation, a “button press” so to speak.

pkill … so for example, if you run, “ping 8.8.8.8” on another tab, you can see the process running. If you do, “pkill 9 ping” this will kill that ping process on the other tab, according to code 9, which is a regular, “stop process,” type kill.

pkill (continued) … so if you use, “pkill -15 ping” this is a different type of killing process, a “graceful shutdown.” These are just different ways of shutting down applications, though -15 is the default.

popd … to know popd, you have to know pushd, which is basically a way to push directories to a list. You can push directories to a list with, “pushd directory_name” then, look at list of those directories with, “dir -l -v” . popd can remove those dirs by name or number.

printenv … prints out all of your environmental variables.

ps … shows all of your currently running processes, along with their PID numbers, etc. Similar to running, “docker ps” but for your actual machine.

pushd … see popd directly above.

pv … monitor the progress of data through a pipe. If you are outputting a file to another, or if you’re doing some kind of operation after a pipe, you can view the progress on a scale of 0-100%.

pwd … print out the current directory.

quota … display a user’s disk usage and limits, if they are set. This would be used together with, “setquota” which sets those quota levels with multiple options, “setquota -u [username] [soft disk limit] [hard disk limit] [soft inode limit] [hard inode limit] [partition]”.

quotacheck … similar to quota.

quotactl … similar to setquota

rcp … is the legacy remote copy protocol tool for copying files between machines. This has been replaced by scp.

read … reads from stdin, allows the capability to create an interface, pass input to a variable.

readarray … you can actually store items in an array within bash, and you can read items from a file into an array in bash with readarray as shown below.

readarray (continued) … bash has arrays just like scripting languages such as python. Here’s a good guide that involves a bunch of array scripting. https://opensource.com/article/18/5/you-dont-know-bash-intro-bash-arrays

readonly … you can make variables readonly.

readonly (continued) … you can’t unset a readonly variable. The purpose of a readonly variable is to make it set permanently until the terminal terminates!

reboot … reboots the system.

rename … rename files in bulk according to a regex pattern and substitution command. Careful! There’s no fallback on this command, so once things are renamed, that’s it!

renice … similar to nice, sets the nice value of a process, re:

rsync … kind of like Dropbox, allows you to sync folders between two machines. https://www.geeksforgeeks.org/rsync-command-in-linux-with-examples/

return … at the end of a function, “return” the value and exit with the given return value.

rev … reverse lines in a file, as shown:

rm … remove, or rather delete, the file in question…fairly common command.

rmdir … this is for removing directories (folders). However if it’s a folder with a bunch of stuff in it, you have to use, “rm -rf ", the r and f standing for recursive, forced.

rsync … can be used to make one directory sync up, or be equivalent in terms of the files and directories in it with another. Can be used remotely, but need to make sure it’s over ssh, or basically, secured.

scp … secure copy. This is a command parallel to ssh, which allows you to securely copy a file from one server/computer to another, remotely. So whereas you would login with ssh by doing, “ssh name@server” and then have some sort of passphrase or keyless entry…

scp (continued) …with scp, you do something similar, but you would also include a filename, e.g.: “scp filename.txt name@server” or if there’s a port involved, “scp -P 1234 filename.txt name@server” - you can also specify the location that the file goes on the receiving server.

sdiff … merge two files interactively. If you do sdiff -l it shows ( on the right side which are identical. If you do sdiff -o OutputFile file1 file2, it brings up a prompt which allows you to merge the two files.

sdiff (continued) … the prompt looks like the following. So for example if you choose, “l” then the left side will go into the output file.

sed … stream editor, like awk but different options and more meant for streams of characters rather than tabular data. You can use a sed builder such as this: https://sed.js.org/

sed (continued) … here we’re showing a replacement command, which takes names and replaces them with, “Spanky.” But you can also do loops, flags, prints, appending, match patterns, and so on. Here’s a cheat sheet. https://quickref.me/sed

screen … you can use this while using ssh, to run multiple remote shells while logged in to ssh.

select … allows a human to select an an array of options. The way to code select is quite simple because select generates a list for you based upon the array.

sensors … used to read sensor chips showing temperature and other environmental stuff. It’s not available on the computer I’m on but here’s a guide and example. https://www.commandlinux.com/man-page/man1/sensors.1.html

seq … print out numeric sequences according to different patterns and with some different options available.

set … allows you to create all sorts of settings of bash itself, there are lots of options. For example, if you set to, “-v” verbose, it prints out the command you entered in every time.

sftp … ftp but uses ssh message protocol. You can do: sftp -oPort=customport user@servername, then you will get to a prompt, sftp> at which point you can use get/put, e.g. sftp> get remote-file local-file or sftp> put -r local-directory with files and directories.

shift … is a way to cycle through arguments as shown in the attached image … you can cycle through various arguments fed in to a command.

shopt … shell options. You can use -s to set the options. Here are a bunch of explanations on what the options are. https://www.gnu.org/software/bash/manual/html_node/The-Shopt-Builtin.html

shutdown … shut down or restart linux.

sleep … suspends the calling process for a specified amount of time. Basically, the next command can’t be called until the sleep process is done. You can call by seconds, minutes, hours, etc. with “sleep 5s” or “sleep 5m”

sort … sort stuff, for example - you can fill in ls

sort which will sort all of the files from ls alphabetically. Lots of sorting options. You can also sort the contents of files with, “cat” - for example.

source … executes the contents of a file, similar to “.” - note that source is a builtin command, and that . just uses source to run. However in /bin/sh shells, . behaves differently than source.

split … split a file based upon byte size, lines, suffixes or numbers of chunks. In this example I split the whatever.txt file into two chunks. The default output names of the new files are xaa, xab, xac, etc.

ssh … encrypted communication between servers. Multiple types of encryption can be used such as rsa, dsa, ed25519 etc. The basic format is ssh user@server - the setup involves creating a key/pair on the remote server, or using a password (though key/pair is more secure).

stat … gives info on a file in terms of when it was created, accessed, modified, the size, etc. You can also have the output given in terse format as shown.

sum … can be used to do a checksum on a file, used to help check to see if the files are the same.

suspend … suspends the execution of the shell. Login shells cannot be suspended. Suspend is the equivalent of doing CTRL+Z

strace … traces out the commands of a program, line by line, including timing information. Note, the timing information of strace is not accurate to actual machine or human-perceived time, because strace itself slows down the running of the program.

strace (continued) … I go into great detail about how strace works here in these notes: https://github.com/pwdel/codepractice/blob/main/MLOps/Codeperform/codeperform.md#testing-out-strace-on-code

su … substitute user - if your user has sufficient powers, it can execute commands as another user.

sudo … execute commands as though you are the root user. Note - works differently in different linux versions.

tail … show the last part of an output - you can specify the number of lines.

tar … you can zip using tar - the options are to create, list or extract.

tar (continued) … remember, a tar file has overhead because it contains a header with information on how to extract the file. So your tar file may be larger than the original, if the original is very small.

tee … usually if you redirect the output of a command into a file, then stdout will show nothing. tee allows you to push the output in two or multiple directions, the output and the file. Pushing the output to a file using > would be equal to tee’ing to a file and >/dev/null

systemctl … you can use this to stop, start and control various applications and services. https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units

test … tests expressions against each other. The -eq options tests if equal, there are other options.

time … measure the time to run a routine. You can use it to measure whatever program in any language. I wrote a simple shell script to measure the time to execute various routines against each other: https://github.com/pwdel/codepractice/blob/main/MLOps/Codeperform/app/codeperform.sh

timeout … run the command, but kill it if still running after duration. Can use –signal=SIGNAL to specify a signal to be sent upon timeout. The duration is a floating point number with suffix s for second, m for minutes, h, d, etc.

times … fairly self-explanatory help file and output. Print the accumulated user and system times for processes run from the shell. Note it’s the process time, not the logged-in time, so it’s a sum of the time all processes ran since that shell started.

touch … is an interesting one, I thought it just created a new, blank file, which it does, but really it’s there to open and modify the date edited of a file without changing it. So here’s a file I, “touched” on July 5th which was last modified May 16th.

top … is really more of a linux thing analogous to Windows Task Manager or OSx Activity Monitor, but well, basically, it’s a bash command that allows you to see the top process information on that particular machine at that time.

traceroute … the internet is a series of routers and servers, in order to get from your machine to a remote server, a signal has to pass through a variety of routers. Traceroute shows the route that it goes, demarked by IP addresses, as shown in the example below.

trap … Signals are software interrupts that are sent to a program to notify it of an important occurrence, such as the violation of a policy, a manual interrupt, etc. Traps activates rules for what to do with such signals. We can list signals with trap -l

trap (continued) … so we can set a command to clear a file, “temp.txt” upon exit. Here we set the trap to remove (rm) the file upon exit. Upon returning back to the shell, we can see that the temp.txt file is gone.

trap (continued) … So how to use these signals? Well, for example, “SIGINT” - interrupt, is 2, so if you feed in trap 2 ... the command will execute with an interrupt, CTRL+C for example as shown below.

trap (continued) … so in short, trap can be used to accomplish any automated task based upon a generated signal.

tr … translate, squeeze, delete characters – basically, you can shorten a string based upon the size of an input, or use all sorts of options to manipulate that string, turn it upper or lower case, etc.

true … do nothing, successfully. Produces an exit code of 0. See, “false” -

tty … print the filename of the terminal connected to standard input. Here are some of the differences between terminal, console, shell and command-line. Whereas bash is the shell at /bin/bash, the terminal is a layer down and is at /dev/ttys005

type … gives information about a command or file, presuming that the command is within $PATH.

umount … unmount a device. In linux/unix, everything is a file, including devices. If you mount a USB drive, it is thought of within linux as a file. https://www.debian.org/releases/wheezy/amd64/apds01.html.en

The mount/umount commands basically makes the, “file” accessible to the directory structure.

unalias … the un-doing of alias. If you no longer find the inside joke command name funny, you can always take off the alias with, “unalias.”

uname … prints off the system information. So for MacOS it’s going to be Darwin. On Linux it’s going to be Linux. You can show more info with –all or various pieces of –all with the different options uname provides.

unexpand … replaces spaces and tabs in the line with the assumption that a tab causes the terminal used to display the line to move to the next tabstop. The tabstop is the number of spaces per tab character.

unexpand (continued) … so for example, you can set the tabsize of the tabstop to 4 spaces per tab with, set tabsize 4; set tabstospaces .

unexpand (continued) … The first 8 characters, A-7 in the result are one tab interval. That leaves a space (where the 8 was) at the first tabstop, (sp in octal). The unexpand program does not add a space; that is left over after unexpand replaces the spaces in 1-7 with a tab.

unexpand (continued) … A file with many lines beginning with spaces can be much larger than one using tabs. Files with different tabstop sizes are also messy, so unexpand can be used to shrink a file or clean it up.

uniq … filter out the unique lines in a file. With -c prints out counts of repeats, with -i ignores differences in character case.

units … converts all sorts of unit measurements (yes, actual scientific unit measurements) into other units. You can use your own custom conversion file with -f or just use the database built into the units command.

rar … another compression format like tar. Here we compressed a file from 25602 bytes (25.6kb) down to 5.1 kb

unrar … un-compress a rar file.

unset … opposite of set.

until … useful for loops or conditions in shell scripts, e.g., “do this until X happens,”

uptime … show the machine uptime

useradd / userdel … create and delete linux users.

users … show the users currently logged in.

uuencode / uudecode … are encoding tools, used to encode and transfer over mediums that do not support ASCII.

mail … I just became aware of this command now. You can send email from a bash shell. https://stackoverflow.com/questions/5155923/sending-a-mail-from-a-linux-shell-script

vdir … verbosely list directory contents

vi / vim … text editors. I’m not going to go into these because I don’t want to start a fight between these and emacs.

wait … wait for a condition to occur. You can add this into shell scripts.

watch … execute or display a program periodically. So for example if you do, watch -b false it will beep every two seconds, super annoying.

wc … very useful command, already used quite a bit in this whole thread, outputs counts of either characters, lines or bytes depending upon the option used.

which / whereis … searches the user $PATH for a program. Whereis does not appear to work on Unix/MacOS. Here I searched kustomize as an example.

while … used in a while loop within a script. Doing a quick search to see where I have used while in this thread already, looks like I first used it on May 6th, 2022 in the, “continue” example.

who … print out all of the users logged in. Different than, “users” which just prints out the name, “who” shows what type of user and when last seen.

whoami … prints out your user name.

wget …. retrieve web pages, url’s via HTTP, HTTPS, FTP. This is great for downloading binaries and packages that may not be available with apt-get.

write … another old-school messaging command, first appeared on unix in 1993, seems like it’s similar to mail in that it would need a decent mount of setup to make it work with an actual SMTP server.

xargs … build and execute commands from a stdin. The -a option can be used to read from files. -p prompts the user yes/no, and -r skips running if the input is empty.

xargs (continued) … you can take the results of ls -l for example, and compress that all into a compact list by piping it into xargs. The output of xargs can also be used as an input directly to other commands.

xargs (continued) … stands for extended arguments, so it’s basically a way to help create arguments which will then be used as an input to another function which actually accepts input arguments. That being said, it has its own exit codes to say if it was successful.

xargs … So, you have to make sure you feed the output of xargs to commands that accept input arguments, which could be existing commands or your own bash programs / commands, which include $1, $2, etc. input arguments.

xargs (continued) … to further demonstrate this, look at how grep outputs arguments and compare that to how xargs outputs arguments. Grep creates a columnar list vs. xargs creates a stream suitable for stdin. So outputting from xargs into wc is more informative than from grep.

yes …creates y indefinitely until interrupt. Be careful with yes, it could fill up your disk space fast. I interrupted this command after 5 seconds and it had created a file that was >603MB of just the letter y. A 500GB hard drive would fill up in about 80 mins.

yes (continued) … it can also be used for testing, e.g. fill up your computer’s CPU and see if there is a heat issue, for example.

zip … another compression method, like tar and rar.

I’ve reached the end of the alphabet, so I’m going to move on to symbols, which account for the control and redirection operators … https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_113

and https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Redirections

Control Operators … & && ( ) ; ;; newline

… so starting out with & single ampersand, causes commands to execute asynchronously.

Redirection Operators … < > >

« » <& >& «- <> … perform redirection functions and is one of those symbols. More info: https://www.gnu.org/software/bash/manual/html_node/Redirections.html

Among the various symbols there are also: Compound Commands … { [command list]; } ( [command list] ) … https://mywiki.wooledge.org/BashSheet#Compound_Commands

Expressions … (( [arithmetic expression] )) $(( [arithmetic expression] )) [[ [test expression] ]] … https://mywiki.wooledge.org/BashSheet#Expressions

Special Parameters … 1,2 addheaders.sh command-line-a-to-z-new.md command-line-a-to-z.md getimagelinks.sh getimagelinks_special.sh imagedownload.sh imagelinks.txt images replaceimageurls.sh replaceimageurls_special.sh @ # ? - $ ! _ … https://mywiki.wooledge.org/BashSheet#Special_Parameters

Parameter Operations … “$var”, “${var}” there are a lot of these … https://mywiki.wooledge.org/BashSheet#Parameter_Operations

To understand Control Operators, we can go through and review how they each behave, but basically they control the sequence of command execution. Redirection Operators deal with the inputs and outputs to the terminals, as well as to and from files. About inputs/outputs:

So we covered &, which means execute commands simultaneously. Moving on, && … means only execute the next command if the first command was successful.

… is a logical OR operator, but it’s a sequential OR, which means that per bash spec, if true exit status is fed in firstly, the next statement will not execute. It’s important to not look at it purely as a mathematical expression, but an operator. https://unix.stackexchange.com/questions/632670/the-paradox-of-logical-and-and-or-in-a-bash-script-to-check-the-succes

… also known as pipe, takes the standard output (stdout, or 1) and feeds it into the next expression as an input (stdin or 0). This has already been used a lot in this guide.

; … semicolon is the bulldozer of control statements, the next statement executes no matter the output of the previous command, in order, non-asynchronously.

;; … double semicolon, from the man pages, is used at the end of a, “case” statement to create a hard stop in evaluating a case. Otherwise if you used ; the next line would execute. https://www.man7.org/linux/man-pages/man1/bash.1.html

;& and ;;& … note from the above tweet, you can build complex logic in case statements with these tools.

[ ] ( ) [[ ]] (( )) … Basically if you see a double, it’s an extended bash feature, whereas the single is the original /bin/sh feature, which means single is for testing and expressions, double is for setting exit codes and math https://unix.stackexchange.com/questions/306111/what-is-the-difference-between-the-bash-operators-vs-vs-vs

… as mentioned in the above tweet, the doubles are for expressions, either test expressions or arithmetic expressions.

… whereas the singles are for commands.

So that covers the control statements, next are redirection operators.

vs > … these redirect stdout output, 1 of a command to a file. You can do set -o noclobber to prevent files from being overwritten. If you use > it over-rides that setting while > will follow whatever the settings are.

vs > … (continued) … attempting to use > with noclobber set on will result in an error, with a stderr output of 1, whereas using > will have a stderr output of 0 (no error).

< … the contents of a file get pushed into the input (stdin, 0) of a command.

« … known as a here-document, redirects the output of the file until a particular string is found, such as “EOF” . Basically, read input from the source until a line containing only the key/delimiter is seen. This can be done in a script as well as interactive mode.

«< … here-string pushes a string to a program input. Instead of typing in text, you give a pre-made string of text to a program. So this is an alternate method of doing echo ‘5+1’

&> … long explanation, it’s essentially the same as 2>&1, a way to redirect errors to be able to output to a file. This is a way to view or store errors in a way that can be reviewed later. &> is the preferred form of >&

… this one is pretty simple, append. Basically it does the same thing as > but rather than overwriting the file completely, it just adds whatever you sent into it to the end of the file.

«- … this is similar to here-document, but it will ignore leading tabs.

<> … is similar to < but it will work no matter what, as shown. From stack overflow, a possible use case is listed.

Now, on to special parameters in bash 1,2 addheaders.sh command-line-a-to-z-new.md command-line-a-to-z.md getimagelinks.sh getimagelinks_special.sh imagedownload.sh imagelinks.txt images replaceimageurls.sh replaceimageurls_special.sh @ # ? - $ ! _ …

(aside) … a lot of these special characters are described here: https://tldp.org/LDP/abs/html/special-chars.html: … outputs 0, similar to, “true” – can be used in logic.

addheaders.sh command-line-a-to-z-new.md command-line-a-to-z.md getimagelinks.sh getimagelinks_special.sh imagedownload.sh imagelinks.txt images replaceimageurls.sh replaceimageurls_special.sh … the catch-all character, or wildcard which means, “anything.” For example, we can use it in grep to catch anything with certain leading characters.

$* vs $@ vs $# … if you run ./script.sh a b c ‘d e’ then “$@” will pay attention to the ‘d e’ as a separate argument specifically while “$*” will act as a wildcard, outputting the whole thing as one long string input. Without quotes they are the same. # shows number of args.

(continued) … note that “@” means “at” so it’s being used to designate specific, typed strings between ‘’ whereas “*” being the, “whatever” symbol is just throwing everything together as one. Taking away the quotes “” takes away their special powers.

$? … shows the exit status of the previous command, pretty straightforward. So if you want to create an if statement in a script for example, which runs if the last thing was in error, you can use $?.

$... shows the process id for the script in which it appears. So if you just do, "echo$” it will show you the PID for that particular run of echo.

vs ~+ … besides being minus, - also can designate the old previous working directory, OLDPWD, which is the previous previous working directory, whereas ~+ can designate the current PWD, and is equivalent to the pwd command.

So that basically covers the vast majority of symbols and special characters in bash. Bash is an interactive interface language, with a lot of pre-set settings running which are already in place when you use it. If you run commands from another location, it’s different.

If you’re using something like Python or Golang, different assumptions may be made in their various os or exec libraries about what settings are being used, and you can’t use the redirection and control operators native to bash.

So next I’ll go through some of the things I have learned about using the os package in Golang https://pkg.go.dev/os

to run what would be similar to bash scripts. I could also do similar exercises using Python.

https://pkg.go.dev/os#pkg-examples

… The os package as a whole does several fundamental things that bash can do in terms of interfacing with a unix-like machine. For example, if you want to do mkdir, you could use Golang os’ Mkdir

golang os package (continued) … in the example shown, this would be the equivalent of running in bash mkdir testdir and then echo "Hello, Gophers!" > testdir/testfile.txt https://pkg.go.dev/os#example-Mkdir

golang os/exec … basically, a part of the os package, this is the way to actually execute bash commands (vs. manipulate the directory structure and environmental variables). https://pkg.go.dev/os/exec

You can write your own bash scripts (or other language scripts) and then have exec.LookPath() see if it exists within path, which is like your $PATH variable. Once you executed LookPath you can then use(path)

Once you use(path) then there’s a Cmd struct field Path that is a string which gets set to the path in question. https://pkg.go.dev/os/exec#Cmd

The Cmd struct represents an external command being prepared or run. It has Args, Stdin, Stdout and Stderr, just like in bash. Args can be set dynamically since it’s a string list, you can, “build” your arguments. Stdin/Stdout/Stderr can be forwarded to different variables in Go.

So basically, while in bash everything is on the terminal, in Golang you can have inputs and outputs go to the terminal, or you can have them go into variables in Golang (which would be like sending outputs to an $ENV in bash).

Note that the Stdin is of type io.Writer and Stdout is of type io.Reader https://pkg.go.dev/io#Writer

& https://pkg.go.dev/io#Reader

… the io library provides access to, “io primitives,” which in my basic reading is a way to write to disk (via io).

How does writing to disk have anything to do with Stdout/Stdin/Stderr? Well in unix-like systems, “everything is a file,” and that includes the standard inputs/outputs. So simplistically io.Reader and io.Writer are just ways of classifying that type of file.

tr Example - use exec.Command() to create a Cmd struct, feeding in Args as strings in a []string (string array). Then you set the Stdin to an io.Reader using NewReader, which outputs a io.Reader type to be used as input with string “some input”

We fed, “some input” into the Stdin by setting Stdin to that io.Reader, generated by strings.NewReader() which takes in a string and outputs an io.Reader. Within exec.Command() we ran “tr” with args, “a-z” and “A-Z”, which will run tr on the stdin with those args.

So turning this into a diagram, we used strings and string lists (shown in red) as the command and argument, and generated an input that stdin would understand with NewReader(). The stdout output was set to bytes.Buffer, which means print out on the terminal.

If you wanted to convert to a string output rather than print out stdout just to the buffer, then you can create a strings.Builder object, then do, “io.Copy()” to copy that outReader into the buf, then finally print buf.String() to a variable which will be a string!

/bin/bash A-to-Z was originally published by Patrick Delaney at Patrick Delaney on August 05, 2022.

Simple Bash-Based Code Performance Measuring Tool

2022-03-10T00:00:00-06:00

Project Summary:

A lengthy how-to guide on writing a shell script that measures code execution time.

Codeperform Project

The purpose of this project is to build a simple code performance measuring tool which includes an output that captures the relative time of execution of two different programs, ideally of similar output and purpose, to understand on a minute scale which execution path is faster.

The idea here is to create a benchmark which helps determine what type of program runs faster on a particular machine. Part of the impetus for this project is just basic computer language benchmarks, such as the Computer Language Benchmarks Game.

Often times in benchmarking we may think of in terms of performance benchmarks which roughly compare two different languages across multiple different benchmark algorithms, such as fannkuck-redux, and may compare two or more different languages against each other, such as C++ and Rust.

Well, what if you wanted just a quick tool to compare two little quick programs that you built yourself?

Existing Tools

Of course it’s always temping to think that one is the first one to have thought of an idea, but that is almost never the case.

Built-in Timer Tools

Most languages likely have built-in timing functions. If you look at the built in time module in Python, then you’ll notice several functions that can measure time:

monotonic()
perf_counter()
process_time()
time()

These types of programs can measure time within the interpretive language itself, but what about just a simple shell script which can measure the relative time of any program, regardless of the language it was written in?

Rough Outline

The command line tool, “strace” can be used to monitor application performance throughout the execution of a program.

The pseudocode version of what we wish to build would be the following:

# start program

# input files should have been inputs to the shell program

# run the first program using strace

# A do an awk or sed on the output on the first line to grab the timestamp

# B do an awk or sed on the output on the first line to grab the timestamp

# subtract the last_timestamp from the first_timestamp

# run the second porgram using strace

# repeat A and B above on the second program

# output metrics between the two programs

Working through the Bash Script

Creating the File, Initiating with #! Shebang Operator

We can implement the above as a bash script by creating a file, “codeperform.sh,” and adding the shebang “#!/bin/bash/” operator at the very start of the file to indicate that this script is going to run as bash (as opposed to zshell or /bin/sh).

Of course, this is easier said than done, as an absolute path is needed specifying from the root directory of a linux or unix based system, meaning that we can’t just put, “#!/bin/bash” we have to put either:

#!/usr/bin/env bash

…which has the benefit of looking for whatever the default version of the program is running in your current environment, whereas:

#!/usr/bin/bash

would be useful if it was a one-line command in which more than one argument needed to be passed in at the onset, with a command such as:

/user/bin/env awk -f

So basically, the form, “#!/usr/bin/env bash” is more portable, but has limitations in terms of what executables are called.

Adding Arguments to the Bash Script

Bash uses a tool called, “positional parameters,” to provide a means of etnering data into a Bash program when it is run from the command line. There are ten possible positional parameters that run from $0 to $9.

After entering in a simple positional parameter, “echo $0” (and making the script executable by applying chmod +x filename.sh), we run the script with no parameters to start off with:

$ ./codeperform.sh 1
./codeperform.sh

Basically, the $0 position is reserved for the predefined name of the running script and can’t be used for anything else. So instead starting with $1 as the positional parameter, and using two parameters to start off with:

$ ./codeperform.sh -h -f
-h

As you can see, the flag -h just repeats and the second parameter, -f does nothing because we only have one parameter input at this point. If the code instead used two parameters and we inserted those paremters, we would get the actual output of both parameters as shown below:

#!/usr/bin/env bash
echo $1
echo $2

and then:

$ ./codeperform.sh -h -f
-h
-f

Creating a Help Function

It’s a best practice to always include a help function within a shell script. Something to note about running functions on bash is that functions can be essentially inserted into memory as a variable enclosed within the {} brackets just like a variable. Whereas a variable declaration may be something along the lines of:

EXAMPLE_VARIABLE=2
...
echo $EXAMPLE_VARIABLE
2

Which would store the EXAMPLE_VARIABLE, equaling 2 into memory for super fast access, the same can be done with a function itself using:

help(){ echo "This is the unhelpful help file. Goodbye."; }

Just as you could use, “printenv” to print all variables in an environment, you can do the same for functions with, “compgen -A function” and of course filter out the result with grep:

compgen -A function | grep help
help

Running the -help file itself would be just a matter adding the help function into the script, and then calling in the, “main” part of the script.

#!/usr/bin/env bash
# -------------------- help --------------------
help(){ echo "This is the unhelpful help file. Goodbye.";}

# -------------------- main --------------------
help

This will of course run the help file as a default rather than as an option flag.

Turning Help into a Flag Rather than a Default

The help flag, as well as other flags, is accomplished with a while loop including different cases within the main part of the program.

# -------------------- main --------------------

while getopts ":h" option; do
   case $option in
      h) # display Help
         help
         exit;;
      *) # invalid cases
         echo 'Invalid option. Find options with flag: -h'
         exit;;
   esac
done

By including -h as a flag within the function, this access the help function, which prints out that help file we created above.

Running strace on One Program

We have created a simple c program called, “hello.c” which essentially does the following:

#include <stdio.h>
int
main(int argc, char *argv[])
{
  printf("hi!\n");
  return 0;
}

We compile this into an executable called, “whatever” using the command, “gcc hello.c” and then we can run the outputted file (after changing its name) with

./whatever

Basically all the program does is say, “hi.”

So now if we want to run strace on this, a linux box is required. Attempting to use this on MacOS fails, as attempting to isntall with brew results in:

strace: Linux is required for this software.
linux-headers@4.4: Linux is required for this software.
Error: strace: Unsatisfied requirements failed this build

So, we can simply set up a lightweight ubuntu docker distro and mount the shell script we are writing here as a volume (or rather, as a bind mount on our local machine so that as we change the code, we can keep re-running it, but it also saves locally in our github repo).

A perhaps more easily recognizable name for this type of setup is a, “dev mode container.”

In order to set up a dev mode container, we first have to establish a folder structure under which we can hold a Dockerfile and our application files, so that we can point to the source code to copy into the Image which will be the basis for the container. Secondly we set up a run command which bind mounts the appropriate folder which contains our code so that we can access and edit it within the running container.

So first off, our folder structure will look like the following:

project
│   codeperform.md
│   Dockerfile
│
└───app
    │
    └───codeperform.sh
    |___otherfiles.files

Basically, “/app” is where we we keep everything that needs to be, “bind mounted,” meaning, we can change the files on our disk and it will reflect on the container, and vice-versa.

The Dockerfile itself, in order to copy the file from the local machine into the Docker Image during the build process, must include a, “COPY” clause, as follows:

# syntax=docker/dockerfile:1
FROM ubuntu:latest

# copy the local app file into the image
COPY app /usr/destination

So, “app” is the source file, whereas “/usr/destination” is the destination file. Of course without opening a container directory structure first, it’s impossible to know what the best place to put these files might be on the destination Docker Image, so a good practice would be to simply build and run the image with an, “exec” funtion to essentially log in, or rather, exec in to the container to explore and find the right place first. From within the same directory as the Dockerfile:

docker build -t codeperform_ubuntu_image:latest .
docker run  -t -d --name codeperform_ubuntu_container codeperform_ubuntu_image
docker exec -t -i codeperform_ubuntu_container /bin/bash

From here we see that the directory structure is:

bin  boot  dev  etc ~  lib  lib32  lib64  libx32  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

…and ~” seems like a good place, so we can copy our files to, ~/app” by modifying the Dockerfile:

# copy files from local directory
COPY app ~/app

After deleting the running Docker Container, running through the above commands again puts the files we need in, ~/app”.

The Dockerfile defines how an image is built, not how it’s used, so you can’t specify the bind mount in a Dockerfile. We could create a declarative specification for including the Dockerfile in question, and including a bind mount or volume, allowing us to simply run, “docker-compose up -d” rather than a complex Docker command line, by including a docker-compose.yml file with the following:

version: '3.1'

services:
  codeperform_ubuntu_container:
    image: codeperform_ubuntu_image:latest
    build: .
    volumes:
      - type: bind
        source: ./app
        target: ~/app

We can run the above by navigating into the folder where this docker-compose.yml file sits and then running, “docker-compose up -d” - which follows the commands shown above and then runs the container, mounting the volume with at the specified directory noted as a canonical path ending with the ./app folder (with the ./ included to indicate a relative path), and connecting that to ~/app on the container.

Of course, when we run, “docker-compose up -d” or “docker-compose up” the container exits after the process has completed.

If we look at the verbose version of this command, we get some feedback:

docker-compose --verbose up

...

 'Config': {'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['bin/bash'],
            'Domainname': '',
            'Entrypoint': None,

...

Basically, there was a ‘Cmd’ being issued to use bin/bash, but no terminal, so trying to use bash without a terminal makes the container exit immediately. When bash starts up with no terminal attached, it has no script or command to by design, it exits.

In order to attach the terminal, we use tty: true. We can also add, “command: ‘bin/bash’” for good measure to be explicit to our docker-compose.yml:

    tty: true
    command: 'bin/bash'

After this was fixed successfully, we can run the following to get everything up and going:

docker-compose up -d
docker exec -t -i codeperform_ubuntu_container /bin/bash

This will compose the entire image, build it if not already built, and then exec into the container with bash.

Side note, we could have added a command or entrypoint to the Dockerfile itself rather than the compose file. These commands would basically tell the image by default to run an entrypoint or to bash any time the image runs. If we make any changes to the Dockerfile, we may need to re-build the image with, “docker-compose up –build” e.g., using the –build option.

Ensuring that Changes in the Container Reflect in the Code on Local

So the reason we have set up this fancy Dev Mode Container is to be able to make changes within our Container and have them save on our local so that we can push to Github, essentially to be able to save our work.

So to run a simple test, we can do the following from within the container:

~/app# echo 'hello worlds' >> hellotest.txt

Which we can see saves a file on our local. Removing this file also removes it on local.

Checking the Shell Script with strace

So now we have a working Ubuntu container, so we should be able to install strace. Once we have figured out how to install it manually within the container, we can go back and re-compose the container with “docker-compose build up -d”.

So we can start out by running:

# apt-get update

...

# apt-get install strace

...

# strace -V
strace -- version 5.5
Copyright (c) 1991-2020 The strace developers <https://strace.io>.

When we opt to add this into our Dockerfile, we can opt in if we so choose to explicitly install strace 5.5 for compatibility purposes.

Testing out Strace on Code

So having installed strace, we can install our, “whatever” pre-compiled C code on our new Ubuntu container.

strace -ttT ./whatever
19:26:11.555004 execve("./whatever", ["./whatever"], 0x7ffc856beb38 /* 9 vars */) = -1 ENOEXEC (Exec format error) <0.004814>
strace: exec: Exec format error
19:26:11.563988 +++ exited with 1 +++

However when we try to do this, we get the above error. This is because originally the file was compiled on a Mac with OSX rather than on a linux system, without a cross-compiler. There is a way to compile code such that it works across systems with the, “binutils” library and gcc and a string of commands. However since we have the Ubuntu container up and running, we can simply re-compile on Ubuntu with Gnu Compiler Collection.

apt-get update
...
apt-get install build-essential
...
gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.

GNU Compiler Collection (GCC)

We can now overwrite our, “whatever” file by using gcc and hello.c:

gcc hello.c -o whatever

Now having compiled the file properly according to Ubuntu’s binary requirements, we can run, “./whatever” which just prints out, “hi!” - so we can then use strace:

strace -ttT ./whatever
19:34:22.804636 execve("./whatever", ["./whatever"], 0x7ffe65523658 /* 9 vars */) = 0 <0.001316>
...

Which does print out a timestamped output, one line of which is shown above.

Using Strace and Awk to Output

Looking at, “strace -h” we can see that we also have the option of printing out just statistics with the -c flag:

strace -c ./whatever
hi!
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 28.72    0.000407         407         1           execve
 20.54    0.000291          41         7           mmap
 12.00    0.000170          85         2         1 arch_prctl
  8.33    0.000118          19         6           pread64
  7.76    0.000110          36         3           brk
  6.49    0.000092          46         2           openat
  6.07    0.000086          43         2           close
  2.96    0.000042          42         1           munmap
  2.61    0.000037          12         3           mprotect
  1.69    0.000024           8         3           fstat
  1.13    0.000016          16         1         1 access
  0.99    0.000014          14         1           write
  0.71    0.000010          10         1           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.001417                    33         2 total

Note that the total calculated seconds is shown at the bottom of, “seconds,” with 0.001417 seconds to be precise, which may be a much easier piece of data to grab than actually subtracting and calculating the time to execute. Sure enough, with the right command we can directly extract that value onto the stdout:

strace -c ./whatever 2>&1 >/dev/null | awk 'END{print $2}'
0.001196

Note that the timed value, 0.001196 is different than 0.001417. Through the process of building the above command, it was obvious that each time strace was run, the timed value for the command was different. Reading further into what strace really does, it turns out that strace actually actively slows down the running of a program while it executes the program, by up to 10 times! Essentially every time that strace runs a step in the application, it exits back into its own shell to do some calculations, then it jumps back in and runs the next step of the application again.

Ultimately, it makes sense that the time measurements are different every time, as the computational load on a CPU may differ depending what else a machine may be doing at any given time. However, how reliable or useful strace may be as just a, “bulk measurement,” of a program may be questionable.

Using the GNU time Command

The GNU Time Command…

runs the specified program command with the given arguments. When command finishes, time writes a message to standard error giving timing statistics about this program run.

Essentially,

Note that:

Note: some shells (e.g., bash(1)) have a built-in time command that provides similar information on the usage of time and possibly other resources. To access the real command, you may need to specify its pathname (something like /usr/bin/time).

Looking at our /usr/bin, we find:

:/usr/bin# ls | grep time
timeout
uptime

Basically, we don’t have the, “real” time installed, so we can try to do that with:

apt-get install time

...

:/usr/bin# ls | grep time
time
timeout
uptime

Once we have this installed, we can invoke the actual, “real” time command with:

/usr/bin/time time -p ./whatever

hi!
real 0.00
user 0.00
sys 0.00

Note, this is different than if we just use, “time ./whaever” which does actually print out a different set of results:

time ./whatever
hi!

real	0m0.004s
user	0m0.002s
sys	0m0.001s

Playing around with this time function, we see that there is a help flag with –help, as well as –verbose, –quiet, which supresses non-zero exit status and –format=format which formats the output according to rules given on the help page, which specify a, “printf like way.” So to replicate the results of -p, we can format as follows:

/usr/bin/time time -f "real %E\nuser %U\nsys %S\n" ./whatever

hi!

real 0:00.00
user 0.00
sys 0.00

Looking at the precision of this output, what we see is unfortunately the maximum precision we can get with is centiseconds, whereas the bash time command can work with a bit more precision. We can find documentation for the builtin bash time function by searching under, “TIMEFORMAT” within here. Basically the only way we can adjust the time format is with “TIMEFORMAT=%3R” as a setting within bash.

We can also reset the timeformat to the original -p value with:

TIMEFORMAT=$'\nreal\t%3lR\nuser\t%3lU\nsys\t%3lS'

time ./whatever
hi!
0.003

Here with this function of, “time” the only option avaialble is -p, which shortens the precision.

So therefore, pushing this into awk:

{ time ./whatever ; } 2> result.txt
hi!
...

awk '{ print }' result.txt
0.003

So basically, rounding this out, we have the following command which cleanly prints out the real time in seconds:

{ time ./whatever ; } 1> /dev/null | awk '{ print }'
0.003

Putting This Back Into codeperform.sh

So going back into our functions, we insert:

timetest()
{
   # format time to seconds only, real time, 3 significant digits
   TIMEFORMAT=%3R
   # run time function on
   # use $@ as a general variable input
   THETIME=$({ time "$@" ; } 1> /dev/null | awk '{ print }');
}

Which actually creates a shellcheck error, saying that the redirection overrides the output pipe, so we should use “tee” to prevent that. So we change the above to:

timetest()
{
   # format time to seconds only, real time, 3 significant digits
   TIMEFORMAT=%3R
   # run time function on
   # use $@ as a general variable input
   THETIME=$({ time "$@" ; } | tee 1> /dev/null awk '{ print }');
}

Which, this takes away the error, but our variable THETIME is not being reported down under the functon call below. The reason we continued to use /dev/null is to get around awk ‘{ print }’ going and printing a new line as the output of the time function. We can get around that quirk by using printf:

{ time ./whatever ; } | awk 'BEGIN{ printf "" }'
0.003

So now we don’t have to use tee to send the results to awk, and we should have a result with no new line if we put this back in our script. However, it just results in more complications of course with confusions between stdout and stderr. So instead the better route was to simply place the stderr into an environmental variable, and then run awk on that.

timetest()
{
   # format time to seconds only, real time, 3 significant digits
   TIMEFORMAT=%3R
   # run time function on
   # use $@ as a general variable input
   TIMETOSTDR=$({ time "$@" ; } 1> /dev/null);
   THETIME=$(awk '{ print }' "$TIMETOSTDR");

}

However, if we do some debugging, we see that what we expect to be happening with variable assignment is not happening at all. Basically, what we had expected was that our TIMETOSTDR was being assigned the stderr of the command, “time $@” but instead what was happening was that it was getting assigned stdout. This was found by placing some, “echo” statements within the above function and observing the outputs of variables at different points.

The real way to assign the stderr output into a variable would be the following. Side note, using /dev/null results in a faster response time than just some random variable such as, “output” - not sure why, but something computationally intensive on the order of 5 microseconds is going on.:

WAKA=$({ time ./whatever > /dev/null ; } 2>&1)
...
echo $WAKA
0.003

So replacing our faulty variable assignment code, we then have the following, with echo "$@" thrown in for good measure to make sure we’re indeed evaluating both input applications.

timetest()
{
   # format time to seconds only, real time, 3 significant digits
   TIMEFORMAT=%3R
   # run time function on
   # use $@ as a general variable input
   TIMETOSTDERR=$({ time "$@" > /dev/null ; } 2>&1)
   echo "$@"
}

The above runs the code flawelessly, showing the time difference for each test.

Now, if we re-do our hello10.c code so that it prints out “hello” a million times and re-compile it, testing it with our evaluator tool, we get:

# ./codeperform.sh './whatever' './whatever10'
APP1 execution time was 0.002 seconds.
APP2 execution time was 0.035 seconds.

Now we are talking about some serious difference between the two real execution times for this one operation, on the order of centiseconds vs. milliseconds.

Creating Mathematical Output

Bash does not support floating-point arithmetic, so we need an external tool, bc, which needs to be installed.

differencetest()
{
   # run difference on two variables
   # use $@ as a general variable input
   THEDIFFERENCE=$(echo "$1-$2" | bc)
   echo "$1"
}

The bc function removes the trailing zero off of the result, but interestingly, awk can do floating point arithmetic as well as customize the significant digits:

THEDIFFERENCE=$(echo "$1-$2" | bc | awk '{printf "%.3f", $0}')

So we may actually consider just awk for compatibility, but this code looks cleaner so we will keep it.

Besides difference, it would also be helpful to have a percentage difference.

Executing on Python Code

So the first thing to do is install a python3-minimal base in our container:

apt-get update -y
apt-get install -y python3-minimal

In order to operate on a python script that we create, basically on a stock, “hello world” function, we have to put python within brackets, which essentially opens up a command shell, executing the code, and then pushing the output of that to time.

time { python3 hello.py; }
hello world
0.022

Translating this to our own bash script is a bit problematic, because if we pump in that command, basically the functions we wrote have no idea what to do with it:

./codeperform.sh { python3 hello.py; }

If we enter the function name as an argument into our bash script, we can echo it back out as an output, so we know that we can actually put a string into the our script which at least is the title of the script. From there, we can hypothetically run an if function to look for the .py extension and then have a conditional that pushes the evaluation into the proper function which measures the time of execution, including those {} if necessary.

We can test out with grep rather than regex, since grep is a lot easier to use:

if echo "hello.py" | grep -q .py; then
    echo "MATCH!"
fi
...

MATCH!

Testing an alternative case:

if echo "hello.rb" | grep -q .py; then
    echo "MATCH!";
fi
    echo "NO MATCH!";
...
NO MATCH!

So hypothetically an if statement could assign a variable based upon whether a .py exists within the input filename, which using the grep operator, is pretty easy to accomplish:

apptype()
{
   # if the input program has .py, then it's python
   if echo "$@" | grep -q .py; then
      APPTYPE="python"
   else
      APPTYPE="unknown"
   fi
}

Once we have the proper application type labeled, we can then enter into the proper if statement to execute the code in the way it needs to be executed, so in the case of python:

   elif [ "$APPTYPE" == "python" ]; then
      TIMETOSTDERR=$({ time python3 "$@" > /dev/null ; } 2>&1)

Finally running the code with a compiled c application vs. our hello.py:

./codeperform.sh './whatever10' hello.py
./whatever10 execution time was 0.037 seconds.
hello.py execution time was 0.019 seconds.
the difference between (./whatever10) - (hello.py) is -0.018 seconds.

Comparing a Vectorized Operation vs. Non Vectorized Operation

Python has the ability to vectorize through numpy. That being said, python-slim does not have the capability to install numpy with pip3, so we may need to manually install a binary.

Starting out, a brief python program which calculates the dot product manually via a for loop can be created as follows:

import array

# 8 bytes size int
a = array.array('q')
for i in range(100000):
    a.append(i);

b = array.array('q')
for i in range(100000, 200000):
    b.append(i)

dot = 0.0;

for i in range(len(a)):
      dot += a[i] * b[i]

print("dot_product = "+ str(dot));

Attempting to run this through our new codeperform.sh results in:

# ./codeperform.sh dotproduct.py
dotproduct.py execution time was 0.088 seconds

88 ms is not very much! It would be great to have closer to a tenth of a second perhaps, to see if we can get some serious performance difference between two different ways of doing this operation. Upping the order of magnitude on the array sizes to 10^6 gets us in the 0.6 second range.

So that being said, we have to figure out how to install numpy and compile the binary manually. To find out which version of python we’re using, simply do python3 –version:

Python 3.8.10

Looking on the numpy.org/news site, we see that Numpy 1.22.0 is compatible with Python 3.8. We have to remember to install pip.

pip --version
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)

So creating a requirements.txt, we explicitly call out numpy==1.22.0 and then compile from binary with:

apt-get update && \
    apt-get install -y \
        build-essential \
        make \
        gcc \
    && pip install -r requirements.txt \
    && apt-get remove -y --purge make gcc build-essential \
    && apt-get autoremove -y \
    && rm -rf /var/lib/apt/lists/*

After this and testing the python shell, we are able to see that importing numpy is no problem.

Now, comparing the two methods in our new shell tool, codeperform, we get (using the 10^6 order of magnitude):

# ./codeperform.sh dotproductvect.py dotproduct.py
dotproductvect.py execution time was 0.444 seconds.
dotproduct.py execution time was 0.633 seconds.
the difference between (dotproductvect.py) - (dotproduct.py) is 0.189 seconds.

Running this again, with the 10^5 order of magnitude:

./codeperform.sh dotproductvect.py dotproduct.py
dotproductvect.py execution time was 0.176 seconds.
dotproduct.py execution time was 0.088 seconds.
the difference between (dotproductvect.py) - (dotproduct.py) is -0.088 seconds.

Whoa! This was not at all the performance boost that I was expecting. Python-centric articles online recommend using the embedded python, “tic/toc” function from the time library, which show that in the 10^5 order of magnitude range, the vectorization method is supposed to be 100 times faster, however in our case, it’s 2 times slower! Of course in the 10^6 range, it’s faster, but only by 1.4. Of course that performance adds up over time, but what’s interesting is that the vectorization, from a real-time, “outside of the python evnironment,” world, only seems to make sense past a certain quantity of bytes being added into a variable. It’s almost as though adding those values into an array has a fixed cost, below which it’s not really worth it to use vectorization and you might as well use a for-loop if it’s not at least a million or so values (or whatever the byte equivalent might be).

If we go back to our trusty old strace, and run the following two commands with our values at the 10^5 level:

strace python3 dotproduct.py 2> linecount.txt
strace python3 dotproductvect.py 2> linecountvect.txt

We can see that the dotproduct.py application executes 598 lines of code while the dotproductvect.py application executes 2939 lines of code!

If we change our values in the files back to the 10^6, on dotproduct.py and dotproductvect.py, we get:

674 lines executed on dotproduct.py
3039 lines executed on dotproductvect.py

Of course, running the timer again on both functions does come up with a similar ratio of timing, with the vector version being about 1.4 times faster than the for-loop version.

So basically, it’s not that the for-loop version executes way more lines of code either way, it’s just that one of those executions is a command to, “do the thing a lot of times,” whereas the overhead generally for using that numpy array is really high.

Updating the Help File

So now that we have our tool essentially built and functioning with binaries and python, it’s time to write a help file.

In short, the help file is visible with the -h flag. I wanted to make it look a bit cool and hacker-ish, so I added some separator brackets and bars.

Error Processing, Other Shell Best Practices

Unfortunately this is a very simple project, so I didn’t add any error processing or handling.

Creating a Dynamic Link to This Shell Command

It’s kind of annoying to keep having to type out, “./whatevercommand.sh” every time one has to run a program. So basically we’re going to copy this into our ~/bin folder and then ensure the path to that folder is included in our bash profile.

So first, we just want this command available to our user, so we can create a file:

mkdir ~/bin

Then we can copy our codeperform.sh into that file, give it a new name and change the permissions with chmod.

cp codeperform.sh ~/bin/codeperform
chmod +x ~/bin/codeperform

Next, we can add an export to put the path variable into our .bashrc profile:

echo "export PATH=\$PATH:~/bin" >> ~/.bashrc

Then, we can restart / source bash:

source ~/.bashrc

So finally we can use codeperform as simply a command:

codeperform hello.py
hello.py execution time was 0.022 seconds.

We can figure out how to add this into the docker image if we so choose.

Updating the Docker File to Include Needed Functions

First, we should explicitly call out the Ubuntu version for the Docker image.

cat /etc/issue
Ubuntu 20.04.4 LTS \n \l

Which would correspond to:

ubuntu:20.04

Next, we need to install our dependencies, the first of which was bc, which was used in a mathematical operation.

The versions we used were bc 1.07.1, Python 3.8.10

RUN apt-get update                                                 \
    # install shell tools, bc, python
    apt-get install -y --no-install-recommends                     \
        bc=1.07.*                                                  \
        python3-minimal=3.8.10                                     \
        python3-pip=20.0.2                                         \
    apt-get clean                                               && \
    rm -rf /var/lib/apt/lists/*

# install python requirements.txt via binary
RUN apt-get update && \
    apt-get install -y \
        build-essential \
        make \
        gcc \
    && pip install -r requirements.txt \
    && apt-get remove -y --purge make gcc build-essential \
    && apt-get autoremove -y \
    && rm -rf /var/lib/apt/lists/*

After adding the above to the Dockerfile, and deleting the old image and container, we can run Docker-compose again to build this image and get it up and running.

docker-compose up -d
docker exec -t -i codeperform_ubuntu_container /bin/bash

The first problem we have with the above method is, executor failed running [/bin/sh -c apt-get update which is because the Docker runtime which is running the installation is running /bin/sh rather than /bin/bash.

Once we have a, RUN /bin/bash statement included in the Dockerfile, this error is cleared, but then we have a problem with our apt-update transition to apt-install, which can be fixed by adding && \, with the “&&” term being used to mean, “execute the next command only if the preceding command exited without errors. This is of course different than ending a line with “;” which means, “just do the next lineno matter what,” and “ ” means, exit if the previous command failed (so it’s kind of like a fixit). So, we’re sing dependencies, so each line is dependant upon the last one executing properly.

RUN apt-get update && \

The other thing we’re trying to do with this Dockerfile, is to be as explicit we can be with the versions of packages that we’re installing. Of course, this leads to issues if the version numbers are not exactly correct, as with:

#12 4.920 E: Version ‘3.8.10’ for ‘python3-minimal’ was not found
#12 4.920 E: Version ‘20.0.2’ for ‘python3-pip’ was not found
Searching online through the Ubuntu packages repo at packages.ubuntu.com for Ubuntu 20.04, even though the python version is 3.8.10, the actual python3-minimal version is, “3.8.2-0ubuntu2”.
As far as python-pip goes, it appears there is not a dedicated python-pip for Ubuntu 20.04, but there is one for Ubuntu 18.04, “9.0.1-2.3~ubuntu1.18.04.5” - so we can try that, although there is a, python-pip-whl with version, “20.0.2-5ubuntu1.5” as well.
Looking a bit closer, even though the, “pip” command is used in our Dockerfile, the actual package we’re trying to install is “python3-pip=20.0.2-5ubuntu1.5” which is compatible with Ubuntu 20.04. Ideally, the alias, “pip” will work and the bash shell within Docker will not be required to use, “pip3.”

When we use this package, python3-pip, then we get the error that we’re missing the “whl” version, so likely it’s required for installing python3-pip, as a whl (zip) type package:

#12 4.915 The following packages have unmet dependencies:
#12 5.003 python3-pip : Depends: python-pip-whl (= 20.0.2-5ubuntu1.5) but 20.0.2-5ubuntu1.6 is to be installed

Basically, it’s not clear which version of python3-pip is used, and what kinds of dependency errors are created when attempting to install, so instead of listing out the exact version, we do:

python3-pip=20.0.*                                      && \

Which basically installs whatever the latest python3-pip verson of 20.0 is, minimizing the chance of erros while being as explicit as possible. This could cause problems later, but not as likely as just doing, “python3-pip” with no version.

Finally, we get an error that Docker can’t access the requirements file.

ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'

To fix this, we simply have to move into the proper working directory within the Dockerfile in order to able to access that requirements.txt file.

# move into the proper working directory
WORKDIR /home/app

Finally, it installs. However when attempting to run the container, we get another error:

OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "bin/sh": stat bin/sh: no such file or directory: unknown

For some reason, the docker-compose command command: 'bin/bash' won’t allow us to immediately bash into the container. However, if we do the following command after the container is built, it works. This evidently could be in part because we do: RUN /bin/bash in the Dockerfile whereas previously we did not.

docker exec -t -i codeperform_ubuntu_container /bin/bash

So, upon logging in, we can test to see if our various tools work:

$ docker exec -t -i codeperform_ubuntu_container /bin/bash
python3 --version
Python 3.8.10
bc --version
bc 1.07.1
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006, 2008, 2012-2017 Free Software Foundation, Inc.
pip --version
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)
python3
Python 3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>>

Basically, everything we expected is working. Since docker works on layers, it’s good to have the Dockerfile design such that all of these, “heavier weight” things, basically installing dependencies, happen in a, “lower layer,” meaning higher up on the Dockerfile, because they don’t have to run again and again, we just have to do quick fixes on the last bit of code at the end. The only thing that’s not as certain is that, since we are copying files from the local, /app/ folder and mounting them, would it be appropriate to move the COPY command lower, if for example, we add additional dependencies to the requirements.txt file? Likely it would be best to add this command at least below the previous

The only thing that’s not working is of course, our alias for our custom command. This requires running the above commands that we had earlier established to install, “codeperform” into ~/bin, as well as the bash profile, and then make it executable. So we add that:

RUN mkdir ~/bin                                                 && \
    cp codeperform.sh ~/bin/codeperform                         && \
    chmod +x ~/bin/codeperform                                  && \
    echo "export PATH=\$PATH:~/bin" >> ~/.bashrc                && \
    source ~/.bashrc

So what we found is that, we weren’t really using bash the whole time at all, we were using /bin/sh, which is why “source” didn’t work but using “. ~/.bashrc” did work. Source is not included in /bin/sh. Basically the command at the top of the Dockerfile, “RUN /bin/bash” wasn’t really running bash for the whole Dockerfile, it was just doing it for that one command. Instead, to switch actual shells, we have to use:

SHELL ["/bin/bash", "-euo", "pipefail", "-c"]

However, this causes a bunch of errors dealing with environment variables, since now we’re in bash rather than bin/sh. So instead of adding the above, we just removed, “RUN /bin/bash” and changed, “source” to “.”.

So after that, everything works as expected!

Conclusion

One thing to note is that, if a changes is made to codeperform.sh, it does not automatically change the, “codeperform” tool, this is only done at initialization or setup of the image.
Also, since in order to change the setup image requires a change in the Dockerfile, even if you change the codeperform.sh file, it may not change the base image, because the Dockerfile wasn’t changed.

Ideas for Future Improvement

For c code, rather than requiring binaries, have the tool compile the whatever.c file and then run the binary.
Include functionality for rust, go and other relevant languages.
Compare the execution time on a 10^5 and 10^6 dot product within c code vs. python.

Sources

Simple Bash-Based Code Performance Measuring Tool was originally published by Patrick Delaney at Patrick Delaney on March 10, 2022.

Text Generator with Human Editors

2021-07-13T00:00:00-05:00

Project Summary:

Flask Web App wrapper for GPT2, with administrative protection capabilities and dual-user type, including a, “sponsor” and an, “editor” which can interact with machine generated text.

Language Generator Programs Documentation

Author

Patrick Delaney, July 2021

Dependencies

Description / Purpose	Badge	Note
Language Models
DNN Toolkit
Web App
Database Mapper
Container Image
Database Adapter		Do not use binary for production.

Repos Covered and General Description
Getting Started - How to Install and Run
- Quickstart on Ubuntu
Project Structure for Machine Learning
- Word on Monolithic Applications
- This Application Project Structure
  - Alternative Data Organization Open Source Projects
  - How GPT2 Comes Into Play
Credits
License

Repos Covered and General Description

Repo Description

Link to Repo

The main link to the repo being referenced in this documentation is immediately below.

Item	Link	Description
src flask (or source flask)	src flask	A fully functional application which serves as a breadcrumb along the journey to creating a fully secured, deployable language generation application using python-flask and postgres.

That being said, there are several, “breadcrumb,” applications which preceded this application, building in various milestones of functionality along the way. For anyone who uses this application but may wish to fork off at a prior point along the build, they may want to reference one of the following endpoints.

Note, the following endpoints are ordered in reverse-sophistication, with the lower repos on the list being more basic and, “earlier,” in the process:

Item	Link	Description
flasksecurity	flasksecurity	Admin dashboard with login approval as well as a review and implementation of a security plan.
textgeneratornotes	textgeneratornotes	Text generator Google Colab Notebook with experiments.
userlevelmodelsflask	userlevelmodelsflask	Dual user type login functionality.
postgresloginherokudockerflask	postgresloginherokudockerflask	Flask application running on Docker, with Postgres and basic login.
herokudockerflask	herokudockerflask	Flask on Docker deployed on Heroku

Functionality Description

Simply put, this application generates text using GPT2 into an organized system that would allow humans to edit the odd, information-less machine generated text into working information. In other words, it’s a cyborg text generation application.
The app uses a combination of GPT2 and Flask integrated with Postgres to accomplish the above. The reason Flask was used is because of its flexibility and suitability for Machine Learning.
The main challenge (but at the same time advantage) in designing this app to the point where it is at is that Flask is more or less a blank slate, which while allowing a developer to implement their own database structures and source code wherever they would like into the ad-hoc platform, also requires a lot of dependencies and structure to be built from scratch. While getting flask up and going is relatively easy to begin with, there is not a lot of documentation on how to really architect many types of applications, so work and thought needs to be put in.
In the future this application could be further adapted to write an arbitrarily defined length of text, fine-tuned from a group of text bodies scraped from the web. Basically the idea would be to have a sponsor scrape a bunch of text using a search functionality, auto-generate machine text on a defined knowledgebase, and then pay for a human editor to perfect said text…hence, “cyborg text generation.”

Breakdown of Features

Built in python-flask, which is a largely, “from scratch,” platform which allows source code to be embedded within the application project structure, making it superior for machine learning or language processing/generation applications. [Python is widely used in machine learning applications, flask is a web application platform which does not restrict database architecture or project folder structure to any particular layout, as would be the case in for example, python-django.]
User classes separated into admin, sponsor and editors, with the admin having the capability to approve or reject the other two types of users prior to their activation within the system, allowing built-in resource protection.
Built-in common security vulnerability prevention, which flask does not come with, “out of the box,” with a full analysis of which security protections have been put in place and which have not at this stage of the app, and recommendations for further steps including but not limited to: 1. Decorated Routes which restrict access to certain routes to specified user types with a simple decorator. 2. 403 Error handling. 3. XSS, Cross Site Scripting protection. 4. CSRF Cross-Site Request Forgery protection. 5.SQL Injection Prohibition. 6. Directory Transversal Protection. 7. XSS Uploaded Files. 8. JSON Security. 9. Flask Security Headers. 10. Cookie Protection. 11. X content Type Options. 12. X-Frame Options. 13. X-XSS Protection. 14. HTTP Public Key Pinning. 15. Terminal Copy Paste Protection.
Full review of all security considerations can be found here.
Flash shell to allow arbitrary database commands.
SQLAlchemy integration for use with Postgres.
Initial non-user state for greater security, e.g. initial user only created on the server itself via human command.
CPU based operation of tensorflow - GPU/CUDA not required to generate text - therefore the development version can be run on essentially any modern 64 bit machine which can run docker.

System Requirements

Highlighted Dependencies

Breakdown of Highlighted dependencies shown at the top of this readme file.

Docker

The main desktop programs required to run this application in development mode are Docker Engine and Docker Compose, the requirements of which can be read about within the links provided.

This application was built on an HP-Omen 870-244, 16GB of DDR4, 2.6GHz i7 Processor.
With the dependencies listed in this readme file, one sample of machine-generated text could be created within about 30 to 60 seconds of processing time.
Note - with GPU/CUDA installed, this time goes down to about 3-5 seconds, however that has not been included in this application.

Behind the Scenes

Motivation

The world of statistical language processing has grown significantly in the 2010’s with the introduction of cheap deep neural network learning. Neural networks allow for the focus to be on, “word embeddings,” which basically means assigning codes to words and watch for the frequency with which those codes emerge in proximity to other codes, rather than, “rule based,” language processing, which strictly defines the order or grammar of words.

Much of the attention surrounding Natural Language Processing (NLP) has been concentrated around the capability to analyze and sense intent in either open or restricted data environments, either with financial gain ramifications or cost savings ramifications. For example, in healthcare, much diagnosis data is protected from being read or analyzed by humans due to privacy concerns, but data processing may be fair game for the interests of public health improvement. NLP is one of many methods that public health may be analyzed, by reading and analyzing diagnosis information in bulk from text, health outcomes may be improved.

The reverse of language processing is language generation. If a sufficient knowledgebase of words used in particular situations can be obtained, then sentences may be written responding to cues based upon statistical models. Today these models may not contain much substantial information, and may not be able to diagnose a problem, healthcare-related or otherwise, but in the future, with sufficient rule bases and proper text generation techniques, at the very least text generation may potentially be able to serve as a labor saving aid to humans, allowing authors to select and edit pre-written paragraphs around topic matter, rather than having a person write everything themselves.

Why Build this Project

Since text generation is such an interesting area, and in particular within the healthcare domain, it makes sense to open source some concepts using standardized web and server technologies available today so that others may be able to build more sophisticated applications in the future.

What Problems this Solves

There are many text generation applications, and at the simpliest level, this application is simply a wrapper for GPT2 at this point. The purpose of this application is not to re-invent text generation, but rather to suggest a direction for managing three key challenges:

The human interaction between automatically generated texts, including a, “generating” function which is wholly machine-generated and an “editing” function done by humans.
The challenge of organizing a set of application folders into a structure which mimics data science. More specifically than that, Python-Flask tends to be a standard way to build sophisticated machine learning applications today. However it is a very open, non-strictly architectured platform, it is more or less just scaffolding. This project attempts to architect a helpful folder structure which mimics good data science project design, basically putting the machine learning source code in an appropriate server-side location and laying out a philosophy for how to organize folders and files that interact with data science projects as they scale.
The challenge of resource protection and security is addressed by an administrative function. Access to sophisticated GPUs at least at present is not free, or at least not in a custom application sense. Working with expensive computing resources requires some type of way to administrate and protect said resources so they are not completely open to the world.

Learned Along the Way

Some helpful tips I learned while building this project:

Design database relational models in a way that will prevent you from having to do custom configuration. If you can link tables, link them.
For moving to production, sometimes there may be environmental variable name conflicts, where the new production system might demand that it uses a certain variable name and value, and this could cause problems in running the application.
Using a cheap, 3-year old GPU is roughly 10 times faster than using a CPU for two sentences of generated text on GPT2.
Sometimes there are variable name conflicts when moving to production from development, where a server, such as Heroku, has a fixed environmental name that it may use for something, that a dependency may also use as a convention. There are hacks to get around this which had to be employed to push an older version of this application to production on Heroku. Watch out!

Getting Started - How to Install and Run

All of the above repos run on Docker. Installation instructions for Docker can be found here.

Quickstart on Ubuntu

Ubuntu Install

The quick way to get started and install on Ubuntu terminal is the following, as of 11 July 2021:

# run updates

$ sudo apt-get update

# install docker

$  sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

# install docker GPG key

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# setup stable repository

$  echo \
  "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# install docker repo - run an update again just to be sure!

$ sudo apt-get update
$ sudo apt-get install docker-ce docker-ce-cli containerd.io

# test out to ensure you have installed docker with the basic hello world app

$ sudo docker run hello-world

Beyond the docker engine, docker-compose is also necessary. To install docker-compose, follow the instructions for Ubuntu at this link.

Or, for a quickstart guide, follow the below (as of 11 Jul 2021).

# download the current stable release of docker-compose.

$ sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

# apply executable permissions to the binary

$ sudo chmod +x /usr/local/bin/docker-compose

# to test installation, check the version. The version and build should pop up if installed correctly.

$ docker-compose --version

Application Build

Once you have installed docker-compose correctly, you can then run the actual application, by navigating to the root folder within your terminal and running the below command. The root folder is defined as the folder which contains docker-compose.yml:

# build the web service

$ sudo docker-compose up --detach --build web

The above should download all dependencies and build the entire, “container environment” along with the application which gets built on top of that container environment. If you’re not familiar with containers, I recommend reading this. In short, a container environment is a slimmed down simulation of a linux environment, which can be run on any operating system, whether it be Windows, MacOS, Linux, etc. assuming that said operating system has the capability to run the container environment. In our case we’re using the container environment called, “docker.”

The reason for using containers is to eliminate some of the problems associated with moving an application from a development environment, such as a laptop, to a production environment, such as a Platform as a Service type server, such as Docker or AWS-EC2.

Once you have, “composed” the web service in detached mode using the command above, you can then inspect which, “docker images,” now exist on your machine and their respective sizes. The terms, “compose, images” are specific to docker and can be read about more within the docker documentation.

$ sudo docker images

REPOSITORY         TAG               IMAGE ID       CREATED          SIZE
userlevels_flask   latest            2c8777dd50f9   29 minutes ago   2.53GB
python             3.8-slim-buster   0e0d73ddd34d   12 days ago      114MB
postgres           13-alpine         d3a70afcf848   2 weeks ago      191MB

The “python” image is a base image upon which our application image, userlevels_flask was constructed on.

Essentially, “python,” is a slimmed down version of a linux system which has python installed.
The, “postgres,” image follows the same concept, but with a postgres database installed.
“userlevels_flask” is our custom application which leverages these two below base images, as well as some other dependencies, including tensorflow, which is part of the reason for the large size of 2.53GB.
A, “normal” flask docker application may only be perhaps in the 100MB or less range, but because we’re leveraging tensorflow, which has a base size of around 2.5GB or so, depending upon the version, we have a fairly large size to our image shown above.

The images form instruction sets upon which, “processes,” get built, the processes run the application. To inspect the processes running, run the command in terminal:

# process inspection command

$ sudo docker ps -a

| CONTAINER ID | IMAGE              | COMMAND                | CREATED           | STATUS           | PORTS                                     | NAMES |
|--------------|--------------------|------------------------|-------------------|------------------|-------------------------------------------|-------|
| b45d45623de2 | userlevels_flask   | "/usr/src/theapp/ent…" | About an hour ago | Up About an hour | 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp | flask |
| c294603339c1 | postgres:13-alpine | "docker-entrypoint.s…" | About an hour ago | Up About an hour | 5432/tcp                                  | db    |

The processes shown above are:

The database, named, “db” and run using postgres on alpine linux.
The application, named “flask” with image name “userlevels_flask”

The endpoint for the application itself is found at:

http://0.0.0.0:5000/login

Which, if you visit in your browser should bring up the main front-end, endpoint of the application, which should look like this:

Usage of Web Interface in Development Mode

There are a couple of ready-made pre-approved sponsor and editor users on development mode which can be used to log in immediately:

# sponsor user
user: test@test.com
password: 123456

# editor user
user: editor@test.com
password: 123456

Generating Text

The sponsor user has the capability to, “create” a Document. A Document includes a title, body and an attached Editor-User. When a Document is created, the machine automatically creates an, “Autodoc,” which is an automatically created text snippet, created by GPT-2, reflecting the text of the Document as input.

How Documents/Autodocs get created is shown below. Note that the amount of time it takes for a CPU to create an Autodoc has been sped up significantly. :

Viewing and Editing Previously Created Documents

Once a document is created, Sponsor-Users have the capability to go back in and edit the Document Text, Title and Editor-User, as shown below:

Playing the Editor Role

Editor-Users are simply secondary accounts which do not have access to the Autodoc capability, but have the ability to edit the Document text and title, if assigned by a Sponsor, as shown below.

Role Authentication

The application is designed to be secured through a multi user-type categorization, which means that an administrator user type has the capability to accept or reject other user types prior to their application access. In development mode, we have a pre-set administrator user which can be accessed via the /login page with the following credentials:

user: admin@test.com
password: password

Upon logging in as an administrator, the main dashboard shows a couple different options:

However, in order to see any pending issues or data within the, “Signup Requests Dashboard” option, we have to go back to the main sign-in page and request to sign up either as a sponsor or as an editor.

Logging out and returning to the front page, by clicking the, “Sign Up as Sponsor,” link we can go in and create a requested sponsor-user.

Pending users can be viewed on the, “Signup Requests Dashboard.”

Allowing Pending Users

When a user signs up, they do not automatically have access to the application. The application protects its resources by only allowing certain paths to be accessible by certain user types.

In order for a, “new” user to gain access to a restricted resource, the admin must go in and, “Approve,” them as a user, as shown in the below video.

Project Structure for Machine Learning

Word on Monolithic Applications

This application is monolithic, in that it is designed to completely run and fit on one server. Technically, there are two, “containers” - one for the application itself, and one for the database, postgres.

As an application grows in size, or as it becomes more of an, “enterprise grade,” application with teams working through it, it starts to take more of the shape of a graph, with different parts of the application doing different things in various containers, essentially using a, “microservices,” architecture.

This Application Project Structure

The main project structure for this application, including the web application, but not zooming in on the data science portion of the project looks like the following:

├── .env.dev
├── .env.prod
├── .env.prod.db
├── .gitignore
├── docker-compose.prod.yml
├── docker-compose.yml
└── services
	├── nginx
	│   	├── Dockerfile
	│   	└── nginx.conf
	└── web
	    	├── Dockerfile
    		├── Dockerfile.prod
    		├── entrypoint.prod.sh
    		├── entrypoint.sh
    		├── manage.py
     		├── requirements.txt
    		├── project
    			├── __init__.py
    			├── assets.py
    			├── auth.py
    			├── forms.py
    			├── models.py
    			├── routes.py
    			├── config.py
    			└── static
	    			├── /css
	    			├── /dist
	    			├── /img
	    			├── /src
		    			└── js
	    			      └── style.css	    			
    			           └── /templates

The data science portion of the project goes under static/src, even though the data itself is not static, the code which manipulates the data is static.

The paradigm here is that all of the, “web” type stuff which touches HTML and controls routes should remain within the main web-type folders that are familiar to web developers, such as the assets, auth, routes, config and models.

The models may lay out the data within relational data tables, but the static/src is where the actual data processing goes.

└── src
    │	├── features
    │	├── preperation
    │	├── preprocessing
    │	├── evaluation
    │	└──	js...
    └── tests
    │	└──	unit_tests
    └── models
    │	├── seedmodels
    │	└──	retrainedmodels
    └── data
    │	├──	raw_data
    │	├──	processed_data
    │	└──	user_input_data
    └── pipeline
    │	└──	model_retraining_automation_scripts
    └── docs
	    ├──	Documentation
	    └──	Notebooks

So for example, if raw data is inputted into the system, the folder, “/src/static/data/raw_data” does not contain the actual data itself, e.g. the data is not uploaded directly onto the server, but rather that folder would contain some file, raw_data_control.py or something similar, which would manipulate a relational database or otherwise to control where that data goes.

Other folders, for example pre-build models which output a result based upon input data, would hold code that points to those models, also stored in a relational database for example, and so on.

Alternative Data Organization Open Source Projects

The above described way of working with data only goes so far and is meant for prototyping and getting an application going. There are more sophisticated data management systems which are designed to work with existing cloud services platforms to keep data storage affordable and traceable.

The best one appears to be the following:

Data Version Control

However, there are others:

Quilt PyPi or Quilt Github is designed to create versioned datasets with S3.
VespaAI (also open source)
Polyaxon
Seldon.io

There is also an alternative cookie cutter data science project structure format, although not integrated with flask.

How GPT2 Comes Into Play

This application does not use, “All of GPT2,” it uses a, “GPT2 head model,” as well as a, “GPT2 tokenizer,” which is available through the “transformers,” dependency by huggingface.

The meat of the actual code for that can be found here in this autodockwriter.py file.

GPT2 as a whole is a set of parameters which have been trained based upon a, “transformer model” type algorithm on a large corpus of text gathered from the web. When you use the huggingface GPT2 head model, it’s not some kind of API access to a cloud service spitting out an answer, it’s a slice of a much larger GPT2 model (of which there are three sizes). The head model has essentially been, “map reduced,” into a pattern, and we’re merely, “calling,” on that model.

There are three ways of talking about how a language model can be dealt with:

Training a model.
Fine Tuning a model.
Calling or Evoking a model.

We are merely calling the model, not training or fine tuning it.

“Training,” GPT2 costed an estimated tens of thousands of dollars in 2018/2019, and involved inputting upwards of 1.5 billion parameters (basically many bodies of text from web sources) into a massive neural network to output the different GPT2 models.
“Fine Tuning,” involves taking the, “Head Model,” (of TFGPT2LMHeadModel namesake) which is really a dense system of decoders, and adjusting that based upon a new set of input text. This fine tuning may take a significant amount of resources, perhaps better to run using GPUs rather than a CPU, but does not necessarily require tens of thousands of dollars of resources (as of 2020).
Evoking a model requires adjusting the Head Model similar to “Fine Tuning,” but is not as resource intensive, particularly after the first call. It’s basically just a way of applying the model against a small number of words.

The GPT2 Model - is defined as the bare GPT2 Model transformer outputting raw hidden-states without any specific head on top. In other words, this is the model output of the, “Training,” stage, straight from OpenAI.

There are several sizes of “architectures” of GPT2 Models available, including 1558M (extra large), 774M (large), 355M (medium), 124M (small). Using Transformers Configuration, one can select parameters which will be equivalent to the different sizes of models, but without selecting any parameters, which we did not, it defaults to small.

So in short, our application evokes the GPT2-small model with a header, without any fine tuning, and looks at the few words which were put into the, “text” body and computes the next few words in a sentence or two.

Rough Outline of the Math

When evoking GPT2, there are a few options for which head model algorithm type to use to search for the next word in a sequence. One option is, “Greedy Search,” which essentially just picks the next highest probability word after the previous word (or sequence of words). For example:

In the above scenario, the string of words put together would be, “the nice woman,” because those words are the next highest probability at each step.

Beam search instead looks at an entire sequence of words and optimizes for the highest sequence of outputs, for example:

In the above scenario, the string of words put together would be, “the dog has,” because the values ascribed to the sequence of words would sum up to 0.5+0.9 = 1.4 whereas with Greedy Search, “the nice woman,” would have only scored 0.5+0.4 = 0.9.

Greedy search tends to be much more repetitive than beam search and is of, “higher quality.” The flasksrc application used beam search, however there are other options for models which produce arguably better results, including top-k sampling and top-p sampling, discussed at length in the below article.

Image credit and reference

Credits

Created by Patrick Delaney.
Thank you to all of the authors and contributors of the various open source dependencies used in this application, including Tensorflow, Transformers, OpenAI, Flask and many more.

License

TBD

Check out my Portfolio

Text Generator with Human Editors was originally published by Patrick Delaney at Patrick Delaney on July 13, 2021.

How to Start a Webinar

2020-12-21T00:00:00-06:00

by Patrick Delaney

How do you open your webinar or conference talk? In an age of constant webinars and online video presentations, there is a need to understand how to better communicate through this medium. Over the past nine months, I have spent a considerable amount of time building a new online webinar and conference system called, “Confrnz.” Through this process I was able to glean some fairly interesting insights about general, “product demo,” type webinars, by collecting huge amounts of data observing how people watch webinars.

I was able to collect around 4.5 person-months worth of straight watching time, which amounted to over 3000 hours of content viewing across 200+ individual demos and videos. The following is a summary of a Twitter thread that I put together at an account dedicated to this project over at @Confrnz.

People don’t pay attention for very long, in general. Think of the opening of a webinar like the opening to a movie.

We have found through capturing over 4 person-months worth of viewing time, over hundreds of presentations, most viewers only watch for about 18 minutes or less. And the vast majority only watch for 2.5 minutes. You have two minutes!

Start out with your branding/logo. This is sort of like, “meta establishment” - who are you? Who is telling me this story?

Establish location - like, really…a physical location. Give people a sense of place. Having a sense of place for people is rare these days, we all spend all day long in our homes, we don’t get to go anywhere anymore. Give people that gift, like a movie does.

Introduce the main character - is it you? Show yourself on the screen. Don’t put yourself in a little box in the upper righthand corner. Don’t show a screen with your name or worse, a screen name like jSmith. Is it someone else? Paint a picture. Show, don’t tell.

Introduce intrigue - you have to keep building interest. You can’t just show a bunch of random slides and expect people to stay interested, you have to design in interest section by section. Show what you’re going to get into, hint at it, get them to want to see what’s next.

Tease out the main concept while adding value, maintaining interest. You’re walking down a hallway closer and closer to the concept of interest, a little less abstract now. It’s bright and shiny, amid darkness. What drama will occur along the way? Let’s find out…

Now you can get into the detail. What is useful and why? What’s exciting? How will this prevent a future problem and what will happen next? How do we get back to where we came from? You have to introduce these concepts into the viewers’ minds.

By the way, the timestamp we are at with both Apple’s most recent Event and Raider’s of the Lost Ark are 7:42 and 7:37 respectively. Coincidence? No - these presentations were designed by the best directors & cinematographers in the world. They know what they are doing.

They know to keep the main message it under the 10 minute mark, they know most people won’t pay attention after that. Now - you’re not LucasFilm or Apple…do you have 10 minutes?

Well, maybe you do, maybe you don’t, it depends upon your objective. Do you need to capture a huge amount of attention? Millions of people? Then you have to put more into your webinar. Do you need to keep 5 to 10 people watching and paying attention at a time? Less worry.

Apple and Lucasfilm MUST capture millions of peoples’ attentions, anything less is a failure for them. You may have a different audience, a subsegment, interested in a particular niche topic. You may have people listening because they are interested in you specifically.

If your material is for general broadcast, then compress down what you have to say in under two minutes. If it’s more for a small group of colleagues and customers, you can stretch it out toward 20.

Finally, don’t kid yourself, it takes more work to make things short and sweet and 2 minutes, it takes less work to do a 45 minute presentation because you can just blab on and on. It’s harder to make things shorter. Be ready to put in the work to get your message out there.

Follow more of our work helping people put together better presentations and building further insights on how to give better webinars at @Confrnz on Twitter We’re building our startup in public. #buildinpublic

One more thing… just doing a comparison to NVIDIA’s recent GEFORCE event. Less production value, same pattern…done in much less time…

They teased out everything they wanted to talk about, the whole pattern within…that’s right – 2 minutes and 45 seconds, or under 2.8 minutes…

The rest of the GEFORCE talk goes deeper into detail for the gaming audience, who they know ahead of time is going to be more interested in the detail of their content. However the broader audience got their entire message in 2 minutes.

For their expanded video, they consistently tease out concepts using a similar pattern. They even use a marble rolling down a sort of Rube Goldberg machine environment to maintain interest, making the viewer continually ask, “What’s next?”

Loading…

Check out my Portfolio

How to Start a Webinar was originally published by Patrick Delaney at Patrick Delaney on December 21, 2020.

Utilizing Video Views to Create Recommendations for Marketing Funnels

2020-12-07T00:00:00-06:00

by Patrick Delaney

Download the full paper here.

The Role of a Digital Marketer

Marketers, data scientists and generally those tasked with characterizing entire industries and sectors are working in a world of ever-expanding and overwhelming data and choices. Mapping out audience or industry stake- holder interests is increasingly complex, and there is need to filter, prioritize and efficiently deliver relevant information in order to alleviate the problem of information overload.

At the same time, 2020 has ushered in a year of online conferences, videos, connections far beyond what has ever been seen in the past. Interestingly, over the past decade, many peer-reviewed, high-quality studies have been carried out in relation to leveraging online video for learning, brand loyalty and engagement. Much of this rests on the fact that a user must choose wisely how they spend their time online viewing technical videos - video viewing time on a per person basis is a scarce commodity.

In other words, video viewing behavior can be considered to be, “similar” to purchasing or deciding, from a psychological standpoint. But what does this mean from a data science or trend-following perspective?

Translating Video View Data to Insights

Translating the above psychological assumption into a mathematical framework, I created an algorithm which uses a traditional cluster analysis technique to create serendipitous market segmentation recommendations for a digital marketer to use.

Example Clusters From Marketing Funnels from Video Views Paper Above

Distance and Cluster Formula Used in Paper

Cluster analysis is the mathematical task of grouping observations in such a way that items which are close to each other in value, distance, measurement or by some pre-defined method are in the same group. The paper I put together on how this was all constructed goes into detail about how this cluster analysis works, as well as the mathematics behind it.

Measuring Algorithm Performance

Simply Making clusters from data is not enough. Any kind of clustering algorithm will always yield a result. The utility of that result depends upon its function and the efficiency of that function. Typically when the type of math used described in my paper is used, efficiency can be measured in increased sales, or increased user satisfaction. In this case, the situation is reversed, in that additional performance measurements must be put in place which measure return on investment.

The first part of this algorithm would need to be a ranking system, which ranks topics or titles to a particular audience segment or cluster. Basically, percolating the topic up to the top of the list for a given cluster is going one step beyond merely segmenting out the audience - it’s recommending what that audience may most enjoy.

Cluster Ranking Function Graphical Model

From this point, a digital marketer would need to create and redistribute content, or distribute the same content in a different medium. This means a third layer of software or at least manual process would need to be performed which measures advertising spend using a clustering, audience segmentation vs. just randomly feeding content to a non-clustered audience. Metrics would need to be selected either based upon user response rate to new content, click rate, clickthrough rate, or sales and cart amounts of the segmented audience vs. the non-segmented audience. Weights could be applied on an ongoing basis to respond to how effective automated user segmentation is in terms of driving further sales.

Conclusion

I’m always interested in finding out new applications for this framework. Please contact me or sign up for my mailing list for more information on this and other similar interesting applications using machine learning and internet of things for various market segments.

Loading…

Check out my Portfolio

Utilizing Video Views to Create Recommendations for Marketing Funnels was originally published by Patrick Delaney at Patrick Delaney on December 07, 2020.

Virtual Conferences are Dead - Rethinking Community Knowledge Worker Development

2020-08-14T00:00:00-05:00

by Patrick Delaney

The Former Function of Mass Events in the “Old World”

Understanding that there is no certainty in terms of if, how or when the pandemic may taper off, in terms of its affects on mass events, I believe it is important to take a step back and look at an abstract view on the function of conferences and events in society. When approaching an important engineering or design problem, one of the first questions we should ask is, “How does this serve humans?”

The initial knee jerk reaction of many conference organizers around the world, including myself, was to create a solution and “go virtual.” Many of my sponsors and attendees were excited about this, and believed in the idea of building something new together, making lemondate in the face of about a million tons of lemons - for which I am greatful. I went back to my attendees and offered refunds, or the capability to convert their ticket prices into cheaper, “Digital Subscriptions,” at $19.95 as opposed to hundreds of dollars. I was surprised to find that many attendees simply decided to just donate their formally physical ticket revenue toward the cause. Things were going great, and I’m not afraid to say that I take a lot of pride in being able to convert a completely physical event into a completely virtual, software-based event within an about a month timeframe, using my own software, much of which I wrote myself. Personally, I happened to have created some simple software over the past three years which was utilized in sharing post-conference video material. I was able to leverage that and run a, “Virtual Conference,” version of my annual largest conference, IoTFuse. Attendeance was down about 50%, and some of the survey information showed me afterward that attendees felt a bit…shall we say…underwhelmed. While I did recieve many comments noting that the experience was exciting, inspiring and, “better than [they] expected it would have been,” - the overwhelming majority of the sentiment was that it was, “just not the same.”

Check out my Portfolio

Virtual Conferences are Dead - Rethinking Community Knowledge Worker Development was originally published by Patrick Delaney at Patrick Delaney on August 14, 2020.

Conferences and Events in Our Modern Pandemic

2020-08-13T00:00:00-05:00

by Patrick Delaney

Estimated Reading Time: 32 Mins

Main Findings of This Article:

Conference organizers of virtual conferences can no longer solidly promise to sponsors a defined return on their conference spend investment as they would have been able to with physical conferences
Virtual conferences have in a sense, become like Over the Counter (OTC) Penny Stocks. By that I mean, they are highly risky, they do offer some possibilities for a high return, but they are essentially no where near the solid investments that physical conferences were.
It is unknown at this time when things will change in terms of the public’s risk perception and willingness to engage in physical conferences again
A new risk model for conferences may be built which applies across many new ongoing digital marketing activities - this new way of thinking may provide opportunities for building future success in the virtual event space.

Main Findings of This Article:
Where We Are Today
My Background and Why I’m Writing This Article
The Mechanisms of Viral Spread
An Enhanced Pandemic in a Technologically Enhanced World
Past Performance Does Not Guarantee Future Results
Reframing the Scenario We Are In
- Conference Industry Scenario Matrix
- What The Market Has Looked Like in 2020
The Business Model of Conferences
Longer Term Projections on Conference industry
Conclusion

Where We Are Today

The $100 billion world of conferences and events has been without a doubt, completely devastated by COVID19. The losses from the string of tech conference cancellations from earlier this year have likely climbed up into the tens of billions of dollars at this point, with some of the world’s largest conferences being continuously postponed, cancelled or converted to virtual events.

The business model behind events and conferences - bringing people together for serendipitous meetings - represents the perfect breeding ground for viral spread, along with other similar, “mass gathering” type industries including large sporting events, concerts, large indoor facilities such as malls or indoor theme parks, movie theaters, restaurants and so on.

My Background and Why I’m Writing This Article

COVID19 or rather the public health response to the virus, has affected my business and life greatly, as I run a company called Knowledge Conferences and a nationally recognized conference called IoTFuse. While I have not yet been infected with COVID19, nor has anyone I know well yet died from the virus, I have had several friends and professional contacts contract it, and some of whom have had family members die from it. I am sure that as time goes on, sadly, this number will go up for many of us around the globe. I converted my conference into a completely virtual event, in April 2020, when it was scheduled to take place. This pandemic inspired digital transformation was perhaps the singular most harrowing experience of my professional life, and I will talk more about what happened in a future post, but the purpose of this post is to really dig deep into the environment under which we are all operating now.

Conferences and events play an important role in society. Much like restaurants, which have been devastated as well, conferences and events represent excellent entry-level jobs, and the job outlook for this sector had been projected to grow around 7% between 2018 and 2028, according to the US BLS. Much like restaurant jobs have been termed as providing a social safety net for much of US Society in terms of being a job source that anyone can jump into in dire situations, conferences and tech conferences in particular have represented a step or rung on a ladder for many young professionals or underrepresented professionals to get themselves going on a career path, either by speaking at, working for or organizing a conference around an important topic. Event planning has represented a fairly decent paying job at around $40,000 to $60,000 depending upon location, for those without a high amount of expertise or education in a sector. Tech conferences, or at least quality independent conferences, essentially neutral ground for various tech topics, have created ways for industry to flourish and explore, safe spaces for serendipitous meetings and exploratory activities which represent low cost research and development that goes uncounted and off the books of private or public companies around the world, while at the same time providing upwards of half of marketing qualified lead generation activities for service providers. In short, conference organizers have largely been the, “salt of the earth,” of the tech world, providing greater velocity of knowledge transference, while not necessarily having had to originate from purely technical educational backgrounds or roles.

That being said - holding conferences and mass gatherings for the time being, represents a large public risk. In this blog post, I would like to take a hard headed approach to this analysis of the Conference and Events industry. I have noticed over the past several months an emerging pattern in the world of tech conferences marked by overpromises, unrealistic objectives and even downright fraud.

My essay will go through several sections, to try to do the best possible job of investigating a range of possible outcomes for the industry:

Mechanisms of how coronaviruses spread, and in particular what is known about COVID19
Brief historical analysis of the previous large-scale pandemic in 1918,
Project a range of possibilities for the near future,
Brief overview of the business model behind tech conferences and events,
Possible ramifications for the far future of conferences and events.

Part of why I have an interest in taking a hard-headed approach to understand a variety of possibilities of what may happen comes from my career, during the last economic crisis. In 2008 and 2009 I worked as an optical engineer and image recognition software developer in the film and class industry, and one of my former bosses told me regarding the prediction of future trends relying on scientific phenomena, “if the business can’t be backed by the fundamental physics, it’s a scam.” He was referring to the company, “Nanosolar” which he called, “Scamosolar.” Nanosolar in 2009 had received over $100 million dollars in venture funding, but my boss was insistent that it was essentially a case of physicists claiming to financiers who didn’t know any better, that they would be able to coat plastic with extremely precise, multi-micron thick layers of chemicals on plastic roll to roll sheets to manufacture solar panels which would be orders of magnitude cheaper than existing forms. Of course Scamosolar did in fact, go defunct, because physics dictate that you can never really pull plastic sheets taught enough between rollers at micron precision. I suppose a modified version of what my boss told me would be, which is the hypothesis of this blog post is that, “If the business can’t be backed by the fundamental biology, it’s a scam.”

The Mechanisms of Viral Spread

At the time of writing this blog post, the world has gone through about eight months of observing generally how COVID19 spreads, however there have been stringent interventions and changes in human behavior which have perhaps left a lot of us scratching our heads and wondering what the precise mechanisms are behind how the virus spreads, what is known and what is not known since the virus first popped up in popular conciseness. The average person today may take for granted that we now generally know how the virus spreads based upon what we have been directed to do in terms of washing our hands, wearing masks and so fourth. However there is continuous and changing scientific and medical consensus based upon new ongoing studies since the virus first emerged. One unresolved topic, on which it appears that evidence is still preliminary and requires further assessment, is the question of whether the virus spreads through just droplets, which is known and widely agreed upon, but aerosols, or “airborne spread” in addition to those droplets.

The United States Centers for Disease Control (CDC) States:

“The virus that causes COVID-19 is thought to spread mainly from person to person, mainly through respiratory droplets produced when an infected person coughs, sneezes, or talks. These droplets can land in the mouths or noses of people who are nearby or possibly be inhaled into the lungs. Spread is more likely when people are in close contact with one another (within about 6 feet).”

This statement is very direct about referring to what is widely agreed upon, that COVID19 spreads through droplet spread, as opposed to aerosol spread. However the World Health Organization (WHO) goes a bit further in addressing aerosol spread:

“Some medical procedures can produce very small droplets (called aerosolized droplet nuclei or aerosols) that are able to stay suspended in the air for longer periods of time. When such medical procedures are conducted on people infected with COVID-19 in health facilities, these aerosols can contain the COVID-19 virus. These aerosols may potentially be inhaled by others if they are not wearing appropriate personal protective equipment. Therefore, it is essential that all health workers performing these medical procedures take specific airborne protection measures, including using appropriate personal protective equipment. Visitors should not be permitted in areas where such medical procedures are being performed.”

The WHO further goes on to state on this same page:

“There have been reported outbreaks of COVID-19 in some closed settings, such as restaurants, nightclubs, places of worship or places of work where people may be shouting, talking, or singing. In these outbreaks, aerosol transmission, particularly in these indoor locations where there are crowded and inadequately ventilated spaces where infected persons spend long periods of time with others, cannot be ruled out. More studies are urgently needed to investigate such instances and assess their significance for transmission of COVID-19.”

On the surface, the difference between aerosol or airborne spread sounds stark, and the ramifications of each type of spread mechanism for the conference world could be massive. Since the WHO is saying that aerosol spread can’t be ruled out, let’s take a look at what the difference between aerosol spread and droplet spread means:

Image source National Institutes of Health Paper.

Essentially, from a reading of this National Institutes of Health paper, there is no clear standard size cutoff between a droplet and aerosol, and there is not a globally precise definition of an aerosol vs. a droplet, but rather they are defined as needed, however there are some common size ranges (1 to 5 micrometers for aerosols) that have been used across similar studies. The main important point is to be able to study and predict spread in different environments, and come up with clear public health recommendations based upon empirical data.

Conferences are all about talking and meeting others. We may have ways of creating forms of, “security theater,” which essentially addresses a low-level understanding of how to prevent biological spread. We may invest in hand washing stations, plexiglas barriers, masks, and any other manner of protective measures, but as of now, we don’t know if the simple act of talking may actually lead to the spread of COVID19 in an enclosed setting. Talking releases anywhere from a few dozen to a few hundred droplets and aerosols ranging in size from 1 to 100 microns in size, and includes mostly aerosols ranging in size from 1 to 2 micrometer. For that matter, sneezing can produce anywhere from a few hundred thousand to a few million droplets and aerosols, mostly in the aerosol size range. That one guy (or gal) sitting thirty feet away at a conference who lets out a sneeze, who happens to be infected with COVID19 yet asymptomatic, could end up infecting hundreds of people in an enclosed space with a single sneeze.

An Enhanced Pandemic in a Technologically Enhanced World

There are perhaps few precedents in any other point in history where a set of industries has been faced with such severe devastation. One might look at the previous large pandemic, the influenza virus from 1918-1920 - but that pandemic just doesn’t quite compare for a few reasons. For one thing, the world population has since then increased by 2800%, and beyond that we as a species have achieved massive transformations in our our public transportation infrastructure, global communications speed, forecasting capability as well as knowledge about what drives infectious transmission.

Basically the “step function,” of the COVID19 happening and then the public and government bodies reacting does not look like the 1918 pandemic at all. During the previous global pandemic, there was a much slower reaction speed. There was less understanding about how to prevent the disease and how it transmitted - James Watson and Francis Crick had not yet discovered the genome. Arguably we as a species, at least in my locale, the Twin Cities of Minnesota have reacted much more efficiently during the COVID19 pandemic than the 1918 Influenza Pandemic. In addition to the general lack of know-how during the pandemic over a century ago, the Twin Cities lacked nurses and doctors in 1918, because a lot of them were deployed in World War 1. This National Institutes of Health study goes through how Minneapolis and St. Paul reacted to the initial stages of the 1918 Influenza pandemic - there was considerable confusion about how many underlying cases may have already existed by the time Minnesotan authorities gained awareness that the flu virus was even there. Within the first 10 days of the Pandemic starting on September 30th, 1918, the count of hospitalized cases went from 1 to over 800. Compare that to the COVID19 pandemic, where the number of positive cases detected went from 1 to 90 from March 5th, 2020 to March 15th, 2020. As far as hospitalizations in Minnesota go, the Minnesota Department of Health shows that hospitalizations data went from 1 to 75 from March 19th, 2020 to March 29th, 2020. From my understanding, much of how the nation as a whole during COVID19, has in fact acted with a far better response than we did in 1918, contrary to popular belief. However, some of the same strengths that we now have in terms of communication efficiency have lead to weaknesses. Whereas we now have the capability to more effectively study and disseminate information about the virus, leading to superior response times and methods, we also unfortunately are experiencing a much faster proliferation of misinformation. Where much of the same pattern of denialism, lack of solutions and blaming did occur during the 1918 pandemic, what we appear to be experiencing now is a fractalization of information streams. Whereas in 1918, there were fewer information authorities, meaning the public looked to a narrower set of information sources to determine how to act, today with the web, we are experiencing many competing clusters of information authorities, coming competing interests originating in political, religious, professional, or national influences and being disseminated by discussion boards, social media, online video, traditional media or even just some unimportant blogger like myself. This has resulted in a truly what one might term, “An Enhanced Pandemic in a Technologically Enhanced World.”

None of this is of course to diminish the current pandemic in any way, or suggest that we are on, “the right track and that things will be fine,” but rather to show that there are stark differences between how the two events unfolded, and that we cannot simply look to the 1918 influenza pandemic as a predictor for how things may play out during the current pandemic.

Past Performance Does Not Guarantee Future Results

I believe that it is important to point all of this out because there seems to be a dominant belief in industry at this time that the previous pandemic will be demonstrative of what will happen this time around. However, arguably this is not really a 100-year event, like a 100-year flood, or history repeating itself, this is a flavor of viral-economic calamity that has never happened before, period.

This pandemic has sliced across a specific set of industries, hobbling the economy as a whole by shooting part of it in the leg. The only analog which is comparable is likely a single country experiencing a massive natural disaster, war or invasion. However even in those cases, the industry disruption tends to be short-lived, rather than semi-permanent. There is absolutely no concrete indicator that we have on our hands at the time of writing this article which will allow humanity to safely scale up large events at a known date. While it is possible that a vaccine may be developed over the coming months or years - at this point, it is unknown when that may occur, or even if that may occur. Public infectious disease experts seem to be in agreement that vaccines should not be considered not be a silver bullet for allowing mass activity, and that even if a vaccine is developed within the shortest timeframe known to human history, COVID19 may be with us for a very long time, and it may be that we need to adapt society in ways we perhaps did not consider at the onset of this disease.

Reframing the Scenario We Are In

The typical model that I believe sitting in most people’s mind at this time is that, “things will generally get better.” As an industry, and perhaps even the world at large have been referring to what is going on as a, “crisis,” a word which signifies that a story will be written which includes a defined beginning, middle and end, within a perhaps month long or multi-month long time period. Instead what we may be faced with, is a variety of different risk scenarios defined by possible vaccination timeframes and effectiveness, as well as continuously developing human risk assessment of the seriousness of virus situation. In short, we really don’t know what’s going to happen, and a continuous appraisal of the situation on a monthly or even weekly basis is probably what is warranted, rather than a long-term, multi-month prediction.

This becomes incredibly important in the event industry in particular, where the fundamental requirement for holding a mass gathering event, besides of course the attendees, is a physical space. Physical spaces typically need to be reserved many months in advance, if not over a year in advance. I believe that the event industry, based upon its legacy operating procedures and just how events get financed, has in many cases opted mostly toward postponing or, “punting” events to take place at a future date, perhaps a year in the future. However a closer consideration of the state of the world shows that many events may not take place for a very long time, if at all, depending upon what happens.

Below, I have created a scenario matrix outlining possible scenarios in terms of vaccine development vs. human behavior in the face of various perceptions of risk. Within the cells of the matrix, I have written common sense outcomes based upon the scenarios outlined on the different axes.

Conference Industry Scenario Matrix

Virus Minimization vs. Human Behavior Response Toward Mass Gatherings	Diminished Cost of Exposure to Individuals	Medium Cost of Exposure to Individuals	High Cost of Exposure to Many Individuals
Strongly Preventative Vaccine Widely Distributed By January 2021	Large Gatherings May Resume at Lower Attendance Volumes with Late 2021 and Ramp Up Beyond	Large Gatherings May Resume but with Limited Capacity in Late 2021, Ramp Up Over Time. Increase in Cost to Run Events.	Large Gatherings Likely Will Restart Late 2021, but Expensive Operational Changes Needed to Protect At Risk Individuals
90 Day Effective Vaccine Largely Distributed by June 2021	Large Gatherings May Resume at Lower Attendance Volumes with Very Strict Health Measures Early 2022 but with Limited Capacity, Trend May Backtrack If Infections from Events Occur	Large Gatherings May Be Allowed to Proceed Pending Certification Against Expensive Operational Regulatory Requirements and at Lower Capacities	Gatherings May Tentatively Proceed at Lower Capacity in Late 2021, But Attendance Volume is Likely to be Lower, Due to Risks, Costs Higher Due to Health Requirements
Somewhat Effective Vaccine Partially Distributed by January 2022	Large Gatherings May Resume with Limited Capacity Mid to Late 2022 with Expensive Operational Changes Needed to Prevent Infections	Gatherings May Tentatively Proceed at Lower Capacity in Late 2022, Attendance Difficult to Predict if at All, Possible Regulatory Costs	Large Gatherings Will Likely Not Happen Until Late 2022, Or Transition to Limited Capacity Events, Which Is Not a Mass Gathering
Slightly Effective Vaccines, Continuous Aggressive Social Health Measures	Gatherings May Tentatively Proceed Depending Upon How Much Risk Tapers Off, or Transition to Smaller Gatherings	Large Gatherings Will Probably Just Not Take Place, Maybe Smaller Gatherings Allowed	Large Gatherings Will Probably Just Never Take Place at any Level of Low Risk, or Just Not At All

To summarize the above table, the farther to the right and below that we go in terms of both health risks to individual customers, longer timelines and lower effectiveness of solutions, the worse outcome for the industry in terms of previous business models. This table represents a, “risk model,” approach to where things could go in terms of the conference industry, rather than making a definitive prediction. Multiple scenarios could arise in the coming months and years, depending upon where science leads us, and what more we learn about COVID19. For example, over the last month it has become increasingly clear that while the death rate of those infected with COVID19 in the US has hovered around 1% to 2% as a large overall average, affecting older populations at a greater rate the effects of, “Long COVID”, or health affects as a result of contracting COVID19, including brain fog, lung damage, heart damage and hypertension, are largely still unknown in terms of severity or how long these effects may last.

What The Market Has Looked Like in 2020

Virtual events are largely still in their infancy, but some of the weaknesses of virtual events are definitely showing through. In a post titled, Rethinking Community Knowledge Worker Development, I will go through some of the user experience factors behind what has been driving massive variations in attendance levels, and in particular, lack of willingness to participate, and differences in ways that attendees participate in virtual events as opposed to physical events, but for now I will just segment virtual event types into three main types. I don’t have statistics on how many events of each type are occuring right now, but rest assured, I have personally found and analyzed events that are fraudulently reporting their virtual attendance numbers by looking at their Alexa web traffic ranking, and have heard from sponsors and previous sponsors of mine of an event that had reported to have over 100,000 attendees signed up, and instead had attendance in the low hundreds.

Virtual Conference Classifications - Post-COVID19
Virtual Conference Type	What Happened	Why This Happened
Hyperinflation	Highly reported attendee numbers, attendance much lower than reported.	Not enough time to transition to virtual conference, not properly marketed, and people's general unwillingness to convert to virtual. There may have been a variety of both marketing factors and promotional factors at play, the conference material may not have ever been extremely interesting to the attendees in the physical realm in the past, but face to face networking and socializing at the physical conference may have been a large benefit.
The Engager	Conference expectations for virtual conference managed, not much in the way of attendance but some marketing objectives possibly met for sponsors.	Conference may have already had sufficient momentum to convince community members to attend, may be an ongoing yearly tradition with a lot of emotional attachment, so attendees are at least partially willing to engage or tune in for a bit. The conference may have used sufficient technology to allow networking or some kind of peripheral benefit to attendees.
Gold Rush	Attendance much higher than physical conferences previously, interest levels through the roof - incredible success.	There have been very few, but have existed instances of conferences which seem to have hit at the right time, due to all of the transitions which have been going on - becoming extremely relevant during the time they were held. For example, there was a food safety and security conference which was already scheduled to take place during the time when there were food shortages and the national conversation began to turn to food. This conference had formally been a regularly attended but not out-of-this-world conference in previous years, but in 2020 was attended far beyond where it had been previously because of the digital venue creating increased accessibility, and the topic matter being extremely relevant.

The Business Model of Conferences

Types of Large Conferences

Large Consolidated Expo

Many Large Consolidated Expo companies have their origins in news, namely financial and stock trading informational markets. In the past, large publishing companies, which may have started as far back as the 1800’s, got into the business of putting together conferences, or acquired conferences (or companies that held conferences). These large scale conference companies may be extremely large scale private companies, or even public companies such as Informa. These companies, being public, may have access to bonds and loans specifically for the purposes of running expos that other type of conferences may not, simply because they have a many decade track record of successfully running profitable conferences, and have a large diversified portfolio of conferences.

Corporate Events

Corporate Events are essentially a company running an event for perhaps some of the same purpose why they may also attend an event, but rather than merely attending, they are branding the event and using it as a marketing outlet. In these scenarios, the purpose of the event may not even be to make a profit, but rather to be a, “loss leader” for other products and services, or to break even in order to add value to vendors, partners and customers.

Community Conferences

Community Conferences are the small businesses of the conference world - these could be started by anyone, for any purpose, and may range in size from a company or non-profit event that has grown in size over a decade to include thousands of attendees, to perhaps just a group of 100 or so people who are interested in a particular software. These community conferences may be businesses in it of themselves, like the Large Consolidated Expos discussed above, or they may serve as some kind of loss leader or marketing generation vehicle for a group of folks, service providers, or small companies - or they may be simply run for passion reasons, or for some combination. These are the, “entrepreneurs,” which provide spice and diversity of thought into the information transference that goes on in the world of conferences. Larger consolidated companies may not be willing to take risks starting up a conference on a niche topic, but smaller conferences do it all the time.

The Basic P and L Statement

Regardless of what type of conference being run, the Profit and Loss Statement of a physical conference which may have occured in the, “old world” prior to the COVID19 pandemic can be simplified as follows:

Often larger expos and conferences may model out their business model as square footage rather than as, “Conference Units,” as I have termed above in this example. Conference Units are meant to demonstrate simply that there are similar sets of activities and resources across all different categories of conferences, whether they be large consolidated events or public companies, corporate events or community events.

Conference Unit Profit and Loss
Attendee Ticket Sale Income	$250,000
Sponsor Income	$500,000

Total Income		$750,000

Loaded Space Cost	$250,000
Staff and Setup Costs Over Year	$250,000

Total Costs		$500,000

Net		$250,000
Net Percentage of Sales		~33%

The above model is farily generalized and this will of course vary from conference to conference, and among the conference types. However, generally if you compare the net margins before interest and taxes of conference companies to parallel industries such as restaurants and hospitality, the general benchmark for higher end hospitality is around 20% to 30%, and conferences are in a sense, a high-end hospitality function mixed with marketing. Hence a generalized business model of about 1/3rd net revenue before interest and taxes for a, “Conference Unit” is generally acceptable.

In most cases, conferences are going to have sponsors of some type. While conferences may be also run completely on attendee revenue, this is a much risker way of running a conference, because you have to wait until tickets are sold to be able to know for sure that the conference is going to be able to cover its bills. Human ticket buying nature dictates that 80% of the tickets purchased for a physical conference do not occur until the last couple of weeks prior to the date of the conference. Conferences which run 100% on ticket revenue have opportunities to build much more attendee-focused experiences, and buy in from the community, but suffer from market effects.

It is important to consider how sponsors think about conferences, since sponsorship revenue may actually drive a significant amount of conference activity.

Typically it is the marketing department of an organization which spends money on sponsorships. In many companies, it is marketing’s job to support sales, particularly if that company has an actual sales force. Of course this will vary from company to company and industry to industry, but generally holds true. Essentially, the marketing department, by sponsoring a conference, is supporting the sales department by filling the sales funnel, and of course assisting sales personnel in moving contacts and leads down an existing pipeline by building face to face trust.

Marketers choose to invest in a conference among a broad array of market engagement opportunities. While some companies depend upon conferences much more than others, Marketing Directors do not make their decisions in a vacuum. They are typically looking at their entire sales and marketing funnel, and comparing the effectiveness rates across many different marketing opportunities which includes conferences and expos. As you can see in the below chart, which draws off of survey data from over 13,000 marketers over 10 years from a public relations firm, average rates for marketing effectiveness differ depending upon the channel used.

Lead Generation Effectiveness for Various Activities

Brand Building Effectiveness for Various Activities

Source: White Paper from Public Relations firm Cision

In addition to the, “overall effectiveness” of a given channel, marketing directors may also look at raw engagement vs. conversion rates for a particular channel strategy, and compare them to each other. The below chart shows a survey of B2B marketing directors and their findings on effectiveness of different channels as of January 2020, immediately prior to the COVID19 pandemic.

Source: Marketer.com

Basically, sponsors do not sponsor physical shows, “out of the goodness of their heart,” they do so because it supports ongoing sales activities. Going forward in our new socially distant world, it may be reasonable to speculate that the conversion rate of virtual events may be significantly lower than in-person trade shows and events, perhaps more in line with what one might see on a website conversion rate.

When marketing directors narrow things down and compare conferences to one another, they typically may look at the number of, “Marketing Qualified Leads,” that came out of a conference, which may be measured by engaged meetings.

Conference Engagement Rates
Number of Engaged Prospects	Pe
Number of Passing Prospects	Pp
Marketing Effectiveness Percentage	= Pe/Pp
Overall Sponsorship Costs
Sponsorship Cost	$5,000
Other Costs - Flight, Hotel, Staff Time	$5,000
Total Costs	$10,000
Cost Per Engaged Face to Face Meeting
Engaged Face to Face Meeting Cost	$10,000/Pe
Engaged Face to Face Meeting Range	$50 to $200

In summary, a marketing director may, at the end of the day, try to understand the cost of getting a salesperson into a meeting, based upon all marketing spend, and then try to optimize that cost on a quarterly or yearly basis by investing in:

The right marketing channels.
The right conferences, if conferences are the right channel.

The Work Involved in Put a Conference Together

Putting together a conference is a lot of work, and in our idealized model above, the staff and other costs, or the, “behind the scenes,” represents upwards of 50% or more of the costs throughout the year of organizing a physical conference. Below is a high level list of all of the major tasks involved in making a conference work.

Staff and Setup Conference Activities
Sponsorship Sales
Marketing, Emails, Social Media
Website (Hosting, Maintain, Content) & Software Selection - for Layout, Communication, Etc.
Graphic Design, Design
Purchasing, Logistics, Technical Setup
Inviting Speakers, Managing Speaker Communications
Attendee Customer Service
Building the Conference Program, Managing Speaker Times, Themes, Content
Financial Accounting
Venue Relations, Purchasing, Negotiation
Timelines, Project Management

In summary:

Events are often financed through pulling together a critical number of sponsors. Without these sponsors, events cannot happen.
Events also bring in attendees, which obviously are needed to run a conference, but depending upon the event, the attendees may pay anywhere from nothing to thousands of dollars per attendee to attend, which may range from 0 to 2/3rds of a conference financing, but the higher the dependency on purely attendee revenue, the greater the financial risk as space and staff time is generally much more fixed.
Sponsors decide whether and how to engage with events based upon their internal calculations for engagement rates, or conversion rates. Companies which have a heavy sales strategy and conversion strategy may invest more heavily in conferences, but may also measure conferences against each other with a clearer degree of certainty than companies looking for brand establishment and recognition.

Risk Models When Transitioning To Digital Events

To attempt to predict how virtual conference sector may operate going forward, I would propose a risk based model which takes into account the generalized conference business model discussed in this article and the classifications of virtual events that we have seen in actuality, including, “Hyperinflation,” “the Engager,” and “the Gold Rush,” mentioned above in this article.

Attendance at virtual conferences is known to vary considerably, and may generally be lower and less engaged than physical events. There is no captive audience, attendees float in and float out, like ghosts, and as such this means that conference organizers can no longer solidly promise to sponsors a defined return on their conference spend investment.
On the conference side, one of the inherent risks in transitioning an event organization or company into a digital event company, is first off that staff and setup (promotional) costs may be somewhat similar, depending upon staff running event. As discussed above, much of the cost of running the event is based upon the marketing and gathering of speakers, getting attendees engaged with websites and communication, as well as garnering sponsorships. Without any other serious operational changes, this labor rate could remain fixed between physical conferences and virtual conferences.
Event space costs, food, purchasing, is all but non-existent. However in its place, software costs, website costs, the costs of backing up systems and ensuring that all of the proper accounts are in place to make sure the event does not fail from a technical perspective, have gone up considerably.
In many cases, the type of people who may have previously been involved in running a conference and organizing physical activities, shipping, may not have a lot of familiarity with running virtual events right out of the gate, so there is a huge risk of mistakes and errors or just not knowing what to do at all, which is probably part of the reason why a lot of conferences just outright cancelled during COVID19 rather than even attempt proceeding in a digital format.

What all of this leads up to, is that virtual conferences, at least for the time being have become highly expensive, yet highly risky assets. Virtual conferences have in a sense, become like Over the Counter (OTC) Penny Stocks. By that I mean, they are highly risky, they do offer some possibilities for a high return, but they are essentially no where near the solid investments that physical conferences were.

Per the, “Investment Risk Model” image above, prior to the COVID19 pandemic, conferences were the high yield bond of the marketing world. They were expensive, but they offered a solid return on investment. Now with so many industries being thrust into the digital world, we no longer have as much capacity to simply purchase that trusted, face-to-face value building which so many industries have run on and leveraged to grow and function.

A new risk model for conferences may be built which applies across many new ongoing digital marketing activities - this new way of thinking may provide opportunities for building future success in the virtual event space. Imagine for example that you are a marketing director, tasked with putting on a webinar. What’s the risk profile of doing that webinar? You will need to be able to answer to other team members, as well as whoever allocates the budget above you, why this webinar is worth it to run, and report engagement vs. spend metrics.

One way to set this webinar up would be to just set it up from scratch, and start pumping it out into email lists and social media, basically the, “if you build it they will come,” model. This model may be lower cost, but if it pulls off well and becomes a, “Gold Rush,” you created an extremely valuable penny stock.
Another way to do this, which perhaps lowers the, “risk” of engagement would be perhaps to set up a webinar with a defined set of customers who you already have in place, and then hopefully invite in some newer or earlier customers. This would allow you to create a clearer immediate return on investment by engaging with defined customers, assuming they have the time to attend, but also to possibly convert over new customers.
Finally, you could also hypothetically start up a collection of webinars, working with a team of vendors, partners, customers and providers to set up a variety of content which collectively casts a wide net of topic matter, further ensuring that your singular webinar may be more successful through network effects. You could still invite your customers as shown in the previous model, but you are also using network marketing to ideally pull in more customers, rather than pure social media or email spend.

Longer Term Projections on Conference industry

Sadly, as with any major disruption, there will be companies which fail to adapt and in particular, conferences and events which we know and love which will no longer be able to function. This is an inevitable outcome of the situation we are in - it’s just a matter of timeframes, severity and conference business models before we know which ones will be able to survive.

I am reminded of an article I had read about how COVID19 has put the restaurant industry at the mercy of the tech industry. In it the author Sarah Emerson makes a poignant observation:

But the cost of techno-optimized restaurants could be the disappearance of beloved neighborhood eateries — by empowering delivery apps that demand near-impossible fees, and weakening the need for brick-and-mortar establishments that, while challenging to run even in a healthy economy, are embedded in the fabric of so many local communities.

Just as we see neighborhood restaurants close in favor of chain stores, and we see the small shut their doors in favor of the large, at least in the, “medium term,” next decade timeframe, we may enter into a period of blight in the conference world, where only the largest entities and corporate events may be able to flourish and return, because of their access to low interest financing and capability to work in more highly regulated or health-intensive environments when the opening does eventually happen. The smaller, scrappier conferences will take much longer to come back, just as the neighborhood eateries and restaurants we know and love.

Conclusion

We’ve got to start thinking beyond our previous careers if we are going to achieve the goals of a conference, which is to spread and efficiently disseminate knowledge.

Loading…

Check out my Portfolio

Conferences and Events in Our Modern Pandemic was originally published by Patrick Delaney at Patrick Delaney on August 13, 2020.

Plotly Test Page

2020-08-10T00:00:00-05:00

Plotly is interesting because it can be used to store and display data on a page in the form of pure static HTML page, making it extremely lightweight and fast to use.

This is a test using plotly on this Jekyll-created static page to just test out what it looks like. Absolutely no server apps were used or harmed in the making of this page.

Plotly Test Plot Above

The following data strings were entered into the plot.ly function above to generate the above chart. Below is the entirety of the data you are seeing above.

var trace1 = { x: ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov',
'Dec'], y: [20, 14, 25, 16, 18, 22, 19, 15, 12, 16, 14, 17], type: 'bar', name:
'Primary Product', marker: { color: 'rgb(49,130,189)', opacity: 0.7, } };

var trace2 = { x: ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov',
'Dec'], y: [19, 14, 22, 14, 16, 19, 15, 14, 10, 12, 12, 16], type: 'bar', name:
'Secondary Product', marker: { color: 'rgb(204,204,204)', opacity: 0.5 } };

Check out my Portfolio

Plotly Test Page was originally published by Patrick Delaney at Patrick Delaney on August 10, 2020.

The Future of 'IoT' as a Term

2020-08-08T00:00:00-05:00

Gartner published an article on July 28th, 2020 arguing that, “IoT’s stand-alone marketing impact is disappearing.” As an electrical engineer, software developer, product manager and engineer, this term has been of keen interest to me over the last decade. I have been such a huge, “IoT fanboy,” that I went as far as to cofound a conference, called the IoTFuse Conference in 2015 which grew to be a nationally recognized event. I thought it would be interseting to dissect Gartner’s thoughts because:

I had originally not thought that the term, “IoT” would have as much staying power as it has had.
I’m intersted in where the term, “IoT” will continue to go in the coming years.

Gartner’s Approach

Gartner puts forward a hard-numbers prediction that the term, “Internet of Things,” will serve less as a communication mechanism measured by percentage of service providers using the term. Gartner is asserting that - broadly speaking, the term, “IoT,” will fade as, “[solutions] … are commoditized.” Gartner rests strategy recommendations in this paper on the assertion that, “by 2024, fewer than 30% of technology and service providers with IoT portfolios will include the term “IoT” in their messaging, down from 60% today.” Putting an actual number and date on a prediction is laudable because it’s falsifiable. Falsifiability is important in predictive sciences because if one makes a prediction that doesn’t include a hard timeframe and a way of quantifying or classifying what is being predicted, one’s prediction basically amounts to “hand waving,” meaning it’s not really even a prediction, it’s just talking around a subject using rhetoric.

That being said, the more falsifiable a prediction is, the greater range of possibilities that it could be wrong, and the higher chance for reward if it’s right - which is actually what makes these discussions interesting!

Gartner makes specific strategic recommendations based upon their prediction. An overly simplified way of summarizing their recommendations is that, “The risk capital is gone from IoT.” Gartner asserts that because IoT solutions have been commoditized, any internal IoT projects which get proposed may need to have shorter payback periods associated with them than previous years. I actually agree with this recommendation and have heard many of my IoTFuse subscribers, attendees from engineering divisions across different industries tell me that there has been a shift between 2017 and 2020 or so.

Of course Gartner’s perspective is aimed at a C-Level audience, as their main clients are CIO’s, COO’s and people who are interested in operational technologies from a budgeting and accounting standpoint. They sell information services to large companies and Fortune 500 companies, so there is an incentive to create an image of continuously moving forward, to make falsifiable predictions and also for Gartner to in a sense gain ownership over mnemonic terms. While Gartner may not be as much attempting to predict where the world will go from a bottom up approach as they are sway the opinions of their customers and convince these customers that they need to continue to purchase need Gartner products.

Analysing The Interest of “The Internet of Things” From The Standpoint of Alternate Tech Trends

While of course it is a truism that trends and products do have lifecycles, predicting and timing those lifecycles is incredibly difficult. This difficulty is compounded when you take into account the fact that some products and concepts are very narrow and mean a definite thing to certain audiences, for example the concept of a Virtual Private Network, or VPN, while other concepts are much more broad, as with the Internet of Things. The Internet of Things is really more of an abstraction meaning different things to different professions whereas a VPN is a defined communications system. Some industries and sectors do not even use the term, “The Internet of Things,” such as the medical and healthcare sector, which tend instead to use the term, “Telehealth,” as a catch all to include many health sensing devices which are indeed internet connected and use many of the Internet of Things protocols and platforms that people keenly interested in the term, “Internet of Things,” work on all the time.

Looking more closely at the term, “VPN,” - there aren’t really any conferences leveraging the term, “VPN,” anymore, even though it is much more widely used than ever.

That being said, VPNs are very much the domain of folks who are hyper-interested in that one narrowly defined communications protocol, so everyone who is at the center of that, “Interest Sphere,” probably already knows each other and has known each other for 20 years or more. There would not really be much point in holding a conference or putting out networking resources and vastly creative knowledge articles around a narrow concept.

Contrast that with the Internet of Things, which has widely different spheres of interest encompassing different professions and people.

If you compare the usage of the term, “IoT” vs. VPN over time using Google Trends as a proxy for interest level, obviously there is a huge underlying usage rate of VPNs, even though it is not a sexy concept that one might build a conference about, or put an advertisement for on the side of a bus. The exception to the usage of VPNs as a sexy marketing term might be perhaps talking about VPNs in the context of protecting one’s personal privacy, essentially, “Personal VPNs as a Service,” which is really only a tiny minority of all use cases of VPNs.

So since the year 2000, when the last VPN Conference seems to have taken place, “VPNCon 2000,” as far as can be gleaned from a quick online search, the sexiness of the term VPN has died down, but the usage of the term has continued. If you compare this to other technology trends within the same timeframe:

“Smartphones” grew from 2009 through around 2013 or so, then tapered off since then.
“Smartwatches” seem to have this seasonal spike and seemed to have grown in interest until around 2017.
“Mobile Apps,” grew in interest at the advent of the smartphone, and then kind of plateaued but have continued at that higher level of interest.

Similarly the term, “IoT” seems to have plateaued. However, if we consider who uses that term, and what kinds of professions are interested in what is for one crowd, “Connectivity for Electronic Stuff,” and for another crowd, “Lots and lots of machine generated data,” you could see the term perhaps having less, “IoT corporate risk capital,” in favor of, “IoT projects which demand a defined IRR.”

Conclusion

Arguing that the term, “IoT” will go down in usage by X percentage over Y timeframe is guesswork, at best. In reality, no one believes that there are going to be less connected devices, just like no one believes there are going to be fewer smartphone apps over the next five years. The entire problem space from connectivity to data analysis to standardization of data types is by no means solved, and therefore we could see more platforms come up, you could see a resurgence of the term based upon breakthroughs in the electronics industry with the advancement of RISC V, or you could see a huge die off as finance becomes tightened due to the slowing down of the economy. The best approach is likely a continual appraisal of where the terms are every quarter or so, rather than making an extremely bold prediction going out five years.

Check out my Portfolio

The Future of 'IoT' as a Term was originally published by Patrick Delaney at Patrick Delaney on August 08, 2020.

Understanding Two Sides of the Coin: Internet of Things (IoT) Hype and Concreteness

2020-06-01T00:00:00-05:00

Project Summary:

Guest Lecture at the Technological Leadership Program at the University of Minnesota.

Background Rationale

Dr. Tariq Samad, Senior Fellow and Director of Graduate Studies in the Management of Technology Program at the Technological Leadership Program at the University of Minnesota contacted me to request that I give a guest lecture on, “Internet of Things,” to a class of executive engineering leaders who were taking the course. A Management of Technology Program is sort of like an Master’s in Business Administration, but geared towards Engineers, Software Developers, Information Technology Administrators and individuals in technical roles who are looking to expand their careers.

Benefit and to Whom

Project Status or Outcome

Check out my Portfolio

Understanding Two Sides of the Coin: Internet of Things (IoT) Hype and Concreteness was originally published by Patrick Delaney at Patrick Delaney on June 01, 2020.

The AgroFuse.io Conference

2020-02-01T00:00:00-06:00

Project Summary:

The AgroFuse.io Conference

Background Rationale

Benefit and to Whom

Project Status or Outcome

Check out my Portfolio

The AgroFuse.io Conference was originally published by Patrick Delaney at Patrick Delaney on February 01, 2020.

Translation of Matlab to Python

2020-01-01T00:00:00-06:00

Project Summary:

Converting various common Matlab statistical and machine learning tools into Python, located at this Github Repo.

Background Rationale

My personal motivation from this project comes from having studied electrical and computer engineering at the University of Minnesota, which involves the use of Matlab. Matlab was in fact my first foray into software development, other than C and C++ for embedded devices, though I didn’t even realize I was really developing software at the time - as perhaps many users of Matlab may themselves not realize.

As computing will inevitably continue to become more powerful over the years, the use of terms such as, “Machine Learning,” and “Artificial Intelligence,” has become democratized and expended, and those who work with servers will likely continually be interested in working with those with serious math skills. The terms, “Machine Learning,” or “Artificial Intelligence” to a certain degree is about the capacity to model the world in terms of math and practice said math at volume.

Benefit and to Whom

The, “at volume” side of that statement has to do with the capability to deploy software onto a server. While many of the most talented mathematicians, statisticians and scientists in various fields are still today trained in Matlab in academia, Matlab is not really designed to deploy, “at scale,” in the sense that it is a paid development environment, designed to run on a single Personal Computer or Machine, and includes a yearly fee with upgrades, rather than an open-source computing language such as Python which can be infinitely copied from server to server with no licensing fees.

There are of course advantages and disadvantages to both Python and Matlab, which is discussed in the project page on Github.

Project Status or Outcome

This is an ongoing hobby project. I maintain the repo and update it over time, with the idea being to create a way to assist others in translating from Matlab to Python and hopefully build their projects more intelligently and faster.

Check out my Portfolio

Translation of Matlab to Python was originally published by Patrick Delaney at Patrick Delaney on January 01, 2020.

The MedFuse Conference

2019-12-01T00:00:00-06:00

Project Summary:

MedFuse Conference

Background Rationale

Benefit and to Whom

Project Status or Outcome

Check out my Portfolio

The MedFuse Conference was originally published by Patrick Delaney at Patrick Delaney on December 01, 2019.

IoTHackDay.MN

2019-11-01T00:00:00-05:00

Project Summary:

IoTFuse Conference

Background Rationale

Benefit and to Whom

Project Status or Outcome

Check out my Portfolio

IoTHackDay.MN was originally published by Patrick Delaney at Patrick Delaney on November 01, 2019.

The IoTFuse Conference

2019-09-01T00:00:00-05:00

Project Summary:

IoTFuse Conference

Background Rationale

Benefit and to Whom

Project Status or Outcome

Check out my Portfolio

The IoTFuse Conference was originally published by Patrick Delaney at Patrick Delaney on September 01, 2019.

Patrick Delaney

Python Packages for Machine Learning

Why Create a Python Package for Machine Learning?

Creating a Python Package for Machine Learning Purposes

Reserving the Namespace

Use Recent Documentation

The Two Essential Things That Happen

How Twine Knows What It Knows, Security Considerations

Enjoying the Fruits of Your Labor

Conclusion and Packaging Up Other Considerations

Creating a Simple ML Server from Scratch with CherryPy

Creating a Simple Machine Learning Server with CherryPy

1. Import Necessary Libraries

2. Load the Trained Model and Columns

3. Define the Server and API Classes

4. Implement the query() method in the API class

5. Define the main() Function and Server Configuration

6. Run the Server

Miscellaneous Concerns and Wrapping Up

Future Concerns

Subscribe to Patrick Delaney’s Email List

Check out my Portfolio

Wanna Bet?

Bets Ending End of 2024

Bets Ending End of 2023

Destination Site Definition

Sign Up For My Email List

AI/ML Architectures Mythology

Contents and TLDR

Introduction

Framing Mythological Nicknames for AI/ML Pipeline Construction

The Persephone Protocol

The Daedalus Dataflow

The Prometheus Pipeline

The Zeus Zonal

Nemesis Normalization

The Odysseus Orchestration

Long Term Importance of Odysseus Orchestration

Epimetheus Endpoint

Building an Endpoint From Scratch

Hephaestus Hold

Kubeflow Kraken

Rough Breakdown of Kubeflow

Closing Up

Appendix, Further Assumptions, Prerequisites

Building a Simple Plagiarism Detector

/bin/bash A-to-Z

Simple Bash-Based Code Performance Measuring Tool

Project Summary:

Codeperform Project

Existing Tools

Built-in Timer Tools

Rough Outline

Working through the Bash Script

Creating the File, Initiating with #! Shebang Operator

Adding Arguments to the Bash Script

Creating a Help Function

Turning Help into a Flag Rather than a Default

Running strace on One Program

Ensuring that Changes in the Container Reflect in the Code on Local

Checking the Shell Script with strace

Testing out Strace on Code

Using Strace and Awk to Output

Using the GNU time Command

Putting This Back Into codeperform.sh

Creating Mathematical Output

Executing on Python Code

Comparing a Vectorized Operation vs. Non Vectorized Operation

Updating the Help File

Error Processing, Other Shell Best Practices

Creating a Dynamic Link to This Shell Command

Updating the Docker File to Include Needed Functions

Conclusion

Ideas for Future Improvement

Sources

Text Generator with Human Editors

Project Summary:

Language Generator Programs Documentation

Author

Dependencies