• CSS only submenu
• Photo Gallery

CSS only submenu

Create a new file /#jekyll-theme-hydejack/_includes/body/sub-menu.html

{% assign menu = include.menu %}
{% for node in menu %}
    {% assign url = node.url | default: node.href %}
    <li {% if node.menu.size > 0 %} class="has_submenu" {% endif %}>
        <a
        {% if forloop.first %}id="_drawer--opened"{% endif %}
        {% if node.url %}href="{% include_cached smart-url url=url %}"{% endif %}        
        class="sidebar-nav-item {% if node.external  %}external{% endif %}"
        {% if node.rel %}rel="{{ node.rel }}"{% endif %}
    >
        {{ node.name | default:node.title }}
    </a>
    {% if node.menu %}
        <ul class="sub-menu">
            {% include body/sub-menu.html menu=node.menu %}   
        </ul>     
    {% endif %}
    </li>
{% endfor %}

Edit /#jekyll-theme-hydejack/_includes/body/nav.html and replace the content of this if statement with this:

  {% if site.menu %}
   {% include body/sub-menu.html menu=site.menu %}
  {% else %}
  

Add a new sass file sub-menu.scss:

// file: '/#jekyll-theme-hydejack/_includes/body/sub-menu.html'
.sub-menu{
    padding: 0;
    margin: 0;
    display: block;
    transition: max-height 0.3s ease-out, opacity 0.2s linear;
    max-height: 0rem;
    position: relative;
    margin-left: 0.5rem;
    list-style: none;
    overflow: hidden;
    opacity: 0;
    margin-left: 20%;
    a {
        font-size: 90%;
        margin-bottom: 0.2rem;
        font-weight: 100;
        line-height: 1.3;
        font-family: 'Noto Sans';
        text-decoration-thickness: 1px;
    }
  }
  
  /* Show the dropdown menu on hover */
  .has_submenu:hover > .sub-menu{
      transition: max-height 0.3s ease-out, opacity 0.2s linear;
      display: block;
      max-height: 10rem;
      opacity: 1;
  }

  .has_submenu > a::after{
    font-family: 'icomoon' !important;
    display: inline-block;
    position: absolute;
    content: "\ea29";
    margin-left: 1.5rem;
    margin-top: 0.7%;
    font-size: 90%;
    vertical-align: middle;
}

.sidebar-nav > ul > .has_submenu > .sub-menu{
    margin-left: -0.2rem;  
    margin-top: -0.1rem;
  } 

Import the new file adding this at the end of my-style.scss:

@import "sub-menu";

Finally, add your submenu (and sub-submenus etc…) to your _config.yml:

menu:
  - title:             Blog
    url:               /blog
  - title:             Photo Gallery
    url:               /photo/
    menu:
      - title:  test1
        url:    /photo/
      - title:  test2
        url:    /photo/
        menu:
        - title:  test21
          url:    /photo/
        - title:  test22
          url:    /photo/
  - title:             Résumé\CV
    url:               /resume/
  - title:             Publications
    url:               /resume/#publications
  - title:             Projects
    url:               /projects/

Photo Gallery

I also like to have full control over my website and my server, which is why I set up Jekyll in a dedicated server through a cheap hosting service and my own domain from Google Domains.

When self-hosting your website however, adding new posts or even making small changes to your website can be rather time consuming. This is especially true when working from different machines.

I eventually came across Coder and the free cloud-editing service behind it code-server.

• Getting your dedicated server up and running
  • Self hosting
  • Domain
• Setting up your server
  • Setting up SSH access
• Setting up NGINX and code-server
• Configuring NGINX
• Setting up code-server
• Configuring code-server
• Exposing code-server with nginx
  • Opening the server’s ports
  • Forwarding port to subdomain
  • Get an SSL certificate with Let's Encrypt!

Getting your dedicated server up and running

Self hosting

After some research I ended up going with Vultr as a hosting provider. Being mine a static, lightweight website, I do not really need a powerful server with lots of space.

I opted for the last option with 25gb SSD and 1GB of RAM which is fairly cheap at AU$ 5 per month.

Domain

In order to make your website secure and trusted you need to set up SSL and HTTPS, and this can only be done with a domain and not with pure IP addresses.

I chose Google Domains in this case as it is fairly cheap and managing DNS records is easy.

Once you set up your domain you can also set up your subdomain for whatever services you might need. This is what my DNS records table look like:

DNSSEC is disabled and I am using Google Domains servers.

Setting up your server

Now it’s time to set up your server and environment. The first thing you want to do is to set up SSH.

Setting up SSH access

HEADS UP: NEVER share your SSH public and/or private keys on the internet! The keys down here have been generated for the purpose of this tutorial and promptly discarded!

This will differ depending on your provider. On Vultr you can go to https://my.vultr.com/settings/#settingssshkeys, click on the “+” button and then “Add SSH Key”. Give it a name to remind you what is the key for, something like vultr-server-blog.

Now you need to create your SSH key, I found this guide fairly simple and straight to the point Create an SSH Key with OpenSSH.

In short:

• Open PowerShell
• Type in ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
• Confirm the file location
• Enter a passphrase for your key
• Enter the same passphrase again

PS C:\Users\Giaga> ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
Generating public/private rsa key pair.
Enter file in which to save the key (C:\Users\Giaga/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\Giaga/.ssh/id_rsa
Your public key has been saved in C:\Users\Giaga/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:gn18jblysbS0XESUadoWM/2pEFJzSYSCEq1lvstPWJQ your_email@example.com
The key's randomart image is:
+---[RSA 4096]----+
|     .o . .+*B.  |
|     . = o.+X..  |
|      *  Eo+o+ ..|
|     + o. .*o  ..|
|    . o S.B.+ .  |
|       +o= B .   |
|      ..o.B      |
|       o.o       |
|        ..       |
+----[SHA256]-----+

Now it’s usually a good idea to make a backup of the private key as they cannot be recovered if lost! Copy the file somewhere safe, possibly NOT on the cloud!

You can now get your public key from the PowerShell by typing:

(base) PS C:\Users\Giaga> type $env:USERPROFILE\.ssh\id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCsQ/tUy8BcGpoWAMCjHHrucZKJLCuOJpFGFxTxhPDy5xet6khpxpfci/cLgyIUm4DEZSGQpCMszl3MXGJ7QuFDoSBue+MebKH9ZC1sK27+DFaFJXvDdD3OhFyTgqwrEIGZ7FKr7GNdMpER+fItUsAKx3ylwBLU4/dh13jsKE968RtApRkluch8M/L3MfzFBaDWJUPO0E+boQv8nOhpZqPU+WL2mzM4j/QA5BGssCGxWOm3zV4m4FTJGlJetN++5P67qTkMyu3U/UW4Y1ItStwnU8Wz/QvRGppOChJgYoAtWfR6MLIXd2iJ1G6IVwtLokVuFnJZFRnOo4BrdabPmqIPD+t3DtBfrsRhXF75IshMrR14r3EYFZKOsX3iJGEWfe3hBmvkoAOmtrV0n70cERcRQdnb8kdgjP4jT60gkt9GMz9UMmAMsMGfhRl3JSMv1bbLoe849xS5T9j/SCMSxQeb6XI6V3mBpFkmIaINs6VwV4/AJGZ9JA/tA175nDrCLsyyo5cGtShNNCYnHSGcFkCc4AWipTTPLdbak3jHetPyK6xn1ExRHKnJd/n65AUFdHNkNj2qj1fSFaWGxOJ706Am6nDWppUVSgWys1J+reXvAt7CFO8lPNhkcWxfJsAfcPpHkvF6fBnPScL8HXa9DwMzbfDNkN7aMkF35sY1z/c34w== your_email@example.com

Copy everything from ssh-rsa to your email included and paste it onto the textbox from the previous point.

That’s it! You can now test your SSH on your PowerShell with

(base) PS C:\Users\Giaga> ssh -i C:\Users\Giaga/.ssh/id_rsa username@123.45.678.910
username@123.45.678.910's password:
[...]
username@vultr-server:~$

If you had already set up your domain and DNS you can also start an SSH session using your domain:

(base) PS C:\Users\Giaga> ssh -i C:\Users\Giaga/.ssh/id_rsa username@gmarini.com
The authenticity of host 'gmarini.com (45.76.124.120)' can't be established.
ED25519 key fingerprint is SHA256:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.
This host key is known by the following other names/addresses:
    C:\Users\Giaga/.ssh/known_hosts:4: 45.76.124.120
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'gmarini.com' (ED25519) to the list of known hosts.
username@vultr-server:~$

Setting up NGINX and code-server

The two most popular choices for web servers are apache and nginx. I have tried them both but personally I find nginx easier to deal with and more intuitive.

Installing nginx is relatively simple:

sudo apt update
sudo apt install nginx

You can then adjust the Firewall rules with ufw. First list all available apps:

username@vultr-server:~$ sudo ufw app list
[sudo] password for username:
Available applications:
  Nginx Full
  Nginx HTTP
  Nginx HTTPS
  OpenSSH

Then allow the ones you need:

sudo ufw allow 'Nginx HTTP'
sudo ufw allow 'Nginx HTTPS'

You can always manage the status of nginx with:

sudo systemctl status nginx
sudo systemctl start nginx
sudo systemctl stop nginx
sudo systemctl restart nginx
sudo systemctl reload nginx
sudo systemctl enable nginx
sudo systemctl disable nginx

Configuring NGINX

Once NGINX is set up you can start serving your own static website by pointing at the folder with the file index.html.

The way NGINX works is by using configuration files contained in two main folders sites-available and sites-enabled.

The usual process is this:

• Create a new configuration file for your wite in the sites-available folder

sudo nano /etc/nginx/sites-available/gmarini.com

• Edit your configuration to point to your website’s root folder. E.g.:

server {
        listen 80;
        listen [::]:80;

        root /var/www/gmarini.com/html;
        index index.html index.htm index.nginx-debian.html;

        server_name gmarini.com www.gmarini.com;

        location / {
                try_files $uri $uri/ =404;
        }
}

• Create a symlink to the file you just created into the sites-enabled folder

sudo ln -s /etc/nginx/sites-available/gmarini.com /etc/nginx/sites-enabled/

• Test your file to make sure there are no errors:

sudo nginx -t

• Restart nginx:

sudo systemctl restart nginx

Setting up code-server

Now that you know how to configure NGINX is time to set up Visual Studio Code! While Visual Studio Code is not officially made to be run in a browser, the fact that it’s developed on top of GitHub’s Electron means that it is pretty much a webapp itself. In fact VScode uses Chromium engine for rendering and node.js to handle local APIs, making it easily portable to the web!

As explained in the intro, a web port of VSCode called code-server can be found here.

So first things first, let’s install code-server by using the official install script:

curl -fsSL https://code-server.dev/install.sh | sh

The documentation is quite detailed and well done! For reference, the next two steps are a quick summary of these two bits:

1. Expose code-server
2. HTTPS and self-signed certificates

Configuring code-server

Code-server uses password authentication by default. The password can be find in its config file located at ~/.config/code-server/config.yaml We can also set the binding address and port.

sudo nano `~/.config/code-server/config.yaml`

My config file looks something like this

bind-addr: 0.0.0.0:8080
auth: password
password: xxxxxxxxxxxxxxxxx
cert: false

Exposing code-server with nginx

Opening the server’s ports

Let’s also make sure that we our Vultr server allows connections through the port we just chose. Most providers will handle Firewall rules through security groups. You can create a new security group accepting TCP and UDP connections from the port you have chosen (8080 in this case) and from any ip/subent usually denoted as 0.0.0.0/0. Sometimes you might have to create two different rules, one for TCP and one of UDP connections, with the same port and IP range.

Forwarding port to subdomain

Now that code-server is set up we can link it with the subdomain we created earlier code.gmarini.com.

Let’s create a new nginx configuration file for code-server

sudo touch /etc/nginx/sites-available/code-server.conf

The server configuration should look something like this:

server {
    listen 80;
    listen [::]:80;
    server_name code.gmarini.com www.code.gmarini.com;

        location / {
                proxy_pass http://127.0.0.1:8080/;
                proxy_set_header Host $host;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection upgrade;
                proxy_set_header Accept-Encoding gzip;
        }
}

Now let’s enable the new configuration

sudo ln -s ../sites-available/code-server /etc/nginx/sites-enabled/code-server.conf

And reload nginx

sudo nginx -t && sudo nginx -s reload

Get an SSL certificate with Let's Encrypt!

Now that the server is ready and the IP is linked to the domain, we need to set up an SSL certificate for the domain itself. We can self-validate our domain by using certbot.

First let’s install certbot

sudo add-apt-repository ppa:certbot/certbot

Then let’s make sure the right ports are open

sudo ufw allow 80
sudo ufw allow 443

Finally, we can certify our domain by providing our email address:

Let’s encrypt

sudo certbot --non-interactive --redirect --agree-tos --nginx -d code.gmarini.com -m giaga7@gmail.com

Just follow the prompts and you should be ready to go! Certbot will automatically add the certificate to your nginx configuration file code-server.conf which should look like something like this in the end:

server {
    server_name code.gmarini.com www.code.gmarini.com;

        location / {
                proxy_pass http://127.0.0.1:8080/;
                proxy_set_header Host $host;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection upgrade;
                proxy_set_header Accept-Encoding gzip;
        }

    listen [::]:443 ssl; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/code.gmarini.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/code.gmarini.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
    if ($host = code.gmarini.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    listen 80;
    listen [::]:80;
    server_name code.gmarini.com www.code.gmarini.com;
    return 404; # managed by Certbot

Finally, we can restart code-server and test it out by navigating to our subdomain!

sudo systemctl restart code-server@$USER

• PhD
• Internship
• Life in Australia
• Other Stuff

PhD

Well I haven’t been dealt the best hand when it comes to my PhD. My project was all about tracking people’s location indoor, studying their interactions and how they move around buildings. Thanks to COVID, buildings were pretty much empty in 2020 and 2021 so most of my work went out of the window. I’m working on my second big first-author paper and hoping to get another one before the end of the year!

As of now I would like to submit my thesis my Jan/Feb 2022 but who knows! As of 3 months ago we were enjoying life again, travelling everywhere. And now we’ve been on an off lockdown since June, too unpredictable to say!

Internship

Summer Data Science Internship at Nokia Bells Lab, Cambridhge, UK

Overall the biggest update career-wise is my internship at Nokia Bells Lab. Despite it being a remote internship, it was a very interesting and eye opening experience nevertheless. As I found out during these three months, research in industry is quite a bit different from research in Academia. For instance one of the things I noticed is the ethics process: In some Universities this process can take anywhere from a few weeks to a few months, and it is extremely important to not go out of the lines defined by your ethics application. In industry however it would appear that things are a bit more streamlined and open-ended, considerably speeding up the process of getting ethics and starting the data collection.

The initial project proposal was along the line of sensing and social dynamics. The idea was to match the “offline” interaction of a team with its “online” counterpart. In this scenario the offline part would be tracked through an indoor localization system while the online aspect constitutes of Slack messages, channels and interactions.

We then set out to deploy the indoor localization system consisting of RFID-based active badges and Ethernet receivers. In the few weeks since the beginning on the internship I received a lot of interesting devices, tinkering with all sorts of systems and programming languages. A quick list of what I’ve worked on in the first month or two of the internship:

• Developed a Slack Bot to download messages and workspace meta-data automatically (GitHub Link)
  • The exporter is multi-threaded and takes care of load balancing the calls to avoid getting rate limited by the Slack API
• Adapted, fixed and implemented a set of NLP tools to automatically analyse the Slack corpus:
  • LIWC
  • 10 Dimensions
  • Integrative Complexity
  • Sentimnent Analysis through Flair
• Implemented the above NLP tools in an encrypted, secure and multi-threaded Python Flask Server service (GitHub Link)
• Implemented a server to collect RFID data from the active badges
• Configure the Active Badges through Serial Port
• Learnt Data Analysis and Machine Learning on Python, Pandas and MatPlotlib!
  • Not sure whether I enjoy it more than R
  • I have written some pretty cool functions to export graphics and LaTeX code for density and correlation plots (GitHub Link)
• Written a not-so-simple multi-threaded R script to clean up and complete LaTeX references in .bib files (GitHub Link)

Life in Australia

As of July 14th 2021 I’m officially a Permanent Resident (PR) in Australia!!

I submitted an EOI (Expression of Interest) for the Global Talent Visa Subclass 858 on the 29th of January 2021. I then got the invitation to apply on the 1st of June 2021. I prepared all the documents proving my “talent” and the ability to earn quite a high salary eventually, plus police clearance and criminal record from Italy. Once I submitted the actual application with all the documents I had to do a Visa medical check. I booked the first available appointment, which was the 14th of August 2021!! Quite late and I was running the risk of lockdown being in place at that time, further delaying my PR. As a matter of fact, we were in lockdown on that day, and still are at the time of writing :(

I was obviously not going to wait until August and I wrote a little Python+Selenium script to automatically check the BUPA medical services website and modify my appointment whenever an appointment closer to the current day would be available. I ran the script for a few hours on the 28th of June morning and what do you know, it automatically changed my appointment to the day after,t he 29th of June 2021!

Once I did the medical check, I waited 2 weeks and I eventually got my PR! What a relief, at least I’ll be able to live in Australia beyond my student visa expiration date. But most importantly, I’ll be more easily hirable by any company in Australia!

Other Stuff

I updated the website with a few new features. Most importantly I ended up buying the awesome Hydejack Pro theme for Jekyll

Well this part is the one that got affected the most and not just because of the pandemic! I have managed to lose 14kg since the beginning on 2020, went from 80kg to 66kg and I finally learnt the handstand properly (I can still walk for quite a few steps!).

Differently from Europe, the public transport in Australia just sucks. It’s expensive, rather slow, and the stations do not seem to be well spread out. I got a bike as soon as I moved to Melbourne and it has been essential. But I started to feel the need for more adventures, especially in those few weekends that Victoria was not in lockdown! So I got myself a cheap-ish second hand car, a Toyota Rav4 2011 (Manual) to drive around Australia and go camping and hiking!

• I chucked a mattress and two pillows in it. Got all the camping and cooking gear for when I’m on the road!
• So far I drove ~30 000km already.
• Went to Mungo desert twice in NSW
• Drove all along NSW and Queensland all the way up to Noosa (absolute heaven)
• Drove back to Melbourne alone on a 19h trip from Noosa!
• Drove a few times to the Great Ocean Road, on the snow in Mt. Donna Buang, and of course Phillip Island, Mornington Peninsula and Wilson’s Prom!

Since I got a car of my own, I thought it’d be nice to do more water-based activities so I also bought a Stand-Up Paddleboard (SUP)! I brought it all the way to Queensland during my end of the year trip in 2020/2021. Unfortunately the paddleboard was defective but I got it replaced for free with a newer model (Aztron Terra)!

I then got stuck in Queensland due to borders closure caused by a COVID outbreak. I took the chance to get my PADI open water scuba diving license while in the Gold Coast, Queensland!

Photography wise, I got myself a Sony 70-200m f2.8 G-master lens! I thought I would end up not using it much but oh boy! It’s incredibly sharp and fast! Perfect for portraits :)

I’m not sure whether I have more updates since the past year! I’ll add them whenever I remember them :)

For this example, I’ll scrape data from a Plant DNA C-Values Database website. The website has a search function which dynamically loads data in a table displaying the search result.

• Static crawling with Javascript
• Dynamic web scraping with Python and Selenium

Static crawling with Javascript

The first and easiest way of getting some data from a table is definitey with Javsacript. It runs on the browser and with the addition of JQuery everything becomes extremely easy. We could use Javascript on the Chrome developer console.

First and foremost, we need to load JQuery into our Javascript session. Let’s open up the Chrome Developer console with CTRL + SHIFT + J and type in the following lines

var jqry = document.createElement('script');
jqry.src = "https://code.jquery.com/jquery-3.3.1.min.js";
document.getElementsByTagName('head')[0].appendChild(jqry);
jQuery.noConflict();

The page might already have JQuery loaded but it might be assigned to a differnt symbol than the usual $, so noConflict() is helping us in this regard by not overriding the $. We can still call jQuery by using the jQuery() function.

Next up we need a function to convert the table to a csv file. I have taken inspiration by a pretty nice jQuery plgin called table2csv. I have added aa download function to download the file automatically and edited che code to call it and some default options.

jQuery( document ).ready(function($) {
    function download(filename, text) {
        var element = document.createElement('a');
        element.setAttribute('href', 'data:text/csv;charset=utf8,' + encodeURIComponent(text));
        element.setAttribute('download', filename);

        element.style.display = 'none';
        document.body.appendChild(element);

        element.click();

        document.body.removeChild(element);
    }

    jQuery.fn.table2CSV = function(options) {
            var options = jQuery.extend({
                separator: ',',
                header: [],
                headerSelector: 'th',
                columnSelector: 'td',
                delivery: 'download', // popup, value, download
                filename: 'table.csv', // filename to download
                transform_gt_lt: true // make &gt; and &lt; to > and <
            },
            options);

            var csvData = [];
            var headerArr = [];
            var el = this;

            //header
            var numCols = options.header.length;
            var tmpRow = []; // construct header avalible array

            if (numCols > 0) {
                for (var i = 0; i < numCols; i++) {
                    tmpRow[tmpRow.length] = formatData(options.header[i]);
                }
            } else {
                $(el).filter(':visible').find(options.headerSelector).each(function() {
                    if ($(this).css('display') != 'none') tmpRow[tmpRow.length] = formatData($(this).html());
                });
            }

            row2CSV(tmpRow);

            // actual data
            $(el).find('tr').each(function() {
                var tmpRow = [];
                $(this).filter(':visible').find(options.columnSelector).each(function() {
                    if ($(this).css('display') != 'none') tmpRow[tmpRow.length] = formatData($(this).html());
                });
                row2CSV(tmpRow);
            });
            if (options.delivery == 'popup') {
                var mydata = csvData.join('\n');
                if(options.transform_gt_lt){
                    mydata=sinri_recover_gt_and_lt(mydata);
                }
                return popup(mydata);
            }
            else if(options.delivery == 'download') {
                var mydata = csvData.join('\n');
                if(options.transform_gt_lt){
                    mydata=sinri_recover_gt_and_lt(mydata);
                }
                download(options.filename, mydata);
                //var url='data:text/csv;charset=utf8,' + encodeURIComponent(mydata);
                //window.open(url);
                return true;
            } 
            else {
                var mydata = csvData.join('\n');
                if(options.transform_gt_lt){
                    mydata=sinri_recover_gt_and_lt(mydata);
                }
                return mydata;
            }

            function sinri_recover_gt_and_lt(input){
                var regexp=new RegExp(/&gt;/g);
                var input=input.replace(regexp,'>');
                var regexp=new RegExp(/&lt;/g);
                var input=input.replace(regexp,'<');
                return input;
            }

            function row2CSV(tmpRow) {
                var tmp = tmpRow.join('') // to remove any blank rows
                // alert(tmp);
                if (tmpRow.length > 0 && tmp != '') {
                    var mystr = tmpRow.join(options.separator);
                    csvData[csvData.length] = mystr;
                }
            }
            function formatData(input) {
                // double " according to rfc4180
                var regexp = new RegExp(/["]/g);
                var output = input.replace(regexp, '""');
                //HTML
                var regexp = new RegExp(/\<[^\<]+\>/g);
                var output = output.replace(regexp, "");
                output = output.replace(/&nbsp;/gi,' '); //replace &nbsp;
                if (output == "") return '';
                return '"' + output.trim() + '"';
            }
            function popup(data) {
                var generator = window.open('', 'csv', 'height=400,width=600');
                generator.document.write('<html><head><title>CSV</title>');
                generator.document.write('</head><body >');
                generator.document.write('<textArea cols=70 rows=15 wrap="off" >');
                generator.document.write(data);
                generator.document.write('</textArea>');
                generator.document.write('</body></html>');
                generator.document.close();
                return true;
            }
        };
});

Now for the final touch, just call this line on the result of your search and it will conver the table to a .csv file and automatically download it or you. Pretty cool uh?

jQuery( document ).ready(function($) {
     jQuery("table.search-results").table2CSV();
});

There are quite a few limitations with this approach:

• Everytime the page reloads we need to re-run the code above
• We need to do this manually for each page of the table

Dynamic web scraping with Python and Selenium

Now that we sort of know where the data is and how to extract it we can think of how to automate this process. The most common tools for web scraping are selenium and beautifulSoup packages for python.

Getting Selenium to work should be a post of its own but for now let’s assume you followed this guide and that everything is working. I am using Anaconda3 so I simply ran:

conda install -c anaconda beautifulsoup4 request selenium

I then downloaded the Chrome WebDriver from here and extracted it to C:\chromedriver\chromedriver.exe.

HEADS UP: Make sure your Chrome driver version matches your installed Chrome version. For instance: I was using Chrome 84 but the chrome driver was for Chrome 85, I just updated my Chrome and it all worked perfectly.

Now that we all set up, let’s start by opening our page:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException
from bs4 import BeautifulSoup
import re
import pandas as pd
import os
import time
import csv


url = "https://cvalues.science.kew.org/search"
# Create a new Chrome session
driver = webdriver.Chrome('C:\\chromedriver\\chromedriver.exe') # I am using Chrome on Windows so I need to specify the path
driver.implicitly_wait(0.1)
driver.get(url) # Navigate to the page
titme.sleep(30) # Give the user some time to see the page at least
driver.close()

This code is fairly simple: Create a new Chrome session, navigate to the page, wait for 30 seconds and then close it. But what about implicitly_wait()? This line sets the amount of time Selenium waits for an element before throwing an Exception. So calling findElement() will throw an exception after 0.1 seconds if the element is not found. Remembe that this line sets the waiting time. So you only need to call it once to sets the time for the whole session until the browser is closed.

When we open the link we provide you will notice there is no table but only a bunch of paramters we can set to filter our search. Now we could:

• Program our script to fill in these parameters and let selenium click the search button
• Let the user fill the search parameters on the web page and then click search

The second approach has the pros of saving us a lot of time and headaches especially in case the structure of the form changes. The only thing we should be careful about is to make sure Selenium knows when the search button has been clicked and when to start scraping the data from the table.

The easiest way I found is to just have a loop that constantly checks for the presence of the table of results.

while True:
    tabResults = None # This will contain the table container, it's initially None since we haven't found it yet
    try:
        tabResults = driver.find_element_by_id("tabResults") # Try to find the tabResults which is the table container
    except Exception as e:
        continue # If there was an exception then the table was not found so we should skip the next part and re-start the loop
    ...# extract the data

The code is pretty self explanatory: Try to get the table container and if this throws an exception (due to the implicitly_wait()) restart the loop. If there is not exception and tabResults is not None then do something.

Let’s create some functions to extract the data from the table:

def rowToCsvSel(columns, csvwriter):  
    tdRow = []  
    for td in columns:
        tdRow.append(td.text)
        referenceData = td.find_elements_by_css_selector("a")
        if len(referenceData) > 0:
            tdRow.append(referenceData[0].get_attribute("data-content"))

    if len(tdRow) > 0:
        csvwriter.writerow(tdRow)
        return 1
    return 0

def getTableDataSel(tabResults):
    global currentResult
    global looping
    global currentFileIndex
    global csvfile
    global wr
    numbers = re.findall(r'\d+', tabResults.find_element_by_css_selector("h1").text)
    # print(numbers)
    fromRes = int(numbers[0])
    toRes = int(numbers[1])
    total = int(numbers[2])  
    if fromRes <= 1:
        currentResult = fromRes
        looping = True
        csvfile = open('plants_data{}.csv'.format(currentFileIndex), 'w', newline='', encoding='utf-8') # Getting the handle to the file, I need to specify UTF-8 since the table contains UTF characters
        wr = csv.writer(csvfile) # Getting the csv writer with the file associated
    table =  tabResults.find_element_by_class_name("search-results")
    if looping:
        for row in table.find_elements_by_css_selector('tr'):
            driver.implicitly_wait(0) # Without this it will just hang since the td elements might not have been loaded
            print("Result {} of {}".format(currentResult, total))
            if currentResult <= 1:                
                rowToCsvSel(row.find_elements_by_css_selector('th'), wr)
            currentResult += rowToCsvSel(row.find_elements_by_css_selector('td'), wr)
    if toRes >= total:
        if looping:
            print("Data collection completed!")
            currentFileIndex += 1
        looping = False
    return currentResult

There is a lot to explain here but let’s go step by step:

• The global variables are just a way to keep track of what’s happening and

What are regular expressions

Text, especially code, can easily be parsed by defining patterns and looking for them. Think about it as if you are reading a page of a book or even a website: You look for patterns, be it an email, a phone number, the structure of a sentence.

While the semantic of a sentence is not easy to parse and understand for a computer (see the whole branch of NLP), encoded and marked text can easily be captured by specifying patterns. Emails for instance are defined by username@domain and we all know how to read them, we all know that username is anything that preceeds the @ character, and anything after it is the domain.

I find this concept of patterns, text and encoding extremely fascinating and stimulating. I might make a post just about them but for now let’s focus on this little project.

• What are regular expressions
• Downloading GitHub repositories data
  • Downloading the README.md
  • Downloading the images used in the README.md
• How regular expressions work
• RegEx to match markdown image sources
  • Wrapping up

Downloading GitHub repositories data

Downloading the README.md

I wanted to port all of my GitHub projects on my website. Since the README.md files are already markdown, Jekyll will know how to render them. Of course I would also like to get all the images I use in the readmes. As the lazy programmer that I am, I would rather spend 2 hours writing some code that does it for me rather than manually copy-pasting each project’s README.md file.

The first thing I needed to do is to find out whether GitHub had an API or not. Depending on the answer I could have used a python-based browser, for instance beautifulsoup or the request package to send GET and POST.

Luckily GitHub does have an API and of course there already was a package wrapped around it, PyGithub. So let’s start by importing it and getting my own info from github

from github import Github
import oyaml

with open(("github_credentials.yaml"), "r") as cf:
    cred = oyaml.safe_load(cf)
g = Github(cred['username'], cred['password'])

for repo in g.get_user().get_repos():
    print(repo.name)

I made the mistake once of committing a .py file with my credentials in plain text. This time I am not making the same mistake. My username anf file are in a yaml file structured like this:

username: "gabryxx7"
password: "password" # You wish...

I used a wonderful package called oyaml (ordered yaml) which is just a wrapper around yaml. I can’t remember why I stiched to this one, something to do with the ordering, stumbled upon this issue while working on the AI_Dating project. Anyways back to the topic. Once I got my own GitHub details, I am iterating through the list of repositories in my account. As soon as I run the code I saw an enormous amount of new lines being printed. Well that was not because of my awesome curriculum and activities on GitHub, it was because of the new GitHub Classroom that we are using at the University of Melbourne for the COMP30019 subject. Every student in the subject can copy one of our workshop’s template repository in our GitHub organisation/classroom. And since me and the other tutors are the owner of the organisation, we are also owners of all the repositories, which means all of those repositories appear in my list. Besides that, there are lots of forks that I don’t really use. In short, I only want MY repositories, the ones that I created. so with a very simple string comparison:

for repo in g.get_user().get_repos():
    if "workshop" not in repo.name.lower():
        if 'gabryxx7' in repo.owner.login.lower() and not repo.fork:
            print(repo.name)

Now we are talking! If the name workshop appears in the title of the repository we’ll just skip it. Same if the owner is not me. Unfortunately, PyGithub documentation is not great so it’s hard to know what variables or methods the repository object has. In these cases I usually just go to the source code repository and with a simple ctrl + f I look for whatever I need. In this case I was looking for:

• The name of the repository
• The README.md file url
• The repository’s date of creation
• The repository caption and description
• The repository’s link

So after a few ctrl+f here and there I found everything I needed to create a Jekyll project out of the GitHub repo. Let’s set up the yaml header for Jekyll:

header = oyaml.safe_load("""
    layout: project
    title: title
    icon: icon-github
    caption: caption
    date: 2020-01-01
    image: ""
    path: ""
    description: >
    Descr
    links:
    - title: Source
        url: https://github.com/Gabryxx7
    """)

I find it easier to define it this way and then read it with yaml rather than defining the dictionary and adding every (key,value) pair to it.

So now we are ready to download our README.md and use them as jekyll projects!

for repo in g.get_user().get_repos():
    if "orkshop" not in repo.name:
        if 'abryxx7' in repo.owner.login and not repo.fork:
            print(repo.name)
            try:
                print(repo.get_readme().download_url)        
                abs_path = os.path.abspath('../../../_projects/github/')
                file_name = str(repo.name)+".md"                                        
                if os.path.exists(abs_path+'/'+file_name):
                    print("Readme already exists: " +abs_path+'/'+file_name)
                else:
                    # Create the folder tree if it does not exist
                    if not os.path.exists(abs_path):
                        os.makedirs(abs_path)
                    # Download the readme/md
                    r = requests.get(repo.get_readme().download_url, allow_redirects=True)
                    if r.content is not None:
                        readme_text = r.content.decode('utf-8') # decode the binary content as 'utf-8' to allow for emojis
                        if readme_text is not None and len(readme_text.split('\n')) > 5:
                            header['title'] = repo.name
                            header['description'] = repo.description
                            header['caption'] = repo.description
                            header['date'] = repo.created_at
                            header['links'][0]['url'] = repo.html_url
                            header['image']['path'] = first_image

                            header_text = str(oyaml.dump(header, allow_unicode=True))

                            open(abs_path+'/'+file_name, 'w', encoding="utf-8").write("---\n"+header_text+"\n---\n\n"+str(readme_text))
                            first_image = None
            except Exception as e:
                print("No README.md " +str(e))
            print() # new line separator

And there you have it! This little for loop will now dowlonad all of your readmes from GitHub and save them into the _projects folder with a nice yaml header.

But WAIT! the readmes often have images, which are relative to the repository, or they are even external links or links to the raw image file in github. Oh shoot! What’s the point of having this nice project showcase if I then have to manually add the screenshots? And what if I change anything on github, I’d have to do it again manually and it would take ages! Plus I’d have to replace the relative paths in the code as well, besides downloading all the images.

Downloading the images used in the README.md

Now it’s the fun part where regexr come in super handy! Images in markdown can be in a few different formats, including html. This is a few examples of how I used images in my README.md files. I have never really paid much attention to how I put images in it, so they are often in the same line, with wrong spacings, with or without alternative text, sometimes relative paths, sometimes a full url, sometimes even from external websites!

![Blynk App](Blynk1.jpg) ![Blynk App](/assets/gabryxx7/img/GitHub/ShrekIntruderAlert/Blynk2.jpg)

<img  alt="The Tower Logo" width="250" src="https://github.com/Gabryxx7/TheTower/blob/master/Textures/thetower1.png"/><br/>

<img src='https://github.com/Gabryxx7/TheTower/blob/master/Textures/thetower2.png' alt="The Tower Logo" width="250"/> <img src='https://github.com/Gabryxx7/TheTower/blob/master/Textures/thetower3.png' alt="The Tower Logo" width="250"/> 

<img alt="Gabryxx7 | Instagram" height="100px" src="https://gmarini.com/assets/gabryxx7/img/photos/202006/79841217_555270635156572_8260864494112333405_n.jpg" />

![Screenshot](tinderai1.png)

![](tinderai2.png)

![](https://github.com/Gabryxx7/TheTower/blob/master/Textures/thetowerasd.png)

Now let’s say we want to download all of these images on our computer and then maybe reference them in the same README.md we downloaded earlier, but with their new path in our computer. Now that sounds like a mess! We need to extract the image filename, understand whether it’s relative to the repository or not, and then even replace its occurrences with the new local path. Man that’s a lot of different patterns, how to deal with all of this?

Luckily we have our “friends” the regular expressions. It took me around an hour to get the right expression and I saved it here for future references RegExr.

IMG_REGEXR = "(?:!\[[^\]]*\]\(([^\)]*)\))+|(?:<img\s.*?src=(?:'|\")([^'\">]+)(?:'|\"))"

Now that looks like some sort of alien language, does it? Let’s break it down. Before that, a quick explanation on regexp.

How regular expressions work

Regular Expressions are used to define patterns in text. The idea is to define a pattern and then execute it on your string or document, and it will return you all the “matches” or occurrences of that pattern in your string/document.

They have a few important special features and characters, I’ll go through the most common ones:

• .: The dot is a special characters than it matches ANY character except for a new line. This means that it will return as a match anything: spaces, commas, letters, digits, tabs etc…
• |: either/or, a|b matches either an a or a b but not both
• \. \\: Matches the actual character, in this case the actual dot and the actual forward slash respectively, no special meaning.
• \w \d \s: These three match a letter, a digit and a whitespace respectively
• \W \D \S: These three match anything BUT a letter, a digit and a whitespace respectively
• * + ?: These two tell how many times you want a character to be matches, respectively: 0 or more, 1 or more, 0 or 1
• []: Matches any one of the characters inside the square brackets. [ab] matches either a single a or a single b separately as separate matches.
• [^] Matches any character except for the one inside. [^ab] matches anything but a single a or a single b
• [-]: Matches any character in the range. [a-g] will match ay character between a and g separately, this is the same as \w. [0-9] will match any character between 0 and 9 separately, this is the same as \d

So we can now do something simple like .* which matches anything up to a new line. Running this will just tell us whether there is a match or not in the string. And well, this will always be true, even nothing will return true because the * matches 0 or more occurrences of .. gabriele will match the word gabriele for instance. Another example is [^a]+ which matches any character which is not an a but at least once.

An important features are groups, so that we can define a sequence of character

• () Everything in brackets is a “capture group”. Every pattern found inside the capture group can be referenced after running the expression and it’s very useful to reference certain parts of the pattern you want to extract
• (?:) This is a non-capturing group, same as before but cannot be referenced, well what’s the point then? You might ask. This is useful to define repetitions of a sub-pattern or characters

That’s pretty much it for now, let’s break down the regex above:

RegEx to match markdown image sources

First of all, the expression is divided in two by the pipe character |, the first one matches the markdown for images, which is in the form of ![]() while the second one matches the html <img src=... />.

• Markdown (?:!\[[^\]]*\]\(([^\)]*)\))+
  • (?:: Everything is surrounded by a non-capturing group
  • !\[: match an exclamation mark and an open square bracket
  • [^\]]*: MAtches anything up to a closed square bracket, there can also be nothing to match (so an empty square brackets [])
  • \]\(: We are not at the closed square bracket, so let’s match it and look for the open bracket
  • ([^\)]*): Now we are inside the round brackets and we need to capture the image url (or path). So this is a capture group and it matches any character up to a close round bracket [^\)], it can also be empty thanks to the *.
  • \): Now we are up to the closed round bracket, so let’s match it
  • ): Closure of the non-capturing group
  • +: This one is important, it matches the whole pattern we just defined in the non-capturing group, at least once. This is important in case we have two images in the same line such as ![](test.png) ![caption](test2.png). This will retuurn both test.png and test2.png
• HTML (?:<img\s.*?src=(?:'|\")([^'\">]+)(?:'|\"))
  • (?:: Everything is surrounded by a non-capturing group
  • <img: Matches the open html image tag
  • \s.*?: Matches any amount of empty spaces
  • src=: Matches the source attribute in the html image tag
  • (?:'|\"): This is a non-capturing group and it matches either a ' or a " quotation mark, we are opening the source string
  • ([^'\">]+): This is a capturing group and it matches anything but a ' or a " quotation mark
  • (?:'|\"): This is a non-capturing group and it matches either a ' or a " quotation mark to close the source string
  • ): Closure of the non-capturing group

Wrapping up

So there you have it, now we can use the re python package to match all of these in our README.md and replace the captured group with the new file’s local path

def download_and_replace(match):
    global first_image
    print(match.group(0))
    if match.group(1) is not None:
        cap_group = match.group(1).strip()
    else:
        cap_group = match.group(2).strip()
    url = ""
    if len(cap_group) > 0:
        if 'http' in cap_group:
            if 'github.com' in cap_group:                
                url = cap_group.replace('github.com', 'raw.githubusercontent.com').replace('blob/','')
            else:
                url = cap_group
            file_name = os.path.basename(urlparse(url).path)
        else:
            url = "https://raw.githubusercontent.com/" +repo.full_name+"/master/"+cap_group
            file_name = os.path.basename(cap_group)
        path_from_root = '/assets/gabryxx7/img/GitHub/'+repo.name+'/'
        abs_path = os.path.abspath('../../../'+path_from_root)

        if os.path.exists(abs_path+'/'+file_name):
            print("Skipping: " +abs_path+'/'+file_name)
        else:
            if not os.path.exists(abs_path):
                os.makedirs(abs_path)
            print("Downloading: " +abs_path+'/'+file_name +"\tFrom: " +str(url))
            f_img = requests.get(url, allow_redirects=True)
            open(abs_path+'/'+file_name, 'wb').write(f_img.content)

        print("Replaced string: " + match.group(0).replace(cap_group,path_from_root+file_name))
        if first_image is None:
            first_image = path_from_root+file_name
        return match.group(0).replace(cap_group,path_from_root+file_name)
    return cap_group

...

  readme_text = re.sub(IMG_REGEXR, download_and_replace, readme_text) # Calling a function for each match which will download the img and replace the path in the markdown file
  
...

The function is fairly simple but has a few catches:

• First we need to get our captured image path/url:
  • match.group(0) represents the WHOLE pattern, not just the captured groups
  • match.group(1) reference the first capture group (the markdown images)
  • match.group(2) reference the second capture group (the html images)
• If the input is not empty then we can keep going
  • If it’s a link there will be http in the string
  • We cannot download images from github links, so we have to convert those to raw.githubusercontent.com links
  • If it’s not a link then we concatenate the raw github link with the repository full name
• Now that we have our link we can send a request and download the binary blob
• Once we downloaded the file we just need to make sure the path exists, and write the content to our disk
• We then return the replacement for the whole match, this includes also the non-capturing group. So what we do is return the whole match but we substitute the captured group with the now file’s local path

We then return the edited README.md output to the for loop to then write everything to the new README.md file with the yaml header.

The final images’ path it’s now replaced with the local path of the file downloaded on our computer (or server), so lines like these ones:

![Blynk App](Blynk1.jpg) ![Blynk App](Blynk2.jpg)
<img  alt="The Tower Logo" width="250" src="https://github.com/Gabryxx7/TheTower/blob/master/Textures/thetower1.png"/><br/>
<img src='https://github.com/Gabryxx7/TheTower/blob/master/Textures/thetower2.png' alt="The Tower Logo" width="250"/> <img src='https://github.com/Gabryxx7/TheTower/blob/master/Textures/thetower3.png' alt="The Tower Logo" width="250"/> 

Will now look something like this:

![Blynk App](/assets/Gabryxx7/GitHub/img/ShrekIntruderAlert/Blynk1.jpg) ![Blynk App](/assets/gabryxx7/img/GitHub/ShrekIntruderAlert/Blynk2.jpg)
<img  alt="The Tower Logo" width="250" src="/assets/Gabryxx7/GitHub/img/thetower1.png"/><br/>
<img src='/assets/Gabryxx7/GitHub/img/thetower2.png' alt="The Tower Logo" width="250"/> <img src='/assets/Gabryxx7/GitHub/img/thetower3.png' alt="The Tower Logo" width="250"/> 

Putting it all together, the script ends up looking like this:

from github import Github
import requests
import oyaml
import re
import os
from urllib.parse import urlparse

IMG_REGEXR = "(?:!\[[^\]]*\]\(([^\)]*)\))+|(?:<img\s.*?src=(?:'|\")([^'\">]+)(?:'|\"))"
first_image = None

def download_and_replace(match):
    global first_image
    print(match.group(0))
    if match.group(1) is not None:
        cap_group = match.group(1).strip()
    else:
        cap_group = match.group(2).strip()
    url = ""
    if len(cap_group) > 0:
        if 'http' in cap_group:
            if 'github.com' in cap_group:                
                url = cap_group.replace('github.com', 'raw.githubusercontent.com').replace('blob/','')
            else:
                url = cap_group
            file_name = os.path.basename(urlparse(url).path)
        else:
            url = "https://raw.githubusercontent.com/" +repo.full_name+"/master/"+cap_group
            file_name = os.path.basename(cap_group)
        path_from_root = '/assets/gabryxx7/img/GitHub/'+repo.name+'/'
        abs_path = os.path.abspath('../../../'+path_from_root)

        if os.path.exists(abs_path+'/'+file_name):
            print("Skipping: " +abs_path+'/'+file_name)
        else:
            if not os.path.exists(abs_path):
                os.makedirs(abs_path)
            print("Downloading: " +abs_path+'/'+file_name +"\tFrom: " +str(url))
            f_img = requests.get(url, allow_redirects=True)
            open(abs_path+'/'+file_name, 'wb').write(f_img.content)

        print("Replaced string: " + match.group(0).replace(cap_group,path_from_root+file_name))
        if first_image is None:
            first_image = path_from_root+file_name
        return match.group(0).replace(cap_group,path_from_root+file_name)
    return cap_group


with open(("github_credentials.yaml"), "r") as cf:
    cred = oyaml.safe_load(cf)
g = Github(cred['username'], cred['password'])

header = oyaml.safe_load("""
layout: project
title: title
icon: icon-github
caption: caption
date: 2020-01-01
image: 
  path: ""
description: >
  Descr
links:
  - title: Source
    url: https://github.com/Gabryxx7/TheTower
""")

# Then play with your Github objects:
# print(dir(g.get_user().get_repos()[0]))
for repo in g.get_user().get_repos():
    if "workshop" not in repo.name.lower():
        if 'gabryxx7' in repo.owner.login.lower() and not repo.fork:
            print(repo.name)
            try:
                print(repo.get_readme().download_url)
                abs_path = os.path.abspath('../../../_projects/github/')
                file_name = str(repo.name)+".md"                                        
                if os.path.exists(abs_path+'/'+file_name):
                    print("Readme already exists: " +abs_path+'/'+file_name)
                else:
                    if not os.path.exists(abs_path):
                        os.makedirs(abs_path)
                    r = requests.get(repo.get_readme().download_url, allow_redirects=True)
                    if r.content is not None:
                        readme_text = r.content.decode('utf-8')
                        if readme_text is not None and len(readme_text.split('\n')) > 5:
                            readme_text = re.sub(IMG_REGEXR, download_and_replace, readme_text)

                            header['title'] = repo.name
                            header['description'] = repo.description
                            header['caption'] = repo.description
                            header['date'] = repo.created_at
                            header['links'][0]['url'] = repo.html_url
                            header['image']['path'] = first_image

                            header_text = str(oyaml.dump(header, allow_unicode=True))

                            open(abs_path+'/'+file_name, 'w', encoding="utf-8").write("---\n"+header_text+"\n---\n\n"+str(readme_text))
                            first_image = None
            except Exception as e:
                print("No README.md " +str(e))
            print()

And there you have it! A little python script to scrape your own github repository data and export their README.md as jekyll markdown posts, and their images in the correct jekyll assets folder.

• Setting up the R Markdown file
• Knitting it!

local({
  # fall back on "/" if baseurl is not specified
  baseurl = blogdown:::get_config2("baseurl", default = "/")
  knitr::opts_knit$set(base.url = baseurl)
  knitr::render_jekyll()  # set output hooks

  # input/output filenames as two arguments to Rscript
  a = commandArgs(TRUE)
  d = gsub("^_|[.][a-zA-Z]+$", "", a[1])
  knitr::opts_chunk$set(
    fig.path   = sprintf("figure/%s/", d),
    cache.path = sprintf("cache/%s/", d)
  )
  knitr::knit(
    a[1], a[2], quiet = TRUE, encoding = "UTF-8",
    envir = globalenv()
  )
})

So I understood that the knitr options called base.url is a prefix to fig.path, while base.dir is the actual directory where the images will be genereated, base.url is pretty much an alias.

For instance by setting base.url='/', fig.path='assets/r_figures/' and base.dir='C:\Users\Gabryxx7\Documents\GitHub\blog\' the final markdown for a figure will look something like this:

![](/assets/r_figures/nice_plot.png)<!-- -->

And nice_plot.png will actually be located at C:\Users\Gabryxx7\Documents\GitHub\blog\assets\r_figures\nice_plot.png.

This is awesome since Jekyll will interpret the first / as the website root folder where the assets folder actually is!

From the knitr documentation:

Knitr’s settings must be set in a chunk before any chunks which rely on those settings to be active. It is recommended to create a knit configuration chunk as the first chunk in a script with cache = FALSE and include = FALSE options set. This chunk must not contain any commands which expect the settings in the configuration chunk to be in effect at the time of execution.

Another interesting feature is the possibility to set up parameters into the yaml header and use them withing your code. This makes up for parameterized reports

So with all this knowledge, let’s get started.

Setting up the R Markdown file

When on RStudio the first thing you should do is to create a new R Markdown file. So File -> New File -> R Markdown.

Once you get your file, you can add the following lines to your heading and as the first chunk of your markdown file:

---
title: "R Markdown and jekyll"
output:
  md_document:
    variant: gfm
    preserve_yaml: TRUE
params: 
  fig_base_path: "C:\\Users\\Gabryxx7\\Documents\\GitHub\\blog\\assets\\gabryxx7\\r_figures"
knit: (function(input, ...) {
  rmarkdown::render(input=input,
    output_file=paste0(Sys.Date(),'-',xfun::sans_ext(basename(input)),'.md'),
    encoding = encoding,
    output_dir="C:\\Users\\Gabryxx7\\Documents\\GitHub\\blog\\blog\\coding\\_posts")
  })
---

```{r setup, include=FALSE}
knitr::opts_knit$set( base.dir=params$fig_base_path, base.url="/assets/gabryxx7/r_figures/")
knitr::opts_chunk$set(echo = TRUE, fig.path = "")

```

HEADS UP: If you get an error saying Error in setwd(...): cannot change the working directory you probably mispelled the path or pointed at a non existing directory. R Will only create directories specified under opts_chunk$set but anything else (for instance fig_base_path) will be expected to exist already.

So connecting back to blogdown I found out that you can override the rmarkdown::render() function in R Markdown and R Notebook files. The render function takes as parameters an input file path and an encoding string (here it’s included in ...).

By overriding the function we can tell it to render the .rmd input file to a new file with .md extension and the proper Jekyll’s post markdown formatting. The part xfun::sans_ext(basename(input)) is just getting the filename from the input file path and removing its extension. Finally just set the output_dir to be whatever your Jekyll post folder is (some extra freedom for you! Decoupling is always the key!).

A bit of an annoying thing is that you cannot just use a base path and then concatenate the twos since rmarkdown::render() will be called before the YAML header is processed. I mean, params is literally in the same header, unless YAML is interpreted which is not, there is no way for knit to know what other variables are in the yaml header.

For this reason you will have to copy-paste your folder path twice, once as the output_dir parameter in the rmarkdown::render() function and another time as a parameter in your YAML header. But that’s actually not all bad, as I said before now you can output your figures to whatever folder you prefer. This is especially useful in my case where I work on different machines (develope my blog on Windows/Macbook and depploy on Linux server).

So as you can see here:

• The .md markdown file will be generated in C:\Users\Gabryxx7\Documents\GitHub\blog\blog\coding\_posts
• The figure files will be generated in: C:\Users\Gabryxx7\Documents\GitHub\blog\assets\gabryxx7\r_figures
• The figures in the markdown file will all have /assets/gabryxx7/r_figures/ as base url so that jekyll will know where to get the files starting from the root of your website.

We can actually take it a step further and put the figures in each post’s separate folder in the r_figures base folder:

knitr::opts_chunk$set(echo = TRUE,
                      strip.white=TRUE,
                      message=FALSE,
                      warning=FALSE,
                      results="markup",
                      out.format="jekyll",
                      class.output="plaintext",
                      out.extra = "",
                      fig.path = paste0(Sys.Date(),'-',xfun::sans_ext(basename(knitr::current_input())),"/"))

In this way:

• The final path will be C:\Users\Gabryxx7\Documents\GitHub\blog\assets\gabryxx7\r_figures\2020-08-13-ggantt
• The images url will be /assets/gabryxx7/r_figures/2020-08-13-ggantt/nice_plot.png
• Message and warnings won’t be included in the code output. The code output will be wrapped in a code chunk with language plaintext so that we can capture it with javascript and do something with it

HEADS UP: Do not forget to add out.extra = "" otherwise knitr will add images with an ending comment like ![](/assets/gabryxx7/r_figures/2020-08-13-ggantt/test.png)<!-- --> completely messing up the html page in Jekyll

Knitting it!

Well there is not much left to do, just press knit on RStudio with your .rmd file opened and you’re good to go!

Alternatively you can call the knit function from the console but be careful as this won’t call the markdown::render() method in the yaml header so the .md file will be generated in the working directory unless you specify the output file path:

setwd("C:\\Users\\Gabryxx7\\Documents\\GitHub\\whatever\\") # adjust to your preferences.
knitr::knit("rmarkdown_file.rmd") # Generate to working directory
knitr::knit("ggantt.rmd", output = "C:\\Users\\Gabryxx7\\Documents\\GitHub\\blog\\blog\\coding\\_posts\\2020-08-13-rmarkdown_file.md") # Generate to Jekyll's post directory

You can also make your own function in case you have multiple files:

knit_to_jekyll <- function(input, ...) {
  rmarkdown::render(input=input,
    output_file=paste0(Sys.Date(),'-',xfun::sans_ext(basename(input)),'.md'),
    encoding = encoding,
    output_dir="C:\\Users\\Gabryxx7\\Documents\\GitHub\\blog\\blog\\coding\\_posts")
  }
knit_to_jekyll("r_notebook_markdown.rmd")

When you click the Knit button a document will be generated that includes both content as well as the output of any embedded R code chunks within the document. You can embed an R code chunk like this:

• Title 2
  • Including Plots

library(tidyverse)
vlines <- tribble(
  ~Date, ~LineType, ~Size, ~Color,
  "2020-01-01", "solid", 0.6, "grey80",
  "2021-01-01", "solid", 0.6, "grey80")
summary(cars)

##      speed           dist       
##  Min.   : 4.0   Min.   :  2.00  
##  1st Qu.:12.0   1st Qu.: 26.00  
##  Median :15.0   Median : 36.00  
##  Mean   :15.4   Mean   : 42.98  
##  3rd Qu.:19.0   3rd Qu.: 56.00  
##  Max.   :25.0   Max.   :120.00

Title 2

So Imma going to try to add a few extra lines and see what the output is going to be

summary(mtcars)

##       mpg             cyl             disp             hp       
##  Min.   :10.40   Min.   :4.000   Min.   : 71.1   Min.   : 52.0  
##  1st Qu.:15.43   1st Qu.:4.000   1st Qu.:120.8   1st Qu.: 96.5  
##  Median :19.20   Median :6.000   Median :196.3   Median :123.0  
##  Mean   :20.09   Mean   :6.188   Mean   :230.7   Mean   :146.7  
##  3rd Qu.:22.80   3rd Qu.:8.000   3rd Qu.:326.0   3rd Qu.:180.0  
##  Max.   :33.90   Max.   :8.000   Max.   :472.0   Max.   :335.0  
##       drat             wt             qsec             vs        
##  Min.   :2.760   Min.   :1.513   Min.   :14.50   Min.   :0.0000  
##  1st Qu.:3.080   1st Qu.:2.581   1st Qu.:16.89   1st Qu.:0.0000  
##  Median :3.695   Median :3.325   Median :17.71   Median :0.0000  
##  Mean   :3.597   Mean   :3.217   Mean   :17.85   Mean   :0.4375  
##  3rd Qu.:3.920   3rd Qu.:3.610   3rd Qu.:18.90   3rd Qu.:1.0000  
##  Max.   :4.930   Max.   :5.424   Max.   :22.90   Max.   :1.0000  
##        am              gear            carb      
##  Min.   :0.0000   Min.   :3.000   Min.   :1.000  
##  1st Qu.:0.0000   1st Qu.:3.000   1st Qu.:2.000  
##  Median :0.0000   Median :4.000   Median :2.000  
##  Mean   :0.4062   Mean   :3.688   Mean   :2.812  
##  3rd Qu.:1.0000   3rd Qu.:4.000   3rd Qu.:4.000  
##  Max.   :1.0000   Max.   :5.000   Max.   :8.000

##       mpg             cyl             disp             hp       
##  Min.   :10.40   Min.   :4.000   Min.   : 71.1   Min.   : 52.0  
##  1st Qu.:15.43   1st Qu.:4.000   1st Qu.:120.8   1st Qu.: 96.5  
##  Median :19.20   Median :6.000   Median :196.3   Median :123.0  
##  Mean   :20.09   Mean   :6.188   Mean   :230.7   Mean   :146.7  
##  3rd Qu.:22.80   3rd Qu.:8.000   3rd Qu.:326.0   3rd Qu.:180.0  
##  Max.   :33.90   Max.   :8.000   Max.   :472.0   Max.   :335.0  
##       drat             wt             qsec             vs        
##  Min.   :2.760   Min.   :1.513   Min.   :14.50   Min.   :0.0000  
##  1st Qu.:3.080   1st Qu.:2.581   1st Qu.:16.89   1st Qu.:0.0000  
##  Median :3.695   Median :3.325   Median :17.71   Median :0.0000  
##  Mean   :3.597   Mean   :3.217   Mean   :17.85   Mean   :0.4375  
##  3rd Qu.:3.920   3rd Qu.:3.610   3rd Qu.:18.90   3rd Qu.:1.0000  
##  Max.   :4.930   Max.   :5.424   Max.   :22.90   Max.   :1.0000  
##        am              gear            carb      
##  Min.   :0.0000   Min.   :3.000   Min.   :1.000  
##  1st Qu.:0.0000   1st Qu.:3.000   1st Qu.:2.000  
##  Median :0.0000   Median :4.000   Median :2.000  
##  Mean   :0.4062   Mean   :3.688   Mean   :2.812  
##  3rd Qu.:1.0000   3rd Qu.:4.000   3rd Qu.:4.000  
##  Max.   :1.0000   Max.   :5.000   Max.   :8.000

##       mpg             cyl             disp             hp       
##  Min.   :10.40   Min.   :4.000   Min.   : 71.1   Min.   : 52.0  
##  1st Qu.:15.43   1st Qu.:4.000   1st Qu.:120.8   1st Qu.: 96.5  
##  Median :19.20   Median :6.000   Median :196.3   Median :123.0  
##  Mean   :20.09   Mean   :6.188   Mean   :230.7   Mean   :146.7  
##  3rd Qu.:22.80   3rd Qu.:8.000   3rd Qu.:326.0   3rd Qu.:180.0  
##  Max.   :33.90   Max.   :8.000   Max.   :472.0   Max.   :335.0  
##       drat             wt             qsec             vs        
##  Min.   :2.760   Min.   :1.513   Min.   :14.50   Min.   :0.0000  
##  1st Qu.:3.080   1st Qu.:2.581   1st Qu.:16.89   1st Qu.:0.0000  
##  Median :3.695   Median :3.325   Median :17.71   Median :0.0000  
##  Mean   :3.597   Mean   :3.217   Mean   :17.85   Mean   :0.4375  
##  3rd Qu.:3.920   3rd Qu.:3.610   3rd Qu.:18.90   3rd Qu.:1.0000  
##  Max.   :4.930   Max.   :5.424   Max.   :22.90   Max.   :1.0000  
##        am              gear            carb      
##  Min.   :0.0000   Min.   :3.000   Min.   :1.000  
##  1st Qu.:0.0000   1st Qu.:3.000   1st Qu.:2.000  
##  Median :0.0000   Median :4.000   Median :2.000  
##  Mean   :0.4062   Mean   :3.688   Mean   :2.812  
##  3rd Qu.:1.0000   3rd Qu.:4.000   3rd Qu.:4.000  
##  Max.   :1.0000   Max.   :5.000   Max.   :8.000

Including Plots

You can also embed plots, for example:

Note that the echo = FALSE parameter was added to the code chunk to prevent printing of the R code that generated the plot.

So I wanted to give my blog a cool vibe while also looking good. I did not want it to be too overwhelming with effects, sliding animations and such. So I opted for something simple, it’s just text after all! But it’s also quite catchy and fun to watch.

So here is how I have achieved this effect, it’s nothing crazy hard really but I love playing with CSS and Javascript and so I did.

• Creating the terminal typewriter
  • The basic HTML
  • Styling the terminal
  • Start typing!
  • Adding a status bar
  • Extras (looping, list of words, emojis)
• Adding a terminal typewriter to your page or post
  • Adding content to the blog page
  • Allowing for custom featured images in posts
  • Adding the javascript
  • Adding a description to your page terminal

I started by looking for Javascript libraries that implemented the typewriter effect. I would rather go for pure javsacript than JQuery, even better if it’s a lightweight specific library that only includes that effect.

So I found Typewriter.js, which does exactly what I needed!

The challenge was of course to make sure it looked nice and smooth!

Creating the terminal typewriter

The basic HTML

The html is really simple, nothing to explain here. I just added a pipe ▌ character as default in the typing area so it does not look empty when the user loads the page and Typewriter has not started typing yet.

  <div class='typing-window'>
    <div class="typing-area">▌</div>
  </div>

  <div class='typing-window'>
    <div class="typing-area">▌</div>
  </div>

Now that we have our html structure we can start styling it and adding the actual typewriter to it!

Styling the terminal

I wanted to give it a terminal look, but also not a DOS look, something a bit more fancy and modern. So I started with a simple rounded rectangle with a nice dark gray background:

.typing-window {
  transition: opacity 0.5s, max-height 0.5s;
  max-height: 9000px;
  font-size: 1.6rem;
  line-height: 1.2;
  font-weight: 400;
  font-family: var(--code-font-family);
  background-color: #1f1f21;
  border-radius: 0.5rem;
  padding: 0;
  color: #ccc;
  margin-bottom: 2rem;
  box-shadow: 0 1px 1px rgba(0,0,0,0.12), 0 2px 2px rgba(0,0,0,0.12), 0 4px 4px rgba(0,0,0,0.12), 0 8px 8px rgba(0,0,0,0.12), 0 16px 16px rgba(0,0,0,0.12);
  
  strong {
    font-weight: 700;
  }

  .typing-area{
    padding: 1.5rem 1.5rem 1.5rem 1rem;
  }
}

(I LOVE this font, it’s amazing and it has some cool ligatures!) A few CSS tricks I learnt over the years:

• transition: It animates any change to the specified property, in this case I am aimating opacity and max-height so whenever they change, their value is animated in 0.5s
• font-family: I am using the CSS variable that Hydejack uses for the code blocks font family, just to be consistent with the rest of the website
• box-shadow: This is a layered, high quality shadow generated with https://brumm.af/shadows

Start typing!

This is the best part of the project!

Now that we have everything set up we can start typing in text into the terminal. Let’s start by retreiving the container where the text will go

Ok so now that we have our container, we need to look for the typing area. It’s also a good idea to only start typing if the container exists, and do nothing otherwise.

var typerStarted = false;
var startTyper = function() {   
  typerStarted = true;
  var typingWindow = document.getElementsByClassName('typing-window')[0];  
  if(!typingWindow){
    return;
  };
  var app = typingWindow.getElementsByClassName('typing-area')[0];
  var typewriter = new Typewriter(app, {
    loop: true,
    delay: 'natural',
    cursor: '▌'
  });
  typewriter
  .pasteString("Hey, I'm alive!)
  .pauseFor(1200)
  .typeString("<br/> Yup, still alive!")
  .pauseFor(2000)
  .deleteChars(17)
  .typeString(" Noooo don't delete mee!")
  .pauseFor(1000)
  .start();
};

We also need to make sure that the typing only starts when the page content is loaded. In the case of Hydejack (the Jekyll theme I am using), we need to account for dynamic page loading so:

document.addEventListener('DOMContentLoaded', function(){ 
  if(!typerStarted) startTyper();
}, false);
document.getElementById('_pushState').addEventListener('hy-push-state-load', function() {
  if(!typerStarted) startTyper();
});

Alright so let’s test it out, shall we?

Adding a status bar

Let’s try and give it more of a terminal look now by adding a Ubuntu-styled status bar. Let’s add a few divs for the bar and its buttons

  <div class='typing-window'>
    <div class='typing-toolbar'>      
        <div class="toolbar-title">
          Window Title
        </div>
        <div class="toolbar-btns-container">
          <div class="typing-toolbar-btn min">—</div>
          <div class="typing-toolbar-btn size">☐</div>
          <div class="typing-toolbar-btn close disabled">✕</div>
        </div>
    </div>
    <div class="typing-area">▌</div>
  </div>

Let’s style it up a little bit now. We want to use an inset shadow, and give a different color to the buttons. The only clickable button should be the closing button so let’s change it’s :hover: CSS and also add a disabled state.

.typing-toolbar {
  display: flex;
  justify-content: flex-end;
  height: 1.6rem;
  padding: 0.1rem 0.1rem 0.2rem 0.2rem;
  background-color: var(--border-color);
  border-radius: 0.5rem 0.5rem 0rem 0rem;
  box-shadow: inset -6px 3px 10px 0px rgba(0,0,0,0.11);

  .typing-toolbar-btn {
    transition: background-color 250ms;
    display: inline-block;
    height: 1.1rem;
    width: 1.1rem;
    margin-top: 0.1rem;
    margin-right: 0.7rem;
    position: relative;
    z-index: 2;
    border-radius: 100%;
    margin-bottom: 0.2rem;
    text-align: center;
    font-size: 0.8rem;
    padding-top: 0.1rem;
    color: #848282;
    font-weight: 700;
  }
  .typing-toolbar-btn.close {
    background-color: #e01e1e;
  }
  .typing-toolbar-btn.close.disabled {
    background-color: #693737;
    color: #8e8e8e;
  }
  
  .typing-toolbar-btn.min {
    background-color: var(--body-bg);
  }
  .typing-toolbar-btn.size {
    background-color: var(--body-bg);
  }
  .typing-toolbar-btn.close:not(.disabled):hover,
  .typing-toolbar-btn.close:not(.disabled):focus,
  .typing-toolbar-btn.close:not(.disabled):active {
    transition: background-color 250ms;
    background-color: #f94b29;
    cursor: pointer;
  }
}

Now for the final touch, let’s make it so that the close button actually closes the terminal, but we’ll enable the button after 10 seconds so there is time for some text to be typed in the terminal!

var startTestTyper = function() {   
  typerStarted = true;
  var typingWindow = document.getElementsByClassName('typing-window test')[0];  
  if(!typingWindow){
    return;
  };

  //Let's retreive the close button, remember that getElementsByClassName returns an array
  //of elements but we only need the first one
  var closeBtn =  typingWindow.getElementsByClassName('typing-toolbar-btn close')[0];
  setTimeout (function() {
    closeBtn.classList.toggle('disabled');
    closeBtn.onclick = function() {
          typewriter.stop();
          typingWindow.style.opacity = 0.01;
      };
  }, 10000);

  
  //Let's listen for the end of the CSS transition (remember how opacity and max-height are animated in css?)
  typingWindow.addEventListener('transitionend', function(event) {
    //We don't want this to run for all the children, just this specific target
    //Events propagate through the DOM tree
    if(event.target !== event.currentTarget)
      return;
      // Javascript returns max-height as a string "0px" so we need to check it it's "0px" before removing the element
    if (event.propertyName === "max-height" && typingWindow.style.maxHeight === "0px"){
      typingWindow.remove();
    }else if(event.propertyName === "opacity" && typingWindow.style.opacity < 0.1){
      //We FIRST change the opacity to make it invisible and only THEN we scale it down to slide the other content up
      typingWindow.style.maxHeight = 0;
    };
 });
  ...
};

Heads up: Reload the page if you don’t see the terminal or if you closed it already!

The code is fairly simple:

1. Retreive the close button
2. Add an event listener to its click
3. Add an event listener to the typingWindow transitionend
4. When It gets clicked, set the opacity of the window to 0
5. CSS will start the animation, when it’s over it will trigger the event transitionend
6. On the event handler we check if the opacity is below a certain threshold, if it is then the element is invisible and we can scale it down by setting its max-height to 0
7. When the transitionend is triggered again we’ll check if max-height is equal to 0px and if its we can remove the whole typingWindow from the DOM

Extras (looping, list of words, emojis)

Alright so here are a few extra tricks I implemented to save some time. I am also writing it down for me to remember. The way Typewriter.js works is by adding events to a queue. Calling pasteString() deleteChars() or pauseFor() will add those events to the queue, and they will be executed in order whenever start() is called. After that, we won’t be able to add any new event to the queue, we’ll only be able to stop() the typewriter.

I wanted my typewriter to keep changing the last few words in loop so I implemented a little trick without having to rewrite the library:

function appendLeadingZeroes(n){
  if(n <= 9){
    return "0" + n;
  };
  return n;
};

function strip(html){
  var doc = new DOMParser().parseFromString(html, 'text/html');
  return doc.body.textContent || "";
};

var randomDelay = function(){
  return Math.random() * (pause[1] - pause[0] + pause[0]);
};

  var totalLoops = 20;
  var wordList = ["<em>PhD Student</em> 📚", "Programmer 💻", "Photographer 📷", "Pizza Lover 🍕", "Gamer 👾", "Swimmer 🏊", "Traveller 🌏"];
  var pause = [1000, 4500];
  var terminalStarter = "> <span style='color: #5cd400;'> gabryxx7@blog:</span><span style='color: #1c39c7;'>~</span>$ ";
  var current_datetime = new Date();
  var formatted_date = current_datetime.getFullYear() + "-" + appendLeadingZeroes(current_datetime.getMonth() + 1) + "-" + appendLeadingZeroes(current_datetime.getDate()) + " " + appendLeadingZeroes(current_datetime.getHours()) + ":" + appendLeadingZeroes(current_datetime.getMinutes()) + ":" + appendLeadingZeroes(current_datetime.getSeconds());

  var typewriter = new Typewriter(app, {
    loop: false,
    delay: 'natural',
    cursor: '▌'
  });
  typewriter
  .pasteString(terminalStarter +"["+formatted_date+"]")
  .pauseFor(1000);
  .changeDelay(90)
  .pasteString("<br /> " +terminalStarter)
  .pauseFor(1200)
  .typeString("Hey!")
  .pauseFor(800)
  .typeString(" I'm <strong>Gabriele</strong>")
  .pauseFor(1000)
  .pasteString("<br /> " + terminalStarter)
  .typeString(" I'm a...");

  for(var k = 0; k < totalLoops; k++){ 
    for(var i=0; i < wordList.length; i++){
      typewriter
      .typeString(wordList[i])
      .pauseFor(randomDelay())
      .deleteChars(strip(wordList[i]).length);
    };
  };
  typewriter.start();

So the main idea is to have a list of words that keep being written and deleted, but I did not want to manually enter the char to delete every time. So by using a list I can delete the length of the previous word and type in the new one:

• Create a list of words
• Type the first part of the text (non looped)
• Add a loop which types in and deletes each word from the list (including pauses)
• Loop the previous loop for how many times you want. If you have 10 words with a delay of 2 seconds between each word, 100 loops will take ~2000 seconds, around half an hour.

TIP: You can use html to format your text, but don’t forget that deleteChars() of Typewriter expects the number of chars without counting html, so a simple string.length in javascript won’t work. You need to strip the string of the html and that’s where strip() comes in

Heads up: I tried with 100 loops and I noticed it would take a few seconds for the Typewriter to start, slowing down the page loading, so I use 20 loops

Adding a terminal typewriter to your page or post

Adding content to the blog page

So first thing I did was to allow for the Hydejack’s blog page layout to include content besides the list of post, so I copied the blog layout from hydejack and create a new layout file _layouts/blog__custom.html based on the base layout:

TIP: You can avoid Jekyll capturing liquid code by wrapping it in \{\% raw \%\} \{\% endraw \%\}

---
layout: base
---

{{ content | markdownify }}
{% assign plugins = site.plugins | default:site.gems %}

{% if plugins contains 'jekyll-paginate' %}
  {% assign posts = paginator.posts %}
{% else %}
  {% assign posts = site.categories[page.slug] | default:site.tags[page.slug] | default:site.posts %}
{% endif %}
  
{% for post in posts %}
  {% include_cached components/post.html post=post no_link_title=page.no_link_title no_excerpt=page.no_excerpt hide_image=page.hide_image %}
{% endfor %}

{% if plugins contains 'jekyll-paginate' %}
  {% include components/pagination.html %}
{% endif %}

So you can see how I added {{ content | markdownify }} to the top to add whatever content I’ll write to any page using blog-custom as a layout.

Now that we can add content to the top of the blog page we can start creating our structure for the typewriter window:

Allowing for custom featured images in posts

I made a few changes to the Hydejack theme so that I could:

• Display wide images as a post feature image
• Display divs and javascript content as a featured post content (as you can see for this post)

For the first point I simply added custom classes for posts, I added a class property to the post image (aside from path). In order to use this new property I edited the _incudes/component/hy-img.html as below:

{% assign img_class = include.img.class | default:"" %}
{% assign include_class = include.class | default:"" %}
{% classes = include_class | concat: img_class %}
<img
  {{ sources }}
  {% if include.alt %}alt="{{ include.alt }}"{% endif %}
  {% if classes != '' %}class="{{ classes }}"
  {% if include.property %}property="{{ include.property }}"{% endif %}
  {% if include.width %}width="{{ include.width }}"{% endif %}
  {% if include.height %}height="{{ include.height }}"{% endif %}
  {% if include.width and include.height %}loading="lazy"{% endif %}
/>

I simply always add classes to the img tag which are empty by default, in this way I am sure this will work with the pre-existing configuration. If I always added include.img.class when it did not exist I could have incurred into issues later, and I definitely wouldn’t want to remove include.class as it might be used in otehr cases. Furthermore, I did not want to add any style tag simply because I noticed the hy-img always end up with style="opacity: 0; in the final html page and I am not sure if that’s needed or not so I did not want to overwrite that.

Let’s not forget to add the CSS for dealing with wide featured images:

.content .aspect-ratio img.wide-img {
  margin: auto;
  width: 100%;
  /* height: 100%; */
  /* object-fit: scale-down; */
  object-position: center;
  background-color: var(--gray-bg);
  vertical-align: middle;
  object-fit: scale-down !important;
}

For the second point of displaying divss into post featured images, it was actually fairly simple. Sasme as before I added a new post image property called html, I then edited the file _includes/component/post.html so that if image.html is definied it will use the html instead of the image path. so I changed this :

<div class="lead aspect-ratio sixteen-nine flip-project-img">
  {% include_cached components/hy-img.html img=post.image alt=post.title width=864 height=486 %}
</div>

To this:

  {% if post.image.html %}
    <div class="lead aspect-ratio sixteen-nine flip-project-html">
      {{ post.image.html }}
    </div>
  {% else %}
    <div class="lead aspect-ratio sixteen-nine flip-project-img">
      {% include_cached components/hy-img.html img=post.image alt=post.title width=864 height=486 %}
    {% endif %}

Last touch, I added the console effect html to the post property which now looks something like this:

---
layout: post
title: How I made the blog's terminal animation
excerpt_separator: <!--more-->
# categories: [coding]
image: 
  path: /assets/gabryxx7/img/typing-window.png
  class: "wide-img"
  html: |-
    <div class='typing-window post' id='typing-window'>
    <div class='typing-toolbar'>    
        <div class="toolbar-title">
          Window Title
        </div>
        <div class="toolbar-btns-container">
          <div class="typing-toolbar-btn min">—</div>
          <div class="typing-toolbar-btn size">☐</div>
          <div class="typing-toolbar-btn close disabled" id="typing-close-btn">✕</div>
        </div>
    </div>
    <div class="typing-area">▌</div>
    </div>
description: >
  I made a gif of me mashing  the keyboard on hackertyper and it got me the idea of using this terminal as a welcoming animation!
---

Beware that this only works in the post page or in the post preview, it will NOT work in the post card when showed as related post (you’d have to edit _includes/component/post-card.html for that). I would not recommend it as it makes everything more complicated and messy with loading randomg javascript all the time. For this reason I would suggest to have an image to display as a fallback option

Adding the javascript

When it comes to the post terminal I can simply add the javascript shown above in between the <script></script> tags. For the page, I added the javascript to the _includes/my-scripts.html. The event listeners will try to load the terminal at every page but the function returns if there is no element with classes typing-window blog.

IMPORTANT: When deploying jekyll with the command JEKYLL_ENV=production bundle exec jekyll build (to enable the search function) make sure that your javascript is PERFECT as in, add a ; to EVERY statement. Otherwise then it gets minified, something like

var test = 1
var test = 3

Becomes:

var test = 1var test = 3

Generating an error and making your whole javascript code crash

An important note, if you end up using IDs instead of classes, multiple terminials on the same page might not work. The reason is that when Javascript calls getElementById() it always retreives one only element with that id. That’s the purpose of IDs, right? Uniquely identifying elements. If we want it to work for ANY html element with the class typing-window we need to use getElementByClass() as shown above.

Adding a description to your page terminal

You will notice I also added a little description to the typing window to link to this post, I simply wrapped the whole window in another div like this:

<div class="typing-window-wrapper">
  <p class="note-sm">
    Do you want to make one yourself for your website? Check this post out:
    <strong><a href="/blog/blog/coding/2020-08-10-typing-window/" class="flip-title">How I made the blog's terminal animation</a></strong>
  </p>
  <div class='typing-window blog'>
    <div class='typing-toolbar'>
      <div class="typing-toolbar-btn min">—</div>
      <div class="typing-toolbar-btn size">☐</div>
      <div class="typing-toolbar-btn close disabled">✕</div>
    </div>
    <div class="typing-area">▌</div>
  </div>
</div>

I then added some extra CSS to _sass/my-style.scss

.typing-window-wrapper .note-sm{
  position: absolute;
  right: 23rem;
  width: 21rem;
  padding: 0.5rem 1rem 0 1rem;
}