./mickey.sh

Monorepos

2020-04-16T00:00:00+00:00

There has been a renewed buzz about monorepos recently. First of all there was that announcement from Github about git sparse-checkout, an experimental feature in Git 2.25.0 that simplifies the process of partially dowloading a large repository, saving space and improving the performance. I suspect this feature was driven by the growing popularity of of the “microservices in a monorepo”.

Second, the new Software Engineering at Google book was recently released and it often references both the benefits and disadvantages to monorepos.

Over the last couple of years I experienced projects where monorepos organically came into being or were explicitly selected for. Similarly to many others out there I came to the same conclusions in terms of their advantages:

Sharing dependencies, means managing less - if several services depend on eachother but are in the same repository you have the same version number so it’s less likely to encounter issues.
No more cross-repo changes - many features I’ve had to implement, crossed service boundries and would require additional steps (cloning, checkingout, switching contexts), but with a monorepo it’s all in one place so you could make the change in a single commit.
Seeing the bigger picture - if services from the same domain are centralised in a monorepo it’s much easier for someone to understand how they are related, rather than doing haphazard Github searches.

Macroservice:

- not a monolith
- Has no more than 20 devs/3 teams working on the service (5 pizza rule?)
- may or may not have/need monorepo. Dependency management becomes a lot easier (though still non-trivial) the fewer the services/repos
- better observability, debugging
— Cindy Sridharan (@copyconstruct) April 6, 2020

References:

https://danluu.com/monorepo/
https://research.google/pubs/pub45424/
https://medium.com/@chrisnager/case-for-a-monorepo-28cebf26e1aa

image by Owen Davey

Chapterising audiobooks

2020-04-13T00:00:00+00:00

So today is Easter Sunday and we are still on lockdown. I could have spent the weekend building an API client, polishing my CV or make progress on the 30-day LeetCode challenge. Instead I decided to go for my government-sanctioned walk. Going for these long walks by myself can get a bit monotonous, so I wanted to listen to an audiobook. However, the issue was that I had an mp3 file that was 12 hours of continuous audio. I had never bothered to understand why my iOS Music app or iTunes couldn’t remember the current position once I paused or stopped it so I decided to apply some “engineering thinking” to this problem. I had come up with two initial approaches which I cross-validated with Reddit.

Estimation approach

The first one involved finding the table of contents for the book and getting a sense for how many pages each chapter is. Then assuming an average narration speed of about 2 minutes I could get the timings for each chapter. I could then skip to each part and use Audacity, QuickTime or iTunes to split the file into parts.

After spending about 5 to 10 minutes on finding the table of contents and failing I decided to go with my second approach.

AI approach

This second approach involved finding a library on Github which would help me analyse audio files for occurrences of specific words. After a few minutes of searching, I discovered SpeechRecognition. It was essentially an interface for many different audio processing APIs. The only offline API was provided by a library developed at Carnegie Mellon University called CMUSphinx. According to their blog it has been superseded by BERT, Wavenet and others. I chose to use it as it was way simpler at first glance.

After some further Googling and going on StackOverflow I had my process down:

I would convert the into a 16khz mono file to enable processing
```
 ffmpeg -i file.mp3 -ar 16000 -ac 1 file.wav
```
I would build and install the latest version of the a lightweight recognizer library written in C called pocketsphinx library.
```
 brew tap watsonbox/cmu-sphinx
 brew install --HEAD watsonbox/cmu-sphinx/cmu-pocketsphinx
```
If you’re on a Mac using Homebrew should work without any hiccups. The alternative is to try to compile the library from source, which can be a bit more painful as there are all of these transative dependencies and you have to make sure the directory where you cloned the library is called pocketsphinx.
Download the latest modified language model in my case for US English.

The run the detection

 pocketsphinx_continuous -infile file.wav -hmm ~/Downloads/en-us -kws_threshold 1e-10 -keyphrase "chapter" -time yes

Here is an overview of the flags:

 -infile                 Audio file to transcribe.
 -hmm                    Directory containing acoustic model files.
 -kws_threshold          Threshold for p(hyp)/p(alternatives) ratio
 -keyphrase              Keyphrase to spot
 -time                   Print word times in file transcription.

After finally congfiguring everything, I ran the model it wasn’t as accurate as I hoped. Realised it would have taken at least 40 minutes to complete it was time to abandon this approach. It was a good learning experience!

Okay so I still hadn’t partitioned this large mp3 into smaller more managable pieces. It was time to use QuickTime and simply trim the audio to my liking!

QuickTrim approach

Simply open the file in QuickTime Player or any audio editor. In QuickTime Edit > Trim opens the editor menu as seen below

All you need to do is spot the pauses and trim up to that point. It’s not perfect but you would end up with smaller pieces of content that you can consume in one session.

Obvious approach

What I could have done from the start is simply Google for my issue. I would have quickly discovered that there is an option to persist the last playback position. While playing the audiobook simply select Edit/Song Info/Options>Remember Playback Position (and tick the box).

Interview prep

2020-04-12T00:00:00+00:00

As software engineers our environment is ever changing. Things are moving fast. “Once in a lifetime” events seem to happen every couple of years now. That’s why I think one needs to stay sharp and interview at regular intervals.

Hiring platforms:

Otta - my personal favourite. Great companies, great UI/UX, well curated content.
Hired - if you’re looking for startups and FANG companies you’ve already heard of them. The model here is companies reach out to you.
StackOverflow Jobs - I haven’t extensively used it but I think any self-respecting engineering org should post their jobs on here as well.
TripleByte - only available in the US. But you can still take the quiz! Somehow I was in the 83 percentile when I did it. I must be good at guessing.

Coding prep:

LeetCode - loads of problem sets and active community. Read about it on Quora.
HackerRank - I think it has more practice questions than LeetCode but it’s not the community favourite.
AlgoExpert - if you’ve been on YouTube for any period of time you must have seen the adverts.
Codeforces - if you’re hardcore and serious about the coding challenges you do.
CodeChef - haven’t personaly used it but it’s questions can get repetative.
Geektastic - It’s not as slick as everything else but some companies use it so you might as well familiarise yourself. If you use this link you will get 50% your first 3 months.

Coferences:

This will sort of depend on your specialisation. Also they can get quite pricey so if you’d like to attend just volunteer! Here are the ones I like:

PyCon - I’ve attended the UK one, but the one in the US is much bigger.
goto; - volunteered for this one a couple of years ago, it was great!
QCon - another massive conference, which I volunteered for.

Books and blogs:

Cracking The Coding Interview - this comes highly recommended!
Introduction to Algorithms - very famous book but also very dense.
The Algorithm Design Manual - popular yet still quite lengthy.
Grokking Algorithms: An illustrated guide for programmers and other curious people - very light weight way of learning about algorithms
HighScalability Blog
Arden Dertat’s Programming interview questions

YouTube:

I love spending my time on Youtube, that’s how I get most of my information these days. But it’s very important that you actually do prep questions rather that watch these videos for hours (speaking from experience).

Joma Tech - very entertaining and informative
TechLead - I used to love his content now I’m a bit more wary of it.
Gaurav Sen - outstanding videos on interview prep and system design. The images in this post are from his video.
Clément Mihailescu (AlgoExpert) - his content is also quite good but I’ve got brand loyalty to Joma and TechLead, they can be a bit more entertaining.
CS Dojo - haven’t watched his content in a while but he’s got some solid explanation videos. Most everyone else has put their explanations behind a paywall.

Github meta resources:

Redis Key Expiration

2020-04-10T00:00:00+00:00

Not too long ago I was tasked with looking into a performance issue with an internal service using Redis. At the time I had no exposure to it after some time I identified that the issue might be related to issues with key expiration. As part of my investigation I found this great talk by @antirez, the creator of Redis, on The Evolution of Redis Key Expiration Algorithms. This is a short post about my learnings, with links to relevant points in the C code.

Intro

Let’s start with the basics, Redis stands for Remote Dictionary Server, as the name suggests it is a distributed, in-memory key-value database. It uses a hash table, called dict as it’s core data structure, where values can be stored as a linked list. One of the many features it provides is EXPIRE, which sets a timeout for a given key. In other words it allows users to specify how long the keys shoul exist for before being delete.

In order to make this happen Redis needs to store the expiration information with a key.

The obvious approach is to have the existing data structure support expiry for each key. However, this adds unnecessary memory overhead, if only a small subset of keys are set to expire.

The alternative solution is to have a second hash-table that stores the expiry information. The image below attempts to illustrate this.

The time at which a key will expire is set at a unix timestamp. To further conserve space, Redis reuses the pointer for the key from the primary dict, in terms of the expiry, the author also tries to optimize by storing a unix timestamp as a pointer. To conserve space it gets stored as a pointer.

This means that if there are no expires, the expiration hastable is empty.

The keys are shared so there is no waisted memory
The expiration information gets stored in place of the pointer for the value object in the hashtable

How to evict keys:

Passive expiration
- If Redis receives the command GET('foo')
- Down the call chain it calls lookUpKey
- This in turn calls expireIfNeeded which ensures that if the time has passed the key expires
- It returns a NULL to the caller if has expired
Active expiration

In the second approach we apply random sampling to the expires table. We sample 10 times per second and test 20 random keys. If the expire time is less than the current timestamp i can evict the key.

However as the number of keys to be expired becomes smaller we are simply burning CPU.

The less keys are expired the more CPU we burn.

After we find less than given percentage of keys that are expired we stop the expire cycle. In the case of Redis this is 25%

When Twitter upgraded from v2.8 (quite an old version) to v6.0 they noticed a regression. In their case where 25% of keys that were expired was not an acceptable default. That’s why an effort configuration was introduced

New approach

Using a Radix tree

Some people have a usecase where they want to use Redis as a timer. What they would do is listen to the Keyspace notification for expired events

Redis docs:

Specifically this is what Redis does 10 times per second:

Test 20 random keys from the set of keys with an associated expire.
Delete all the keys found expired.
If more than 25% of keys were expired, start again from step 1.

However imagine you have a million keys that are “forgotten” by the application code. That means they won’t be accessed therefore never expired if we only relied on this approach.

To check for an expire he checks the “expire hashtable” for a an entry with that key

KeyDB - Rethinking the Redis Key EXPIRE

CubeDrone Endiannes

https://redis.io/commands/expire

https://github.com/antirez/redis/blob/b73d87f5e59ae68a2b901fe5a158d6e22840214c/src/expire.c#L123

Tech Lead Anti-patterns

2020-04-07T00:00:00+00:00

Sometime ago I watched a talk on People First Engineering Management @ Monzo. It’s full of really interesting and insightful content but my biggest takeaway was the following the tech lead anti-patterns.

Let’s use Monzo’s definition of a Tech lead first:

A fairly experienced engineer
Who is still working as an individual contributor and writing code.
And wears a hat 👒 (this is not a role or a progression level)

They are expected to:

Pair with the product manager on the squad’s success
Provide technical context and direction
Enable and empower other engineers
Foster effective collaboration
Won’t necessarily be a tech lead forever

Now that we have this in mind here are the anti-patterns:

The Faux PM 👩🏻‍🏫
- Tries to get involved by prioritising the backlog
- Doing their own research
- Writing their own tickets (team’s tickets)
- Finding features on their own
The Dictator 👮🏻‍♂️
- This is where they define the scope of the project and hand it off to somebody else
- Over specifying and not doing any implementation themselves
- Only ever doing the most interesting work
The Martyr 🧝🏻‍♀️
- Tries to do everything, all the small and boring tasks
- Working late, trying to save the day, so no one else has to
- Not knowing when to delegate
The Over-Empathiser 👩🏽‍⚕️
- Not giving constructive feedback
- Not holding people to account
- Fixing other people’s code themselves

Exhaustive list of books to read

2020-04-02T00:00:00+00:00

Over the last year I’ve been compiling a list of books that my colleagues and friends have either, bought, read or recommended. The checkboxes represent that I either own it, have skimmed it or found it online. Why not pick one of them up while you have the time? (As of writing this we are still in the early days of COVID-19 lockdown)

I have defined a top 5 list:

Everything else

- [ ] Hands-On Machine Learning with Scikit-Learn and TensorFlow - [ ] Python High Performance: Build high-performing concurrent, and distributed applications - [ ] Practical Convolutional Neural Networks: Implement advanced deep learning models using Python - [ ] Machine Learning with Python Cookbook - [ ] Designing for the Real World: Human Ecology and Social Change - [ ] Building Evolutionary Architectures - [ ] In the Bubble: Designing in a complex World (The MIT Press) - [ ] The Oxford Handbook of Innovation - [ ] Clean Architecture: A Craftsman’s Guide to Software Structure and Design - [ ] Designing Distributed Systems: Patterns and Paradigms for Scalable Reliable Services - [ ] Mastering Regular Expressions - [x] The Phoenix Project: A Novel about IT, DevOps and Helping Your Business Win - [ ] Peopleware: Productive Projects and Teams - [ ] No Bullshit Guide to Mathematics - [ ] Introduction to Algorithms - [ ] The Linux Programming Interface: A Linux and Unix System Programming Handbook - [ ] Incremental Software Architecture: A Method for Saving Failing IT Implementations - [x] Go in Practice - [ ] Amazon Web Services in Action, 2E - [ ] Ghost In The Wires: My Adventures as the World’s Most Wanted Hacker - [ ] How Linux Works, 2nd Edition: What Every Superuser Should Know Second Edition - [ ] Learning Amazon Web Services (AWS): A Hands-On Guide to the Fundamentals of AWS Cloud - [ ] UNIX and Linux System Administration Handbook - [ ] Kali Linux Cookbook - Second Edition: Effective penetration testing solution - [ ] Mastering Go: Create Golang production application using network libraries, concurrency, and advanced Go data structures - [ ] Web Penetration Testing with Kali Linux - Third Edition: Explore the methods and tools of ethical hacking with Kali Linux - [ ] Statistics for Data Science: Leverage the power of statistics for Data Analysis, Classification, Regression, Machine Learning, and Neural Networks - [ ] The Creativity Code: How AI is learning to write paint, and think - [ ] New Dark Age: Technology and the End of the Future - [ ] The AWK Programming Language - [ ] Modern Operating Systems - [ ] The Computer Music Tutorial - [ ] Pattern Recognition and Machine Learning - [ ] Cracking the Coding Interview, 6th Edition - [ ] MACHINE LEARNING (Int’l Ed) - [ ] Network Routing: Algorithms, Protocols, and Architectures (The Morgan Kaufmann Series in Networking) - [ ] Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems - [ ] Theory of computing: a gentle introduction - [ ] Algorithms, 4th Edition - [ ] Python High Performance: Build high-performing, concurrent, and distributed applications, 2nd Edition - [ ] The Cucumber Book - [x] No Bullshit Guide to Mathematics - [ ] What is Mathematics - [ ] Refactoring: Improving the Design of Existing Code - [ ] Mastering Postgresql in Application Development - [ ] Practical Monitoring - [ ] The AWK Programming Language - [x] The Algorithm Design Manual - [ ] Introduction to Algorithms - [ ] Site Reliability Engineering: How Google Runs Production Systems - [ ] Programming Clojure, 3e (The Pragmatic Programmers) - [ ] Practical Common Lisp - [ ] The Site Reliability Workbook: Practical Ways to Implement SRE - [X] Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable and Maintainable Systems - [ ] Prometheus - Up & Running - [x] Domain-driven Design: Tackling Complexity in the Heart of Software - [ ] The Go Programming Language (Addison-Wesley Professional Computing Series) - [ ] Humble Book Bundle: Dev Ops by O’Reilly - [ ] Advanced Programming in the UNIX Environment (Addison-Wesley Professional Computing Series) - [x] Godel, Escher, Bach: An Eternal Golden Braid - [ ] Distributed Systems - [ ] Continuous Delivery: Reliable Software Releases through Build, Test and Deployment Automation - [x] Radical Candor: Be a Kick-Ass Boss Without Losing Your Humanity - [x] Site reliability workbook - [ ] TCP/IP Illustrated, Volume 1 (Second Edition)

I'm back

2020-03-27T00:00:00+00:00

No idea how long this will continue for but since we are all working from home during this COVID-19 epidemic. I thought I might as well try to add some content on here.

I started by registering the mickey.sh domain name, I’m a developer after all 👨‍💻. Got it discounted on Namecheap with the use of the Honey Chrome extension.

Decided to stick with the Jekyll static site generator and the Lanyon theme for now. I spent quite a while updating the configuration but it finally works! Although I am currently considering using Hugo and hosting it via Netlify.

Added a few words about myself on my about page.

(image by MarkusM)

Impossible list

2018-05-16T00:00:00+00:00

Inspired by Thomas Frank’s impossible list which in turn was inspired by Joel Runyon the creator of the IMPOSSIBLE list and defined the difference between it and a bucket list

I will add certain things retroactively as a reminder of my achievements. They will serve as a launch point for my new goals and achievements

Last 5 Completed Goals

Fitness/Health Goals

Run a 5K
Cycle 100km in a day

Learn to cruise on a skateboard (September 2015)

Visit the US (March 2016) - New York, trip with Yoan and Ivan

Visit France (April 2017) - Grenoble, France (Florent’s bday) - Visit Paris (May 2017) Visit Sweden (September 2017) - Stockholm, Sweden Visit Asia (June 2017) - Bali, Indonesia

Go to a Drake concert (March 2017) Go to festival (August 2017) - WeWork Summer Camp

Go to a wedding (March 2018) Move to London (August 2018)

Automated my Quora answer

2016-10-05T00:00:00+00:00

Answer can be found here.

Disclaimer I generally think this is not advisable as this just reduces the quality of what has the potential to be a very good learning resource.

Code I used is simple. It uses the BeautifulSoup Python library to extract all the headings from the article and generate markdown links. Time was of the essence as there is a relationship between the length of your answer and the time to answer.

from bs4 import BeautifulSoup
import urllib2

url= 'https://colorlib.com/wp/free-css3-frameworks/'
page = urllib2.urlopen(url)
soup = BeautifulSoup(page.read(), "html.parser")

css_frameworks = soup.findAll("h2")

for framework in css_frameworks:

    try:
        name = framework.a.text
        url = framework.a.get('href')
        print('[{}]({})'.format(name, url))
    except:
        pass