Distributed Systems Software Engineer

Apache Hadoop & Flink automation in OpenStack with Karamel

2016-04-10T18:26:30+02:00

This is a special post as it remarks my first contribution to a small open-source project and also to my ex-colleagues during my brief time at SICS.

This project is called Karamel. Karamel is a tool that simplifies big data deployments on a selection of cloud providers like AWS and Google Cloud Engine, in addition to baremetal cluster setups. It is quite easy to get started, you simply need to define your cluster in a file, submit it to the application and do one click!

With this tool you can easily have al big data cluster up and running in a few clicks which will have preinstalled hadoop , Apache Flink or even Apache Spark. In addition, it is the preferred tool for testing out a new hadoop distribution named Hops which try to address the challenges and issues you could face when you work with very large scale hadoop clusters (for example, how to achieve high availability of the namenode)

Deploying Clusters with Karamel

In the rest of this article, I will describe the process of creating a simple cluster running apache Flink in order to make a deployment using Karamel towards an Openstack environment. We will also go over how you can write a cluster file to be deployed using this tool and how you need to configure the Karamel in order to communicate with your cluster.

Before we start

In order to deploy our Hadoop cluster on Openstack, first we will need to get a copy of the latest version of Karamel, you can do it here. Here you will find guidelines to get started with other providers like AWS. You may download one of the available versions that have support for Openstack (version 0.2.0 onwards should do it) or download the source code and build the application with maven.

Working with Karamel

Getting started with Karamel is quite easy, before launching our cluster, we will need to define the configuration of a cluster. In a cluster definition file, we can identify 4 core elements to describe you cluster: Provider, Cookbooks, Attributes and Groups.

In order to understand the role of each core component, we will go through a simple example cluster, let’s imagine that I want to deploy a simple Hadoop cluster with Apache Flink running a namenode and 20 datanodes using an Openstack Infrastructure.

So how can I express this information in Karamel? In addition, if we want to deploy a wordcount job when the cluster is ready, how can we achieve this? How can Karamel allow us to express this business needs? This is how you would do it, as shown on the following file:

name: flinknova
nova:
  flavor: 3
  image: "99f4625e-f425-472d-8e21-7aa9c3db1c3e"

cookbooks:
  hadoop:
    github: "hopshadoop/apache-hadoop-chef"
    branch: "master"
  flink:
    github: "hopshadoop/flink-chef"
    branch: "master"

groups:
  namenodes:
    size: 1
    recipes:
        - hadoop::nn
        - flink::jobmanager
        - flink::wordcount
  datanodes:
    size: 20
    recipes:
        - hadoop::dn
        - flink::taskmanager

In this cluster, we can see 3 different code blocks gathering all the information we need for Karamel to satisfy our cluster needs.

Provider

The first segment of the file contains the provider specific information, for Openstack; we make use of the keyword nova in order to tell Karamel that this is an Openstack cluster and we are going to make use of the Openstack Nova Controller.

After the keyword nova, it follows key parameters that Openstack will make use to deplou the necessary resources to run our software on.

Flavor: This corresponds to a specific hardware configuration for the VM, this is similar to Amazon’s EC2 instance descriptors. Openstack refers to them as flavors and each one specifies a configuration (CPU, RAM) and are not the same for everyone, as these are configured by the needs of the organization. To simplify this, Karamel makes use of the flavor id attached to your VM configuration.
Image: This corresponds to the VM image you want to launch stored in your Openstack system. In this case, we make use of the generated ID when you store your image in your Openstack project.

Cookbooks

The following code block, gathers the software that we will want to install on our nodes. For this purpose, Karamel makes use of Chef Cookbooks that get will be processed on the nodes by running Chef Solo. To simplify the transfer of the Cookbooks, they should be accessible by the application through Git so it can clone them and execute the recipes on the nodes.

For this example, we want to install Apache Hadoop and Apache Flink so we indicate this Karamel by the keyword Cookbook and specifying the repositories where our Cookbooks are stored plus the branch we want to checkout.

Groups

Under the groups sections, we define the cluster structure around groups of components and services. Here, we specify the number of nodes and the recipes that will install the services to run on those nodes.

Summary

In this, blog post we went over roughly how Karamel now has support to deploy clusters in Openstack based infrastructures. This gives you the opportunity to play with hadoop, flink or spark directly in your private cloud reducing the time you need to configure a whole cluster.

Apache Hadoop & Flink automation in OpenStack with Karamel was originally published by Alberto Lorente Leal at Distributed Systems Software Engineer on April 10, 2016.

Developing Applications With Docker

2015-08-13T14:05:10+02:00

Recently, I had the opportunity to get my hands dirty playing with docker containers in my workplace with the purpose of containerazing our applications and increase our velocity when deploying running software into our hardware. I have been focusing on how to simplify handling a distributed applications deployment where different services (Frontend services, Backend Services and Data services) need to interact in isolated and repeatable environments.

So far the experience has been quite promising on how easy it is to create a container, if we compare it to building for example a simple AMI (Amazon Image) in Amazon Web Services where it involves running Amazon EC2 tools in the Virtual Machine we want to create the image from.

In this post, I will go over the process of working with docker in your local machine.

What is Docker?

Docker is a virtualization technology that focuses on the simple premise of allowing developers and devops to package their applications and dependencies in a medium which allows easier portability among different machines & environments while maintaining a high level of flexibility.

We can tend to think, that docker containers are like normal virtual machines, yet in reality they are more light weight compared to normal vms, as it fine tunes the resources available in the virtualized instance by removing guest OS dependencies and the need of using of a hypervisor, which in involves in an extra overhead to deal with.

So containers deployed in the same machine, tend to share external resources provided by the host OS where docker is running like the kernels OS, filesystem and disk usage.

Why is it becoming so popular?

One thing that I believe, docker is doing great is to focus on isolation and keep the behaviour identical of your application along multiple phases environments.

Once you create your docker image and deploy it in a container, it will behave the same in your testing, staging and production environments like virtual machines, yet allowing faster deployments. For example, if we take a full ubuntu VM; it size will be more than 650MBs at least with everything configured and our application embedded in it. Taking the docker case, its footprint will be around the same size as if we wanted to deploy the builded package itself.

In addition, it has its own version control of the docker images, allowing incremental updates of its content (similarly to a version control like git); which help to keep track of the versions of your applications.

Getting started with docker

It is quite simple to get started with docker and begin creating containers, if you use Intellij Idea you can use a quite nice plugin.

You can start creating docker images easily by bundling it up with a dockerfile. This file acts like a script which docker follows and will generate the image for your docker container.

We will see that in a following post, where I will try to go over how docker works with a working example.

Developing Applications With Docker was originally published by Alberto Lorente Leal at Distributed Systems Software Engineer on August 13, 2015.

Welcome to My New Page!

2015-07-22T11:13:39+02:00

I was always quite keen in finding something more personalizable and that could support better my needs as a software engineer and tech interests. I had an initial try with my old site in wordpress where I tried to create something personal, but still I never got satisfied with how wordpress handled things.

Luckily, been an active user of github, I was surprised with their support for personal sites with github pages. In that sense, I was curious to what it could offer and I already saw that you can have some nice things going on with jekyll and ruby.

So this is my second try to create a personal site, hopefully thanks to this nice theme for jekyll; I will be able to satisfy all my needs on customization and giving the site a personal touch I always wanted. Of course, I am not a great frontend guru (I have deeper background in backend/distributed systems) but I really like how this jekyll theme works.

Hopefully, this second try on my personal site, will be more rewarding than the first one I made. I will try to migrate some material from back there.

Github pages HPSTR theme from MadeMistakes (Mademistakes)

Background images from Subtle Patterns (Subtle Patterns) / CC BY-SA 3.0

Welcome to My New Page! was originally published by Alberto Lorente Leal at Distributed Systems Software Engineer on July 22, 2015.

Looking forward to code? Improve your coding skills, not kidding.

2014-07-14T22:29:09+02:00

Computers have been around us for along time, and even now more electronic devices in the forms of tablets, mobile phones, smart watches, etc are starting to get popular bringing together this new notion that is starting to be defined as the internet of things. More and more, we have more devices are starting to be interconnected between each other through the internet on a large broad of networks which start to produce large amounts of information. This were the concept of Big data processing is also gaining a big momentum.

Most of you will ask, what does this have to do with the title of the post? Well, all this technologies have something in common and is that most of it services are based on software elements which where coded by someone in the end. This is why coding is important in technology as it allows us to create all this cool things. Not even that, one of the most demanding media entertaining markets; videogame are based on software and coding skills to bring virtual worlds to life.

This brief introduction is to clearly state that if you are someone approaching to achieve a major in computer science, information technology or you want to become a software engineer, ~~you will need to code at sometime~~ coding will be your greatest and most valuable tool and you need to become good at it. Why is that? Well lets take for example a carpenter, his role is to build great and elaborate things out from wood by using the tools available to it. What makes the difference of a good from a bad carpenter is that no matter the tools you give him, his art quality is directly related to how good, his carpenter skills are. So In order for this carpenter to become an expert, it mostly depends on how much dedication and effort he focuses on improving his carpenter skills.

This is similar to software, in order to become better at coding; you need to practice along the way. Luckily with the ourburst of the internet, there is a wide range of resources where we can get started ourselves quite easily.

Here I give out some tips:

- Learn a language which you like: Luckily, I learned Java at university which was a good choice to start with and also you get grasp on Object Oriented Programming (OOP). There are other nice options like Python (quite simple also). C++ is also interesting but from my point of view it might be to rough to those who are a bit unsure how to program. The important here is to motivate yourself using a tool that you feel like home. Also now with Massive Online Courses like coursera offers, one can easily pick a language and try it. Even better, have a look at this link and pick your language! http://programming-motherfucker.com/become.html

-Follow some tutorials/books: There are many websites with a vast amount of resources, if you take Java; I recommend the oracle Java tutorial which it is quite good. If you go for python, read "learn python the hard way" very good resource.

-Compete in programming competitions: This is one of the best ways to learn programming and in the same way, you learn and develop problem solving skills. There are plenty of puzzles out there that do not need very complex solutions but they do require some previous knowledge in algorithms. Still try to solve the problem and break it up, is hard and there is a reward in the end which is receiving your own satisfaction after been capable of solving extreme challenges you never thought before. There are some sites which allows you to solve problems of this type, I recommend TopCoder, CodeChef and CoderCharts to start with.

-Take an algorithms course: This courses are fun in the end as you learn techniques to solve problems that seem impossible with a couple of lines of code! Also it is key for solving complex problems in programming competitions. Now with coursera, there are many courses in this area and they are of course free to take!

This few tips, will give you a head start on your career path on IT and technology!

Looking forward to code? Improve your coding skills, not kidding. was originally published by Alberto Lorente Leal at Distributed Systems Software Engineer on July 14, 2014.

How my KTH Thesis looks like after the approval

2013-08-18T15:33:12+02:00

A month has passed since my thesis defense and now I finally got my grade after so much time. Now, my next step is to prepare my thesis in order to be accepted back in my home university (yeah, this means tedious and boring paperwork...). Although it seems a boring stage, I am still having some fun around at SICS in kista in the BioBankCloud project where my architecture will be put into practice. This was one of the main motivations of the development of the KTHFS architecture, a distributed file system in order to store the biological information stored in a DNA sample. This is translated in handling possibly petabytes of metadata corresponding to DNA strands.

So finally, for the people interested; you may find a copy of my thesis here. You may find it also in KTH thesis portal, once they upload it. In future posts I will try to explain how the technologies related to my thesis work together. I hope you enjoy it.

EDIT: It has been finally uploaded on diva KTH portal, you can access it from here

How my KTH Thesis looks like after the approval was originally published by Alberto Lorente Leal at Distributed Systems Software Engineer on August 18, 2013.

How to win friends and influence people, improve your professional life with easy rules.

2013-08-07T13:21:13+02:00

A couple of weeks has passed since my last post, but still I am trying to keep a reasonable trend when writing my posts so I do not forget that this blog exists.

Now lets go to the meat of today's update, as I mentioned previously in one of my posts; I just recently finish my Master studies and now it is time to look further on how I want to develop my professional career.

While in my search of further professional and interpersonal development, this book's recommendation made its appearance after some search around the internet.

"How to win friends and influence people" originally written by Dale Carnegie brings you an extended comprehension of how social beings like our selfs tend to react in our routines, and; from this initial point it starts to show a series of rules focusing on different social aspects in which improvement can be done. A small summary of the books points is as follows, of course they are the titles of each chapter but in the end there is a recap of the main idea behind each title:

Fundamental techniques in handling people
1. "If you want to gather honey, don't kick over the beehive."
2. The big secret of dealing people.
3. "He who can do this has the whole world with him. He who cannot, walks a lonely way."
Six ways to make people like you
1. Do this and you'll be welcomed anywhere.
2. A simple way to make a good impression.
3. If you don't do this. You are headed for trouble.
4. An easy way to become a good conversationalist.
5. How to interest people.
6. How to make people like you instantly.
Twelve ways to win people to your way of thinking
1. You can't win an argument.
2. A sure way of making enemies and how to avoid it.
3. If you're wrong admit it.
4. The high road to a Man's reason.
5. The secret of Socrates.
6. The safety valve in handling complaints.
7. How to get co-operation.
8. A formula that will work wonders for you.
9. What everybody wants.
10. A appeal that everybody likes.
11. The movies do it. Radio does it. Why don't you do it?
12. When nothing else works, try this.
Nine ways to change people without giving offence or arousing resentment
1. If you must find fault, this is the way to begin.
2. How to criticize and not be hated for it.
3. Talk about your own mistakes first.
4. No one likes to take orders.
5. Let the other man save his face.
6. How to spur men on to success.
7. Give the dog a good name.
8. Make the fault seem easy to correct.
9. Making people glad to do what you want.
Letters that produced miraculous results
Seven rules for making you home life happier
1. How to dig your marital grave in the quickest possible way
2. Love and let live
3. Do this and you'll be looking up the time-tables to Reno
4. A quick way to make everybody happy.
5. They mean so much to a woman.
6. If you want to be Happy, don't neglect this one.
7. Don't be a "Marriage illiterate"

As in most technical/engineer degrees like in my case, it is quite possible that we focus to much in the technical aspect and we tend to treat less the social skills like team communication and comprehension. In most scenarios as the book tries to emphasize, 15 percent of one's financial success is due to one's knowledge while about 85 percent comes from its skills in human engineering-to personality and ability to lead people.

This is were this book shines and makes its greatest contribution, definitely a good read that definitely helps one to get a broader view and opens the mind to social improvement. As the author of the book reafirms through the chapters of the book, most of the techniques are possible to be meaningless memorised without any sense. In order for these techniques to be effecive, it requires the interested person using them to adjust itself to a new way of life. The guidelines for this new way of life, are also described along the chapters of the book.

If someone is willing to give it a chance of reading it and looking for professional improvement, I definetely recommend it. At least in my case, I did learn a ton of helpful tips.

How to win friends and influence people, improve your professional life with easy rules. was originally published by Alberto Lorente Leal at Distributed Systems Software Engineer on August 07, 2013.

Laser Sensor Based Dance Dance Revolution Mat

2013-07-06T12:54:08+02:00

This was one of the first projects that made me become involved on the activities of the IEEE Student Branch from the ETSIT UPM during my first years. It allowed me to get my first experience handling my own mini project from scratch. It has passed a long time since this project was done, but; I think it is good to remember my time during a member of the Student Branch at ETSIT.

The problem that we had to handle at that time was to design a new Dance Dance Revolution (DDR) mat. This was due to the fact that the previous working version was permanently damaged because of extreme use, so; the branch was looking for people to come up with a solution.

In my case, surfing through the internet we managed to find an interesting post which contained a brief description of their experience building this kind of device. Luckily they had also information on how to build one and a good guide here.

The first thing was to gather all the materials specified on the material list on the guide and get hands to work! Next, we decided to see which materials we could reuse from the previous dance pad.In this case the wood base and the metal scraps were okay. Also the big springs of the previous board could be reused.

Let's move to some dirty hands work making the paths for the lasers, we lacked the correct tools so as good engineers in action, we grabbed the best tool available during that day. It took a lot of effort but we did manage to do the paths for the lasers and the wires. This is how it looked so far.

One laser on it place, 7 to go! It took a long time, after a couple of wirings this is how the big picture looked like.

Next, was to take the USB game controller PCB board and wire the controller inputs with the laser PCB board. This would allow us to use the laser and the photoresistors to act as triggers of a user pressing the buttons in the controller. A little testing using the basic interface that windows XP has when detecting the mappings of our input device, allowed us to debug the connections of the pad.

It is alive!! Now, we had to put everything together and do some further working, this of course was quite straight forward and the system got better picture.

With the last tweaks in the pad structure, we managed to get a full working mat and we managed to play some games of DDR in the Student Branch. This was the first version of our working laser dance pad which showed to have better response than normal pads. Still, from time to time, we had to open to recalibrate the laser to maintain the performance of the system.

Once we had the first version working, we started to think how to improve our design and we figured that it would be nice that we could have a modular pad when we wanted to carry the pad from one place to another. This second design involved having ethernet cables connected between the modules to transfer the energy for the lasers to work. Unfortunately, I was not able to do any photographs of that next version.

Finally, to conclude this post; I would like to thank all the people that helped me during the development of this project during my stay in the Student Branch as without their help, this project would have not been possible.

Laser Sensor Based Dance Dance Revolution Mat was originally published by Alberto Lorente Leal at Distributed Systems Software Engineer on July 06, 2013.

End of my Master Thesis!

2013-06-24T23:21:28+02:00

Last Wednesday it was an important day and the end of my long journey to become a Telecommunications Engineer at the UPM. The journey ended with my Master Thesis presentation in the Swedish Institute Of Computer Science (SICS) in Kista, Stockholm. It was an important date as I showed the results of what I had been working for the last six months.

In addition, this Master Thesis will allow me to obtain also the M.Sc. of Software Engineering in Distributed Systems from KTH, a Master programme I followed during this last 2 years.

In this case, my thesis topic centered on achieving Platform as a Service Orchestration for Hadoop, specifically for the new highly available Hadoop architecture been developed in SICS. My goal was to enhance the Dashboard application that allowed the system administrators to monitor the Hadoop cluster in order to launch and configure it. This might seem easy intially but in the end I had to face numerous problems during this work. Still I managed to design and integrate this functionality and we could see the results during the demo that took place in the presentation.

I must say, that these last six months at SICS it has been a great and marvellous experience involved in this project at SICS. It was an excellent learning experience were I had a lot of freedom and decision when designing my solution. In addition, the environment in the workplace was outstanding and motivated me to do my best with my work.

After finishing this phase in my life; a new adventure opens upon my feet and it is time to decide where I would go next. For now, I will still be working for a couple of months in the project to improve my work and hopefully make it soon for a release of our platform.

Further these days, I will dedicate a couple of posts regarding the technologies I worked through and also notify when it is possible to access publicly to the contents of my Thesis.

End of my Master Thesis! was originally published by Alberto Lorente Leal at Distributed Systems Software Engineer on June 24, 2013.

Think you can code? IEEExtreme 6.0 starts this night!

2012-10-19T15:58:12+02:00

Another year is starting to come to its end in a few months and as usually, another edition of IEEExtreme is starting. This is a special date for all computer science geeks around the IEEE Student Branches as we have the opportunity to improve our coding skills with this competition.

Teams are formed by 3 members were they need to collaborate and solve together the different challenges that will be posted during the 24 hour schedule of this competition.

Now we are a few hours before the start of the 6th edition of the competition (20th October at 00:00 UTC time) and its time to warm up those coding skills.

Although actually I am living in Stockholm finishing my master studies at KTH (there is actually a IEEE Student Branch where you can attend the IEEExtreme). For this occasion, I decided to take a plane back to Madrid to attend this 6th Edition in the Student Branch of my home university so I could compete once again with my fellow partners of my Student Branch. Also it was a great time to visit some friends from Madrid and see my family once again. Last year I was unlucky to not attend the 5th Edition and missed that experience with my friends, this time I made sure it didn't happen again, as probably this will be my last time attending this activity offered by the IEEE.

In previous editions I was unlucky to solve some problems during the competition, so I decided to make the effort to train myself and improve my theoretical background in computer algorithms. I hope that this time, it makes a difference when solving the challenges we will probably face.

For now, time to relax before the competition and good luck to all the teams attending this edition.

Further Information on IEEExtreme:

http://www.ieee.org/membership_services/membership/students/competitions/xtreme/index.html

Think you can code? IEEExtreme 6.0 starts this night! was originally published by Alberto Lorente Leal at Distributed Systems Software Engineer on October 19, 2012.