What is hard about trading machine or
any model that does well? Let's look at some key challenges. One framework that I
hope you keep in
mind when developing machine learning systems is that, AI systems of machine
learning systems comprise both code, meaning the algorithm or
the model as well as data. There's been a lot of emphasis in the last
several decades on how to improve the code. In fact a lot of a I research had grown
up
by researchers downloading data sets and trying to find an overall model
that does well on the dataset. But for many applications, you have the flexibility
to change
the data if you don't like the data. And so, there are many projects
where the algorithm or model is basically a solved problem. Some model you download
off
Github will do well enough, and they'll be more efficient to spend
a lot of your time improving the data because the data usually has been
much more customized to your problem. This is a view that will carry throughout
this week and next week's materials. Diving into more detail,
when building a machine learning system, you may have an algorithm or a model,
this would be your code and some data. And it's by training your
algorithm on the data that you then have your machine learning
model that can make predictions. And of course hyperparameters
are an additional input to this process. It is important for many applications
to make sure you have a well to learning rates and regularization
parameter and maybe a few other things. The hyperparameters are important, but
because the space of hyperparameters
is usually relatively limited, I'm going to spend more of our time
focusing on the code and on the data. So model development is
a highly iterative process. You usually start off with some model and
hyperparameters and data training model, and then take the model to carry error
analysis, and use that to help you decide how to improve the model or
the hyperparameters or the data. Because machine learning it's such
an empirical process, being able to go through this loop many times very quickly,
is key to improving performance. But one of the things that will
help you improve performance to is, each time through the loop, being able
to make good choices about how to modify the data or how to modify the model or
how to modify the hyperparameters. After you've done this enough times and
achieve a good model, one last step that's often useful is to
carry out a richer error analysis and have your system go through a final
audit to make sure that it is working before you push it to
a production deployment. So why is model development hard? When building a model,
I think there are three key milestones that most projects should
aspire to accomplish. First is you probably want to make sure
you do well, at least on the training set. So, if you're predicting housing prices
as a function of the size of a house, are you at least able to fit a line
that is your training set quite well? After you've done well
on the training set, you then have to ask if your algorithm
does well on the development set or the holdout cross validation set,
and then also the test set. If your algorithm isn't even
doing well on the training set, then it's very unlikely to do well
on the dev set or the test set. So I think of step one as
something you have to do first as a milestone on your way
towards achieving step two. And then after you do well
on the dev set or test set, you also have to make sure that
you're learning algorithm does well according to the business metrics or
according to the project's goals. Over the last several decades,
a lot of machine learning development, was driven by the goal of doing
well on the dev set or test set. Unfortunately for many problems,
having a high test set accuracy is not sufficient for
achieving the goals of the project. And this has led to a lot of frustration
and disagreements between the machine learning team, which is very good at
doing this and business teams which care more about the business metrics or
some other goals of the project. So you may be wondering: "Hey Andrew,
how is it possibly true that achieving low average test set error
isn't good enough for a project?" There are few common patterns that I've
seen across many projects where you need something beyond low
average test set error, and people spot these issues will help you
be more efficient in addressing them. Let's dive more into this
topic in the next video.
