Ziqi's Homepage

Guide To Optimize Machine Learning System

Sun, 17 Nov 2024 00:00:00 +0000

WIP

In this series of blog, I am going to recap what I learned about optimizing performance in my previous work, especially for machine learning systems. And I also want to introduce some newest techniques on accelerating LLMs.

Introduction
Where You Are: How To Measure and Profile A Program
Where The Peak Is: How To Calculate The Theoretical Performance Upper Bound
1. Roofline Model
2. Get Your Own Benchmark For Hardware
What You Can Do
1. Maximize The Utility Of Hardware
2. Add or Upgrade Hardware
3. Less Work
  1. Quantization
4. Beyond Von Neumann
  1. Quantum computing
  2. Computing with Memory
Trending Applications
1. Fast Attentions
2. Distributed Training and Inference
My Previous Work

Guide To Optimize Machine Learning System: Introduction

Sun, 17 Nov 2024 00:00:00 +0000

WIP

In this article, I am going to introduce some basic concepts and methods on performance engineering.

Basic Workflow

Measure the performance of Program A
Calculate the theoretical peek performance of Program A with current input and hardware
Profile Program A
Think about possible changes and how much they would theoretically improve the performance
Make a change to Program A to produce a hopefully faster Program A*
Measure the performance of Program A*
Compare the output of A* to that of A
If A* get the same output as that of A and is faster than A, set A = A*
If A is still not fast enough, go to Step 3

Measure and Profiling

The “performance” here specifically refers to the latency and throughput of a program. I will not cover other metrics like QPS.

Programming Language Course Notes

Thu, 04 May 2017 00:00:00 +0000

Introduction

These notes are compiled from my studies of the Programming Languages course offered by the University of Washington. The course primarily introduces programming language concepts by exploring three different languages, examining how various language characteristics influence program writing. For developers already familiar with common languages like Java/C/C++/Python/JavaScript, this course offers broader insights into language expressiveness and limitations.

The course is currently available on Coursera: https://www.coursera.org/learn/programming-languages

These notes won’t delve deeply into basic syntax, but will focus on significant features. For additional reference, you can check the course lectures and assignments: https://github.com/novakoki/Programming-Language

Programming Language Course Notes (Final)

Thu, 04 May 2017 00:00:00 +0000

Static vs Dynamic Comparison

A language is termed static or dynamic based on when type checking occurs.

ML vs Racket

There are two perspectives:

ML is somewhat a subset of Racket; Racket supports more ways of writing code because some expressions allowed in Racket would not pass ML’s static checks. For instance, an if statement with branches returning different types.
Variables in Racket are not “typeless” but have “one type”.

datatype theType = (* All variable types are the same, so no need to annotate or check *)
  Int of int | (* Runtime values have different tags *)
  String of string |
  Pair of theType * theType |
  Fun of theType -> theType |
  ...

Static Checking

Static checking occurs after syntax analysis and before program execution, hence called “compile-time checking” (though this is unrelated to whether the language uses a compiler or interpreter). How static checking works is part of the language definition. Some languages do more, some do less. Given checking rules, you can also implement tools to perform desired checks.

Programming Language Course Notes (Part Three)

Wed, 03 May 2017 00:00:00 +0000

Ruby Section

Ruby has the following important characteristics:

Pure Object-Oriented Programming (pure OOP): In Ruby, all values are objects, and every expression results in an object (unlike some object-oriented languages like Java, where primitive types like integers, booleans, and floating-point literals are not objects)
Class-Based Object-Oriented Programming: Every object is an instance of a class. Classes determine the methods an object possesses. (Some languages are not class-based, such as JavaScript, which is prototype-based)
Mixins: A compromise between multiple inheritance (C++) and interfaces (Java). In Ruby, each class has only one parent class, but can have any number of mixins. Unlike interfaces, mixins can define methods (not just declare method signatures)
Dynamic: Ruby allows calling any method with any arguments on any object and can change class and object attributes at runtime
Reflection: The ability to determine an object’s class and its methods during runtime
Closures (block and closure): Blocks are almost closures and can be conveniently used with higher-order functions in libraries. Due to numerous iteration functions, Ruby rarely uses explicit loops. Ruby also supports complete closures (proc)
Scripting Language: While there’s no precise definition, it implies the ability to quickly write short programs, easily manipulate files and strings, with less emphasis on performance
Popular in Web Application Development: Ruby on Rails is an extremely popular server-side framework

Class-Based Object-Oriented Programming

Object-oriented programming follows these rules:

Programming Languages Course Notes (Part Two)

Tue, 02 May 2017 00:00:00 +0000

Racket Section

Racket is a dynamically typed, functional language derived from Scheme.

P.S. In fact, compared to Scheme, it does have classes and objects.

Syntax

Racket’s syntax is particularly distinctive, characterized by two main features: parentheses and prefix notation, with examples to follow.

In Racket, everything can be classified into two categories:

Atomic Types (atoms):
- Literals: #t, 11, "hi", null, etc.
- Variable names: x
- Keywords: define, lambda, if, etc.
A sequence within parentheses
- The first element in each sequence affects the subsequent elements
- If the first element is not a keyword and the sequence is part of an expression, it is called as a function (including +, -, *, / which are all functions)
- The entire sequence represents the corresponding abstract syntax tree with no ambiguity

Delayed Evaluation and Thunks

A key semantic design in language: when are sub-expressions evaluated?

Programming Languages Course Notes (Part One)

Mon, 01 May 2017 00:00:00 +0000

Introduction

The course is currently available on Coursera: https://www.coursera.org/learn/programming-languages

Reflections on Upgrading from Vue1 to Vue2

Tue, 10 Jan 2017 00:00:00 +0000

I spent approximately one week upgrading the membership card module I was responsible for from Vue1 to Vue2 at my company. During this process, I encountered numerous challenges, most of which stemmed from the original code’s significant deviation from Vue2’s best practices (primarily regarding unidirectional versus bidirectional data flow), and partly from fundamental changes in Vue2’s underlying architecture (notably the introduction of Virtual DOM).

Motivations and Considerations for Upgrading

The considerations for upgrading essentially boil down to two aspects: whether the upgrade meets current and future requirements, and weighing the benefits against the costs.

Netease Front-End Summer Internship Interview Experience (Spring 2016)

Fri, 15 Apr 2016 00:00:00 +0000

Written Test

Multiple-choice questions omitted

Essay Questions:

Implement a native JavaScript interface that can upload files via Ajax, display upload progress, and receive JSON data from the server upon completion
Implement a three-column layout
How to prevent CSRF (Cross-Site Request Forgery)
List methods to reduce HTTP requests and resource file size Reference
List methods for cross-domain requests

First Interview

Started with self-introduction, during which the interviewer reviews the resume

Q: Are you more familiar with CSS or JS?

A Note of Computer Network

Thu, 20 Aug 2015 00:00:00 +0000

I. Introduction

1. Computer Network and Distributed System

Example: World Wide Web, Hadoop

2. Uses

Business Application
- Resource Sharing (Client-Server Model)
- Communication Medium
- Electric Commerce
Home Application
- Connectivity (Peer-to-Peer)
- Person-to-Person communication
  - Instant Messaging (Twitter)
  - Social Network (Facebook)
  - Wiki (Wikipedia)
- Electric Commerce (B2C, B2B, G2C, C2C, P2P)
- Entertainment
- Ubiquitous Computing (RFID, IOT)
Mobile Users
- Wireless Network
- Text Messaging (SMS)
- Mobile Commerce
- Sensor Network
- Wearable Computers
Social Issues

Memory Leak

Wed, 04 Feb 2015 00:00:00 +0000

Those who cannot remember the past are condemned to repeat it.

The title seems to suggest I was about to write a technical article, but given my limited knowledge (I haven’t even studied operating systems, yet here I am rambling!), this is just an ordinary essay playing with concepts. Why “Memory Leak”? Because I’m genuinely in a state of memory leakage.

Since the summer after my freshman year, the following code describes my actual state:

Getting Started On Android

Wed, 05 Mar 2014 00:00:00 +0000

This term I ,or should be we,decided to develop an android app.So recently I have been searching for some resources online.For example,the class I mentioned ever on Cousera. However, sometimes I would be fed up with the speed of the teacher and I can’t still watch the videos. Maybe I need a book.Luckily I found a set of documents on the android official website.Then I realize that once you have any problem,your first choice to solve it should be the android official reference.The introduction of all APIs and classes you may use is just there,waiting for you.But sometimes I also feel it would be a little hard to fetch what you need because of the large quantity of information. How lucky!Google has written a tutorial for the beginner like me.You can just follow its steps,introduction,and tips. Well,let me make a summary of my beginning. First,I learn how to create an android application project.And in this process some points as followings:

Websites Recommendation

Sun, 02 Mar 2014 00:00:00 +0000

Last night, I read a passage on Zhihu Daily.It recommended lots of wonderful websites to learn online.I also noted some websites that was fresh for me.So here I decide to try to sort out some excellent websites I have already known.Definitely that means I will update this list when I find more.And I’m sorry that this list may almost match my own hobbies,such as coding.

MOOCs(Massive Open Online Course):

Coursera(Maybe the most popular MOOC website)
edX(Almost from Harvard and MIT)
Udacity(The course archive is always there)
XuetangX(Chinese courses offered by Tsinghua University)

Some others:

Mon, 01 Jan 0001 00:00:00 +0000

Blog Series

Mon, 01 Jan 0001 00:00:00 +0000