• Introduction
• 05 Web server
  * 5.1 Regex
• 04 Multitasking
  * 4.1 Thread
  * 4.2 Process
  * 4.3 Coroutine
• 03 UDP and TCP
  * 3.1 UDP
  * 3.2 TCP
• 02 Object Oriented Programming(OOP)
  * 2.1 OOP basic grammer
  * 2.2 Encapsulation (One feature of OOP)
  * 2.3 Inheritance (One feature of OOP)
  * 2.4 Multiple inheritance
  * 2.5 Polymorphism (One feature of OOP)
  * 2.6 Class Attributes
  * 2.7 Class Methods
  * 2.8 Singleton Pattern (One of design Pattern)
  * 2.9 Errors and Exception
  * 2.10 Module
  * 2.11 Read and write
  * 2.12 Manipulate folder and file (OS)
• 01 Procedure Oriented
  * 1.1 Basic
  * 1.2 Branch
  * 1.3 Loop
  * 1.4 Function
  * 1.5 Module
  * 1.6 List and Tuple
  * 1.7 Dictionary
  * 1.8 String
  * 1.9 For
  * 1.10 [Project] Name Card System
  

This is the repository contains the code and difficulties I face when I learn python.

In this folder, it talks about mini-web.

Fontend --(HTTP)--> Server --(WSGI)--> framework --(mysql)--> database

1. Demo (LN_01)

1. Divide static and dynamic data
   1. Server for getting request, and if found the request is for dynamic data, pass it to dynamic server (LN_01)
   2. dynamic server deal with the dynamic request and return to server (LN_02)

1. What is WSGI?

   It is a simple calling convention for web servers to forward requests to web applications or frameworks written in the Python programming language

   
   Process:
   

1. Browser send request - > Server
2. Server check if request is dynamic or static
   1. Static: get the data from harddisk
   2. Dynamic: use WSGL to tell framework, framework return the data to server
3. Sever combine header and body into response and send the response back to Browser

1. What is closure?

   A closure is a nested function which has access to a free variable from an enclosing function that has finished its execution. Three characteristics of a Python closure are: it is a nested function. it has access to a free variable in outer scope. it is returned from the enclosing function.

2. demo

   def line(k, b):
     def create_y(x):
       print(k*x+b)
     return create_y
   
   line_1 = line(1,2)
   line_1(0)
   line_1(1)
   line_1(2)

   

1. What is decorator?

   A decorator in Python is any callable Python object that is used to modify a function or a class. A reference to a function "func" or a class "C" is passed to a decorator and the decorator returns a modified function or class.

2. demo (no arguments and no return)

   def set_func(func):
     def call_func():
       print("-----1-----")
       func()
     return call_func
   
   @set_func  # this is equal to test1 = set_func(test1)
   def test1():   
     print("-----test1-----")
     
   test1 = set_func(test1)
   test1()

3. demo1 (have argument and no return)

   def set_func(func):
     def call_func(num):
       print("-----1-----")
       func(num)
     return call_func
   
   @set_func
   def test1(num):   
     print("-----test1-----")
     
   test1(100)

4. Demo2 (have arguments and no return)

   def set_func(func):
     def call_func(*args, **kwagrs):
       print("-----1-----")
       func(*args, **kwargs)
     return call_func
   
   @set_func
   def test1(num, *args, **kwargs):   
     print("-----test1-----%d" % num)
     print("-----test1-----" args)
     print("-----test1-----" kwargs)
     
   test1(100)
   test1(100, 200)
   test1(100, 200, 300, mm=100)

5. Demo3 (have arguments and have return)

   def set_func(func):
     def call_func(*args, **kwagrs):
       print("-----1-----")
       return func(*args, **kwargs)
     return call_func
   
   @set_func
   def test1(num, *args, **kwargs):   
     print("-----test1-----%d" % num)
     print("-----test1-----" args)
     print("-----test1-----" kwargs)
     return "ok"
     
   ret = test1(100)
   print(ret)
   test1(100, 200)
   test1(100, 200, 300, mm=100)

6. Demo4 (have several decorator)

   def set_add(func):
     print("start1")
     def call_func():
       print("-----1-----")
       func()
     return call_func
   
   def set_func(func):
     print("start2")
     def call_func():
       print("-----2-----")
       func()
     return call_func
   
   @set_add
   @set_func
   def test1():   
     print("-----test1-----")
     
   test1()
   # -----1-----
   # -----2-----
   # -----test1-----

7. Demo5 (have several decorator and have return)

   def set_add(func):
     print("start1")
     def call_func():
       return "<h1>" + func() + "</h1>"
     return call_func
   
   def set_func(func):
     print("start2")
     def call_func():
       return "<td>" + func() + "</td>"
     return call_func
   
   @set_add
   @set_func
   def test1():   
     return "haha"
     
   test1()
   # <h1><td>haha</td></h1>

8. Demo6 (decorator with parameter)

   def set_level(level_num):
   	def set_func(func):
   		def call_func(*args, **kwargs):
   			if level_num == 1:
   				print("----level 1----")
   			elif level_num == 2:
   				print("----level 2----")
   			return func()
   		return call_func
   	return set_func
   
   
   @set_level(1)
   def test1():
   	print("-----test1---")
   	return "ok"
   
   @set_level(2)
   def test2():
   	print("-----test2---")
   	return "ok"
   
   test1()
   test2()

1. What is log?

   A log file is a file that records either events that occur in an operating system or other software runs, or messages between different users of a communication software. Logging is the act of keeping a log. In the simplest case, messages are written to a single log file.

2. Log levels(python for critical and java for fatal)

   

3. Demo1(print in terminal)

   import logging  
   
   logging.basicConfig(level=logging.WARNING)  #WARNNING IS DEFAULT, CAN CHANGE
   logging.basicConfig(level=logging.WARNING,format='%(asctime)s - %(filename)s[line:%(lineno)d] - %(levelname)s: %(message)s') 
   
   logging.debug('This is loggging debug message') 
   logging.info('This is loggging info message')
   logging.warning('This is loggging a warning message')  
   logging.error('This is an loggging error message')  
   logging.critical('This is loggging critical message')

4. Demo2(output to a file)

   import logging  
   
   logging.basicConfig(level=logging.WARNING,  
                       filename='./log.txt',  
                       filemode='w',  
                       format='%(asctime)s - %(filename)s[line:%(lineno)d] - %(levelname)s: %(message)s')  
   # use logging  
   logging.info('This is loggging info message')  
   logging.debug('This is loggging debug message')  
   logging.warning('This is loggging a warning message')  
   logging.error('This is an loggging error message')  
   logging.critical('This is loggging critical message')

5. Demo3(print in terminal and output to a file)

   import logging  
   
   # 1. create a logger
   logger = logging.getLogger()  
   logger.setLevel(logging.INFO)    # for all
   
   # 2. create a logger for file
   logfile = './log.txt'  
   fh = logging.FileHandler(logfile, mode='a')  
   fh.setLevel(logging.DEBUG)  # for file
   
   # 3. create a logger for terminal
   ch = logging.StreamHandler()  
   ch.setLevel(logging.WARNING)  # for terminal
   
   # 4. set the format  
   formatter = logging.Formatter("%(asctime)s - %(filename)s[line:%(lineno)d] - %(levelname)s: %(message)s")  
   fh.setFormatter(formatter)  
   ch.setFormatter(formatter)  
   
   # 5. add for file and for terminal in the overall logger
   logger.addHandler(fh)  
   logger.addHandler(ch)  
   
   # log  
   logger.debug('This is logger debug message')  
   logger.info('This is logger info message')  
   logger.warning('This is logger warning message')  
   logger.error('This is logger error message')  
   logger.critical('This is logger critical message')

1. What is metaclass?

   A metaclass in Python is a class of a class that defines how a class behaves. A class is itself an instance of a metaclass. A class in Python defines how the instance of the class will behave.

2. Globals() in ipython

   {'__name__': '__main__',
    '__doc__': 'Automatically created module for IPython interactive environment',
    '__package__': None,
    '__loader__': None,
    '__spec__': None,
    '__builtin__': <module 'builtins' (built-in)>,
    '__builtins__': <module 'builtins' (built-in)>,
    '_ih': ['', 'globals()'],
    '_oh': {},
    '_dh': ['/Users/lin/Downloads/github/python_learning'],
    'In': ['', 'globals()'],
    'Out': {},
    'get_ipython': <bound method InteractiveShell.get_ipython of <IPython.terminal.interactiveshell.TerminalInteractiveShell object at 0x111b33f10>>,
    'exit': <IPython.core.autocall.ExitAutocall at 0x111b335b0>,
    'quit': <IPython.core.autocall.ExitAutocall at 0x111b335b0>,
    '_': '',
    '__': '',
    '___': '',
    '_i': '',
    '_ii': '',
    '_iii': '',
    '_i1': 'globals()'}

3. Type() can make a class which is a metaclass God in python

   class animal(object, metaclass):  # equal to animal = type('animal', (object,), {"age":2, "height":1})
      age = 2
   	 height = 1
       
   MyDog = type('MyDog', (), {"age":2, "height":1})
   MyCat = type('MyDog', (animal,), {})    #inherit
   
   def eat(self):
     print("eat")
   MyPig = type('MyPig', (), {"eat": eat})  # instance method
   
   @classmethod
   def sleep(cls):
     print("class method")
   MyPig = type('MyPig', (), {"eat": eat, "sleep": sleep})   # class method
   
   @staticmethod
   def drink(cls):
     pass
   MyPig = type('MyPig', (), {"eat": eat, "sleep": sleep, "drink": drink})   # static method

4. specify the metaclass(upper_attr is the metaclass)

   def upper_attr(class_name, class_parents, class_attr):
   
       new_attr = {}
       for name,value in class_attr.items():
           if not name.startswith("__"):
               new_attr[name.upper()] = value
   
       return type(class_name, class_parents, new_attr)
   
   class Foo(object, metaclass=upper_attr):
       bar = 'bip'
   
   print(hasattr(Foo, 'bar'))
   print(hasattr(Foo, 'BAR'))
   
   f = Foo()
   print(f.BAR)

1. What is ORM?

   Object-relational mapping in computer science is a programming technique for converting data between incompatible type systems using object-oriented programming languages. This creates, in effect, a virtual object database that can be used from within the programming language.

2. Demo

class ModelMetaclass(type):
    def __new__(cls, name, bases, attrs):
        mappings = dict()
        for k, v in attrs.items():
            if isinstance(v, tuple):
                print('Found mapping: %s ==> %s' % (k, v))
                mappings[k] = v

        for k in mappings.keys():
            attrs.pop(k)

        attrs['__mappings__'] = mappings
        attrs['__table__'] = name
        return type.__new__(cls, name, bases, attrs)


class Model(object, metaclass=ModelMetaclass):
    def __init__(self, **kwargs):
        for name, value in kwargs.items():
            setattr(self, name, value)

    def save(self):
        fields = []
        args = []
        for k, v in self.__mappings__.items():
            fields.append(v[0])
            args.append(getattr(self, k, None))

        args_temp = list()
        for temp in args:
            if isinstance(temp, int):
                args_temp.append(str(temp))
            elif isinstance(temp, str):
                args_temp.append("""'%s'""" % temp)
        sql = 'insert into %s (%s) values (%s)' % (self.__table__, ','.join(fields), ','.join(args_temp))
        print('SQL: %s' % sql)


class User(Model):
    uid = ('uid', "int unsigned")
    name = ('username', "varchar(30)")
    email = ('email', "varchar(30)")
    password = ('password', "varchar(30)")


u = User(uid=12345, name='Michael', email='test@orm.org', password='my-pwd')
u.save()

In this folder, it talks about mysql.

1. start mysql

   mysql -uroot -p

2. exit mysql

   exit/quit/ctrl+d

3. show version of mysql

   select version();

4. show all databases

   show databases; 

5. create database

   create database python2;
   create database python2 charset=utf8;

6. drop database

   drop database python2;

7. show created database

   show create database python2;

8. show database in use

   select database();

9. use database

   use python2;

1. show all tables of current database

   show tables;

2. [demo] create students table(id、name、age、high、gender、cls_id)

   constraint :auto_increment, not null, primary key, default

   create table students(
           id int unsigned not null auto_increment primary key,
           name varchar(30),
           age tinyint unsigned default 0,
           height decimal(5,2),
           gender enum("male", "female", "secret") default "secret",
           cls_id int unsigned
   );

3. show table description

   desc students;

4. [demo] drop table

   drop table students;

5. [demo] show create table comment

   show create table students;

6. [demo] add colume in students

   alter table students add birthday datetime;

7. [demo] modify colume type in students

   alter table students modify birthday date;

8. [demo] change colume name in students

   alter table students change birthday birth date default "2000-01-01";

9. [demo] drop colume in students

   alter table students drop height;

1. [demo] insert into students (all)

   insert into students values(0, "will", 20, "male", 1, "1990-01-01");
   insert into students values(null, "will", 20, "male", 1, "1990-01-01");
   insert into students values(default, "will", 20, "male", 1, "1990-01-01");
   --enum
   insert into students values(default, "will", 20, 1, 1, "1990-01-01");

2. [demo] insert into students (partial)

   insert into students (name, gender) values ("will", 1);

3. [demo] insert into students (multi-line)

   insert into students (name, gender) values ("will", 1),("jasmine", 2);
   insert into students values(default, "will", 20, 1, 1, "1990-01-01"), (default, "jasmine", 20, 2, 1, "1990-01-01");

4. [demo] change in students id = 3

   update students set gender=1 where id=3;
   update students set age=22, gender=1 where id=3;

5. [demo] search in students

   select * from students;
   select * from students where name="will";
   select * from students where id>3;
   select name,gender from students;
   select name as n,gender as g from students;
   select id, gender, name from students;

6. [demo] delete in students

   delete from students;  --delete all
   delete from students where name="will";
   
   alter table students add is_delete bit default 0; -- do not real change, 0 for not delete, 1 for delete 
   update students set is_delete=1 where id=3;

1. use distinct can find the unique of a data

   select distinct gender from students;

2. “where” is condition

Comparison operator: =, >, >=, <, <=, !=, <>

Logical operator: and or not

Fuzzy match:

* like: normal like
* rlike: regex like
* %: unlimited char(0- unilimited)
* _: one char(1)

Range match: in, between...and..., not between...and...

Null match: is null, is not null null is not equal to ""

select * from students where id=1;
-- Comparison operator
select * from students where id > 3;
-- Logical operator
select * from students where id > 3 and gender=0;
-- Fuzzy match
select * from students where name like 'w%';
select * from students where name like 'w_';
select * from students where name rlike '^w.*';
-- Range match
select * from students where id in(1,3,8);
select * from students where id between 3 and 8;
-- Null match
select * from students where height is null;
select * from students where height is not null;

3. sort

   select * from students where is_delete=0 order by name asc;
   select * from students where gender=1 and is_delete=0 order by name asc, id desc;

4. count, max, min, sum, avg, round

   select count(*) from students;
   select max(id) from students where gender=2;
   select min(id) from students where is_delete=0;
   select sum(age) from students where gender=1;
   select avg(id) from students where is_delete=0 and gender=2;
   select round(sun(age)/count(*),2) from students

5. group

   select gender from students group by gender;
   
   +--------+
   | gender |
   +--------+
   | male   |
   | female |
   | secret |
   +--------+
   
   select gender,group_concat(name) from students group by gender;
   
   +--------+-----------------------------------------------------------+
   | gender | group_concat(name)                                        |
   +--------+-----------------------------------------------------------+
   | male   | will, william                                						 |
   | female | jasmine														                       |
   | secret | anonimous                                                 |
   +--------+-----------------------------------------------------------+
   
   select gender,count(*) from students group by gender having count(*)>1;
   +--------+----------+
   | gender | count(*) |
   +--------+----------+
   | male   |        2 |
   +--------+----------+

6. limit

   select * from students limit 5;
   select * from students limit 6,2;
   select * from students where is_delete=0 limit (n-1)*m,m;

7. connect two tables(inner join, left join, right join)

   -- inner join
   select * from students inner join classes on students.cls_id = classed.id;
   select student.*, classes.name from students as s inner join classes as c on s.cls_id = c.id;
   -- left join
   select * from students as s left join classes as c on s.cls_id = c.id;
   select * from students as s left join classes as c on s.cls_id = c.id having c.id is null; 
   -- right join
   select * from classes as c right join students as s on s.cls_id = c.id;

8. Normal Form

   1. 1NF
   2. 2NF: base on 1NF, must have a primary key, others need to depend on primary key, can not depend partial
   3. 3NF: base on 2NF, can not be indirect depend
   4. ER model:
      1. one to many: add a key in the many table
      2. many to many: add a new table

1. install PyMySql

   python3 -m pip install PyMySql

2. Connect to mysql (LN_01)

   1. start
   2. create a connection
   3. get cursor
   4. do the sql
   5. close cursor
   6. close connection
   7. Finish

3. Python search in mysql (LN_02)

4. Python add, delete, change in mysql (LN_03)

5. Sql injection: do not combine string, let execute function to combine that.

1. backup database on master server

   mysqldump -uroot -p --all-databases --lock-all-tables > ~/master_db.sql

2. Copy master_db.sql to slave server and do the restore command

   mysql –uroot –p < master_db.sql

3. the log_bin on master server must be on and server id should be 1.

   sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf     #to check the config

   

4. server id of slave server must be different with master one

5. restart slave and master

   sudo service mysql restart

6. On master, create a mysql account for slave

   GRANT REPLICATION SLAVE ON *.* TO 'slave'@'%' identified by 'slave';
   FLUSH PRIVILEGES;

7. Use the command and remember the file and position, will be use in step 8

   show master status;
   +---------------+----------+--------------+------------------+-------------------+
   | File          | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
   +---------------+----------+--------------+------------------+-------------------+
   | xxxx          |    0000  |              |                  |                   |
   +---------------+----------+--------------+------------------+-------------------+

8. Test on slave check if it is done

   change master to master_host='[masterip]', master_user='slave', master_password='slave',master_log_file='xxxx', master_log_pos=0000;

   show slave status;
   Slave_IO_Running: Yes          # this one shows done
   Slave_SQL_Running: Yes         # this one shows done

In this folder, it talks about advanced python.

1. Single thread endless loop on the laptop (LN_01)
2. Two threads endless loop on the laptop (LN_02)
3. Two processed endless loop on the laptop (LN_03)
4. GIL only allow one thread at a time. Global Interpreter Lock, is a boolean value in the Python interpreter, protected by a mutex.
5. I/O bound: Threads/ Gevent Compute bound: Multiprocesses
6. Use C to avoid GIL (LN_04)

1. Immutable variable: tuple, int, string, float, bool

2. Mutable variable: dict, set, list

3. If copy something from a all immutable variable( can not be like list inside tuple, need all to be immutable variable), shadow copy and deepcopy will be reference.

4. If copy something outside most is immutable variable, shadow copy is only for reference, but deepcopy copy outer most and all mutable variable inside.

5. If copy something outside most is mutable variable, shadow copy copy outer most, inside still be reference, but deepcopy copy outer most and all mutable variable inside.

6. Shadow copy (outer most mutable, outer most will be copy, inside is still reference)

   

7. Deep copy (outer most mutable, all will be real copy)

   

8. if the original one change after the shodow copy, the shadow copy one will not change if the original one append a new item, only reference the copy items

   

9. Slice list

   

10. Dict copy

    

1. import will import the whole module, and can change variable inside that module

   

2. from import will only use a variable to reference that value of the import thing, use equal to change only change the local one rather than the one in the module import

   

1. Super() based on _mro_ method. Parent do not use __mro__ method. Recommend super()
2. Super() also can set parameter, super(specified class in MRO, self), in this way, super will inherit based on the specified class.
3. Use parent to inherit, if two parent inherit from a same grandparent, the grandparent will be run for two times. (LN_01)
4. Use super() to inherit, it will inherit based on the __mro__method, which will avoid two times running grandparent. (LN_02)

1. @property in python is easy to use because in this way, we do not need to consider about the parameters.

   class Foo:
       @property
       def prop(self):
           pass
         
   foo_obj = Foo()
   foo_obj.prop

2. First way to create @property (Decorator)

   Three methods in @property, class need inherit object (get, set, delete)

   class Goods(object):
   
       def __init__(self):
           self.original_price = 100
           self.discount = 0.8
   		#get
       @property
       def price(self):
           new_price = self.original_price * self.discount
           return new_price
   		#set
       @price.setter
       def price(self, value):
           self.original_price = value
   		#del
       @price.deleter
       def price(self):
           del self.original_price
   
   obj = Goods()
   obj.price      	# get
   obj.price = 200	# set
   del obj.price		# del

3. Second way to create @property (class attribute)

   class Foo(object):
       def get_bar(self):
           print("getter...")
           return 'laowang'
   
       def set_bar(self, value): 
           """two parameters"""
           print("setter...")
           return 'set value' + value
   
       def del_bar(self):
           print("deleter...")
           return 'laowang'
   
       BAR = property(get_bar, set_bar, del_bar, "description...")
   
   obj = Foo()
   
   obj.BAR                # first parameter in property
   obj.BAR = "alex"       # second parameter in property
   desc = Foo.BAR.__doc__ # fourth parameter in property
   print(desc)
   del obj.BAR						 # third parameter in property

4. Third way to create @property (Django) (LN_01)

5. demo1 pages (LN_02)

6. demo2 get set in class attributes (LN_03)

7. Demo3 get set in decorator (LN_04)

1. __doc__ method shows the description of the class

   class Foo:
       """ description of Foo """
       def func(self):
           pass
   
   print(Foo.__doc__)
   # [output]description of Foo

2. __class__ method shows which class create current oject

   class T(object):
     """description of T"""
     
   t = T()
   t.__class__
   # [output]__main__.T

3. __module__ method shows which module create current oject

   class T(object):
     """description of T"""
     
   t = T()
   t.__module__
   # [output]__main__

4. _init_ method and __new__ method together create an object

   class Person: 
       def __init__(self, name):
           self.name = name
           self.age = 18
   
   obj = Person('William')  

5. _del_ method delete object

   class Foo:
       def __del__(self):
           pass

6. _call_ method

   class Foo:
       def __init__(self):
           pass
   
       def __call__(self, *args, **kwargs):
           print('__call__')
   
   
   obj = Foo()  
   obj() # call method

7. _dict_ method can check all attributes in a class or object

   class City(object):
       country = 'Australia'
   
       def __init__(self, name, count):
           self.name = name
           self.count = count
   
       def func(self, *args, **kwargs):
           print('func')
   
   print(City.__dict__)
   # [output]{'__dict__': <attribute '__dict__' of 'City' objects>, '__module__': '__main__', 'country': 'Australia', '__doc__': None, '__weakref__': <attribute '__weakref__' of 'City' objects>, 'func': <function City.func at 0x101897950>, '__init__': <function City.__init__ at 0x1018978c8>}
   
   obj1 = City('Sydney', 10000)
   print(obj1.__dict__)
   # [output]{'count': 10000, 'name': 'Sydney'}
   
   obj2 = City('Canberra', 20000)
   print(obj2.__dict__)
   # [output]{'count': 20000, 'name': 'Canberra'}

8. _str_ method

   class Foo:
       def __str__(self):
           return 'William'
   
   
   obj = Foo()
   print("obj is %s" % obj)
   # [output]obj is William

9. __getitem__ __setitem__ __delitem__ method

   class Foo(object):
   
       def __getitem__(self, key):
           print('__getitem__', key)
   
       def __setitem__(self, key, value):
           print('__setitem__', key, value)
   
       def __delitem__(self, key):
           print('__delitem__', key)
   
   
   obj = Foo()
   
   result = obj['k1']      # run __getitem__
   obj['k2'] = 'william'   # run __setitem__
   del obj['k1']           # run __delitem__

10. __getslice__ __setslice__ __delslice__ method

    class Foo(object):
    
        def __getslice__(self, i, j):
            print('__getslice__', i, j)
    
        def __setslice__(self, i, j, sequence):
            print('__setslice__', i, j)
    
        def __delslice__(self, i, j):
            print('__delslice__', i, j)
    
    obj = Foo()
    
    obj[-1:1]                   # run __getslice__
    obj[0:1] = [11,22,33,44]    # run __setslice__
    del obj[0:2]                # run __delslice__

1. with open

   def m3():
       with open("output.txt", "r") as f:
           f.write("Python")

2. Every object contains __enter__ and __exit__ is a context manager

   class File():
   
       def __init__(self, filename, mode):
           self.filename = filename
           self.mode = mode
   
       def __enter__(self):
           print("entering")
           self.f = open(self.filename, self.mode)
           return self.f
   
       def __exit__(self, *args):
           print("will exit")
           self.f.close()
           
           
   with File('out.txt', 'w') as f:
       print("writing")
       f.write('hello, python')

3. Use contextmanager Recommend

   before yield is enter method, after yield is exit method

   from contextlib import contextmanager
   
   @contextmanager
   def my_open(path, mode):
       f = open(path, mode)
       yield f
       f.close()
       
       
   with my_open('out.txt', 'w') as f:
       f.write("hello , the simplest context manager")

In this folder, it talks about web server.

1. All language have
   1. Match (LN_01, LN_02)
2. In python
   1. Search -> the first one found
   2. Findall -> a list of all found
   3. Sub -> change the found with the parameter, also can call function
   4. Split -> split a string and return a list of substring

1. The response header(baidu as an example)

   HTTP/1.1 200 OK
   Bdpagetype: 2
   Bdqid: 
   Cache-Control: private
   Connection: keep-alive
   Content-Encoding: gzip
   Content-Type: text/html;charset=utf-8
   Date: 
   Expires: 
   Server: BWS/1.1
   Set-Cookie: 
   Traceid: 
   X-Ua-Compatible: IE=Edge,chrome=1
   Transfer-Encoding: chunked

2. The request header(baidu as an example)

   GET / HTTP/1.1
   Host: www.baidu.com
   Connection: keep-alive
   Cache-Control: max-age=0
   Upgrade-Insecure-Requests: 1
   User-Agent: Chrome Safari
   Accept: 
   Accept-Encoding: gzip, deflate
   Accept-Language: en,zh-CN;q=0.9,zh;q=0.8
   Cookie: 
   cp-extension-installed: Yes

3. Three handshakes

   

4. Four waves

   

5. Write a static webserver (LN_01)

6. Write an active webserver (LN_02)

7. Use multiprocessing to write an active webserver (LN_03)

8. Use threading to write an active webserver (LN_04)

9. Use gevent to write an active webserver (LN_05)

10. One process, one thread, non blocking (LN_06)

11. One process, one thread, non blocking, long connection (LN_07)

12. Epoll (work in linux Recommend) (LN_08)

    1. Kernel share memory with server, kernal do not need to copy.

    2. select -> poll -> epoll

       """
       create an epoll object
       """
       epl = select.epoll()
       # register listen socket fd into epl
       epl.register(listen_socket.fileno(), select.EPOLLIN)
       
       # make a dict to save the fds
       fd_event_list = dict()
       
       while True:
         fd_event_list = epl.poll()
         for fd, event in fd_event_list:
           if fd == listen_socket.fileno():
             serve_socket, client_address = listen_socket.accept()
             epl.register(serve_socket.fileno(), select.EPOLLIN)
             fd_event_listp[serve_socket.fileno()] = serve_socket
          	elif event == select.EPOLLIN:
             recv_data = fd_event_list[fd].recv(1024).decode("utf-8")
             if recv_data:
               serve_client(fd_event_list[fd], recv_data)
             else:
               fd_event_list[fd].close()
         

1. tcp/ip transport layer

   

2. tcp/ip encapsulation and decapsulation

   

3. two computers connect

   

4. Computers connect by hub (out-of-date)

   

5. Computers connect by switch

   

6. Computers connect by router

   

7. Computer connect in internet

   

8. In web

   1. MAC address: A media access control address is a unique identifier assigned to a network interface controller for use as a network address in communications within a network segment.
   2. IP address: a numerical label assigned to each device connected to a computer network that uses the Internet Protocol for communication.
   3. IP subnet mask: A subnetwork or subnet is a logical subdivision of an IP network.
   4. Default gateway: A default gateway is the node in a computer network using the internet protocol suite that serves as the forwarding host to other networks when no other route specification matches the destination IP address of a packet.
   5. Hub(out-of-date): A hub is a physical layer networking device which is used to connect multiple devices in a network.
   6. Switch: A switch is a device in a computer network that connects other devices together.
   7. Router: A router is a networking device that forwards data packets between computer networks.
   8. DNS server(Domain Name System): The Internet's system for converting alphabetic names into numeric IP addresses.

In this folder, it talk about multiytasking.

1. Parallel programming and Concurrent programming
   • Parallel programming: executing simultaneously
   • Concurrent programming: in progress at the same time

1. Threads are not in order, everyone can be the first (LN_01)

2. Use thread to do a multitasking (LN_02)

3. show the current threads (LN_03)

   threading.enumerate()

4. Threads share global variables (LN_04)

5. Threads share global variables in arguments (LN_05)

6. Threads share global variables sometimes will cause error (LN_06)

   Because sometimes one thread still do not finish their works, but the cpu just swich to another thread.

7. How to fix the error in multitasking(thread) by using mutex (LN_07)

8. Know about mutex deadlock

9. [Project] UDP multitasking chatting program by thread (LN_08)

1. The different of program and process is that process is active and can use the hardware on the labtop, but the program is static

2. Use process to do a multitasking (LN_01)

3. The difference of threads and processes

   • thread depends on process

   • process allocates memory, threads are working in this memory

     eg. Process is a production line in a factory and threads are the workers working in that production line. Once the tasks is too much for one production line. the second production line(process) will be create and more workers(threads) will be hired.

4. Get pid of thread (LN_02)

5. Processes are not in order (LN_03)

6. Pass args to process (LN_04)

7. Processes are independent with each other, do not share global variable (LN_05)

8. Use queue to share between processes (LN_06)

   **Mac python3.0 by default uses "spawn" instead of "fork" when start, The following need to add at the start of main **

   multiprocessing.set_start_method("fork")

   from https://stackoverflow.com/questions/60518386/error-with-module-multiprocessing-under-python3-8

9. Processing Pool (LN_07)

10. [Project] Copy folder multitasking program by process (LN_08)

11. Process states

    

1. Iterator (LN_01, LN_02, LN_03)

   1. Make an object iterable

      1. had __iter__(self) method inside the class which create the object
      2. The __iter__(self) method inside 1 must return an Iterator
      3. There must have __iter__(self) and __next__(self) method in the Iterator

   2. Justify if an object can Iterate

      from collections import Iterable
      isinstance(OBJECT, Iterable)

   3. Justify it is an Iterator

      from collections import Iterator
      isinstance(OBJECT, Iterator)

   4. xrange() is using iterator and range() is using memory (in Python2)

   5. range() is using iterator (in Python3)

   6. Iterator only need the memory for code, but the normal way need the memory to save all the numbers

2. Generator (LN_04, LN_05, LN_06)

   1. Generator is a kind of Iterator (but do not have iter and next method)
   2. Generator can stop in the yield and restart, the variable inside will not change
   3. If yield is in a function, then this is not function anymore and become a generator. Also, can not using this function anymore but create a generator
   4. next(Generator) and Generator.send("string") can start generator
      • NEXT can not pass args but SEND can pass args
      • do not use SEND for the first one when there is no things outside of the loop to get the args

3. Yield Multitasking, Greenlet Multitasking, Gevent Multitasking (use this) (LN_07, LN_08, LN_09, LN_10)

   1. Greenlet is an encapsulation of yield, is more concise
   2. Gevent is an encapsulation of greenlet
   3. Use monkey and joinall in gevent Recommend this

4. Coroutine is one process, one thread, but just use the sleep time to do another thing, which is the most efficient

In this folder, it talk about UDP and TCP.

• Server:
  1. Create a udp server socket
  2. bind local information(for client to find it)
  3. receive the data from client (recvfrom()), use a while loop STUCK here if do not get the data from client
  4. send the data to the client if client request (sendto())
  5. close the udp server socket
• Client
  1. create a udp client socket
  2. send the request to the server by the server ip and port(sendto())
  3. receive data from server if there is (recvfrom())
  4. close udp client socket

1. udp and tcp can bind a same port (eg. 8080) at the same time
2. udp do not need to get a connect whereas tcp need

• Server:
  1. Create a tcp server socket(listen socket)
  2. bind local information(for client to find it)
  3. listening for the client
  4. wait for the data and create the client socket to serve client (accept()) STUCK here if do not get the data from client
  5. receive the request from the client and do some response
  6. send the data to the client if client request
  7. close the client socket
  8. close the tcp server socket(listen socket)
• Client
  1. create a tcp client socket
  2. specify the server ip and port
  3. connect server
  4. send the request to the server
  5. receive data from server if there is
  6. close tcp client socket

1. In the example above, the server can serve many clients and also serve multiple times in the same time, need to specify the parameter in the listening method.

   tcp_server_socket.listen(128) in this way the server can serve 128 clients.

2. The server must bind the information for the client to find whereas the client do not need to.

3. listen method can make the server get the information from client

4. Client socket is used to serve the client

5. if a client use close method, server will know and close the client socket for this client

In this folder, it talk about all of the python OOP.

• Create first class 🐱 (LN_01)
• when create the second 😺 , their addresses are different (LN_02)
• parameter "self" inside method points to the Class itself
• use parameters to specify the attribute of an object (LN_04)
• __init__, When the object made, init autoly run (LN_03)
• __del__, When object go, del autoly run (LN_05)
• __str__, return specified output of print(Object) (LN_06)

• Two objects do not affect each other (LN_01, LN_02)
• The attribute of an object can be another object (LN_03)
• Python do not have real private, we can add "object._class__attr" or "object._class__method()" to get the private method and attr NOT RECOMMEND (LN_04)

• Child class and Grandchild class can inherit all method from Father class (LN_01)
• When the method of father is not good enough , overide that (LN_01)
• Use super() to get the method of father class (LN_01)
• Use father.method(self) Python 2.x Not recommend (LN_01)
• Child can not get the private attr and private method of its father (LN_02)
• Child can get the private attr and private method by the public method of its father (LN_03)

• Child class can inherit all Methods and all attrs of its all fathers (LN_01)
• If fathers have the same method, avoid inheritance (LN_01)
• Object.__mro__ can show method resolution order in python (LN_02)
• Python 3.x inherit from object class automatically, whereas Python 2.x not and need to specify object as father class. Recommend specify object as father class in order to adapt both 2.x and 3.x (LN_02)

• Different child class call the same method of father class, the outcome is differernt. (LN_01)
• This is based on inherit and overide the father class

• An instance is an object in memory, made by class
• Every instance has its own memory and have different instance attribute
• Every Class only be created once in memory
• Every Method inside class only be create once in memory. The objects create by same class share memory of the method.
• Class is also an object, so it also has class attributes and class method (LN_01)
• Can use object to get father class attribute from grandchild -> father -> grandparent -> ... -> Object Not recommend (LN_02)
• If want to use child.dadattr = to set an attr, this only add a new attr to the grandchild, can not change the attr of its dad. So not recommend for the previous one (LN_03)

• There are three method, instance method, class method and static method
• Class method example, need @classmethod (LN_01)
• Static method example, need @staticmethid, do not need instance just class.staticmethod() (LN_02)
• Combine class attributes and class method (LN_03)

• __new__ method is used to allocate address for object, and need to return object to __init__ (LN_01)
• Can change _new_ method in order to satisfy Singleton Pattern (LN_02)
• Can change _init_ method in order to make init only execute once (LN_03)

• Use try except to get the error (LN_01)
• Use different except to get different error (LN_02)
• Use "except Exception as result" to get unknown error (LN_03)
• The code inside else will occur when no error, the code inside finally will always occur (LN_04)
• Exception pass can let us only try except in the main, to make us easy (LN_05)
• Can use Exception class to make a user-define exception (LN_06)

• Review import module (LN_01, LN_02, LN_03)
• can use "import xxxx as xx" to give the module a alias (LN_01)
• can use "from xxxx inport xx" to specify the Function or class which want to import (LN_04)
• If import same function from different module, the last one will be valid. Or use as to avoid this conflict (LN_05)
• can use "from xxxx inport *" to import all from module Not recommend (LN_06)
• The order import module local folder -> system folder, use random_file_ to see the absolute path of that module (LN_07)
• Make sure every file is able to import
• __name__ == "__main__" is used to test. when others inport, the code inside main will not execute Very Recommend (LN_08, LN_09)
• How to create package: two ways (LN_10_package)
• We must specify the module we want to let outside import inthe init.py file (LN_10_package, LN_10_import_package)
• Use setup if we want to share our module to other developers

• Read file (LN_01)
• When call read(), the read pointer will go to the end of file (LN_02)

• When call readline(), only the first line will be read. (LN_05)
• Copy small: Read a file and write it to a new file. (LN_06)
• Copy large: Readline a file and write it to a new file. (LN_07)

• Manipulate file

• Manipulate folder

In this folder, it talk about all of the python basic.

• Basic grammer for python.

• if, elif and else are the keyword for branch.

• While loop is a type of loop which can run typical code for several time.

• Def is the key word to define a function in python.

• Every .py file is a module and we can use import to use the function of other module.

• LIst and Tuple are two data type of python
• They are ordered and list can be change and tuple can not
• There are many function for list and only two for tuple which is index and count.

• Dictionary is one data type of python
• It has key and value. Key and value are in pair. Dictionary is inordered.

• String is one data type of python. There are many functions for String.

• For each is a typical type of for loop, the code inside the each will occur if the for loop finish without break.

• This is a Name Card System which can add, show and search the cards.