This article is a summary of modern Python development: syntactical fundamentals and engineering practices. See the Python Learning & Practice Resource Index for more Python related materials. This article references Python Crash Course Cheat Sheets, Pysheeet, etc. This article only includes the key knowledge points and syntax that the author often uses in daily work. If you want to further learn Python or are interested in the direction of machine learning and data mining, you can refer to the Data Science and machine Learning Manual of The Programmer.

Basic grammar

Python is a high-level, dynamically typed, multiparadigm programming language; When defining a Python file, we usually declare the file encoding first:

# specify how the script is called #! /usr/bin/env python # configure utF-8 encoding # -* -coding: <encoding-name> -*- # Vim :fileencoding=<encoding-name>Copy the code

Life is short, use Python, and lots of powerful syntactic sugar makes Python code look like pseudo-code a lot of the time. For example, the simple quicksort we implement in Python is much smaller than Java:

def quicksort(arr): if len(arr) <= 1: return arr pivot = arr[len(arr) / 2] left = [x for x in arr if x < pivot] middle = [x for x in arr if x == pivot] right = [x for x in arr if x > pivot] return Quicksort (left) + middle + quicksort(right) print quicksort([3,6,8,10,1,2]) # Prints "[1, 1, 2, 3, 6, 8, 10]"Copy the code

Console interaction

The __name__ keyword is used to determine whether a script is executed directly with a Python command or by an external reference. Google’s open source Fire is also a great framework for quickly encapsulating a class as a command-line tool:

import fire

class Calculator(object):
  """A simple calculator class."""

  def double(self, number):
    return 2 * number

if __name__ == '__main__':
  fire.Fire(Calculator)

# python calculator.py double 10  # 20
# python calculator.py double --number=15  # 30Copy the code

In Python 2 print is an expression, whereas in Python 3 print is a function. If you want print to be used as a function in Python 2, you need to introduce a custom:

from __future__ import print_functionCopy the code

We can also use pprint to beautify the console output:

import pprint stuff = ['spam', 'eggs', 'lumberjack', 'knights', 'ni'] pprint. Pprint (stuff) # pp = pprint. Printer(depth=6) ('knights', ('ni', ('dead',('parrot', ('fresh fruit',)))))))) pp.pprint(tup)Copy the code

The module

Modules in Python are Python source files that export classes, functions, and global variables. When importing variables from a module, the function name is usually a Namespace. Packages in Python are modules’ folders, and __init__.py usually identifies a folder as a Package:

Py # siblingmodule.py def siblingModuleFun(): print('Hello from siblingModuleFun') def siblingModuleFunTwo(): print('Hello from siblingModuleFunTwo') import siblingModule import siblingModule as sibMod sibMod.siblingModuleFun() from siblingModule import siblingModuleFun siblingModuleFun() try: # Import 'someModuleA' that is only available in Windows import someModuleA except ImportError: try: # Import 'someModuleB' that is only available in Linux import someModuleB except ImportError:Copy the code

Package can set a uniform entry for all files in a directory:

someDir/ main.py subModules/ __init__.py subA.py subSubModules/ __init__.py subSubA.py # subA.py def subAFun(): print('Hello from subAFun') def subAFunTwo(): print('Hello from subAFunTwo') # subSubA.py def subSubAFun(): print('Hello from subSubAFun') def subSubAFunTwo(): print('Hello from subSubAFunTwo') # __init__.py from subDir # Adds 'subAFun()' and 'subAFunTwo()' to the 'subDir' namespace from .subA import * # The following two import statement do the same thing, they add 'subSubAFun()' and 'subSubAFunTwo()' to the 'subDir' namespace. The first one assumes '__init__.py' is empty in  'subSubDir', and the second one, assumes '__init__.py' in 'subSubDir' contains 'from .subSubA import *'. # Assumes '__init__.py' is empty in 'subSubDir' # Adds 'subSubAFun()' and 'subSubAFunTwo()' to the 'subDir' namespace from .subSubDir.subSubA import * # Assumes '__init__.py' in 'subSubDir' has 'from .subSubA import *' # Adds 'subSubAFun()' and 'subSubAFunTwo()' to the 'subDir' namespace from .subSubDir import * # __init__.py from subSubDir # Adds 'subSubAFun()' and 'subSubAFunTwo()' to the 'subSubDir' namespace from .subSubA import * # main.py import subDir subDir.subAFun() # Hello from subAFun subDir.subAFunTwo() # Hello from subAFunTwo subDir.subSubAFun() # Hello from subSubAFun subDir.subSubAFunTwo() # Hello from subSubAFunTwoCopy the code

Expression and control flow

To choice

Python uses if, elif, and else for basic conditional selection operations:

if x < 0:
     x = 0
     print('Negative changed to zero')
 elif x == 0:
     print('Zero')
 else:
     print('More')Copy the code

Python also supports ternary conditional Operators:

a if condition else bCopy the code

You can also use a Tuple to achieve a similar effect:

FalseValue (trueValue)[TEST] FalseValue (trueValue)[test= = True] # Or falseValue (trueValue)[bool(<expression>)]Copy the code

To iterate over

For-in can be used to iterate through arrays and dictionaries:

Words = ['cat', 'window', 'defenestrate'] for win words: print(w, len(w)) # if len(w) > 6: words.insert(0, w) # words -> ['defenestrate', 'cat', 'window', 'defenestrate']Copy the code

If we want to traverse a sequence of numbers, we can use Python’s built-in range function:

a = ['Mary', 'had', 'a', 'little', 'lamb']

for i in range(len(a)):
    print(i, a[i])Copy the code

Basic data types

You can use the built-in function to cast:

int(str)
float(str)
str(int)
str(float)Copy the code

Number: indicates the value type

x = 3 print type(x) # Prints "<type 'int'>" print x # Prints "3" print x + 1 # Addition; prints "4" print x - 1 # Subtraction; prints "2" print x * 2 # Multiplication; prints "6" print x ** 2 # Exponentiation; Print x # prints "4" x *= 2 print x # prints "8" y = 2.5 print type(y) # prints "<type 'float'> Y, y + 1, y * 2, y ** 2 # Prints"Copy the code

Boolean type

Python provides a common logical operators, but it’s important to note that Python does not use &&, | |, etc., but the direct use of English words.

t = True f = False print type(t) # Prints "<type 'bool'>" print t and f # Logical AND; prints "False" print t or f # Logical OR; prints "True" print not t # Logical NOT; prints "False" print t ! = f # Logical XOR; prints "True"Copy the code

String: a String

Python 2 supports Ascii STR (), a separate Unicode () type, and no byte; The default string in Python 3 is utF-8 and contains both byte and bytearray:

Type ("Guido") # string type is STR in PYTHon2 # <type 'STR '> # Use the module provided in __future__ to degrade the use of Unicode from __future__ import unicode_literals type("Guido") # string type become unicode # <type 'unicode'>Copy the code

Python strings support sharding, template strings, and other common operations:

var1 = 'Hello World! ' var2 = "Python Programming" print "var1[0]: ", var1[0] print "var2[1:5]: ", var2[1:5] # var1[0]: H # var2[1:5]: ytho print "My name is %s and weight is %d kg!" % ('Zara', 21) # My name is Zara and weight is 21 kg!Copy the code

str[0:4]
len(str)

string.replace("-", " ")
",".join(list)
"hi {0}".format('j')
str.find(",")
str.index(",")   # same, but raises IndexError
str.count(",")
str.split(",")

str.lower()
str.upper()
str.title()

str.lstrip()
str.rstrip()
str.strip()

str.islower()Copy the code

Re.sub ('[^ a-za-z0-9]+', ', myString)Copy the code

If you need to determine whether a substring is included or search for a substring:

# in operator can determine string if "blah" not in someString: continue # find can search subscript s = "This be a string" if s.type ("is") == -1: print "No 'is' here!" else: print "Found 'is' in the string."Copy the code

Regex: indicates a regular expression

Re.match (r'^[aeiou]', STR) # Replace re.sub(r'^[aeiou]', '? ', STR) re. Sub (r '(xyz), r' \ 1, STR) # compiler to generate independent object of regular expression expr = re.com running (r '^... $') expr.match(...) expr.sub(...)Copy the code

Common expressions are listed below:

. # check for HTML tags re search (' < ^ / > [^ >] * > ', '< a href = "# label" >') # common user name password re. The match (' ^ [a zA - Z0-9 - _] 3 dec} {$', 'Foo') is not None. Re match (| '^ \ w / - _ 3 dec} {$', 'Foo') is not None. # Email re match (' ^ ([a - z0-9 _ \. -] +) @ (/ \ \. Da - z - +) \. ([a-z \.] {2, 6}) $', '[email protected]') # Url exp = re.compile(r'''^(https?:\/\/)? # match http or https ([\da-z\.-]+) # match domain \. ([a-z \] {2, 6}) # match domain (/ \ \ / \ w - *) \ /? $# match API or the file "', Re. X) exp. Match (' www.google.com ') # IP address exp = re.com running (r "' ^ (? : (? : 25 [0 to 5] | 2 [0 to 4] [0-9] | [1]? [0-9] [0-9]?) \.) {3} (? : 25 [0 to 5] | 2 [0 to 4] [0-9] | [1]? [0-9] [0-9]?) $" 're. X) exp. Match (' 192.168.1.1)Copy the code

Collection types

List the List:

Operation: Creates additions and deletions

List is the basic sequence type:

Str.split (".") # if you want to split an array into a string, Join list1 = ['1', '2', '3'] str1 = ". Join (list1) # List1 = [1, 2, 3] str1 = ". Join (STR (e) for e in list1)Copy the code

You can use Append and EXTEND to insert elements or concatenate arrays

X = [1, 2, 3] x.a ppend ([4, 5]) # [1, 2, 3, [4, 5]] x.e xtend ([4, 5]) # [1, 2, 3, 4, 5], pay attention to the extend the return value to NoneCopy the code

You can remove elements from a list using pop, slices, del, remove, and so on:

MyList = [10,20,30,40,50] # pop(1) # 20 # myList Mylist.pop (1) # Default to pop the last element mylist.pop () # using slices to delete elements a = [1, 2, 3, 4, 5, MyList = [10,20,30,40,50] rmovIndxNo = 3 del  myList[rmovIndxNo] # myList: [10, 20, 30, 50] Letters = ["a", "b", "c", "d", "e"] numbers.remove(numbers[1]) print(*letters) # used a * to make it unpack you don't have toCopy the code

Iteration: Index Iteration

You can use the basic for loop to iterate over the elements of a list, as in this example:

animals = ['cat', 'dog', 'monkey']
for animal in animals:
    print animal
# Prints "cat", "dog", "monkey", each on its own line.Copy the code

If you want to get the index of the current element while looping, use the enumerate function:

animals = ['cat', 'dog', 'monkey']
for idx, animal in enumerate(animals):
    print '#%d: %s' % (idx + 1, animal)
# Prints "#1: cat", "#2: dog", "#3: monkey", each on its own lineCopy the code

Python also supports Slices:

nums = range(5)    # range is a built-in function that creates a list of integers
print nums         # Prints "[0, 1, 2, 3, 4]"
print nums[2:4]    # Get a slice from index 2 to 4 (exclusive); prints "[2, 3]"
print nums[2:]     # Get a slice from index 2 to the end; prints "[2, 3, 4]"
print nums[:2]     # Get a slice from the start to index 2 (exclusive); prints "[0, 1]"
print nums[:]      # Get a slice of the whole list; prints ["0, 1, 2, 3, 4]"
print nums[:-1]    # Slice indices can be negative; prints ["0, 1, 2, 3]"
nums[2:4] = [8, 9] # Assign a new sublist to a slice
print nums         # Prints "[0, 1, 8, 9, 4]"Copy the code

Comprehensions: transform

Python also uses map, reduce, and filter. Map is used to transform arrays:

Item = [1, 2, 3, 4, 5] squared = list(lambda x: Def multiply(x): return (x*x) def add(x): return (x+x) funcs = [multiply, add] for i in range(5): value = list(map(lambda x: x(i), funcs)) print(value)Copy the code

Reduce is used for inductive calculation:

From functools import Reduce product = Reduce ((lambda x, y: x * y), [1, 2, 3, 4]) # Output: 24Copy the code

Filter can filter an array:

number_list = range(-5, 5)
less_than_zero = list(filter(lambda x: x < 0, number_list))
print(less_than_zero)

# Output: [-5, -4, -3, -2, -1]Copy the code

A dictionary type

Create add or delete

D = {'cat': 'cute', 'dog': 'furry'} # create new dictionary print d['cat'] # Dictionary does not support Dot operator valuesCopy the code

If you need to merge two or more dictionary types:

# python 3.5z = {**x, **y} # python 3.5def merge_dicts(*dict_args): """ Given any number of dicts, shallow copy and merge into a new dict, precedence goes to key value pairs in latter dicts. """ result = {} for dictionary in dict_args: result.update(dictionary) return resultCopy the code

The index traversal

Element access can be made directly by key:

Get print 'cat' in D # Check if a dictionary has a given key; Print d['monkey'] # KeyError: print ['monkey'] # KeyError: print ['monkey'] Print d. set ('monkey', 'N/ a ') # get an element with a default; prints "N/A" print d.get('fish', 'N/A') # Get an element with a default; Prints "wet" d.keys() # You can get all keys using the keys methodCopy the code

You can use for-in to iterate over groups of numbers:

For k in dict.keys():... For value in dict.itervalues():... # Python 2.x for key, value in d.items():Copy the code

Other sequence types

A collection of

# Same as {"a", "b","c"} normal_set = set(["a", "b","c"]) # Adding an element to normal set is fine normal_set.add("d") print("Normal Set") print(normal_set) # A frozen  set frozen_set = frozenset(["e", "f", "g"]) print("Frozen Set") print(frozen_set) # Uncommenting below line would cause error as # we are trying to add element to a frozen set # frozen_set.add("h")Copy the code

function

The function definitions

Functions in Python are defined using the def keyword, for example:

def sign(x):
    if x > 0:
        return 'positive'
    elif x < 0:
        return 'negative'
    else:
        return 'zero'


for x in [-1, 0, 1]:
    print sign(x)
# Prints "negative", "zero", "positive"Copy the code

Python supports runtime creation of dynamic functions, known as lambda functions:

Def f(x): return x**2 # equivalent to g = lambda x: x**2Copy the code

parameter

Option Arguments: Indefinite Arguments

def example(a, b=None, *args, **kwargs): print a, b print args print kwargs example(1, "var", 2, 3, word="hello") # 1 var # (2, 3) # {'word': 'hello'} a_tuple = (1, 2, 3, 4, 5) a_dict = {"1":1, "2":2, "3":3} example(1, "var", *a_tuple, **a_dict) # 1 var # (1, # {'1': 1, '2': 2, '3': 3}Copy the code

The generator

def simple_generator_function(): yield 1 yield 2 yield 3 for value in simple_generator_function(): Our_generator = simple_generator_function() next(our_generator) # 1 Next (our_generator) Def get_primes(number): while True: if is_prime(number): yield number number += 1Copy the code

A decorator

Decorators are very useful design patterns:

# Wraps import wraps def decorator(func): @wraps(func) def wrapper(*args, **kwargs): print('wrap function') return func(*args, **kwargs) return wrapper @decorator def example(*a, **kw): Pass example.__name__ # attr of function preserve # 'example' # Decorator # With input wraps decorator_with_argument(val): def decorator(func): @wraps(func) def wrapper(*args, **kwargs): print "Val is {0}".format(val) return func(*args, **kwargs) return wrapper return decorator @decorator_with_argument(10) def example(): Print "This is example function." example() # Val is 10 # This is example function. print "This is example function." example = decorator_with_argument(10)(example) example() # Val is 10 # This is example  function.Copy the code

Classes and objects

The class definition

The definition of a class in Python is also straightforward:

class Greeter(object): # Constructor def __init__(self, name): self.name = name # Create an instance variable # Instance method def greet(self, loud=False): if loud: print 'HELLO, %s! ' % self.name.upper() else: print 'Hello, %s' % self.name g = Greeter('Fred') # Construct an instance of the Greeter class g.greet() # Call an instance method; prints "Hello, Fred" g.greet(loud=True) # Call an instance method; prints "HELLO, FRED!"Copy the code

Ex = 10 isinstance(ex,int)Copy the code

Managed Attributes: Managed Attributes

Class Example(object): def __init__(self, value): self._val = value @property def val(self): return self._val @val.setter def val(self, value): if not isintance(value, int): raise TypeError("Expected int") self._val = value @val.deleter def val(self): del self._val @property def square3(self): return 2**3 ex = Example(123) ex.val = "str" # Traceback (most recent call last): # File "", line 1, in # File "test.py", line 12, in val # raise TypeError("Expected int") # TypeError: Expected intCopy the code

Class methods versus static methods

class example(object):
  @classmethod
  def clsmethod(cls):
    print "I am classmethod"
  @staticmethod
  def stmethod():
    print "I am staticmethod"
  def instmethod(self):
    print "I am instancemethod"

ex = example()
ex.clsmethod()
# I am classmethod
ex.stmethod()
# I am staticmethod
ex.instmethod()
# I am instancemethod
example.clsmethod()
# I am classmethod
example.stmethod()
# I am staticmethod
example.instmethod()
# Traceback (most recent call last):
#   File "", line 1, in
# TypeError: unbound method instmethod() ...Copy the code

object

instantiation

Attribute operation

Object attributes in Python, unlike dictionary keys, can be evaluated using the dot operator. Using in directly causes problems:

class A(object):
    @property
    def prop(self):
        return 3

a = A()
print "'prop' in a.__dict__ =", 'prop' in a.__dict__
print "hasattr(a, 'prop') =", hasattr(a, 'prop')
print "a.prop =", a.prop

# 'prop' in a.__dict__ = False
# hasattr(a, 'prop') = True
# a.prop = 3Copy the code

It is recommended to use hasattr, getattr, setattr to operate on object attributes:

class Example(object):
  def __init__(self):
    self.name = "ex"
  def printex(self):
    print "This is an example"


# Check object has attributes
# hasattr(obj, 'attr')
ex = Example()
hasattr(ex,"name")
# True
hasattr(ex,"printex")
# True
hasattr(ex,"print")
# False

# Get object attribute
# getattr(obj, 'attr')
getattr(ex,'name')
# 'ex'

# Set object attribute
# setattr(obj, 'attr', value)
setattr(ex,'name','example')
ex.name
# 'example'Copy the code

Exceptions and Tests

Exception handling

Context Manager – with

With is often used to open or close certain resources:

host = 'localhost'
port = 5566
with Socket(host, port) as s:
    while True:
        conn, addr = s.accept()
        msg = conn.recv(1024)
        print msg
        conn.send(msg)
        conn.close()Copy the code

Unit testing

from __future__ import print_function

import unittest

def fib(n):
    return 1 if n<=2 else fib(n-1)+fib(n-2)

def setUpModule():
        print("setup module")
def tearDownModule():
        print("teardown module")

class TestFib(unittest.TestCase):

    def setUp(self):
        print("setUp")
        self.n = 10
    def tearDown(self):
        print("tearDown")
        del self.n
    @classmethod
    def setUpClass(cls):
        print("setUpClass")
    @classmethod
    def tearDownClass(cls):
        print("tearDownClass")
    def test_fib_assert_equal(self):
        self.assertEqual(fib(self.n), 55)
    def test_fib_assert_true(self):
        self.assertTrue(fib(self.n) == 55)

if __name__ == "__main__":
    unittest.main()Copy the code

storage

File to read and write

Path to deal with

Python’s built-in __file__ keyword points to the relative path of the current file, which can be used to construct absolute paths, or to index other files:

Dir = os.path.dirname(__file__) # SRC \app ## once you're at the directory level you want with the desired directory as the final path node: dirname1 = os.path.basename(dir) dirname2 = os.path.split(dir)[1] ## if you look at the documentation, This is exactly what os.path.basename does. Abspath automatically completes os.path.abspath(os.path.dirname(__file__)) # based on the relative path and the current workspace D:\WorkSpace\OWS\tool\ui-tool- SVN \python\ SRC \app # Get the true path of the current file os.path.dirname(os.path.realpath(__file__)) # D: WorkSpace\OWS\tool\ui-tool-svn\python\ SRC \app #Copy the code

You can use the listdir, Walk, glob modules to enumerate and retrieve files:

From OS import listdir from OS. Path import isfile, join onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, From OS import walk f = [] for (dirpath, dirnames, filenames) in walk(mypath): Import glob print(glob.glob("/home/ Adam /*.txt")) # ['/home/ Adam /file1.txt',  '/home/adam/file2.txt', .... ]Copy the code

Simple file read and write

Mode = 'a' if os.path.exists(writepath) else 'w' # With open("file.dat",mode) as f: f.write(...) . F.close () # message = f.read()Copy the code

Complex format file

JSON

import json

# Writing JSON data
with open('data.json', 'w') as f:
     json.dump(data, f)

# Reading data back
with open('data.json', 'r') as f:
     data = json.load(f)Copy the code

XML

LXML can be used to parse and process XML files, and this section introduces common operations. LXML supports creating Element objects from strings or files:

XML = '<a XMLNS ="test"><b XMLNS ="test"/></a>' root = etree.fromString (XML) Etree.tostring (root) # b'<a XMLNS ="test"><b XMLNS ="test"/></a>' # It is also possible to construct tree = etree.parse("doc/test.xml") # from a file Or a specified baseURL root = etree. Fromstring (XML, base_url = "http://where.it/is/from.xml")Copy the code

Iterators are provided to iterate over all elements:

For tag in tree.iter(): if not len(tag): Print (tag. Tag, tag.text) print (tag. Tag, tag.text) print tree.getpath(e)Copy the code

LXML supports XPath look-ups of elements, though it is important to note that XPath look-ups result in arrays, and namespaces need to be specified in cases where namespaces are included:

Root.xpath ('//page/text/text()',ns={prefix:url}) # Use getparent recursively to find el.getparent()Copy the code

LXML provides insert, append, and other methods for manipulating elements:

St = etree.element ("state", name="New Mexico") co = etree.element ("county", Name ="Socorro") st.append(co)Copy the code

Excel

You can use [XLRD]() to read Excel files, and xlsxWriter to write and manipulate Excel files.

Sh.cell (rx, col).valueCopy the code

Workbook = xlsxwriter.workbook (outputFile) worksheet = workbook.add_worksheet( Iterate over a two-dimensional array and write it to Excel for rowData in array: for col, data in enumerate(rowData): worksheet.write(row, col, data) row = row + 1 workbook.close()Copy the code

The file system

For advanced file manipulation, we can use Shutil built into Python

# recursively delete shutil.rmtree(appName) from shutil.rmtreeCopy the code

Network interaction

Requests

Requests is the elegant and easy-to-use Python network request library:

import requests r = requests.get('https://api.github.com/events') r = requests.get('https://api.github.com/user', auth=('user', 'pass')) r.status_code # 200 r.headers['content-type'] # 'application/json; charset=utf8' r.encoding # 'utf-8' r.text # u'{"type":"User"... ' r.json() # {u'private_gists': 419, u'total_private_repos': 77, ... } r = requests.put('http://httpbin.org/put', data = {'key':'value'}) r = requests.delete('http://httpbin.org/delete') r = requests.head('http://httpbin.org/get') r =  requests.options('http://httpbin.org/get')Copy the code

Data is stored

MySQL

import pymysql.cursors

# Connect to the database
connection = pymysql.connect(host='localhost',
                             user='user',
                             password='passwd',
                             db='db',
                             charset='utf8mb4',
                             cursorclass=pymysql.cursors.DictCursor)

try:
    with connection.cursor() as cursor:
        # Create a new record
        sql = "INSERT INTO `users` (`email`, `password`) VALUES (%s, %s)"
        cursor.execute(sql, ('[email protected]', 'very-secret'))

    # connection is not autocommit by default. So you must commit to save
    # your changes.
    connection.commit()

    with connection.cursor() as cursor:
        # Read a single record
        sql = "SELECT `id`, `password` FROM `users` WHERE `email`=%s"
        cursor.execute(sql, ('[email protected]',))
        result = cursor.fetchone()
        print(result)
finally:
    connection.close()Copy the code