+++ /dev/null
-Developer Notes for Python Compiler
-===================================
-
-Table of Contents
------------------
-
-- Scope
- Defines the limits of the change
-- Parse Trees
- Describes the local (Python) concept
-- Abstract Syntax Trees (AST)
- Describes the AST technology used
-- Parse Tree to AST
- Defines the transform approach
-- Control Flow Graphs
- Defines the creation of "basic blocks"
-- AST to CFG to Bytecode
- Tracks the flow from AST to bytecode
-- Code Objects
- Pointer to making bytecode "executable"
-- Modified Files
- Files added/modified/removed from CPython compiler
-- ToDo
- Work yet remaining (before complete)
-- References
- Academic and technical references to technology used.
-
-
-Scope
------
-
-Historically (through 2.4), compilation from source code to bytecode
-involved two steps:
-
-1. Parse the source code into a parse tree (Parser/pgen.c)
-2. Emit bytecode based on the parse tree (Python/compile.c)
-
-Historically, this is not how a standard compiler works. The usual
-steps for compilation are:
-
-1. Parse source code into a parse tree (Parser/pgen.c)
-2. Transform parse tree into an Abstract Syntax Tree (Python/ast.c)
-3. Transform AST into a Control Flow Graph (Python/newcompile.c)
-4. Emit bytecode based on the Control Flow Graph (Python/newcompile.c)
-
-Starting with Python 2.5, the above steps are now used. This change
-was done to simplify compilation by breaking it into three steps.
-The purpose of this document is to outline how the lattter three steps
-of the process works.
-
-This document does not touch on how parsing works beyond what is needed
-to explain what is needed for compilation. It is also not exhaustive
-in terms of the how the entire system works. You will most likely need
-to read some source to have an exact understanding of all details.
-
-
-Parse Trees
------------
-
-Python's parser is an LL(1) parser mostly based off of the
-implementation laid out in the Dragon Book [Aho86]_.
-
-The grammar file for Python can be found in Grammar/Grammar with the
-numeric value of grammar rules are stored in Include/graminit.h. The
-numeric values for types of tokens (literal tokens, such as ``:``,
-numbers, etc.) are kept in Include/token.h). The parse tree made up of
-``node *`` structs (as defined in Include/node.h).
-
-Querying data from the node structs can be done with the following
-macros (which are all defined in Include/token.h):
-
-- ``CHILD(node *, int)``
- Returns the nth child of the node using zero-offset indexing
-- ``RCHILD(node *, int)``
- Returns the nth child of the node from the right side; use
- negative numbers!
-- ``NCH(node *)``
- Number of children the node has
-- ``STR(node *)``
- String representation of the node; e.g., will return ``:`` for a
- COLON token
-- ``TYPE(node *)``
- The type of node as specified in ``Include/graminit.h``
-- ``REQ(node *, TYPE)``
- Assert that the node is the type that is expected
-- ``LINENO(node *)``
- retrieve the line number of the source code that led to the
- creation of the parse rule; defined in Python/ast.c
-
-To tie all of this example, consider the rule for 'while'::
-
- while_stmt: 'while' test ':' suite ['else' ':' suite]
-
-The node representing this will have ``TYPE(node) == while_stmt`` and
-the number of children can be 4 or 7 depending on if there is an 'else'
-statement. To access what should be the first ':' and require it be an
-actual ':' token, `(REQ(CHILD(node, 2), COLON)``.
-
-
-Abstract Syntax Trees (AST)
----------------------------
-
-The abstract syntax tree (AST) is a high-level representation of the
-program structure without the necessity of containing the source code;
-it can be thought of a abstract representation of the source code. The
-specification of the AST nodes is specified using the Zephyr Abstract
-Syntax Definition Language (ASDL) [Wang97]_.
-
-The definition of the AST nodes for Python is found in the file
-Parser/Python.asdl .
-
-Each AST node (representing statements, expressions, and several
-specialized types, like list comprehensions and exception handlers) is
-defined by the ASDL. Most definitions in the AST correspond to a
-particular source construct, such as an 'if' statement or an attribute
-lookup. The definition is independent of its realization in any
-particular programming language.
-
-The following fragment of the Python ASDL construct demonstrates the
-approach and syntax::
-
- module Python
- {
- stmt = FunctionDef(identifier name, arguments args, stmt* body,
- expr* decorators)
- | Return(expr? value) | Yield(expr value)
- attributes (int lineno)
- }
-
-The preceding example describes three different kinds of statements;
-function definitions, return statements, and yield statements. All
-three kinds are considered of type stmt as shown by '|' separating the
-various kinds. They all take arguments of various kinds and amounts.
-
-Modifiers on the argument type specify the number of values needed; '?'
-means it is optional, '*' means 0 or more, no modifier means only one
-value for the argument and it is required. FunctionDef, for instance,
-takes an identifier for the name, 'arguments' for args, zero or more
-stmt arguments for 'body', and zero or more expr arguments for
-'decorators'.
-
-Do notice that something like 'arguments', which is a node type, is
-represented as a single AST node and not as a sequence of nodes as with
-stmt as one might expect.
-
-All three kinds also have an 'attributes' argument; this is shown by the
-fact that 'attributes' lacks a '|' before it.
-
-The statement definitions above generate the following C structure type::
-
- typedef struct _stmt *stmt_ty;
-
- struct _stmt {
- enum { FunctionDef_kind=1, Return_kind=2, Yield_kind=3 } kind;
- union {
- struct {
- identifier name;
- arguments_ty args;
- asdl_seq *body;
- } FunctionDef;
-
- struct {
- expr_ty value;
- } Return;
-
- struct {
- expr_ty value;
- } Yield;
- } v;
- int lineno;
- }
-
-Also generated are a series of constructor functions that allocate (in
-this case) a stmt_ty struct with the appropriate initialization. The
-'kind' field specifies which component of the union is initialized. The
-FunctionDef() constructor function sets 'kind' to FunctionDef_kind and
-initializes the 'name', 'args', 'body', and 'attributes' fields.
-
-*** NOTE: if you make a change here that can affect the output of bytecode that
-is already in existence, make sure to delete your old .py(c|o) files! Running
-``find . -name '*.py[co]' -exec rm -f {} ';'`` should do the trick.
-
-
-Parse Tree to AST
------------------
-
-The AST is generated from the parse tree in (see Python/ast.c) using the
-function::
-
- mod_ty PyAST_FromNode(const node *n);
-
-The function begins a tree walk of the parse tree, creating various AST
-nodes as it goes along. It does this by allocating all new nodes it
-needs, calling the proper AST node creation functions for any required
-supporting functions, and connecting them as needed.
-
-Do realize that there is no automated nor symbolic connection between
-the grammar specification and the nodes in the parse tree. No help is
-directly provided by the parse tree as in yacc.
-
-For instance, one must keep track of
-which node in the parse tree one is working with (e.g., if you are
-working with an 'if' statement you need to watch out for the ':' token
-to find the end of the conditional). No help is directly provided by
-the parse tree as in yacc.
-
-The functions called to generate AST nodes from the parse tree all have
-the name ast_for_xx where xx is what the grammar rule that the function
-handles (alias_for_import_name is the exception to this). These in turn
-call the constructor functions as defined by the ASDL grammar and
-contained in Python/Python-ast.c (which was generated by
-Parser/asdl_c.py) to create the nodes of the AST. This all leads to a
-sequence of AST nodes stored in asdl_seq structs.
-
-
-Function and macros for creating and using ``asdl_seq *`` types as found
-in Python/asdl.c and Include/asdl.h:
-
-- ``asdl_seq_new(int)``
- Allocate memory for an asdl_seq for length 'size'
-- ``asdl_seq_free(asdl_seq *)``
- Free asdl_seq struct
-- ``asdl_seq_GET(asdl_seq *seq, int pos)``
- Get item held at 'pos'
-- ``asdl_seq_SET(asdl_seq *seq, int pos, void *val)``
- Set 'pos' in 'seq' to 'val'
-- ``asdl_seq_APPEND(asdl_seq *seq, void *val)``
- Set the end of 'seq' to 'val'
-- ``asdl_seq_LEN(asdl_seq *)``
- Return the length of 'seq'
-
-If you are working with statements, you must also worry about keeping
-track of what line number generated the statement. Currently the line
-number is passed as the last parameter to each stmt_ty function.
-
-
-Control Flow Graphs
--------------------
-
-A control flow graph (often referenced by its acronym, CFG) is a
-directed graph that models the flow of a program using basic blocks that
-contain the intermediate representation (abbreviated "IR", and in this
-case is Python bytecode) within the blocks. Basic blocks themselves are
-a block of IR that has a single entry point but possibly multiple exit
-points. The single entry point is the key to basic blocks; it all has
-to do with jumps. An entry point is the target of something that
-changes control flow (such as a function call or a jump) while exit
-points are instructions that would change the flow of the program (such
-as jumps and 'return' statements). What this means is that a basic
-block is a chunk of code that starts at the entry point and runs to an
-exit point or the end of the block.
-
-As an example, consider an 'if' statement with an 'else' block. The
-guard on the 'if' is a basic block which is pointed to by the basic
-block containing the code leading to the 'if' statement. The 'if'
-statement block contains jumps (which are exit points) to the true body
-of the 'if' and the 'else' body (which may be NULL), each of which are
-their own basic blocks. Both of those blocks in turn point to the
-basic block representing the code following the entire 'if' statement.
-
-CFGs are usually one step away from final code output. Code is directly
-generated from the basic blocks (with jump targets adjusted based on the
-output order) by doing a post-order depth-first search on the CFG
-following the edges.
-
-
-AST to CFG to Bytecode
-----------------------
-
-With the AST created, the next step is to create the CFG. The first step
-is to convert the AST to Python bytecode without having jump targets
-resolved to specific offsets (this is calculated when the CFG goes to
-final bytecode). Essentially, this transforms the AST into Python
-bytecode with control flow represented by the edges of the CFG.
-
-Conversion is done in two passes. The first creates the namespace
-(variables can be classified as local, free/cell for closures, or
-global). With that done, the second pass essentially flattens the CFG
-into a list and calculates jump offsets for final output of bytecode.
-
-The conversion process is initiated by a call to the function in
-Python/newcompile.c::
-
- PyCodeObject * PyAST_Compile(mod_ty, const char *, PyCompilerFlags);
-
-This function does both the conversion of the AST to a CFG and
-outputting final bytecode from the CFG. The AST to CFG step is handled
-mostly by the two functions called by PyAST_Compile()::
-
- struct symtable * PySymtable_Build(mod_ty, const char *,
- PyFutureFeatures);
- PyCodeObject * compiler_mod(struct compiler *, mod_ty);
-
-The former is in Python/symtable.c while the latter is in
-Python/newcompile.c .
-
-PySymtable_Build() begins by entering the starting code block for the
-AST (passed-in) and then calling the proper symtable_visit_xx function
-(with xx being the AST node type). Next, the AST tree is walked with
-the various code blocks that delineate the reach of a local variable
-as blocks are entered and exited::
-
- static int symtable_enter_block(struct symtable *, identifier,
- block_ty, void *, int);
- static int symtable_exit_block(struct symtable *, void *);
-
-Once the symbol table is created, it is time for CFG creation, whose
-code is in Python/newcompile.c . This is handled by several functions
-that break the task down by various AST node types. The functions are
-all named compiler_visit_xx where xx is the name of the node type (such
-as stmt, expr, etc.). Each function receives a ``struct compiler *``
-and xx_ty where xx is the AST node type. Typically these functions
-consist of a large 'switch' statement, branching based on the kind of
-node type passed to it. Simple things are handled inline in the
-'switch' statement with more complex transformations farmed out to other
-functions named compiler_xx with xx being a descriptive name of what is
-being handled.
-
-When transforming an arbitrary AST node, use the VISIT macro::
-
- VISIT(struct compiler *, <node type>, <AST node>);
-
-The appropriate compiler_visit_xx function is called, based on the value
-passed in for <node type> (so ``VISIT(c, expr, node)`` calls
-``compiler_visit_expr(c, node)``). The VISIT_SEQ macro is very similar,
- but is called on AST node sequences (those values that were created as
-arguments to a node that used the '*' modifier). There is also
-VISIT_SLICE just for handling slices::
-
- VISIT_SLICE(struct compiler *, slice_ty, expr_context_ty);
-
-Emission of bytecode is handled by the following macros:
-
-- ``ADDOP(struct compiler *c, int op)``
- add 'op' as an opcode
-- ``ADDOP_I(struct compiler *c, int op, int oparg)``
- add 'op' with an 'oparg' argument
-- ``ADDOP_O(struct compiler *c, int op, PyObject *type, PyObject *obj)``
- add 'op' with the proper argument based on the position of obj in
- 'type', but with no handling of mangled names; used for when you
- need to do named lookups of objects such as globals, consts, or
- parameters where name mangling is not possible and the scope of the
- name is known
-- ``ADDOP_NAME(struct compiler *, int, PyObject *, PyObject *)``
- just like ADDOP_O, but name mangling is also handled; used for
- attribute loading or importing based on name
-- ``ADDOP_JABS(struct compiling *c, int op, basicblock b)``
- create an absolute jump to the basic block 'b'
-- ``ADDOP_JREL(struct compiling *c, int op, basicblock b)``
- create a relative jump to the basic block 'b'
-
-Several helper functions that will emit bytecode and are named
-compiler_xx() where xx is what the function helps with (list, boolop
- etc.). A rather useful one is::
-
- static int compiler_nameop(struct compiler *, identifier,
- expr_context_ty);
-
-This function looks up the scope of a variable and, based on the
-expression context, emits the proper opcode to load, store, or delete
-the variable.
-
-As for handling the line number on which a statement is defined, is
-handled by compiler_visit_stmt() and thus is not a worry.
-
-In addition to emitting bytecode based on the AST node, handling the
-creation of basic blocks must be done. Below are the macros and
-functions used for managing basic blocks:
-
-- ``NEW_BLOCK(struct compiler *)``
- create block and set it as current
-- ``NEXT_BLOCK(struct compiler *)``
- basically NEW_BLOCK() plus jump from current block
-- ``compiler_new_block(struct compiler *)``
- create a block but don't use it (used for generating jumps)
-
-Once the CFG is created, it must be flattened and then final emission of
-bytecode occurs. Flattening is handled using a post-order depth-first
-search. Once flattened, jump offsets are backpatched based on the
-flattening and then a PyCodeObject file is created. All of this is
-handled by calling::
-
- PyCodeObject * assemble(struct compiler *, int);
-
-*** NOTE: if you make a change here that can affect the output of bytecode that
-is already in existence, make sure to delete your old .py(c|o) files! Running
-``find . -name '*.py[co]' -exec rm -f {} ';'`` should do the trick.
-
-
-Code Objects
-------------
-
-In the end, one ends up with a PyCodeObject which is defined in
-Include/code.h . And with that you now have executable Python bytecode!
-
-
-Modified Files
---------------
-
-+ Parser/
-
- - Python.asdl
- ASDL syntax file
-
- - asdl.py
- "An implementation of the Zephyr Abstract Syntax Definition
- Language." Uses SPARK_ to parse the ASDL files.
-
- - asdl_c.py
- "Generate C code from an ASDL description." Generates
- ../Python/Python-ast.c and ../Include/Python-ast.h .
-
- - spark.py
- SPARK_ parser generator
-
-+ Python/
-
- - Python-ast.c
- Creates C structs corresponding to the ASDL types. Also
- contains code for marshaling AST nodes (core ASDL types have
- marshaling code in asdl.c). "File automatically generated by
- ../Parser/asdl_c.py".
-
- - asdl.c
- Contains code to handle the ASDL sequence type. Also has code
- to handle marshalling the core ASDL types, such as number and
- identifier. used by Python-ast.c for marshaling AST nodes.
-
- - ast.c
- Converts Python's parse tree into the abstract syntax tree.
-
- - compile.txt
- This file.
-
- - newcompile.c
- New version of compile.c that handles the emitting of bytecode.
-
- - symtable.c
- Generates symbol table from AST.
-
-
-+ Include/
-
- - Python-ast.h
- Contains the actual definitions of the C structs as generated by
- ../Python/Python-ast.c .
- "Automatically generated by ../Parser/asdl_c.py".
-
- - asdl.h
- Header for the corresponding ../Python/ast.c .
-
- - ast.h
- Declares PyAST_FromNode() external (from ../Python/ast.c).
-
- - code.h
- Header file for ../Objects/codeobject.c; contains definition of
- PyCodeObject.
-
- - symtable.h
- Header for ../Python/symtable.c . struct symtable and
- PySTEntryObject are defined here.
-
-+ Objects/
-
- - codeobject.c
- Contains PyCodeObject-related code (originally in
- ../Python/compile.c).
-
-
-ToDo
-----
-*** NOTE: all bugs and patches should be filed on SF under the group
- "AST" for easy searching. It also does not hurt to put
- "[AST]" at the beginning of the subject line of the tracker
- item.
-
-+ Stdlib support
- - AST->Python access?
- - rewrite compiler package to mirror AST structure?
-+ Documentation
- - flesh out this doc
- * byte stream output
- * explanation of how the symbol table pass works
- * code object (PyCodeObject)
-+ Universal
- - make sure entire test suite passes
- - fix memory leaks
- - make sure return types are properly checked for errors
- - no gcc warnings
-
-References
-----------
-
-.. [Aho86] Alfred V. Aho, Ravi Sethi, Jeffrey D. Ullman.
- `Compilers: Principles, Techniques, and Tools`,
- http://www.amazon.com/exec/obidos/tg/detail/-/0201100886/104-0162389-6419108
-
-.. [Wang97] Daniel C. Wang, Andrew W. Appel, Jeff L. Korn, and Chris
- S. Serra. `The Zephyr Abstract Syntax Description Language.`_
- In Proceedings of the Conference on Domain-Specific Languages, pp.
- 213--227, 1997.
-
-.. _The Zephyr Abstract Syntax Description Language.:
- http://www.cs.princeton.edu/~danwang/Papers/dsl97/dsl97.html
-
-.. _SPARK: http://pages.cpsc.ucalgary.ca/~aycock/spark/
-
projects / python / commitdiff