Parrot-Virtual-Machine.pdf

 op double_const(out INT, inconst INT) { $1 = $2 + $2; }



And, in PIR: $I0 = double_const 5 # numeric literal "5" is a constant



invar



The input parameter is a variable,



op my_op(invar PMC)



like a PMC



The type of the argument can also be one of several options: t y pe



m e a n i ng



e x am p l e



INT



integervalue



42or$I0



NUM



floating-point value



3.14 or $N3



STR



string



PMC



PMCvariable



$P0



KEY



Hashkey



["name"]



INTKEY Integerindex LABEL



"Hello"or$S4



[5]



location in code to jump to jump_here:



OP naming and function signatures You can have many ops with the same name, so long as they have different parameters. The two following declarations are okay: op my_op (out INT, in INT) { } op my_op (out NUM, in INT) { }



The ops compiler converts these op declarations similar to the following C function declarations: INTVAL op_my_op_i_i(INTVAL param1) { } NUMBER op_my_op_n_i(INTVAL param1) { }



 Run Core



78



Notice the "_i_i" and "_n_i" suffixes at the end of the function names? This is how Parrot ensures that function names are unique in the system to prevent compiler problems. This is also an easy way to look at a function signature and see what kinds of operands it takes.



Control Flow An opcode can determine where control flow moves to after it has completed executing. For most opcodes, the default behavior is to move to the next instruction in memory. However, there are many sorts of ways to alter control flow, some of which are very new and exotic. There are several keywords that can be used to obtain an address of an operation. We can then goto that instruction directly, or we can store that address and jump to it later. K e yw or d



NEXT()



M e an i n g



Jump to the next opcode in memory



ADDRESS(a) Jump to the opcode given by a. a is of typ e opcode_t*. OFFSET(a)



Jump to the opcode given by offset a from the current offset. a is typically type in LABEL.



POP()



get the address given at the top of the control stack. This feature is being deprecated and eventually Parrot will be stackless internally.



The Opcode Compiler The opcode compiler is located at dev/build/ops2c.pl, although most of it's functionality is located in a variety of included libs, such as Parrot::OpsFile. Parrot::Ops2c::* and Parrot::OpsTrans::*. We'll look at the different runcores in the section below. Suffice it to say, however, that different runcores require that the opcodes be compiled into a different format for execution. Therefore the job of the opcode compiler is relatively complex: it must read in the opcode description files and output syntactically correct C code in several different output formats.



Dynops: Dynamic Opcode Libraries The ops we've been talking about so far are all the standard built-in ops. These aren't the only ops available however, Parrot also allows dynamic op libraries to be loaded in at runtime. dynops are dynamically-loadable op libraries. They are written almost exactly like regular built-in ops are, but



they're compiled separately into a library and loaded in to Parrot at runtime using the .loadlib directive.



Run Cores Runcores are the things that decode and execute the stream of opcodes in a PBC file. In the most simple case, a runcore is a loop that takes each bytecode value, gathers the parameter data from the PBC stream, and passes control to the opcode routine for execution. There are several different opcores. Some are very practical and simple, some use special tricks and compiler features to optimize for speed. Some opcores perform useful ancillary tasks such as debugging and profiling. Some runcores serve no useful purpose except to satisfy some basic academic interest.



 Run Core



79



Basic Cores Slow Core In the slow core, each opcode is compiled into a separate function. Each opcode function takes two arguments: a pointer to the current opcode, and the Parrot interpreter structure. All arguments to the opcodes are parsed and stored in the interpreter structure for retrieval. This core is, as it's name implies, very slow. However, it's conceptually very simple and it's very stable. For this reason, the slow core is used as the base for some of the specialty cores we'll discuss later. Fast Core The fast core is exactly like the slow core, except it doesn't do the bounds checking and explicit context updating that the slow core does. Switched Core The switch core uses a gigantic C switch { } statement to handle opcode dispatching, instead of using individual functions. The benefit is that functions do not need to be called for each opcode, which saves on the number of machine code instructions necessary to call an opcode.



Native Code Cores JIT Core Exec Core



Advanced Cores The two cores that we're going to discuss next rely on a specialty feature of some compilers called computed goto. In normal ANSI C, labels are control flow statements and are not treated like first-class data items. However, compilers that support compute goto allow labels to be treated like pointers, stored in variables, and jumped to indirectly. void * my_label = &&THE_LABEL; goto *my_label;



The computed goto cores compile all the opcodes into a single large function, and each opcode corresponds to a label in the function. These labels are all stored in a large array: void *opcode_labels[] = { &&opcode1, &&opcode2, &&opcode3, ... };



Each opcode value can then be taken as an offset to this array as follows: goto *opcode_labels[current_opcode];



Computed Goto Core The computed goto core uses the mechanism described above to dispatch the various opcodes. After each opcode is executed, the next opcode in the incoming bytecode stream is looked up in the table and dispatched from there. Predereferenced Computed Goto Core



 Run Core



80 In the precomputed goto core, the bytecode stream is preprocessed to convert opcode numbers into the respective labels. This means they don't need to be looked up each time, the opcode can be jumped to directly as if it was a label. Keep in mind that the dispatch mechanism must be used after every opcode, and in large programs there could be millions of opcodes. Even small savings in the number of machine code instructions between opcodes can make big differences in speed.



Specialty Cores GC Debug Core Debugger Core Profiling Core Tracing Core



Memory and Garbage Collection Arenas and Pools Parrot allocates memory in large blocks called arenas. Each arena has enough space to hold exactly N items of size M, typically tightly packed together in consecutive memory locations. Some collectors may also store additional metadata in the arena as well. Multiple arenas of the same type are stored as a linked list inside a pool. A pool may no arenas, one arena, or multiple arenas, but arenas in a given pool all contain objects of the same size. However, each arena may hold a different number of objects. The parrot interpreter structure, which maintains state for the entire virtual machine, contains a pointer to the arena_base structure. The arena base structure contains pointers to all the various pools, and a few other important



data items that are used for memory management.



Garbage Collectors Parrot has a number of garbage collectors available, which can be selected at compile time using compiler directives. The most mature and robust collector at this time is a simple mark & sweep collector, GC_MS. The different collectors can be activated or deactivated prior to compilation in the file include/Parrot/settings.h. That file contains a number of options that you can set to customize the behavior of Parrot.



Mark & Sweep Collector The Mark & Sweep (MS) collector is the only collector for Parrot that is currently mature and stable enough for regular use. However, this collector is the cause of several performance problems in Parrot, and there are active efforts under way to replace it. The mark & sweep collector will likely be deprecated or removed entirely from Parrot by the 1.0.0 release.



 Memory and Garbage Collection



81



Incremental Tricolor Collector Development is currently under way on a new garbage collection scheme which uses a tricolor marking algorithm to mark data objects. The tricolor algorithm will enable an incremental behavior in the collector so that the "stop the world" behavior of the MS collector can be avoided for large memory pools.



Writing new Collectors Writing a new collector is as simple as implementing a particular interface. This interface is defined in detail in PDD 09, but we will cover some of the basics here.



Resources • http://www.parrotcode.org/docs/memory_internals.html • http://www.parrotcode.org/docs/pdd/pdd09_gc.html



PMC System PMCs We've discussed PMCs already -- in the Parrot Virtual Machine/Polymorphic Containers (PMCs) chapter -including how to define new PMC types using the PMC compiler, and how to use them in PIR programs. This chapter is going to go into more detail about how PMCs are actually used in Parrot, including memory management of PMCs, morphing PMCs, and interfacing with PMCs.



PMC System Overview The PMC data structure is deceptively simple, and is designed to be extensible enough for general purpose data and functionality use. Here is the definition of the PMC structure and the associated PMC_EXT structure: struct PMC { Parrot_UInt



flags;



VTABLE



*vtable;



DPOINTER



*data;



struct PMC_EXT *pmc_ext; };



typedef struct PMC_EXT { DPOINTER *data; PMC *_metadata;



struct _Sync *_synchronize; PMC *_next_for_GC; } PMC_EXT;



As we can see in these definitions, the PMC is actually very small. Much of the information about a PMC, including all it's various methods and VTABLE interfaces is stored in the ->vtable pointer. The VTABLE structure is a very large structure that contains function pointers for all the various VTABLE interfaces.



 PMC System



PMC Data Every PMC type also contains a pointer for a data structure that's specific to that PMC. These data structures are defined based upon the inheritance hierarchy of the PMC and the various attributes that it has been defined with. For instance, this PMC definition: pmclass MyPmc { ATTR INTVAL a; ATTR FLOATVAL b; ATTR STRING *c; ATTR PMC *d; ... }



will turn into this C data structure definition: typedef struct Parrot_MyPmc_attributes { INTVAL a; FLOATVAL b; STRING *c; PMC *d; } Parrot_MyPmc_attributes;



This structure is supposed to be contained in the ->data pointer, which should always be accessed using the PMC_data macro. This way if the PMC structure definition changes eventually, all code that uses the macros



properly will be automatically updated because the macro will be updated. Here is an example of an initialization VTABLE that uses these attributes: VTABLE void init () { Parrot_MyPmc_attributes *p = mem_allocate_typed(Parrot_MyPmc_attributes); p->a = 0; p->b = 0.0 p->c = NULL; p->d = PMCNULL; PMC_data(SELF) = p; }



There is another macro which uses the word PARROT and the name of the PMC in all capital letters to retrieve the data structure again and properly cast it (so your compiler doesn't give warnings about using pointers without a cast): Parrot_MyPmc_attributes *attr = PARROT_MYPMC(SELF);



82



 PMC System



PObjects C isn't a class-based (or "object oriented") language, but many lessons from OO programming methodologies have been adapted for use in Parrot's code base. PMCs, STRINGs, and a few other data types are based off the definition of a "PObj", also known as a "Buffer": typedef struct Buffer { Parrot_UInt flags; } Buffer;



Notice how the first two entries in the Buffer are the same as in the PMC? All objects that start off with these two data items are said to be "PObject isomorphic". For short, we say that all pobject isomorphic are simply "PObjects", and there are many types of PObjects. The memory management system, for example, can test the flags of all pobjects to determine what type of PObject a memory object is. A PMC may optionally contain a PMC_EXT structure, which adds additional functionality. PMC_EXT allows a PMC to be shared between multiple threads, or multiple Parrot interpreters without introducing data contention. PMC_EXT also allows a PMC to contain a hash of metadata (attribute value pairs), which are typically added as attributes in PIR.



PMC Management PMCs are allocated from two special pools, a PMC pool and a constant PMC pool. Constant PMCs are considered to be immutable and everlasting, and so are never modified nor collected by the garbage collector. STRINGS are allocated in either a string pool or a constant string pool. The same relationship applies, constant strings are never modified and never collected. PMC_EXT structures are not currently managed by the memory management subsystem. However, since PMC_EXTs are assigned to PMCs in a one-to-one relationship, we always know we can free it when it's PMC is freed. In terms of garbage collection, PMCs are one of the only aggregate data types in Parrot. STRINGS do not contain pointers to other data items of interest to the garbage collector. Stack chunks, which are used internally in some structures are PObjs and are also aggregates, but are marked separately by the collector and are not treated as aggregates directly.



VTables VTables represent a standard interface to PMCs of all types. For every single PMC, there are a series of standard operations that you can perform (or attempt). Not all PMCs support all Vtable operations



VTable Types VTables are complicated data items that contain, in addition to a large series of function pointers, a number of data items to support PMCs. One of the data items is a class PMC, a PMC that represents a particular PMC class. Another item is an enum that differentiates between all PMC classes.



Resources • http://www.parrotcode.org/docs/vtables.html



83



 String System



String System Strings Overview



Exception Subsystem Exception handling has become a staple in most modern programming languages. Parrot, since it intends to host many such languages must support a robust exception system. Not only does Parrot use exceptions for error handling and recovery, but it also encourages the use of control flow exceptions to implement high-level control flow features of those languages as well. What this means is that the exception subsystem is one of the most important for language implementers to become familiar with.



Exceptions: The Basics Exceptions are broken down into two primary parts: the exception object and the exception handler. Exception handlers are like subroutines in many ways, and must be specially registered with Parrot before they can be used. When Parrot detects an error, it creates an exception object by including information about the error, a continuation for the current control flow state, and a few other pieces of information. Exceptions, like most everything else in Parrot, are PMCs and can be stored, manipulated, and used like other PMCs. Once the exception object is created, Parrot looks through it's list of handlers, and passes it's exception object to each. Handlers are stored in a stack-like structure, and the most recently registered handler will have the first access to the exception object. The handler can do one of several things. First, it can handle the exception: It can fix the error, call the return continuation that is within the exception object, and returns control flow back to where it was before the error occurred. Second, it could re-throw the exception, passing it to the next handler in the stack. Third, it could ignore the exception. Exceptions which are ignored, or for which no handlers are available, cause Parrot to exit.



84



 IO Subsystem



IO Subsystem I/O Subsystem Resources • http://www.parrotcode.org/docs/pdd/pdd22_io.html



JIT and NCI JIT and NCI JIT, the Just in time compiler is a system which compiles Parrot opcodes into native machine code for faster execution. NCI, the native call interface is a subsystem in Parrot that allows it to communicate with compiled library functions. These two systems are very important for Parrot and, as we shall see, they are closely related to one another.



NCI



Parrot Embedding Embedding Parrot Because the Parrot Virtual Machine is modular, we can link the libparrot library into other executables, creating programs which contain a Parrot interpreter object. One simple use for such a technology would be to create an executable file which contains data in the form of precompiled bytecode and a simple instantiation of the Parrot interpreter to create a standalone executable for a particular programming language. This is called Native Execution, and we will discuss it in more detail below.



Resources • http://www.parrotcode.org/docs/pdd/pdd10_embedding.html • http://www.parrotcode.org/docs/native_exec.html



85



 Extensions



Extensions Extending Parrot Parrot is not just an executable VM, it's a dynamically-linkable library that exports a Parrot API. The API allows add-ons to be developed to extend the functionality of Parrot. There are two basic APIs or, more specifically, a single API that can be divided into two distinct categories: Those that have access to the internals of Parrot, and those that do not. In general, most extensions will not need deep internal access to Parrot's structures, and most should not rely on them. Parrot's internal structures are subject to change, and relying on a precise format of one for your extension could cause compatibility problems later on.



Packfiles Packfiles Parrot bytecode files are called "packfiles" internally. Access routines to get information from and feed information to a packfile are stored in /src/packfile.c Some things that can affect the way a file is stored are: 1. Endianness. Some computers are called "little endian" and some computers are called "big endian". This has to do with the way that bits are ordered in a byte. Instead of picking one or the other to be the default. 2. Value sizes. Things like pointers and INTVALs are going to be different sizes on different computers. Parrot must translate between 16 bit, 32 bit, and 64 bit values for these and other things. Also, FLOATVALS may be 32bit, 64bit, or 128bit, and need to be translated.



Serialization HLL code is most often used on the computer where it is first compiled. To that end, Parrot is optimized to write packfiles using local settings. If a packfile is read which has been created on some other computer, Parrot must translate it internally so that it can be run on your computer. This translation process adds extra execution overhead, but it only needs to be run once on your computer to get things into the proper local format.



Freeze and Thaw Serialization occurs through two interfaces: Freeze and thaw. Freezing is the process of converting a PMC or other type of data into a format ready for insertion into a packfile. Thawing is the process of reading data out of a packfile and recreate the PMC or other data object. PMCs have optional Freeze and Thaw vtable methods, although good defaults are available if the PMC does not use any additional storage. If you use additional storage, especially storage which has been allocated from the system using malloc, you must supply custom Freeze/Thaw methods to store and retrieve that.



86



 87



Appendices PIR Reference Resources • http://www.parrotcode.org/docs/pdd/pdd19_pir.html



PASM Reference Resources • http://docs.parrot.org/parrot/latest/html/docs/pdds/draft/pdd06_pasm.pod.html • http://docs.parrot.org/parrot/latest/html/docs/pdds/draft/pdd05_opfunc.pod.html • http://docs.parrot.org/parrot/latest/html/ops.html



PAST Node Reference Resources • http://www.parrotcode.org/docs/pdd/pdd26_ast.html



 Languages on Parrot



88



Languages on Parrot Languages on Parrot There are a number of programming languages being implemented on Parrot, some of which are nearing functional completion, some which are still in active development, and some which have been started but are now abandoned. Interested developers may want to help join in the development effort with some of these languages, adopt an abandoned language project, or start a new language project entirely. As of the 1.0.0 release, all language implementations except toy and example languages will be developed and maintained outside of the central Parrot repository. Where available, locations to external project pages will be provided.



Language Projects Rakudo (Perl 6) Rakudo is the name of the Perl6 implementation on Parrot. This is not the only implementation of Perl6, however.



Rakudo development is test-driven. There is a gigantic suite of tests for the Perl6 language that have been developed over the years. The progress of the Rakudo interpreter is measured by the number of specification tests, or "spectests" that pass. There is not a straight-forward way to measure the percentage progress of the project, because the total number of tests is changing regularly as well. Rakudo is under active development by several volunteers. Some developers have even received funding to work on Rakudo more regularly.



abc A basic calculator language.



C99 The implementation of the C programming language, following the C99 specification, has a number of purposes. C is a strongly-typed language, so it isn't necessarily the best candidate for implementation on the dynamically-typed Parrot. However, there are a number of benefits to the Parrot project to having a C parser available. The C99 language parser is being used, at least in part, to help automate the process of generating NCI function signatures for new libraries and extensions. This is under active development by volunteers, some of which have been funded.



Cardinal (Ruby) An implementation of Ruby



Jako A language derived from C and Perl



Pipp (PHP) Pipp is a recursive acronym for



Pipp



is



Parrot's PHP.



This language implementation was previously named



"Plumhead", shorthand for the name " Plum-headed Parakeet". Pipp is being maintained on github at pipp project seems halted: the website is down/missing and the last commit was at 2009-07-22.



[1]



. The



 Languages on Parrot



Partcl (TCL) ParTCL is the TCL compiler for Parrot. The ParTCL project lives at http:/ /code.google.com/p/partcl/



Translator Projects Projects to translate to or from Parrot Bytecode.



Resources • http://www.parrot.org/languages/



References [1] http:// wiki.github.com/bschmalhofer/pipp



HLLCompiler Class HLLCompiler Class The HLLCompiler class is used to help instantiate and operate a compiler for a high-level language written for Parrot. HLLCompiler coordinates the execution of the HLL grammar, and controls the conversion of the HLL from PAST to POST, from POST to PIR, and ultimately to Parrot bytecode. This page is going to serve as a brief reference to the HLLCompiler class, and a description of how the class is used to create a compiler for an HLL.



89



 Command Line Options



Command Line Options Resources • http://www.parrotcode.org/docs/running.html



Built-In PMCs Built-in PMCs Parrot ships with a number of PMC data types built in. This means that these standard types are always available. This page is going to serve as a reference to these PMC types. We will not attempt to cover all the PMC types that are added specifically for other HLLs, libraries, or programs. (For information on using these PMC types, and on defining new PMC types, see the Parrot Virtual Machine/Polymorphic Containers (PMCs) chapter). The entries in this list should (A) contain a link to the relevant PMC documentation, and (B) provide a brief overview of the PMC and it's methods.



AddrRegistry • http://docs.parrot.org/parrot/latest/html/src/pmc/addrregistry.pmc.html



Array • http://www.parrotcode.org/docs/pmc/pmc/array.html A simple array class, serves as the base class for other array PMCs. This type of PMC is rarely used directly. Instead, more versatile array PMC types, such as ResizablePMCArray are used. Array specifies an interface that all other Array classes must share. It also provides a number of defaults that other array-like PMCs may default to.



BigInt • http://docs.parrot.org/parrot/latest/html/src/pmc/bignum.pmc.html A PMC type for storing an arbitrarily large number, or a number with arbitrary precision. Not currently implemented.



Boolean • http://docs.parrot.org/parrot/latest/html/src/pmc/boolean.pmc.html A boolean True/False PMC.



Bound_NCI • http://www.parrotcode.org/docs/pmc/pmc/bound_nci.html



90



 Built-In PMCs



Capture • http://docs.parrot.org/parrot/latest/html/src/pmc/capture.pmc.html



Closure • http://www.parrotcode.org/docs/pmc/pmc/closure.html



Compiler • http://www.parrotcode.org/docs/pmc/pmc/compiler.html A Compiler PMC for a particular language. Can be used to convert an HLL into PIR and eventually into Parrot Bytecode.



Complex • http://www.parrotcode.org/docs/pmc/pmc/complex.html A PMC for Complex numbers.



Continuation • http://www.parrotcode.org/docs/pmc/pmc/continuation.html A Continuation PMC allows Parrot to take a snapshot of the current state of the system to return to later.



Coroutine • http://www.parrotcode.org/docs/pmc/pmc/coroutine.html A sub-like PMC that implements a coroutine.



Default • http://www.parrotcode.org/docs/pmc/pmc/default.html



Deleg_PMC • http://www.parrotcode.org/docs/pmc/pmc/deleg_pmc.html



Delegate • http://www.parrotcode.org/docs/pmc/pmc/delegate .html



Enumerate • http://www.parrotcode.org/docs/pmc/pmc/enumerate.html



Env • http://www.parrotcode.org/docs/pmc/pmc/env.html Allows access to the system's environment variables, as a hash.



91



 Built-In PMCs



Eval • http://www.parrotcode.org/docs/pmc/pmc/eval.html



Exception • http://www.parrotcode.org/docs/pmc/pmc/exception.html An Exception PMC holds information about system errors for recovery.



Exception_Handler • http://www.parrotcode.org/docs/pmc/pmc/exception_handler.html A sub-like routine that catches and resolves exceptions



Exporter • http://www.parrotcode.org/docs/pmc/pmc/exporter.html



File • http://docs.parrot.org/parrot/latest/html/src/dynpmc/file.pmc.html A read/write interface for files



FixedBooleanArray • http://www.parrotcode.org/docs/pmc/pmc/fixedbooleanarray.html An array of fixed size of Boolean values.



FixedFloatArray • http://www.parrotcode.org/docs/pmc/pmc/fixedfloatarray.html An array of fixed size for FLOATVAL floating point numbers



FixedPMCArray • http://www.parrotcode.org/docs/pmc/pmc/fixedpmcarray.html An array of fixed size for PMC values



FixedStringArray • http://www.parrotcode.org/docs/pmc/pmc/fixedstringarray.html An array of fixed size for STRING values



92



 Built-In PMCs



Float • http://www.parrotcode.org/docs/pmc/pmc/float.html A floating point number PMC. Used similarly to a FLOATVAL, except has methods and vtable methods. FLOATVALs become Float PMCs when they are promoted to become a PMC.



Hash • http://www.parrotcode.org/docs/pmc/pmc/hash.html A hash, also known as a "dictionary" or an "associative array". Like an array but indexed with strings instead of integers



Integer • http://www.parrotcode.org/docs/pmc/pmc/integer.html A basic integer number PMC. Used similarly to an INTVAL, but with methods and vtable methods. INTVALS become Integer PMCs when they are promoted to become a PMC.



IntList • http://www.parrotcode.org/docs/pmc/pmc/intlist.html A simple list, or array, of integers.



Iterator • http://www.parrotcode.org/docs/pmc/pmc/iterator.html An Iterator PMC provides a stateful counter that enables you to iterate over the items in one of the array classes, one at a time.



Key • http://www.parrotcode.org/docs/pmc/pmc/key.html A value, typically a string, which is used to look up values in a hash.



LexInfo • http://www.parrotcode.org/docs/pmc/pmc/lexinfo.html



LexPad • http://www.parrotcode.org/docs/pmc/pmc/lexpad.html



ManagedStruct • http://www.parrotcode.org/docs/pmc/pmc/managedstruct.html A low-level structure whose memory is allocated and automatically deallocated by Parrot. Extends UnManagedStruct, but adds automatic memory collection.



93



 Built-In PMCs



MultiArray • http://www.parrotcode.org/docs/pmc/pmc/multiarray.html



MultiSub • http://www.parrotcode.org/docs/pmc/pmc/multisub.html A collection of subroutines with the same name. In Multiple Method Dispatch (MMD) the parameters of the function called determine which subroutine from the collection to call.



Namespace • http://www.parrotcode.org/docs/pmc/pmc/namespace.html Implements a Parrot namespace. Contains information about variables, subroutines, coroutines, and MultiSubs that are stored in that namespace.



NCI • http://www.parrotcode.org/docs/pmc/pmc/nci.html A native call function PMC. Stores interface information to a function which has been written in C.



Null • http://www.parrotcode.org/docs/pmc/pmc/null.html A PMC with a NUL value



Object • http://www.parrotcode.org/docs/pmc/pmc/object.html



OrderedHash • http://www.parrotcode.org/docs/pmc/pmc/orderedhash.html



OS • http://www.parrotcode.org/docs/pmc/pmc/os.html



Pair • http://www.parrotcode.org/docs/pmc/pmc/pair.html An association of a Key PMC with a PMC value. Hashes are typically implemented as an array of Pair PMCs



ParrotClass • http://www.parrotcode.org/docs/pmc/pmc/parrotclass.html



94



 Built-In PMCs



ParrotInterpreter • http://www.parrotcode.org/docs/pmc/pmc/parrotinterpreter.html An interface to the interpreter structure.



ParrotIO • http://www.parrotcode.org/docs/pmc/pmc/parrotio.html A read/write interface to the console



ParrotLibrary • http://www.parrotcode.org/docs/pmc/pmc/parrotlibrary.html A dynamically-loaded library object.



ParrotObject • http://www.parrotcode.org/docs/pmc/pmc/parrotobject.html



ParrotRunningThread • http://www.parrotcode.org/docs/pmc/pmc/parrotrunningthread.html



ParrotThread • http://www.parrotcode.org/docs/pmc/pmc/parrotthread.html A PMC that stores information about a thread



Pmethod_test • http://www.parrotcode.org/docs/pmc/pmc/pmethod_test.html



Pointer • http://www.parrotcode.org/docs/pmc/pmc/pointer.html



Random • http://www.parrotcode.org/docs/pmc/pmc/random.html



Ref • http://www.parrotcode.org/docs/pmc/pmc/ref.html



ResizableBooleanArray • http://www.parrotcode.org/docs/pmc/pmc/resizablebooleanarray.html A resizable array to store Boolean values



95



 Built-In PMCs



ResizableFloatArray • http://www.parrotcode.org/docs/pmc/pmc/resizablefloatarray.html A resizable array to store floating point values.



ResizableIntegerArray • http://www.parrotcode.org/docs/pmc/pmc/resizableintegerarray.html A resizable array to store integer values



ResizablePMCArray • http://www.parrotcode.org/docs/pmc/pmc/resizablepmcarray.html A resizable array to store PMC values



ResizableStringArray • http://www.parrotcode.org/docs/pmc/pmc/resizablestringarray.html A resizable array to store Strings



RetContinuation • http://www.parrotcode.org/docs/pmc/pmc/retcontinuation.html A return continuation. Like a regular Continuation PMC, but can only be used once. Can be promoted to a Continuation using the Clone vtable method.



Role • http://www.parrotcode.org/docs/pmc/pmc/role.html An abstract role, or interface, for a class. Specifies actions and properties of a class, but cannot be instantiated



SArray • http://www.parrotcode.org/docs/pmc/pmc/sarray.html



SharedRef • http://www.parrotcode.org/docs/pmc/pmc/sharedref.html



Slice • http://www.parrotcode.org/docs/pmc/pmc/slice.html



96



 Built-In PMCs



SMOP_Attribute • http://www.parrotcode.org/docs/pmc/pmc/smop_attribute.html



SMOP_Class • http://www.parrotcode.org/docs/pmc/pmc/smop_class.html



STMLog • http://www.parrotcode.org/docs/pmc/pmc/stmlog.html



STMRef • http://www.parrotcode.org/docs/pmc/pmc/stmref.html



STMVar • http://www.parrotcode.org/docs/pmc/pmc/stmvar.html



String • http://www.parrotcode.org/docs/pmc/pmc/string.html A PMC to contain a string value. Like a STRING value, but has methods and vtable methods. STRINGS become String PMCs when they are promoted to PMCs.



Sub • http://www.parrotcode.org/docs/pmc/pmc/sub.html A Parrot subroutine. Implements a basic subroutine (using the sub command in PIR), but also serves as a base class for more intricate sub-like classes



Super • http://www.parrotcode.org/docs/pmc/pmc/super.html A parent PMC class, to support multiple inheritance.



Timer • http://www.parrotcode.org/docs/pmc/pmc/timer.html



TQueue • http://www.parrotcode.org/docs/pmc/pmc/tqueue.html



Undef



• http://www.parrotcode.org/docs/pmc/pmc/undef.html An undefined PMC with no usable type.



97



 Built-In PMCs



98



UnamangedStruct • http://www.parrotcode.org/docs/pmc/pmc/unamangedstruct.html A low-level structure which the programmer must manage manually. Parrot does not automatically collect memory allocated for the struct.



Version • http://www.parrotcode.org/docs/pmc/pmc/version.html



Bytecode File Format Packfile Header Format Packfiles start with a standard header that give some information about how Parrot was configured when it wrote the file. This configuration information is used to determine how the reading Parrot must translate the incoming packfile.



Resources • http://www.parrotcode.org/docs/parrotbyte.html



VTABLE List Vtable List Vt abnlea me absolute add_attribute



D e s c r i p t i on



Returns the absolute value of the PMC, as a PMC Adds an attribute to the PMC object. Attributes are typically stored in thepmc->pmc_ext->_metadata field.



add_method



Adds a new method to the PMC's class



add_parent add_role add_vtable_override assign_pmc



Assigns a PMC value to the PMC



assign_string_native



Assigns a string to the PMC



bitwise_not bitwise_nots can clone clone_pmc decrement



Decrements the integer value of the PMC by 1



defined



Determines if the PMC is defined



defined_keyed defined_keyed_int



 VTABLE List



99



defined_keyed_str delprop



Deletes a property from the PMC



destroy



Destroy the PMC



does does_pmc elements exists_keyed exists_keyed_int exists_keyed_str find_method freeze get_attr get_bignum



Gets the BigNum representation of the PMC



get_bool



Gets the boolean representation of the PMC



get_class get_integer



Gets the integer representation of the PMC



get_integer_keyed get_integer_keyed_int get_integer_keyed_str get_iter get_namespace get_number



Gets the floating-point representation of the PMC



get_number_keyed get_number_keyed_int get_number_keyed_str get_pmc



Gets the PMC representation of the PMC



get_pmc_keyed get_pmc_keyed_int get_pmc_keyed_str get_pointer get_pointer_keyed get_pointer_keyed_int get_pointer_keyed_str get_repr get_string



Gets the string representation of the PMC



get_string_keyed get_string_keyed_int get_string_keyed_srt getprop



Get the value of a certain property from the PMC



 VTABLE List



100



getprops i_absolute i_bitwise_not i_bitwise_nots i_logical_not i_net increment



Increments the integer value of the PMC by 1



init



Initializes the PMC. This method is called when the new keyword is used to create a new PMC.



init_pmc inspect inspect_str instantiate invoke



The invoke vtable method is called when the PMC is called like a function. In the following code: .local pmc mypmc = new 'MyPMCType' mypmc()



The invoke vtable method is called on the second line when the PMC is treated as a function call. For string functions, for example, the string class uses the value of the string to look up a f unction with the same name, and then invokes that function. Subroutine PMCs invoke the given function when they are called. is_same isa isa_pmc logical_not mark



Mark the PMC and all it's children as alive for the memory manager. This prevents children of the PMC from being collected prematurely by the garbage collector.



morph name neg new_from_string nextkey_keyed nextkey_keyed_int nextkey_keyed_str pop_float



If the PMC is an array, pops a floating point value off the top of it



pop_integer



If the PMC is an array, pops an integer value off the top of it



pop_pmc



If the PMC is an array, pops a PMC value off the top of it



pop_string



If the PMC is an array, pops a string value off the top of it



push_float



If the PMC is an array, pushes a floating point value onto the top of it



push_integer



If the PMC is an array, pushes an integer onto the top of it



push_pmc



If the PMC is an array, pushes a PMC onto the top of it



push_string



If the PMC is an array, pushes a string onto the top of it



remove_attribute



Removes an attribute from the PMC



 VTABLE List



101



remove_method remove_parent remove_role remove_vtable_override set_attr



Sets an attribute value for the given PMC



set_attr_keyed set_attr_keyed_str set_bugnum_int set_bignum_num set_bignum_str set_bool



Sets the value of the PMC as if it were a boolean



set_integer_keyed set_integer_keyed_int set_integer_keyed_str set_integer_native



Sets the value of the PMC as if it were an integer



set_number_keyed set_number_keyed_int set_number_keyed_str set_number_native



Sets the value of the PMC as if it were a floating point value



set_number_same set_pmc



Sets the value of a PMC to the value of another PMC



set_pmc_keyed set_pmc_keyed_int set_pmc_keyed_str set_pointer set_pointer_keyed set_pointer_keyed_int set_pointer_keyed_str set_string_keyed set_string_keyed_int set_string_keyed_str set_string_native



Sets the value of the PMC as if it were a string



set_string_same setprop share share_ro shift_float



If the PMC is an array, shifts a f loating point value onto the bottom of it



shift_int



If the PMC is an array, shifts an integer onto the bottom of it



shift_pmc



If the PMC is an array, shifts a PMC onto the bottom of it



 VTABLE List



shift_string



102 If the PMC is an array, shifts a string onto the bottom of it



slice splice substr substr_str thaw thawfinish type type_keyed type_keyed_int type_keyed_str unshift_float



If the PMC is an array, unshifts a floating point valuefrom the bottom of it



unshift_integer



If the PMC is an array, unshifts an integer from the bottom of it



unshift_pmc



If the PMC is an array, unshifts a PMC from the bottom of it



unshift_str



If the PMC is an array, unshifts a string from the bottom of it



visit



 103



"Squaak" Language Tutorial Squaak Tutorial Squaak Language Tutorial Episode 1: Introduction Episode 2: Poking in Compiler Guts Episode 3: Squaak Details and First Steps Episode 4: PAST Nodes and More Statements Episode 5: Variable Declaration and Scope Episode 6: Scope and Subroutines Episode 7: Operators and Precedence Episode 8: Hashtables and Arrays Episode 9: Wrap-Up and Conclusion



This tutorial for the toy language "Squaak" was first presented on http:/ / www. parrotblog. org. The tutorials have been released into the public domain by the author, and so have been adapted for use here on Wikibooks. The initial upload revisions for each of these pages are released into the public domain. Subsequent edits made to these pages are released under the GFDL. File names used in these pages are relative to the Parrot source distribution. "PDD" are the Parrot Design Documents, a series of explanatory documents which are located in POD format in the Parrot repository, or on the



http:/ / www. parrot. org website. For more information about downloading the Parrot repository, see the chapter on Building Parrot. These pages have been changed as necessary to fit Wikibooks, and are being improved like any other book or module. If you appreciate this tutorial and would like to leave comments for its srcinal author, you can do so at http://www.parrotblog.org. Notice Since this tutorial has been released, a number of changes have been made to the underlying tools: PCT, PGE, and NQP. Because of this, the code discussed in the tutorial might not be accurate and will likely not operate correctly anymore. These pages will be updated eventually to reflect these changes, but they are not currently up to date.



 Introduction



104



Introduction Squaak Language Tutorial Episode 1: Introduction Episode 2: Poking in Compiler Guts Episode 3: Squaak Details and First Steps Episode 4: PAST Nodes and More Statements Episode 5: Variable Declaration and Scope Episode 6: Scope and Subroutines Episode 7: Operators and Precedence Episode 8: Hashtables and Arrays Episode 9: Wrap-Up and Conclusion



Introduction This is the first episode in a tutorial series on building a compiler with the Parrot Compiler Tools. If you're interested in virtual machines, you've probably heard of the Parrot virtual machine. Parrot is a generic virtual machine designed for dynamic languages. This is in contrast with the Java virtual machine (JVM) and Microsoft's Common Language Runtime (CLR), both of which were designed to run static languages. Both the JVM and Microsoft (through the Dynamic Language Runtime -- DLR) are adding support for dynamic languages, but their primary focus is still static languages.



High Level Languages The main purpose of a virtual machine is to run programs. These programs are typically written in some High Level Language (HLL). Some well-known dynamic languages (sometimes referred to as scripting languages) are Lua, Perl, PHP, Python, Ruby, and Tcl. Parrot is designed to be able to run all these languages. Each language that Parrot hosts needs a compiler to parse the syntax of the language and generate Parrot instructions. If you've never implemented a programming language (and maybe even if you have implemented a language), you might consider writing a compiler a bit of a black art. I know I did when I became interested. And you know what? It is. Compilers are complex programs, and implementing a language can be very difficult. The Facts: 1) Parrot is suitable for running virtually any dynamic language known, but before doing so, compilers must be written, and 2) writing compilers is rather difficult.



The Parrot Compiler Toolkit Enter the Parrot Compiler Toolkit (PCT). In order to make Parrot an interesting target for language developers, the process of constructing a compiler should be supported by the right tools. Just as any construction task becomes much easier if you have the right tools (you wouldn't build a house using only your bare hands, would you?), the same is true for constructing a compiler. The PCT was designed to do just that: provide powerful tools to make writing a compiler for Parrot childishly easy. This tutorial will introduce the PCT by demonstrating the ease with which a (simple) language can be implemented for Parrot. The case study language is not as complex as a real-world language, but this tutorial is written to whet your appetite and show the power of the PCT. This tutorial will also present some exercises which you can explore in order to learn more details of the PCT not covered in this tutorial.



 Introduction



Squaak: A Simple Language The case study language, named Squaak, that we will be implementing on Parrot will be a full-fledged compiler that can compile a program from source into Parrot Intermediate Representation (PIR) (or run the PIR immediately). It can also be used as a command-line interpreter. Squaak demonstrates some common language constructs, but at the same time is lacking some other, seemingly simple features. For instance, our language will not have return, break or continue statements (or equivalents in your favorite syntax). Squaak will have the following features: • global and local variables • basic types: integer, floating-point and strings • aggregate types: arrays and hash tables • operators: +, -, /, *, %, <, <=, >, >=, ==, !=, .., and, or, not • subroutines and pa rameters • assignments and various control statements, such as "if" and "while" As you can see, a number of common (more advanced) features are missing. Most notable are: • classes and objects • exceptional control statements such as break and return • advanced control statements such as switch • closures (nested subroutines and accessing local variables in an outer scope)



The Compiler Tools The Parrot Compiler Tools we'll use to implement Squaak consist of the following parts: • Parrot Grammar Engine (PGE). The PGE is an advanc ed engine for reg ular expressions. Besides regexes as found in Perl 5, it can also be used to define language grammars, using Perl 6 syntax. (Check the references for the specification.) • Parrot Abstract Syntax Tree (PAST). The PAST nodes are a set of clas ses defining generic abstract syntax tree nodes that represent common language constructs. • HLLCompiler class. This class is the compiler driver for any PCT-based compiler. • Not Quite Perl (6) (NQP). NQP is a lightweight language inspired by Perl 6 and c an be used to write the methods that must be executed during the parsing phase, just as you can write actions in a Yacc/Bison input file.



Getting Started For this tutorial, it is assumed you have successfully compiled parrot (and maybe even run the test suite). If you browse through the languages directory in the Parrot source tree, you'll find a number of language implementations. Most of them are not complete yet; some are actively maintained actively and others aren't. If, after reading this tutorial, you feel like contributing to one of these languages, you can check out the mailing list or join IRC (see the references section for details). The languages subdirectory is the right spot to put our language implementation. Parrot comes with a special Perl5 script to generate the necessary files for a language implementation. Before it can be run, parrot must be installed for development. From parrot's root directory, type: $ make install-dev



In order to generate a new directory containing files for our language, type (assuming you're in parrot's root directory): $ perl tools/dev/mk_language_shell.pl Squaak



105



 Introduction (Note: if you're on Windows, you should use backslashes.) This will generate the files in a directory "squaak", and use the name "Squaak" as the language's name. After this, go to this directory and type: $ parrot setup.pir test



This will compile the generated files and run the test suite. If you want more information on what files are being generated, please check out the references at the end of this episode. Note that we didn't write a single line of code, and already we have the basic infrastructure in place to get us started. Of course, the generated compiler doesn't even look like the language we will be implementing, but that's ok for now. Later we'll adapt the grammar to accept our language. Now you might want to actually run a simple script with this compiler. Launch your favorite editor, and put in this statement: say "Squaak!";



Save the file (for instance as test.sq) and type: $ parrot squaak.pbc test.sq



This will run Parrot, specifying squaak.pbc as the file to be run by Parrot, which takes a single argument: the file test.sq. If all went well, you should see the following output: $ parrot squaak.pbc test.sq Squaak!



Instead of running a script file, you can also run the Squaak compiler as an interactive interpreter. Run the Squaak compiler without specifying a script file, and type the same statement as you wrote in the file: $ parrot squaak.pbc say "Squaak!";



which will print: Squaak!



What's Next? This first episode of this tutorial is mainly an overview of what will be coming. Hopefully you now have a global idea of what the Parrot Compiler Tools are, and how they can be used to build a compiler targeting Parrot. If you want to check out some serious usage of the PCT, check out Rakudo (Perl 6 on Parrot) or Pynie (Python on Parrot). The next episodes will focus on the step-by-step implementation of our language, including the following topics: • structure of PCT-based compilers • using PGE rules to define the language grammar • implementing operator precedence using an operator precedence table • using NQP to write embedded parse actions • implementing language library routines In the mean time, experiment for yourself. You are welcome to join us on IRC (see the References section for details). Any feedback on this tutorial is appreciated.



106



 Introduction



107



Exercises The exercises are provided at the end of each episode of this tutorial. In order to keep the length of this tutorial somewhat acceptable, not everything can be discussed in full detail. The answers and/or solutions to these exercises will be posted several days after the episode. Problem 1 Advanced Interactive Mode Launch your favorite editor and look at the file src/Squaak/Compiler.pm, still in the directory squaak. This file is written in Not Quite Perl ("NQP") and is compiled to src/gen_compiler.pir when you run parrot setup.pir test. It contains the setup for the compiler. The class HLL::Compiler defines methods to set a command-line banner and prompt for your compiler when it is running in interactive mode. For instance, when you run Python in interactive mode, you'll see: Python 2.6.5 (r265:79063, Apr



1 2010, 05:28:39)



[GCC 4.4.3 20100316 (prerelease)] on linux2 Type "help", "copyright", "credits" or "license" for more information.



or something similar (depending on your Python installation and version). This text is called the command line banner. And while running in interactive mode, each line will start with: >>>



which is called a prompt. For Squaak, we'd like to see the following when running in interactive mode (of course you can change this according to your personal taste): $ ../../parrot squaak.pbc Squaak for Parrot VM. >



Add code to the file Compiler.pm to achieve this. • Hint: Note that only double-quoted strings in NQP can interp ret escape-characters such as '\n'. Answer Given the hints that were provided, it was probably not too hard to find the solution, which is shown below. This code can be found in the file Compiler.pm. The relevant lines are printed in bold. INIT { Squaak::Compiler.language('Squaak'); Squaak::Compiler.parsegrammar(Squaak::Grammar); Squaak::Compiler.parseactions(Squaak::Actions);



Squaak::Compiler.commandline_banner("Squaak for Parrot VM.\n"); Squaak::Compiler.commandline_prompt('> '); }



There are more options provided by the HLL::Compiler class that you can set, you should explore them all.



 Introduction



108



References • Parrot mailing list: [email protected] • IRC: join #parrot on irc.perl.org • Getting started with PCT: docs/pct/gettingstarted.pod • Parrot Abstract Syntax Tree (PAST): docs/pct/past_building_blocks.pod • Operator Precedence Parsing with PCT: docs/pct/pct_optable_guide.pod • Perl 6/PGE rules syntax: Synopsis 5



[1]



Poking in Compiler Guts



Squaak Language Tutorial



Episode 1: Introduction Episode 2: Poking in Compiler Guts Episode 3: Squaak Details and First Steps Episode 4: PAST Nodes and More Statements Episode 5: Variable Declaration and Scope Episode 6: Scope and Subroutines Episode 7: Operators and Precedence Episode 8: Hashtables and Arrays Episode 9: Wrap-Up and Conclusion



In the first episode, we introduced the Parrot Compiler Tools, and generated a very simple language using a shell script provided with the Parrot distribution. We also announced Squaak, a simple programming language developed specially for this tutorial. Squaak will be the case study to show how PCT can be used as a very effective set of tools to implement a language for Parrot. A list of features of Squaak was specified. If you felt lucky, you might even have tried to do the exercise at the end of the previous episode. In this episode, we will take a closer look at the generated compiler. We shall check out the different stages of the compilation process, and show what's going on in PCT-based compilers.



Under The Hood Remember how we invoked our compiler in the previous episode? We can pass a file, or invoke the compiler without a command line argument, in which case our compiler enters the interactive mode. Consider the first case, passing the file test.sq, just as we did before: $ parrot squaak.pbc test.sq



When invoking our compiler like this, the file test.sq is compiled and the generated code (bytecode) is executed immediately by Parrot. How does this work, you might wonder. The interpretation of a script is done through a series of transformations, starting at the script source and ending in a format that can be executed by Parrot. Compilers built with the PCT (based on the HLL::Compiler class) can take a target option, to show one of the intermediate representations. This option can have the following values, corresponding to the four default compilation phases of an HLLCompiler object: • --target=parse • --target=past • --target=post • --target=pir



 Poking in Compiler Guts



109



This is an example of using the target option set to "parse", which will print the parse tree of the input to stdout: $ parrot squaak.pbc --target=parse test.sq



In interactive mode, giving this input: say 42;



will print this parse tree (without the line numbers): 1 "parse" => PMC 'Regex;Match' => "say 42;\n" @ 0 { 2



 => PMC 'Regex;Match' => "say 42;\n" @ 0 {



3



 => ResizablePMCArray (size:1) [



4



PMC 'Regex;Match' => "say 42" @ 0 {



5



 => PMC 'Regex;Match' => "say 42" @ 0 {



6



 => ResizablePMCArray (size:1) [



7



PMC 'Regex;Match' => "42" @ 4 {



8



 => PMC 'Regex;Match' => "42" @ 4 {



9



 => PMC 'Regex;Match' => "42" @ 4



10



 => \parse[0][0]



11



}



12



}



13



]



14



 => PMC 'Regex;Match' => "say" @ 0



15



}



16



}



17 18



] }



19 }



When changing the value of the target option, the output changes into a different representation of the input. Why don't you try that right now? So, a HLL::Compiler object has four compilation phases: parsing, construction of a Parrot Abstract Syntax Tree (PAST), construction of a Parrot Opcode Syntax Tree (POST), generation of Parrot Intermediate Representation (PIR). After compilation, the generated PIR is executed immediately. If your compiler needs additional stages, you can add them to your HLL::Compiler object. For Squaak, we will not need this, but for details, check out compilers/pct/src/PCT/HLLCompiler.pir. We shall now discuss each compilation phase in more detail. The first two phases, parsing the input and construction of the PAST are executed simultaneously. Therefore, these are discussed together.



 Poking in Compiler Guts



Parse phase: match objects and PAST construction During the parsing phase, the input is analyzed using Perl 6's extended regular expressions, known as Rules (see Synopsis 5 for details). When a rule matches some input string, a so-called Match object is created. A Match object is a combined array and hashtable, implying it can be indexed by integers as well as strings. As rules typically consist of other (sub) rules, it is easy to retrieve a certain part of the match. For instance, this rule: rule if_statement { 'if'  'then'  'end' {*} }



has two other subrules: expression and statement. The match object for the rule if_statement represents the whole string from if to end. When you're interested only in the expression or statement part, you can retrieve that by indexing the match object by the name of the subrule (in this case, expression and statement, respectively). During the parse phase, the PAST is constructed. There is a small set of PAST node types, for instance, PAST::Var to represent variables (identifiers, such as print), PAST::Val to represent literal values (for instance, "hello" and 42), and so on. Later we shall discuss the various PAST nodes in more detail. Now, you might wonder, at which point exactly is this PAST construction happening? This is where the special {*} symbol comes in, just below the string 'if' in the if_statement rule shown above. These special markers indicate that a parse action should be invoked. Such a parse action is just a method that has the same name as the rule in which it is written (in this case: if_statement). So, during the parsing phase, several parse actions are executed, each of which builds a piece of the total PAST representing the input string. More on this will be explained later. The Parrot Abstract Syntax Tree is just a different representation of the same input string (your program being compiled). It is a convenient data structure to transform into something different (such as executable Parrot code) but also to do all sorts of analysis, such as compile-time type checking.



PAST to POST After the parse phase during which the PAST is constructed, the HLL::Compiler transforms this PAST into something called a Parrot Opcode Syntax Tree (POST). The POST representation is also a tree structure, but these nodes are on a lower abstraction level. For instance, on the PAST level there is a node type to represent a while statement (constructed as PAST::Op.new( :pasttype('while') ) ). The template for a while statement typically consists of a number of labels and jump instructions. On the POST level, the same while statement is represented by a set of nodes, each representing a one instruction or a label. Therefore, it is much easier to transform a POST into something executable than when this is done from the PAST level. Usually, as a user of the PCT, you don't need to know details of POST nodes, which is why this will not be discussed in further detail. Use the target option to see what a POST looks like.



110



 Poking in Compiler Guts



POST to PIR In the fourth (and final) stage, the POST is transformed into Parrot Intermediate Representation (PIR). As mentioned, transforming a POST into something executable is rather straightforward, as POST nodes already represent individual instructions and labels. Again, normal usage of the PCT does not require you to know any details about this transformation.



And now for the good news... We established the general data flow of PCT-based compilers, which consists of four stages: 1. source to parse tree 2. parse tree to PAST 3. PAST to POST 4. POST to PIR where we noted that the first two are done during the parse stage. Now, as you're reading this tutorial, you're probably interested in using the PCT for implementing Your Favorite Language for Parrot. We already saw that a language grammar is expressed in Perl 6 Rules. What about the other transformations? Well, earlier in this episode we mentioned the term parse actions, and that these actions create PAST nodes. After you have written a parse action for each grammar rule, you're done!



Say what? That's right. Once you have correctly constructed a PAST, your compiler can generate executable PIR, which means you just implemented your first language for Parrot. Of course, you still need to implement any language specific libraries, but that's besides the point. PCT-based compilers already know how to transform a PAST into a POST, and how to transform a POST into PIR. These transformation stages are already provided by the PCT.



What's next? In this episode we took a closer look at the internals of a PCT-based compiler. We discussed the four compilation stages, that transform an input string (a program, or script, depending on your definition) into a PAST, a POST and finally executable PIR. The next episodes is where the Fun Stuff is: we will be implementing Squaak for Parrot. Piece by piece, we will implement the parser and the parse actions. Finally, we shall demonstrate John Conway's "Game of Life" running on Parrot, implemented in Squaak.



Exercises Starting in the next episode, the exercises will be more interesting. For now, it would be useful to browse around through the source files of the compiler, and see if you understand the relation between the grammar rules in Grammar.pg and the methods in Actions.pm. It's also useful to experiment with the --target option described in this episode. If you don't know PIR, now is the time to do some preparation for that. There's sufficient information to be found on PIR, see the References section for details. In the mean time, if you have any suggestions, questions and whatnot, don't hesitate to leave a comment.



111



 Poking in Compiler Guts



112



References 1. PIR language specification: docs/pdds/pdd19_pir.pod 2. PIR book: docs/book/pir



Squaak Details and First Steps Squaak Language Tutorial Episode 1: Introduction Episode 2: Poking in Compiler Guts Episode 3: Squaak Details and First Steps Episode 4: PAST Nodes and More Statements Episode 5: Variable Declaration and Scope Episode 6: Scope and Subroutines Episode 7: Operators and Precedence Episode 8: Hashtables and Arrays Episode 9: Wrap-Up and Conclusion



In the previous episodes we introduced the Parrot Compiler Tools (PCT). Starting from a high-level overview, we quickly created our own little scripting language called Squaak, using a Perl script provided with Parrot. We discussed the general structure of PCT-based compilers, and each of the default four transformation phases. This third episode is where the Fun begins. In this episode, we shall introduce the full specification of Squaak. In this and following episodes, we will implement this specification step by step, in small increments that are easy to digest. Once you get a feel for it, you'll notice implementing Squaak is almost trivial, and most important, a lot of fun! So, let's get started!



Squaak Grammar Without further ado, here is the full grammar specification for Squaak. This specification uses the following meta-syntax: statement



indicates a non-terminal, named "statement"



{statement} indicates zero or more statements [step]



indicates an optional step



'do'



indicates the keyword 'do'



Below is Squaak's grammar. The start symbol is program. program



::= {stat-or-def}



stat-or-def



::= statement | sub-definition



statement



::= if-statement | while-statement | for-statement | try-statement | throw-statement | variable-declaration



 Squaak Details and First Steps



113 | assignment | sub-call | do-block



block



::= {statement}



do-block



::= 'do' block 'end'



if-statement



::= 'if' expression 'then' block ['else' block] 'end'



while-statement



::= 'while' expression 'do' block 'end'



for-statement



::= 'for' for-init ',' expression [step] 'do' block 'end'



step



::= ',' expression



for-init



::= 'var' identifier '=' expression



try-statement



::= 'try' block 'catch' identifier block 'end'



throw-statement



::= 'throw' expression



sub-definition



::= 'sub' identifier parameters block 'end'



parameters



::= '(' [identifier {',' identifier}] ')'



variable-declaration ::= 'var' identifier ['=' expression] assignment



::= primary '=' expression



sub-call



::= primary arguments



primary



::= identifier postfix-expression*



postfix-expression



::= key | index | member



 Squaak Details and First Steps



114



key



::= '{' expression '}'



index



::= '[' expression ']'



member



::= '.' identifier



arguments



::= '(' [expression {',' expression}] ')'



expression



::= expression {binary-op expression} | unary-op expression | '(' expression ')' | term



term



::= float-constant | integer-constant | string-constant | array-constructor | hash-constructor | primary



hash-constructor



::= '{' [named-field {',' named-field}] '}'



named-field



::= string-constant '=>' expression



array-constructor



::= '[' [expression {',' expression} ] ']'



binary-op



::= '+'



| '-'



| '/'



| 'and | 'or' | '>'



| '%'



| '..'



| '>=' | '<'



| '*'



| '<='



| '==' | '!=' unary-op



::= 'not' | '-'



Gee, that's a lot, isn't it? Actually, this grammar is rather small compared to "real world" languages such as C, not to mention Perl 6. No worries though, we won't implement the whole thing at once, but in small steps. What's more, the exercises section contains enough exercises for you to learn to use the PCT yourself! The solutions to these exercises will be posted a few days later (but you really only need a couple of hours to figure them out).



 Squaak Details and First Steps



Semantics Most of the Squaak language is straightforward; the if-statement executes exactly as you would expect. When we discuss a grammar rule (for its implementation), a semantic specification will be included. This is to prevent myself from writing a complete language manual, which could take some pages.



Interactive Squaak Although the Squaak compiler can be used in interactive mode, there is one point of attention to be noted. When defining a local variable using the 'var' keyword, this variable will be lost in any consecutive commands. The variable will only be available to other statements within the same command (a command is a set of statements before you press enter). This has to do with the code generation by the PCT, and will be fixed at a later point. For now, just remember it doesn't work.



Let's get started! In the rest of this episode we will implement the basic parts of the grammar, such as the basic data types and assignments. At the end of this episode, you'll be able to assign simple values to (global) variables. It ain't much, but it's a very important first step. Once these basics are in place, you'll notice that adding a certain syntactic construct becomes a matter of minutes. First, open your editor and open the files src/Squaak/Grammar.pm and src/Squaak/Actions.pm. The former implements the parser using Perl 6 rules, and the latter contains the parse actions, which are executed during the parsing stage. In the file Grammar.pm, you'll see the top-level rule, named "TOP". It's located at, ehm... the top. When the parser is invoked, it will start at this rule (a rule is nothing else than a method of the grammar class). When we generated this language (in the first episode), some default rules were defined. Now we're going to make some small changes, just enough to get us started. Firstly, change the statement rule to this: rule statement {  {*} }



and add these rules: rule assignment {  '='  {*} } rule primary {  {*} } token identifier {   {*} }



115



 Squaak Details and First Steps



116



token keyword { ['and'|'catch'|'do'



|'else' |'end' |'for' |'if'



|'not'|'or'



|'throw'|'try' |'var'|'while']>>



|'sub'



}



Now, change the rule "value" into this (renaming to "expression"): rule expression { |  {*}



#= string_constant



|  {*}



#= integer_constant



}



Rename the rule "integer" as "integer_constant", and "quote" as "string_constant" (to better match our language specification). Phew, that was a lot of information! Let's have a closer look at some things that may look unfamiliar. The first new thing is in the rule "identifier". Instead of the "rule" keyword, you see the keyword "token". In short, a token doesn't skip whitespace between the different parts specified in the token, while a rule does. For now, it's enough to remember to use a token if you want to match a string that doesn't contain any whitespace (such as literal constants and identifiers), and use a rule if your string does (and should) contain whitespace (such as a an if-statement). We shall use the word "rule" in a general sense, which could refer to a token. For more information on rules and tokens (and there's a third type, called "regex"), take a look at synopsis 5. In token "identifier", the first subrule is called an assertion. It asserts that an "identifier" does not match the rule keyword. In other words, a keyword cannot be used as an identifier. The second subrule is called "ident", which is a built-in rule in the class PCT::Grammar, of which this grammar is a subclass. In token "keyword", all keywords of Squaak are listed. At the end there's a ">>" marker, which indicates a word boundary. Without this marker, an identifier such as "forloop" would wrongly be disqualified, because the part "for" would match the rule keyword, and the part "loop" would match the rule "ident". However, as the assertion  is false (as "for" could be matched), the string "forloop" cannot be matched as an identifier. The required presence of the word boundary prevents this. The last rule is "expression". An expression is either a string-constant or an integer-constant. Either way, an action is executed. However, when the action is executed, it does not know what the parser matched; was it a string-constant, or an integer-constant? Of course, the match object can be checked, but consider the case where you have 10 alternatives, then doing 9 checks only to find out the last alternative was matched is somewhat inefficient (and adding new alternatives requires you to update this check). That's why you see the special comments starting with a "#=" character. Using this notation, you can specify a key, which will be passed as a second argument to the action method. As we will see, this allows us to write very simple and efficient action methods for rules such as expression. (Note there's a space between the #= and the key's name).



Testing the Parser It is useful to test the parser before writing any action methods. This can save you a lot of work; if you write the actions immediately after writing the grammar rules, and only later find out that your parser must be updated, then your action methods probably need to be updated as well. In Episode 2 we saw the target command line option. In order to test the parser, the "parse" target is especially helpful. When specifying this option, your compiler will print the parse tree of your input string, or print a syntax error. It is wise to test your parser with both correct and incorrect input, so you know for sure your parser doesn't accept input that it shouldn't.



 Squaak Details and First Steps



And... Action! Now we have implemented the initial version of the Squaak grammar, it's time to implement the parse actions we mentioned before. The actions are written in a file called src/Squaak/Actions.pm. If you look at the methods in this file, here and there you'll see that the Match object ($/) , or rather, hash fields of it (like $) is evaluated in scalar context, by writing " $( ... )".



As mentioned in Synopsis 5, evaluating a Match object in scalar context returns its result object. Normally the result object is the matched portion of the source text, but the special make function can be used to set the result object to some other value. This means that each node in the parse tree (a Match object) can also hold its PAST representation. Thus we use the make function to set the PAST representation of the current node in the parse tree, and later use the $( ... ) operator to retrieve the PAST representation from it. In recap, the Match object ($/) and any subrules of it (for instance $) represent the parse tree; of course, $ represents only the parse tree what the  rule matched. So, any action method has access to the parse tree that the equally named grammar rule matched, as the Match object is always passed as an argument. Evaluating a parse tree in scalar context yields the PAST representation (obviously, this PAST object should be set using the make function). If you're following this tutorial, I highly advise you to get your feet wet, and do the exercises. Remember, learning and not doing is not learning (or something like that :-). This week's exercises are not that difficult, and after doing them, you'll have implemented the first part of our little Squaak language.



What's next? In this episode we introduced the full grammar of Squaak. We took the first steps to implement this language. The first, and currently only, statement type is assignments. We briefly touched on how to write the action methods that are invoked during the parsing phase. In the next episode, we shall take a closer look on the different PAST node types, and implement some more parts of the Squaak language. Once we have all basic parts in place, adding statement types will be rather straightforward. In the mean time, if you have any questions or are stuck, don't hesitate to leave a comment or contact me.



Exercises This episode's exercises are simple enough to get started on implementing Squaak. Problem 1 Rename the names of the action methods according to the name changes we made on the grammar rules. So, "integer" becomes "integer_constant", "value" becomes "expression", and so on. Problem 2 Look at the grammar rule for statement. A statement currently consists of an assignment. Implement the action method "statement" to retrieve the result object of this assignment and set it as statement's result object using the special make function. Do the same for rule primary. Solution method statement($/) { make $( $ ); }



Note that at this point, the rule statement doesn't define different #= keys for each type of statement, so we don't declare a parameter $key. This will be changed later.



117



 Squaak Details and First Steps



118



method primary($/) { make $( $ ); }



Problem 3 Write the action method for the rule identifier. As a result object of this "match", a new PAST::Var node should be set, taking as name a string representation of the match object ($/). For now, you can set the scope to 'package'. See "pdd26: ast" for details on PAST::Var nodes. Solution method identifier($/) { make PAST::Var.new( :name(~$/), :scope('package'), :node($/) ); }



Problem 4 Write the action method for assignment. Retrieve the result objects for "primary" and for "expression", and create a PAST::Op node that binds the expression to the primary. (Check out pdd26 for PAST::Op node types, and find out how you do such a binding). Solution method assignment($/) { my $lhs := $( $ ); my $rhs := $( $ ); $lhs.lvalue(1); make PAST::Op.new( $lhs, $rhs, :pasttype('bind'), :node($/) ); }



Note that we set the lvalue flag on $lhs. See PDD26 for details on this flag. Problem 5 Run your compiler on a script or in interactive mode. Use the target option to see what PIR is being generated on the input "x = 42". Solution .namespace .sub "_block10" new $P11, "Integer" assign $P11, 42 set_global "x", $P11 .return ($P11) .end



The first two lines of code in the sub create an object to store the number 42, the third line stores this number as "x". The PAST compiler will always generate an instruction to return the result of the last statement, in this case $P11.



 Squaak Details and First Steps



119



Some Notes • Help! I get the error message "no result object". This means that the result object was not set properly (duh!). Make sure each action method is invoked (check each rule for a "{*}" marker), and that there is an action method for that rule, and that "make" is used to set the appropriate PAST node. Note that not all rules have action methods, for instance the "keyword" rule (there's no point in that). • While we're constructing parts of Squaa k's grammar, we'll sometimes make a shortc ut, by forgettin g about certain rules for a while. For instance, you might have noticed we're ignoring float-constants right now. That's ok. When we'll need them, these rules will be added.



References • pdd26: ast • synopsis 5: Rules • docs/pct/*.pod



PAST Nodes and More Statements Squaak Language Tutorial Episode 1: Introduction Episode 2: Poking in Compiler Guts Episode 3: Squaak Details and First Steps Episode 4: PAST Nodes and More Statements Episode 5: Variable Declaration and Scope Episode 6: Scope and Subroutines Episode 7: Operators and Precedence Episode 8: Hashtables and Arrays Episode 9: Wrap-Up and Conclusion



The previous episode introduced the full grammar specification of Squaak, and we finally started working on the implementation. If you're doing the exercises, you currently have basic assignments working; strings and integers can be assigned to (global) variables. This episode will focus on implementation of some statement types and explain a few bits about the different PAST node types.



Parrot Abstract Syntax Tree A Parrot Abstract Syntax Tree (PAST) represents a program written in Squaak (or any other Parrot-ported language), and consists of nodes. In the previous episode, we already saw nodes to represent string and integer literals, identifiers and "operator" nodes (PAST::Op), in our case assignment. Other operators represent other high-level language constructs such as conditional statements, loops, and subroutine invocation. Depending on the node type, a PAST node can take child nodes. For instance, a PAST node to represent an if-statement can have up to three child nodes. The first child node represents the condition; if true, the second child node is evaluated. If the condition evaluates to false, and there's a third child node, this third child node is evaluated (the else part). If the PAST represents a subroutine invocation, the child nodes are evaluated in a different way. In that case, the first child node represents the subroutine that is to be invoked (unless the :name attribute was set on this node, but more



 PAST Nodes and More Statements



120



on that in a later episode), and all other child nodes are passed to that subroutine as arguments. It generally doesn't matter of which PAST node type the children are. For instance, consider a language in which a simple expression is a statement: 42



You might wonder what kind of code is generated for this. Well, it's really very simple: a new PAST::Val node is created (of a certain type, for this example that would be 'Integer'), and the value is assigned to this node. It might seem a bit confusing to write something like this, as it doesn't really do anything (note that this is not valid Squaak input): if 42 then "hi" else "bye" end



But again, this works out correctly; the "then" and "else" blocks are compiled to instructions that load that particular literal into a PAST::Val node and leave it there. That's fine, if your language allows such statements. The point I'm trying to make is, that all PAST nodes are equal. You don't need to think about the node types if you set a node as a child of some other parent node. Each PAST node is compiled into a number of PIR instructions.



Go with the control-flow Now you know a bit more on PAST nodes, let's get our hands dirty and implement some more statement types. In the rest of this episode, we'll handle if-statements and throw-statements.



If-then-else The first statement we're going to implement now is the if-statement. An if-statement has typically three parts (but this of course depends on the programming language): a conditional expression, a "then" part and an "else" part. Implementing this in Perl 6 rules and PAST is almost trivial: rule if_statement { 'if'  'then'  ['else' $= ]? 'end' {*} } rule block { * {*} }



rule statement { |  {*} |  {*}



#= assignment #= if_statement



}



Note that the optional else block is stored in the match object's "else" field. If we hadn't written this $= part, then  would have been an array, with block[0] the "then" part, and block[1]



the optional else part. Assigning the optional else block to a different field, makes the action method slightly easier to read.



 PAST Nodes and More Statements



121



Also note that the statement rule has been updated; a statement is now either an assignment or an if-statement. As a result, the action method statement now takes a key argument. The relevant action methods are shown below: method statement($/, $key) { # get the field stored in $key from the $/ object, # and retrieve the result object from that field. make $( $/{$key} ); } method block($/) { # create a new block, set its type to 'immediate', # meaning it is potentially executed immediately # (as opposed to a declaration, such as a # subroutine definition). my $past := PAST::Block.new( :blocktype('immediate'), :node($/) ); # for each statement, add the result # object to the block for $ { $past.push( $( $_ ) ); } make $past; }



method if_statement($/) { my $cond := $( $ ); my $then := $( $ ); my $past := PAST::Op.new( $cond, $then, :pasttype('if'), :node($/) ); if $ { $past.push( $( $[0] ) ); } make $past; }



That's, easy, huh? First, we get the result objects for the conditional expression and the then part. Then, a new PAST::Op node is created, and the :pasttype is set to 'if', meaning this node represents an if-statement. Then,



if there is an "else" block, this block's result object is retrieved and added as the third child of the PAST node. Finally, the result object is set with the make function.



Result objects At this point it's wise to spend a few words on the make function, the parse actions and how the whole PAST is created by the individual parse actions. Have another look at the action method if_statement. In the first two lines, we request the result objects for the conditional expression and the "then" block. When were these result objects created? How can we be sure they're there? The answer lies in the order in which the parse actions are executed. The special {*} symbol that triggers a parse action invocation, is usually placed at the end of the rule. For this input



 PAST Nodes and More Statements



122



string: "if 42 then x = 1 end" this implies the following order: 1. parse TOP 2. parse statement 3. parse if_statement 4. parse expression 5. parse integer 6. create PAST::Val( :value(42) ) 7. parse block 8. parse statement 9. parse assignment 10. parse identifier 11. create PAST::Var( :name('x')) 12. parse integer 13. create PAST::Val( :value(1) ) 14. create PAST::Op( :pasttype('bind') ) 15. create PAST::Block (in action method block) 16. create PAST::Op( :pasttype('if') ) 17. create PAST::Block (in action method TOP) As you can see, PAST nodes are created in the leafs of the parse tree first, so that later, action methods higher in the parse tree can retrieve them.



Throwing Exceptions The grammar rule for the "throw" statement is really quite easy, but it's useful to discuss the parse action, as it shows the use of generating custom PIR instructions. First the grammar rule: rule throw_statement { 'throw'  {*} }



I assume you know how to update the "statement" rule by now. The throw statement will compile down to Parrot's "throw" instruction, which takes one argument. In order to generate a custom Parrot instruction, the instruction can be specified in the :pirop attribute when creating a PAST::Op node. Any child nodes are passed as arguments to this instruction, so we need to pass the result object of the expression being thrown as a child of the PAST::Op node representing the "throw" instruction. method throw_statement($/) { make PAST::Op.new( $( $ ), :pirop('throw'), :node($/) ); }



 PAST Nodes and More Statements



123



What's Next? In this episode we implemented two more statement types of Squaak. You should get a general idea of how and when PAST nodes are created, and how they can be retrieved sub (parse) trees. In the next episode we'll take a closer look at variable scope and subroutines. In the mean time, I can imagine some things are not too clear. In case you're lost, don't hesitate to leave comment, and I'll try to answer (as far as my knowledge goes).



Exercises Problem 1 We showed how the if-statement was implemented. The while-statement and try-statement are very similar. Implement these. Check out pdd26 to see what PAST::Op nodes you should create. Solution The while-statement is straightforward: method while_statement($/) { my $cond := $( $ ); my $body := $( $ ); make PAST::Op.new( $cond, $body, :pasttype('while'), :node($/) ); }



The try-statement is a bit more complex. Here are the grammar rules and action methods. rule try_statement { 'try' $= 'catch'  $= 'end' {*} } rule exception {  {*} } method try_statement($/) { ## get the try block my $try := $( $ );



## create a new PAST::Stmts node for ## the catch block; note that no ## PAST::Block is created, as this ## currently has problems with the ## exception object. For now this will ## do.



 PAST Nodes and More Statements



124



my $catch := PAST::Stmts.new( :node($/) ); $catch.push( $( $ ) ); ## get the exception identifier; ## set a declaration flag, the scope, ## and clear the viviself attribute. my $exc := $( $ ); $exc.isdecl(1); $exc.scope('lexical'); $exc.viviself(0);



## generate instruction to retrieve ## the exception objct (and the ## exception message, that is passed ## automatically in PIR, this is stored ## into $S0 (but not used). my $pir := " ~ "



.get_results (%r, $S0)\n" store_lex '" ~ $exc.name()



~ "', %r"; $catch.unshift( PAST::Op.new( :inline($pir), :node($/) ) );



## do the declaration of the exception ## object as a lexical here: $catch.unshift( $exc );



make PAST::Op.new( $try, $catch, :pasttype('try'), :node($/) ); }



method exception($/) { our $?BLOCK; my $past := $( $ ); $?BLOCK.symbol( $past.name(), :scope('lexical') ); make $past; }



Instead of putting "identifier" after the "catch" keyword, we made it a separate rule, with its own action method. This allows us to insert the identifier into the symbol table of the current block (the try-block), before the catch block is parsed.



First the PAST node for the try block is retrieved. Then, the catch block is retrieved, and stored into a PAST::Stmts node. This is needed, so that we can make sure that the instructions that retrieve the exception



object come first in the exception handler.



 PAST Nodes and More Statements Then, we retrieve the PAST node for the exception identifier. We're setting its scope, a flag telling the PAST compiler this is a declaration, and we clear the viviself attribute. The viviself attribute is discussed in a later episode; if you didn't read that yet, just keep in mind the viviself attribute (if set) will make sure all declared variables are initialized. We must clear this attribute here, to make sure that this exception object is not initialized, because that will be done by the instruction that retrieves the thrown exception object, discussed next. In PIR, we can use the .get_results directive to retrieve a thrown exception. You could also generate the get_results instruction (note the missing dot), but this is much easier. Currently, in PIR, when retrieving the



exception object, you must always specify both a variable (or register) for the exception object, and a string variable (or register) to store the exception message. The exception message is actually stored within the exception object. We use $S0 to store the exception message, and we'll ignore it after that. Just remember for now that if you want to retrieve the exception object, you must also specify a place to store the exception message. There is no special PAST node to generate these instructions, so we use a so-called inline PAST::Op node. We store the instructions to be generated into a string and store that in the inline attribute of a PAST::Op node. Once created, this node is unshifted onto the PAST::Stmts node representing the exception handler. After that, the declaration is stored



in that PAST::Stmts node, so that this declaration comes first. Finally, we have the block representing the try block, and a PAST::Stmts node representing the exception handler. Both are used to create a PAST::Op node whose pasttype is set to the built-in " try" type. Problem 2 Start Squaak in interactive mode, and specify the target option to show the generated PIR instructions. Check out what instructions and labels are generated, and see if you can recognize which instructions make up the conditional expression, which represent the "then" block, and which represent the "else" block (if any). Solution Start Squaak in interactive mode, and specify the target option to show the generated PIR instructions. Check out what instructions and labels are generated, and see if you can recognize which instructions make up the conditional expression, which represent the "then" block, and which represent the "else" block (if any). > if 1 then else end



.namespace .sub "_block16" new $P18, "Integer" assign $P18, 1 ## this is the condition: if $P18, if_17 ## this is invoking the else-block: get_global $P21, "_block19" newclosure $P21, $P21 $P20 = $P21() set $P18, $P20 goto if_17_end ## this is invoking the then-block: if_17: get_global $P24, "_block22" newclosure $P24, $P24



125



 PAST Nodes and More Statements



126



$P23 = $P24() set $P18, $P23 if_17_end: .return ($P18) .end .namespace .sub "_block22" :outer("_block16") .return () .end



.namespace .sub "_block19" :outer("_block16") .return () .end



References • PDD26: AST • docs/art/*.pod for go od introductions to PIR



Variable Declaration and Scope Squaak Language Tutorial Episode 1: Introduction Episode 2: Poking in Compiler Guts Episode 3: Squaak Details and First Steps Episode 4: PAST Nodes and More Statements Episode 5: Variable Declaration and Scope Episode 6: Scope and Subroutines Episode 7: Operators and Precedence Episode 8: Hashtables and Arrays Episode 9: Wrap-Up and Conclusion



Episode 4 discussed the implementation of some statement types, such as the if-statement. In this episode we'll talk about variable declarations and scope handling. It's going to be a long story, so take your time to read this episode.



Globals, locals and default values Squaak variables have one of two scopes: either they're global, or they're local. In order to create a global variable, you just assign some expression to an identifier (which hasn't been declared as a local). Local variables, on the other hand, must be declared using the var keyword. In other words, at any given point during the parsing phase, we have a list of variables that are known to be local variables. When an identifier is parsed, it is looked up and if found, its scope is set to local. If not, its scope is assumed to be global. When using an uninitialized variable, its value is set to an object called Undef. Some examples are shown below. x = 42



# x was not declared, so it is global



var k = 10



# k is local and initialized to 10



 Variable Declaration and Scope a + b



127 # neither a nor b was declared; # both default to the value "Undef"



Scoping and Symbol Tables Earlier we mentioned the need to store declared local variables. In compiler jargon, such a data structure to store declarations is called a symbol table. For each individual scope, there's a separate symbol table. Squaak has a so-called do-block statement, that is defined below. rule do_block { 'do'  'end' {*} }



Each do-block defines a new scope; local variables declared between the "do" and "end" keywords are local to that block. An example to clarify this is shown below: do var x = 1 print(x)



# prints 1



do var x = 2 print(x)



# prints 2



end



print(x) end



# prints 1



So, each do/end pair defines a new scope, in which any declared variables hide variables with the same name in outer scopes. This behavior is common in many programming languages. The PCT has built-in support for symbol tables; a PAST::Block object has a method symbol that can be used to enter new symbols and query the table for existing ones. In PCT, a PAST::Block object represents a scope. There are two blocktypes: immediate and declaration. An immediate block can be used to represent the blocks of statements in an do-block statement, for instance: do block end



When executing this statement, block is executed immediately. A declaration block, on the other hand, represents a block of statements that can be invoked at a later point. Typically these are subroutines. So, in this example: sub foo(x) print(x) end



a PAST::Block object is created for the subroutine foo. The blocktype is set to declaration, as the subroutine is defined, not executed (immediately). For now you can forget about the blocktype, but now that I've told you, you'll recognize it when you see it. We'll come back to it in a later episode.



 Variable Declaration and Scope



Implementing Scope So, we know how to use global variables, declare local variables, and about PAST::Block objects representing scopes. How do we make our compiler to generate the right PIR instructions? After all, when handling a global variable, Parrot must handle this differently from handling a local variable. When creating PAST::Var nodes to represent the variables, we must know whether the variable is a local or a global variable. So, when handling variable declarations (of local variables; globals are not declared), we need to register the identifier as a local in the current block's symbol table. First, we'll take a look at the implementation of variable declarations.



Variable declaration The following is the grammar rule for variable declarations. This is a type of statement, so I assume you know how to extend the statement rule to allow for variable declarations. rule variable_declaration { 'var'  ['=' ]? {*} }



A local variable is declared using the var keyword, and has an optional initialization expression. If the latter is missing, the variable's value defaults to the undefined value called Undef. Let's see what the parse action looks like: method variable_declaration($/) { # get the PAST for the identifier my $past := $( $ );



# this is a local (it's being defined) $past.scope('lexical'); # set a declaration flag $past.isdecl(1);



# check for the initialization expression if $ { # use the viviself clause to add a # an initialization expression $past.viviself( $($[0]) ); } else { # no initialization, default to "Undef" $past.viviself('Undef'); } }



make $past;



Well, that wasn't too hard, was it? Let's analyze what we just did. First we retrieved the PAST node for the identifier, which we then decorated by setting its scope to lexical (a local variable is said to be lexically scoped, hence lexical), and setting a flag indicating this node represents a declaration (isdecl). So, besides representing



variables in other statements (for instance, assignments), a PAST::Var node is also used as a declaration statement.



128



 Variable Declaration and Scope



129



Earlier in this episode we mentioned the need to register local variables in the current scope block when they are declared. So, when executing the parse action for variable-declaration, there should already be a PAST::Block node around, that can be used to register the symbol being declared. As we learned in Episode 4, PAST nodes are created in a depth-first fashion; the leafs are created first, and then the nodes "higher" in the parse tree. This implies that a PAST::Block node is created after the statement nodes (which variable_declaration is) that will be the children of the block. In the next section we'll see how to solve this problem.



Implementing a scope stack In order to make sure that a PAST::Block node is created before any statements are parsed (and their parse actions are executed -- these might need to enter symbols in the block's symbol table), we add a few extra parse actions. Let's take a look at them. rule TOP { {*}



#= open



* [ $ || <.panic: syntax error> ] {*}



#= close



}



We now have two parse actions for TOP, which are differentiated by an additional key parameter. The first parse action is executed before any input is parsed, which is particularly suitable for any initialization actions you might need. The second action (which was already there) is executed after the whole input string is parsed. Now we can create a PAST::Block node before any statements are parsed, so that when we need the current block, it's there (somewhere, later we'll see where exactly). Let's take a look at the parse action for TOP. method TOP($/, $key) { our $?BLOCK; our @?BLOCK; if $key eq 'open' { $?BLOCK := PAST::Block.new( :blocktype('declaration'), :node($/) ); @?BLOCK.unshift($?BLOCK); } else { # key is 'close' my $past := @?BLOCK.shift(); for $ { $past.push( $( $_ ) ); } make $past; } }



Let's see what's happening here. When the parse action is invoked for the first time (when $key equals "open"), a new PAST::Block node is created and assigned to a strange-looking (if you don't know Perl, like me. Oh wait, this is Perl. Never mind..) variable called $?BLOCK. This variable is declared as " our", which means that it is a package variable. This means that the variable is shared by all methods in the same package (or class), and, equally



important, the variable is still around after the parse action is done. Please refer to the Perl 6 specification



[1]



for



 Variable Declaration and Scope more semantics on "our". The variable $?BLOCK holds the current block. After that, this block is unshifted onto another funny-looking variable, called @?BLOCK. This variable has a "@" sigil, meaning this is an array. The unshift method puts its argument on the front of the list. In a sense, you could think of the front of this list as the



top of a stack. Later we'll see why this stack is necessary. This @?BLOCK variable is also declared with "our", meaning it's also package-scoped. However, as we call a method on this variable, it should have been already created; otherwise you'd invoke the method on an undefined ("Undef") variable. So, this variable should have been created before the parsing starts. We can do this in the compiler's main program, squaak.pir. Before doing so, let's take a quick look at the " else" part of the parse action for TOP, which is executed after the whole input string is parsed. The PAST::Block node is retrieved from @?BLOCK, which makes sense, as it was created in the first part of the method and unshifted on @?BLOCK. Now this



node can be used as the final result object of TOP. So, now we've seen how to use the scope stack, let's have a look at its implementation.



A List Class We'll implement the scope stack as a ResizablePMCArray object. This is a built-in PMC type. However, this built-in PMC does not have any methods; in PIR it can only be used as an operand of the built-in shift and unshift instructions. In order to allow us to write this as method calls, we create a new subclass of ResizablePMCArray. The code below creates the new class and defines the methods we need. 1 .namespace 2 .sub 'initlist' :anon :init :load 3



subclass $P0, 'ResizablePMCArray', 'List'



4



new $P1, 'List'



5



set_hll_global ['Squaak';'Grammar';'Actions'], '@?BLOCK', $P1



6 .end 7 .namespace ['List'] 8 .sub 'unshift' :method 9 10



.param pmc obj unshift self, obj



11 .end 12 .sub 'shift' :method 13



shift $P0, self



14



.return ($P0)



15 .end



Well, here you have it: part of the small amount of PIR code you need to write for the Squaak compiler (there's some more for some built-in subroutines, more on that later). Let's discuss this code snippet in more detail (if you know PIR, you could skip this section). Line 1 resets the namespace to the root namespace in Parrot, so that the sub 'initlist' is stored in that namespace. The sub 'initlist' defined in lines 2-6 has some flags: :anon



means that the sub is not stored by name in the namespace, implying it cannot be looked up by name. The :init flag means that the sub is executed before the main program (the "main" sub) is executed. The :load flag makes sure that the sub is executed if this file was compiled and loaded by another file through the load_bytecode instruction. If you don't understand this, no worries. You can forget about it now. In any case, we know for sure there's a List class when we need it, because the class creation is done before running the actual compiler code. Line 3 creates a new subclass of ResizablePMCArray, called "List". This results in a new class object, which is left in register $P0, but it's not used after that. Line 4 creates a new List object, and stores it in register $P1. Line 5, stores this List object by name of " @?BLOCK" (that name should ring a bell now...) in the namespace of the Actions



130



 Variable Declaration and Scope



131



class. The semicolons in between the several key strings indicate nested namespaces. So, lines 4 and 5 are important, because the create the @?BLOCK variable and store it in a place that can be accessed from the action methods in the Actions class. Lines 7-11 define the unshift method, which is a method in the "List" namespace. This means that it can be invoked as a method on a List object. As the sub is marked with the :method flag, the sub has an implicit first parameter called "self", which refers to the invocant object. The unshift method invokes Parrot's unshift instruction on self, passing the obj argument as the second operand. So, obj is unshifted onto self, which is the List object itself. Finally, lines 12-15 define the "shift" method, which does the opposite of "unshift", removing the first element and returning it to its caller.



Storing Symbols Now, we set up the necessary infrastructure to store the current scope block, and we created a datastructure that acts as a scope stack, which we will need later. We'll now go back to the parse action for variable_declaration, because we didn't enter the declared variable into the current block's symbol table yet. We'll see how to do that now. First, we need to make the current block accessible from the method variable_declaration. We've already seen how to do that, using the "our" keyword. It doesn't really matter where in the action method we enter the symbol's name into the symbol table, but let's do it at the end, after the initialization stuff. Naturally, we're only going to enter the symbol if it's not there already; duplicate variable declarations (in the same scope) should result in an error message (using the panic method of the match object). The code to be added to the method variable_declaration looks then like this: method variable_declaration($/) { our $?BLOCK; # get the PAST node for identifier # set the scope and declaration flag



# do the initialization stuff # cache the name into a local variable my $name := $past.name(); if $?BLOCK.symbol( $name ) { # symbol is already present $/.panic("Error: symbol " ~ $name ~ " was already defined.\n"); } else {



}



$?BLOCK.symbol( $name, :scope('lexical') );



make $past; }



 Variable Declaration and Scope



What's Next? With this code in place, variable declarations are handled correctly. However, we didn't update the parse action for identifier, which creates the PAST::Var node and sets its scope; currently all identifiers' scope is set to 'package' (which means it's a global variable). As we already covered a lot of material in this episode, we'll leave this for the next episode. In the next episode, we'll also cover subroutines, which is another important aspect of any programming language. Hope to catch you later!



Exercises Problem 1 In this episode, we changed the action method for the TOP rule; it is now invoked twice, once at the beginning of the parse, once at the end of the parse. The block rule, which defines a block to be a series of statements, represents a new scope. This rule is used in for instance if-statement (the then-part and else-part), while-statement (the loop body) and others. Update the parse action for block so it is invoked twice; once before parsing the statements, during which a new PAST::Block is created and stored onto the scope stack, and once after parsing the statements, during which this PAST node is set as the result object. Make sure $?BLOCK is always pointing to the current block. In order to do this exercise correctly, you should understand well what the shift and unshift methods do, and why we didn't implement methods to push and pop, which are more appropriate words in the context of a (scope) stack. Solution Keeping the Current block up to date : Sometimes we need to access the current block's symbol table. In order to



be able to do so, we need a reference to the "current block". We do this by declaring a package variable called "$?BLOCK", declared with "our" (as opposed with "my"). This variable will always point to the "current" block. As blocks can nest, we use a "stack", on which newly created blocks are stored. Whenever a new block is created, we assign this to $?BLOCK, and store it onto the stack, so that the next time a new block is created, the "old" current block isn't lost. Whenever a scope is closed, we pop off the current block from the stack, and restore the previous "current" block. Why unshift/shift and not push/pop?: When we're talking about stacks, it would seem logical to talk about stack



operations such as "push" and "pop". Instead, we use the operations "unshift" and "shift". If you're not a Perl programmer (such as myself), these names might not make sense. However, it's pretty easy. Instead of pushing a new object onto the "top" of the stack, you unshift objects onto this stack. Just see it as an old school bus, with only one entrance (at the front of the bus). Pushing a new person means taking the first free seat when entering, while unshifting a new person means everybody moves (shifts) one place to the back, so the new person can sit in the front seat. You might think this is not as efficient (more stuff is moved around), but that's not really true (actually: I guess (and certainly hope) the shift and unshift operations are implemented more effectively than the bus metaphor; I don't know how it is implemented). So why unshift/shift, and not push/pop? When restoring the previous "current block", we need to know exactly where it is (what position). It would be nice to be able to always refer to the "first passenger on the bus", instead of the last person. We know how to reference the first passenger (it's on seat no. 0 (it was designed by an IT guy)); we don't really know what is the seat no. of the last person: s/he might sit in the middle, or at the back. I hope it's clear what I mean here... otherwise, have a look at the code, and try to figure out what's happening: method block($/, $key) { our $?BLOCK; our @?BLOCK; if $key eq 'open' { $?BLOCK := PAST::Block.new(



132



 Variable Declaration and Scope



133 :blocktype('immediate'), :node($/) );



@?BLOCK.unshift($?BLOCK); } else { my $past := @?BLOCK.shift(); $?BLOCK



:= @?BLOCK[0];



for $ { $past.push( $( $_ ) ); } make $past; } }



References [1] http:// perlcabal.org/syn/S02.html#Names



Scope and Subroutines Squaak Language Tutorial Episode 1: Introduction Episode 2: Poking in Compiler Guts Episode 3: Squaak Details and First Steps Episode 4: PAST Nodes and More Statements Episode 5: Variable Declaration and Scope Episode 6: Scope and Subroutines Episode 7: Operators and Precedence Episode 8: Hashtables and Arrays Episode 9: Wrap-Up and Conclusion



In Episode 5, we looked at variable declarations and implementing scope. We covered a lot of information then, but did not tell the full story, in order to keep that post short. In this episode we'll address the missing parts, which will also result in implementing subroutines.



Variables In the previous episode, we entered local variables into the current block's symbol table. As we've seen earlier, using the do-block statement, scopes may nest. Consider this example: do var x = 42 do print(x) end end



 Scope and Subroutines



134



In this example, the print statement should print 42, even though x was not declared in the scope where it is referenced. How does the compiler know it's still a local variable? That's simple: it should look in all scopes, starting at the innermost scope. Only when the variable is found in any scope, should its scope be set to "lexical", so that the right instructions are being generated. The solution I came up with is shown below. Please note that I'm not 100% sure if this is the "best" solution, but my personal understanding of the PAST compiler is limited. So, while this solution works, I may teach you the wrong "habit". Please be aware of this. method identifier($/) { our @?BLOCK; my $name



:= ~$;



my $scope := 'package'; # default value # go through all scopes and check if the symbol # is registered as a local. If so, set scope to # local. for @?BLOCK { if $_.symbol($name) { $scope := 'lexical'; } } make PAST::Var.new( :name($name), :scope($scope), :viviself('Undef'), :node($/) ); }



Viviself You might have noticed the viviself attribute before. This attribute will result in extra instructions that will initialize the variable if it doesn't exist. As you know, global variables spring into life automatically when they're used. Earlier we mentioned that uninitialized variables have a default value of "Undef": the viviself attribute does this. For local variables, we use this mechanism to set the (optional) initialization value. When the identifier is a parameter, the parameter will be initialized automatically if it doesn't receive a value when the subroutine it belongs to is invoked. Effectively this means that all parameters in Squaak are optional!



 Scope and Subroutines



135



Subroutines We already mentioned subroutines before, and introduced the PAST::Block node type. We also briefly mentioned the blocktype attribute that can be set on a PAST::Block node, which indicates whether the block is to be executed immediately (for instance, a do-block or if statement) or it represents a declaration (for instance, subroutines). Let us now look at the grammar rule for subroutine definitions: rule sub_definition { 'sub'   * 'end' {*} }



rule parameters { '(' [ [',' ]* ]? ')' {*} }



This is rather straightforward, and the action methods for these rules are quite simple, as you will see. First, however, let's have a look at the rule for sub definitions. Why is the sub body defined as * and not as a ? Surely, a subroutine defines a new scope, which was already covered by  Well, you're right in



that. However, as we will see, by the time that a new PAST::Block node would be created, we are too late! The parameters would already have been parsed, and not entered into the block's symbol table. That's a problem, because parameters are most likely to be used in the subroutine's body, and as they are not registered as local variables (which they are), any usage of parameters would not be compiled down to the right instructions to fetch any parameters. So, how do we solve this in an efficient way? The solution is simple. The only place where parameters live, is in the subroutine's body, represented by a PAST::Block node. Why don't we create the PAST::Block node in the action method for the parameters



rule. By doing so, the block is already in place and the parameters are registered as local symbols right in time. Let's look at the action methods. method parameters($/) { our $?BLOCK; our @?BLOCK; my $past := PAST::Block.new( :blocktype('declaration'), :node($/) ); # now add all parameters to this block for $ { my $param := $( $_ ); $param.scope('parameter'); $past.push($param); # register the parameter as a local symbol $past.symbol($param.name(), }



:scope('lexical'));



 Scope and Subroutines



136



# now put the block into place on the scope stack $?BLOCK := $past; @?BLOCK.unshift($past);



make $past; }



method sub_definition($/) { our $?BLOCK; our @?BLOCK; my $past := $( $ ); my $name := $( $ ); # set the sub's name $past.name( $name.name() ); # add all statements to the sub's body for $ { $past.push( $( $_ ) ); } # and remove the block from the scope # stack and restore the current block @?BLOCK.shift(); $?BLOCK := @?BLOCK[0];



make $past; }



First, let's check out the parse action for parameters. First, a new PAST::Block node is created. Then, we iterate over the list of identifiers (which may be empty), each representing a parameter. After retrieving the result object for a parameter (which is just an identifier), we set its scope to "parameter", and we add it to the block object. After that, we register the parameter as a symbol in the block object, specifying the scope as "lexical". Parameters are just a special kind of local variables, and there's no difference in a parameter and a declared local variable in a subroutine, except that a parameter will usually be initialized with a value that is passed when the subroutine is invoked. After handling the parameters, we set the current block (referred to by our package variable $?BLOCK) to the PAST::Block node we just created, and push it on the scope stack (referred to by our package variable @?BLOCK).



After the whole subroutine definition is parsed, the action method sub_definition is invoked. This will retrieve the result object for parameters, which is the PAST::Block node that will represent the sub. After retrieving the result object for the sub's name, we set the name on the block node, and add all statements to the block. After this, we pop off this block node of the scope stack ( @?BLOCK), and restore the current block ( $?BLOCK). Pretty easy, huh?



 Scope and Subroutines



137



Subroutine invocation Once you defined a subroutine, you'll want to invoke it. In the exercises of Episode 5, we already gave some tips on how to create the PAST nodes for a subroutine invocation. In this section, we'll give a complete description. First we'll introduce the grammar rules. rule sub_call {   {*} }



Not only allows this to invoke subroutines by their name, you can also store the subroutines in an array or hash field, and invoke them from there. Let's take a look at the action method, which is really quite straightforward. method sub_call($/) { my $invocant := $( $ ); my $past



:= $( $ );



$past.unshift($invocant); make $past; }



method arguments($/) { my $past := PAST::Op.new( :pasttype('call'), :node($/) ); for $ { $past.push( $( $_ ) ); } make $past; }



The result object of the sub_call method should be a PAST::Op node (of type 'call'), which contains a number of child nodes: the first one is the invocant object, and all remaining children are the arguments to that sub call. In order to "move" the result objects of the arguments to the sub_call method, we create the PAST::Op node in the method arguments, which is then retrieved by sub_call. In sub_call, the invocant object is set as the first child (using unshift). This is all too easy, isn't it? :-)



What's Next? In this episode we finished the implementation of scope in Squaak, and implemented subroutines. Our language is coming along nicely! In the next episode, we'll explore how to implement operators and an operator precedence table for efficient expression parsing. In the mean time, should you have any problems or questions, don't hesitate to leave a comment!



Exercises Problem 1 By now you should have a good idea on the implementation of scope in Squaak. We haven't implemented the for-statement yet, as it needs proper scope handling to implement. Implement this. Check out episode 3 for the BNF rules that define the syntax of the for-statement. When implementing it, you will run into the same issue as we did when implementing subroutines and parameters. Use the same trick for the implementation of the for-statement. Solution



 Scope and Subroutines



138



First, let us look at the BNF of the for-statement: for-statement ::= 'for' for-init ',' expression [step] 'do' block 'end' step



::= ',' expression



for-init



::= 'var' identifier '=' expression



It's pretty easy to convert this to Perl 6 rules: rule for_statement { 'for'  ','  ? 'do'  'end' {*} } rule step { ','  {*} } rule for_init { 'var'  '='  {*} }



Pretty easy huh? Let's take a look at the semantics. A for-loop is just another way to write a while loop, but much easier in certain cases. This: for var  = , ,  do  end



corresponds to: do var  =  while  <=  do  end



 =  + 



end



If  is absent, it defaults to the value "1". Note that the step expression ( expr3) should be positive; the loop condition contains a <= operator. When you specify a negative step expression, the loop variable will only decrease in value, which will never make the loop condition false (unless it overflows, but that's a different issue; this might even raise an exception in Parrot; this I do not know). Allowing negative step expressions introduces more complexity, which I felt was not worth the trouble for this tutorial language.



 Scope and Subroutines



139



Note that the loop variable  is local to the for loop; this is expressed in the equivalent while loop by the surrounding do/end pair: a new do/end pair defines a new (nested) scope; after the end keyword, the loop variable is no longer visible. Let's implement the action method for the for-statement. As was mentioned in the exercise description, we're dealing with the same situation as with subroutine parameters. In this case, we're dealing with the loop variable, which is local to the for-statement. Let's check out the rule for for_init: method for_init($/) { our $?BLOCK; our @?BLOCK; ## create a new scope here, so that we can ## add the loop variable ## to this block here, which is convenient. $?BLOCK := PAST::Block.new( :blocktype('immediate'), :node($/) ); @?BLOCK.unshift($?BLOCK);



my $iter := $( $ ); ## set a flag that this identifier is being declared $iter.isdecl(1); $iter.scope('lexical'); ## the identifier is initialized with this expression $iter.viviself( $( $ ) ); ## enter the loop variable into the symbol table. $?BLOCK.symbol($iter.name(),



:scope('lexical'));



make $iter; }



So, just as we created a new PAST::Block for the subroutine in the action method for parameters, we create a new PAST::Block for the for-statement in the action method that defines the loop variable. (Guess why we made for-init a subrule, and didn't put in "var  = " in the rule of for-statement). This block is the place to live for the loop variable. The loop variable is declared, initialized using the viviself attribute, and entered into the new block's symbol table. Note that after creating the new PAST::Block object, we put it onto the stack scope. Now, the action method for the for statement is quite long, so I'll just embed my comments, which makes reading it easier. method for_statement($/) { our $?BLOCK; our @?BLOCK;



First, get the result object of the for statement initialization rule; this is the PAST::Var object, representing the declaration and initialization of the loop variable. my $init := $( $ );



 Scope and Subroutines



140



Then, create a new node for the loop variable. Yes, another one (besides the one that is currently contained in the PAST::Block). This one is used when the loop variable is updated at the end of the code block (each iteration).



The difference with the other one, is that it doesn't have the isdecl flag, and it doesn't have a viviself clause, which would result in extra instructions checking whether the variable is null (and we know it's not, because we initialize the loop variable). ## cache the name of the loop variable my $itername := $init.name(); my $iter := PAST::Var.new( :name($itername), :scope('lexical'), :node($/) );



Now, retrieve the PAST::Block node from the scope stack, and push all statement PAST nodes onto it. ## the body of the loop consists of the statements written by the user and ## the increment instruction of the loop iterator.



my $body := @?BLOCK.shift(); $?BLOCK



:= @?BLOCK[0];



for $ { $body.push($($_)); }



If there was a step, we use that value; otherwise, we use assume a default step size of "1". Negative step sizes won't work, but if you Feel Lucky, you could go ahead and try. It's not that hard, it's just a lot of work, and I'm too lazy for that now.... ehm, I mean, I leave it as the proverbial exercise to the reader. my $step; if $ { my $stepsize := $( $[0] ); $step := PAST::Op.new( $iter, $stepsize, :pirop('add'), :node($/) ); } else { ## default is increment by 1 $step := PAST::Op.new( $iter, :pirop('inc'), :node($/) ); }



The incrementing of the loop variable is part of the loop body, so add the incrementing statement to $body. $body.push($step);



The loop condition uses the <= operator, and compares the loop variable with the maximum value that was specified. ## while loop iterator <= end-expression my $cond := PAST::Op.new( $iter, $( $ ), :name('infix:<=') );



Now we have the PAST for the loop condition and the loop body, so now create a PAST to represent the ( while) loop.



 Scope and Subroutines



141



my $loop := PAST::Op.new( $cond, $body, :pasttype('while'), :node($/) );



Finally, the initialization of the loop variable should go before the loop itself, so create a PAST::Stmts node to do this: make PAST::Stmts.new( $init, $loop, :node($/) ); }



Wow, we've done it! This was a good example of how to implement a non-trivial statement type using PAST.



Operators and Precedence Squaak Language Tutorial Episode 1: Introduction Episode 2: Poking in Compiler Guts Episode 3: Squaak Details and First Steps Episode 4: PAST Nodes and More Statements Episode 5: Variable Declaration and Scope Episode 6: Scope and Subroutines Episode 7: Operators and Precedence Episode 8: Hashtables and Arrays Episode 9: Wrap-Up and Conclusion Up till now, we've implemented a great deal of the Squaak language. We've seen assignments, control-flow



statements, variable declarations and scope, subroutines and invocation. Our expressions have been limited so far to singular values, such as string literals and integer constants. In this episode, we'll enhance Squaak so it can handle operators, so you can construct more complex expressions.



Operators, precedence and parse trees We will first briefly introduce the problem with recursive-descent parsers (which parsers generated with the PCT are) when parsing expressions. Consider the following mini-grammar, which is a very basic calculator. rule TOP { * } rule expression {  }



rule term {  [   ]* } token addop { '+' | '-' }



 Operators and Precedence



142



rule factor {  [   ]* }



token mulop { '*' | '/' | '%' } rule value{ |  | '('  ')' }



This basic expression grammar implements operator precedence by taking advantage of the nature of a recursive-descent parser (if you haven't seen the word, google it). However, the big disadvantage of parsing expressions this way, is that the parse trees can become quite large. Perhaps more importantly, the parsing process is not very efficient. Let's take a look at some sample input. We won't show the parse trees as shown in Episode 2, but we'll just show an outline. input: 42 results in this parse tree: TOP expression term factor value number 42



As you can see, the input of this single number will invoke 6 grammar rules before parsing the actual digits. Not that bad, you might think. input: "1 + 2" results in this parse tree (we ignore the operator for now): TOP expression term factor | value |



number



|



1



factor value number 2



Only a few more grammar rules are invoked, not really a problem either. input: "(1 + 2) * 3" results in this parse tree: TOP expression term factor



 Operators and Precedence



143



value | expression |



term



|



| factor



|



|



|



|



|



|



|



term



|



value number 1



factor



|



value



|



number



|



2



value number 3



Right; 16 grammar rules just to parse this simple input. I'd call this slightly inefficient. The point is, implementing operator precedence using a recursive-descent parser is somewhat problematic, and given the fact there are better methods to parse expressions like these, not the way to go. Check out this nice explanation



[1]



or google it.



Bottom-up parsing and stacks: operator tables I would like to explain to you how bottom-up parsing works for expressions (or bottom-up parsers in general; Yacc/Bison are parser generators that generate bottom-up parsers for your grammar specification), taking operator precedence into account. However, it's been about 6 years that I did this in a CS class, and I don't remember the particular details. If you really want to know, check out the links at the end of the previous section. It's actually worth checking out. For now, I'll just assume you know what the problem is, so that I'll introduce the solution for PCT-based compilers immediately. At some point when parsing your input, you might encounter an expression. At this point, we'd like the parser to switch from top-down to bottom-up parsing. The Parrot Grammar Engine supports this, and is used as follows: rule expression is optable { ... }



Note that we used the word "expression" here, but you can name it anything. This declares that, whenever you need an expression, the bottom-up parser is activated. Of course, this "optable" must be populated with some operators that we need to be able to parse. This can be done by declaring operators as follows: proto 'infix:*' is tighter('infix:+') { ... }



This defines the operator "*" (the "infix:" is a prefix that tells the operator parser that this operator is an infix operator; there are other types, such as prefix, postfix and others). The "is tighter" clause tells that the "*" operator has a higher precedence than the "+" operator. As you could have guessed, there are other clauses to declare equivalent precedence ("is equiv") and lower precedence ("is looser"). It is very important to spell all clauses, such as "is equiv" correctly (for instance, not "is equil"), otherwise you might get some cryptic error message when trying to run your compiler. See the references section for the optable guide, that has more details on this. Of course, the expression parser does not just parse operators, it must also parse the operands. So, how do we declare the most basic entity that represents an operand? It can be anything, from a basic integer-constant, a function call, or even a function definition (but adding two function definition doesn't really make sense, does it?). The operands are parsed in a recursive-descent fashion, so somewhere the parser must switch back from bottom-up (expression



 Operators and Precedence



144



parsing) to top-down. To declare this "switch-back" point, write: proto 'term:' is tighter('prefix:-') is parsed(&term)



{ ... }



The name "term:" is a built-in name of the operator bottom-up parser; it is invoked every time a new operand is needed. The "is parsed" clause tells the parser that "term" (which accidentally looks like "term:", but you could also have named it anything else) parses the operands. Note: it is very important to add a "is tighter" clause to the declaration of the "term:" rule. Otherwise your expression parser will not work! My knowledge here is a bit limited, but I usually define it as "is tighter" relative to the tightest operator defined.



Squaak Operators We have defined the entry and exit point of the expression (bottom-up) parser, now it's time to add the operators. Let's have a look at Squaak's operators and their precedence. The operators are listed with decreasing precedence (so that high-precedence operators are listed at the top). (I'm not sure if this precedence table is common compared to other languages; some operators may have a different precedence w.r.t. other operators than you're used to. At least the mathematical operators are organized according to standard math rules). unary "-" unary "not" * / % + - .. < <= >= > != == and or



(".." is the string concatenation operator). Besides defining an entry and exit point for the expression parser, you need to define some operator as a reference point, so that other operators' precedence can be defined relative to that reference point. My personal preference is to declare the operator with the lowest precedence as the reference point. This can be done like this: proto 'infix:or' is precedence('1') { ... }



Now, other operators can be defined: proto 'infix:and' is tighter('infix:or') { ... } proto 'infix:<' is tighter('infix:and') { ... } proto 'infix:+' is tighter('infix:<') { ... } proto 'infix:*' is tighter('infix:+') { ... } proto 'prefix:not' is tighter('infix:*') { ... } proto 'prefix:-' is tighter('prefix:not') { ... }



Note that some operators are missing. See the exercises section for this. For more details on the use of the optable, check out docs/pct/pct_optable_guide.pod in the Parrot repository.



 Operators and Precedence



145



Short-circuiting logical operators Squaak has two logical operators: and and or; and results true if and only if both operands evaluate to true, while or results true if at least one of its operands evaluates to true. Both operands are short-circuited, which means that they don't evaluate both operands if that's unnecessary. For instance, if the first operand of the and operator evaluates to false, then there's no need to evaluate the second operand, as the final result of the and-expression cannot become true anymore (remember: both operands must evaluate to true). Let's think about how to implement this. When evaluating an and-expression, we first evaluate the first operand, and if it's true, only then does it make sense to evaluate the second operand. This behavior looks very much the same as an if-statement, doesn't it? In an if-statement, the first child is always evaluated, and if true, the second child (the "then" block) is evaluated (remember, the third child -- the "else" clause -- is optional). It would be great to be able to implement the and operator using a PAST::Op( :pasttype('if') ) node. Well, you can, using the "is pasttype" clause! Here's how: proto 'infix:and' is tighter('infix:or') is pasttype('if') { ... }



So what about the or operator? When evaluating an or-expression, the first operand is evaluated. If it evaluates to true, then there's no need to evaluate the second operand, as the result of the or-expression is already true! Only if the first operand evaluates to false, is it necessary to evaluate the second child. Mmmmm.... what we're saying here is, unless the first operand evaluates to true, evaluate the second child. Guess what pasttype you'd need for that!



Operators PAST types and PIR instructions In the previous section, we introduced the "pasttype" clause that you can specify. This means that for that operator (for instance, the "and" operator we discussed), a PAST::Op( :pasttype('if') ) node is created. What happens if you don't specify a pasttype? In that case a default PAST::Op node is created, and the default pasttype is 'call'. In other words, a PAST::Op node is created that calls the declared operator. For instance, the "infix:+" operator results in a call to the subroutine "infix:+". This means you'll need to implement subroutines for each operator. Now, that's a bit of a shame. Obviously, some languages have very exotic semantics for the "+" operator, but many languages just want to use Parrot's built-in add instruction. How do we achieve that? Instead of adding a "pasttype" clause, specify a "pirop" clause. The "pirop", or "PIR operator", clause tells the code generator what operator should be generated. Instead of generating a subroutine invocation with the operands as arguments, it will generate the specified instruction with the operator's operands as arguments. Neat huh? Let's look at an example: proto 'infix:+' is tighter('infix:<') is pirop('n_add') { ... }



This specifies to use the "n_add" instruction, which tells Parrot to create a new result object instead of changing one of the operands. Why not just the "add" instruction (which takes two operands, updating the first), you might think. Well, if you leave out this "is pirop" stuff, this will be generated: $P12 = "infix:+"($P10, $P11)



You see, three registers are involved. As we mentioned before, PCT does not do any optimizations. Therefore, instead of the generated instruction above, it just emit the following: n_add $P12, $P10, $P11



which means that the PMCs in registers $P10 and $P11 are added, and assigned to a newly created PMC which is stored in register $P12.



 Operators and Precedence



146



To circumfix or not to circumfix Squaak supports parenthesized expressions. Parentheses can be used to change the order of evaluation in an expression, just as you're probably have seen this in other languages. Besides infix, prefix and postfix operators, you can define circumfix operators, which is specified with the left and right delimiter. This is an ideal way to implement parenthesized expressions: proto 'circumfix:( )' is looser('infix:+') is pirop('set') { ... }



By default, a subroutine invocation will be generated for each operator, in this case a call to " circumfix:( )". However, we are merely interested in the expression that has been parenthesized. The subroutine would merely return the expression. Instead, we can use the pirop attribute to specify what PIR operation should be generated; in this case that is the "set" operation, which sets one register to the contents of another. This solution works fine, except that "set" instructions are a bit of a waste. What happens is, the contents of some register is just copied to another register, which is then used in further code generation. This "set" instruction might as well be optimized away. Currently, there are no optimizations implemented in the PCT. There is an alternative solution for adding grammar rules for the parenthesized expressions, by adding it as an alternative of term. The grammar rule term then ends up as: rule term { |  {*}



#= float_constant



|  {*} |  {*}



#= integer_constant #= string_constant



|  {*}



#= primary



| '('  ')' {*} #= expression }



Of course, although we save one generated instruction, the parser will be slightly more inefficient, for reasons that we discussed at the beginning of this episode. Of course, you are free to decide for yourself how to implement this; this section just explains both methods. At some point, optimizations will be implemented in the PCT. I suspect "useless" instructions (such as the "set" instruction we just saw) will then be removed.



Expression parser's action method For all grammar rules we introduced, we also introduced an action method that is invoked after the grammar rule was done matching. What about the action method for the optable? Naturally, there must be some actions to be executed. Well, there is, but to be frank, I cannot explain it to you. Every time I needed the action method for an optable, I just copied it from an existing actions file. Of course, the action method's name should match the name of the optable (the rule that has the "is optable" clause). So, here goes: method expression($/, $key) { if ($key eq 'end') { make $($); } else { my $past := PAST::Op.new( :name($), :pasttype($),



 Operators and Precedence



147 :pirop($), :lvalue($), :node($/) );



for @($/) { $past.push( $($_) ); } make $past; } }



What's Next? This episode covered the implementation of operators, which allows us to write complex expressions. By now, most of our language is implemented, except for one thing: aggregate data structures. This will be the topic of Episode 8. We will introduce the two aggregate data types: array and hashtables, and see how we can implement these. We'll also discuss what happens when we pass such aggregates as subroutine arguments, and the difference with the basic data types.



Exercises Problem 1 Currently, Squaak only has grammar rules for integer and string constants, not floating point constants. Implement this grammar rule. A floating-point number consists of zero or more digits, followed by a dot and at least one digit, or, at least one digit followed by a dot and any number of digits. Examples are: 42.0, 1., .0001. There may be no whitespace between the individual digits and the dot. Make sure you understand the difference between a "rule" and a "token". Hint currently, the Parrot Grammar Engine (PGE), the component that "executes" the regular expressions (your grammar rules), matches alternative subrules in order. This means that this won't work: rule term { |  |  ... }



because when giving the input "42.0", "42" will be matched by , and the dot and "0" will remain. Therefore, put the  alternative in rule term before . At some point, PGE will support longest-token matching, so that this issue will disappear. Solution token float_constant { [ | \d+ '.' \d* | \d* '.' \d+ ] {*} }



 Operators and Precedence



148



Problem 2 Implement the missing operators: (binary) "-", "<=", ">=", "==", "!=", "/", "%", "or" Solution For sake of completeness (and easy copy-paste for you), here's the list of operator declarations as I wrote them for Squaak: rule expression is optable { ... }



proto 'infix:or' is precedence('1') is pasttype('unless') { ... } proto 'infix:and' is tighter('infix:or') is pasttype('if') { ... } proto 'infix:<'



is tighter('infix:and') { ... }



proto 'infix:<='



is equiv('infix:<') { ... }



proto 'infix:>'



is equiv('infix:<') { ... }



proto 'infix:>='



is equiv('infix:<') { ... }



proto 'infix:=='



is equiv('infix:<') { ... }



proto 'infix:!='



is equiv('infix:<') { ... }



proto 'infix:+'



is tighter('infix:<') is pirop('n_add')



proto 'infix:-'



is pirop('n_sub')



proto 'infix:..'



{ ... }



is equiv('infix:+') { ... }



is equiv('infix:+') is pirop('n_concat') { ... }



proto 'infix:*'



is tighter('infix:+') is pirop('n_mul')



proto 'infix:%'



is pirop('n_mod') proto 'infix:/'



{ ... }



is equiv('infix:*') { ... }



is equiv('infix:*') is pirop('n_div')



{ ... }



proto 'prefix:not' is tighter('infix:*') is pirop('n_not') proto 'prefix:-'



is pirop('n_neg') proto 'term:'



{ ... }



is tighter('prefix:not') { ... }



is tighter('prefix:-') is parsed(&term)



{ ... }



 Operators and Precedence



149



References • docs/pct/pct_optable_guide.pod



References [1] http:// epaperpress.com/oper/download/oper.pdf



Hash Tables and Arrays Squaak Language Tutorial Episode 1: Introduction Episode 2: Poking in Compiler Guts Episode 3: Squaak Details and First Steps Episode 4: PAST Nodes and More Statements Episode 5: Variable Declaration and Scope Episode 6: Scope and Subroutines Episode 7: Operators and Precedence Episode 8: Hashtables and Arrays Episode 9: Wrap-Up and Conclusion



Welcome to Episode 8! This is the second-last episode in this tutorial. After this episode, we'll have a complete implementation of our Squaak language. This episode focuses on aggregate data structures: arrays and hashtables. We'll discuss the syntax to assign to them and to construct them. We'll see that implementing the action methods is really easy, almost trivial. After that, we'll make some notes on aggregates as arguments, and how they differ from the basic data types when passing them around as subroutine arguments.



Arrays and Hashtables Besides basic data types such as integer, floating-point and string, Squaak has two aggregate data types: array and hashtable. An array is an object that can store a sequence of values. The values in this sequence can be of different types, unlike some languages that require all elements of an array to be the same type. An example of using arrays is shown below: grades[0] = "A" grades[1] = "A+" grades[2] = "B+" grades[3] = "C+"



A hashtable stores key-value pairs; the key is used as index to store a value. Keys must be string constants, but the value can be of any type. An example is shown below: lastnames{"larry"}



= "wall"



lastnames{"allison"} = "randal"



 Hash Tables and Arrays



Array constructors Just as there are integer literals (42) and string literals ("hello world") that can be assigned to variables, you can have array literals. Below is the grammar rule for this: rule array_constructor { '[' [  [',' ]*]? ']' {*} }



Some examples are shown below: foo = [] bar = [1, "hi", 3.14] baz = [1, [2, 3, 4] ]



The first example creates an empty array and assigns this to foo. The second example shows the construction of three elements, assigning the array to bar. Note that the elements of one array can be of different types. The third example shows the construction of nested arrays. This means that element baz[1][0] evaluates to the value 2 (indexing starts at 0).



Hashtable constructors Besides array literals, Squaak supports hashtable literals, that can be constructed through a hashtable constructor. The syntax for this is expressed below: rule hash_constructor { '{' [ [',' ]* ]? '}' {*} } rule named_field {  '=>'  {*} }



Some examples are shown below: foo = {} bar = { "larry" => "wall", "allison" => "randal" } baz = { "a" => { "b" => 42} }



The first line creates an empty hashtable and assigns this to foo. The second creates a hashtable with two fields: "larry" and "allison". Their respective values are: "wall" and "randal". The third line shows that hashtables can be nested, too. There, a hashtable is constructed that has one field, called "a", and its value is another hashtable, containing a field "b" that has the value 42.



150



 Hash Tables and Arrays



151



Implementation You might think implementing support for arrays and hashtables looks rather difficult. Well, it's not. Actually, the implementation is rather straightforward. First, we're going to update the grammar rule for primary: rule primary {  * {*} } rule postfix_expression { |  {*} |  {*}



#= index #= key



} rule index { '['  ']' {*} }



rule key { '{'  '}' {*} }



A primary object is now an identifier followed by any number of postfix-expressions. A postfix expression is either a hashtable key or an array index. Allowing any number of postfix expressions allows to nest arrays and hashtables in each other, allowing us to write, for instance: foo{"key"}[42][0]{"hi"}



Of course, you as a Squaak programmer must make sure that foo is actually a hashtable, and that foo{"key"} yields an array, and so forth. Implementing this is actually quite simple. First, let us see how to implement the action method index. method index($/) { my $index := $( $ ); my $past := PAST::Var.new( $index, :scope('keyed'), :viviself('Undef'), :vivibase('ResizablePMCArray'), :node($/) ); make $past; }



First, we retrieve the PAST node for expression. Then, we create a keyed variable access operation, by creating a PAST::Var node and setting its scope to 'keyed'. If a PAST::Var node has keyed scope, then the first child is evaluated as the aggregate object, and the second child is evaluated as the index on that aggregate. But wait! The PAST::Var node we just created has only one child!



 Hash Tables and Arrays



152



Here's where the updated action method for primary comes in. This is shown below. method primary($/) { my $past := $( $ );



for $ { my $expr := $( $_ ); $expr.unshift( $past ); $past := $expr; } make $past; }



First, the PAST node for identifier is retrieved. Then, for each postfix-expression, we get the PAST node, and unshift the (current) $past onto it. Effectively, the (current) $past is set as the first child of $expr. And you know what $expr contains: that's the keyed variable access node, that was created in the action method index.



After that, $past is set to $expr; either there's another postfix-expression, in which case this $past will be set as the first child of that next postfix-expression, or, the current $past is set as the result object.



Implementing Constructors To implement the array and hashtable constructors, we're going to take advantage of Parrot's Calling Conventions (PCC). The PCC supports, amongst others, optional parameters, named parameters and slurpy parameters. If you're Dutch, you might think that slurpy parameters make a lot of noise ("slurpen" is a Dutch verb meaning drinking carefully, which you usually do if your beverage is hot, making noise in the process), but you would be wrong. Slurpy parameters will store all remaining arguments that have not yet been stored in other parameters (implying that there can only be one slurpy (positional) parameter, and it should come after all normal (positional) parameters). Parrot will automatically create an aggregate to store these remaining arguments. Besides positional slurpy parameters, you can also define a named slurpy parameter, which will store all remaining named parameters, after all normal (named) arguments have been stored. You might be confused by now. Let's look at an example, as this issue is worth a few brain cells to store. .sub foo .param pmc a .param pmc b .param pmc c :slurpy .param pmc k :named('x') .param pmc l :named('y') .param pmc m :named :slurpy .end



foo(1, 2, 3, 4, 6 :named('y'), 5 :named('x'), 7 :named('p'), 8 :named('q') )



This will result in the following mapping: a: 1 b: 2 c: {3, 4}



 Hash Tables and Arrays



153



k: 5 l: 6 m: {"p"=>7, "q"=>8}



So, after the positional parameters (a, b), c is declared as a slurpy parameters, storing all remaining positional parameters. Parameters k and l are declared as named parameters, which have the respective names "x" and "y". Using these names, values can be passed. After the named parameters, there's the parameter m, which is both flagged as named and slurpy. This parameter will store all remaining named arguments that have not yet been stored by the normal named parameters. The interesting parameters for us are "c" and "m". For the positional slurpy parameter, Parrot creates an array, while for the named slurpy parameter a hashtable is created. This happens to be exactly what we need! Implementing the array and hash constructors becomes trivial: .sub '!array' .param pmc fields :slurpy .return (fields) .end .sub '!hash' .param pmc fields :named :slurpy .return (fields) .end



Array and hashtable constructors can then be compiled into subroutine calls to the respective Parrot subroutines, passing all fields as arguments. (Note that these names start with a "!", which is not a valid Squaak identifier. This prevents us from calling these subs in normal Squaak code).



Basic data types and Aggregates as arguments All data types, both basic and aggregate data types are represented by Parrot Magic Cookies (PMCs). The PMC is one of the four built-in data types that Parrot can handle; the others are integer, floating-point and string. Currently, the PCT can only generate code to handle PMCs, not the other basic data types. Parrot has registers for each its four built-in data types. The integer, floating-point and string registers store the actual data value, while PMC registers store a reference to the PMC object. This has consequences for how PMCs are handled when passing them as arguments. When passing a PMC as an argument, the invoked subroutine gets access to the PMC reference; in other words, PMCs are passed by reference. This means that the subroutine can change the srcinal argument that was passed by the caller. Of course, it depends what instructions are being generated, what the invoked subroutine does to the references. In Squaak, when passing basic data values, these cannot be changed by the invoked subroutine. When assigning a new value to a parameter, a whole new object is created and bound to the parameter identifier. No changes are made to the srcinal argument. Aggregate data types are handled differently, however. When an invoked subroutine assigns to an index or hashtable field of a parameter, then the srcinal argument is affected. In other words, basic data types have by value semantics, while aggregate data types have by reference semantics. A short example to demonstrate this: sub foo(a,b,c) a



= 42



b[0]



= 1



c{"hi"} = 2 end



 Hash Tables and Arrays



154



var a = 0 var b = [] var c = {} foo(a,b,c)



print(a, b[0], c{"hi"} ) # prints 0, 1, 2



What's Next? This was the last episode to discuss implementation details to make Parrot (run) Squaak. After doing this episode's exercises, your implementation should be fairly complete. Next episode will be the last of this series, in which we'll recap what we did, and demonstrate our language with a nice demo program.



Exercises Problem 1 We've shown how to implement keyed variable access for arrays, by implementing the action method for index. The same principle can be applied to keyed access for hashtables. Implement the action method for key. method key($/) { my $key := $( $ ); make PAST::Var.new( $key, :scope('keyed'), :vivibase('Hash'), :viviself('Undef'), :node($/) ); }



Problem 2 Implement the action methods for array_constructor and hash_constructor. Use a PAST::Op node and set the pasttype to 'call'. Use the "name" attribute to specify the names of the subs to be invoked (e.g., :name("!array") ). Note that all hash fields must be passed as named arguments. Check out PDD26 for doing this, and look for a "named " method. method named_field($/) { my $past := $( $ ); my $name := $( $ ); ## the passed expression is in fact a named argument, ## use the named() accessor to set that name. $past.named($name); make $past; } method array_constructor($/) { ## use the parrot calling conventions to ## create an array, ## using the "anonymous" sub !array ## (which is not a valid Squaak name) my $past := PAST::Op.new( :name('!array'),



 Hash Tables and Arrays



155 :pasttype('call'), :node($/) );



for $ { $past.push($($_)); } make $past; }



method hash_constructor($/) { ## use the parrot calling conventions to ## create a hash, using the "anonymous" sub ## !hash (which is not a valid Squaak name) my $past := PAST::Op.new( :name('!hash'), :pasttype('call'), :node($/) ); for $ { $past.push($($_)); } make $past; }



Problem 3 We'd like to add a little bit of syntactic sugar for accessing hashtable keys. Instead of writing foo{"key"}, I'd like to write foo.key. Of course, this only works for keys that do not contain spaces and such. Add the appropriate grammar rule (call it "member") that enables this syntax, and write the associated action method. Make sure this member name is converted to a string. Hint: use a PAST::Val node for the string conversion. rule postfix_expression { |  {*}



#= key



|  {*}



#= member



|  {*}



#= index



} rule member { '.'  {*} }



method member($/) { my $member := $( $ ); ## x.y is syntactic sugar for x{"y"}, ## so stringify the identifier: my $key



:= PAST::Val.new( :returns('String'), :value($member.name()), :node($/) );



## the rest of this method is the same



 Hash Tables and Arrays



156



## as method key() above. make PAST::Var.new( $key, :scope('keyed'), :vivibase('Hash'), :viviself('Undef'), :node($/) ); }



Wrap-Up and Conclusion Squaak Language Tutorial Episode 1: Introduction Episode 2: Poking in Compiler Guts Episode 3: Squaak Details and First Steps Episode 4: PAST Nodes and More Statements Episode 5: Variable Declaration and Scope Episode 6: Scope and Subroutines Episode 7: Operators and Precedence Episode 8: Hashtables and Arrays Episode 9: Wrap-Up and Conclusion



Welcome to the final Episode of the Parrot Compiler Tools Tutorial! Let's review the previous episodes, and summarize this tutorial.



Review In Episode 1, we introduced the Parrot Compiler Tools (PCT), gave a high-level feature overview of Squaak, the case study language that we are implementing in this tutorial, and we generated a language shell that we use as a foundation to implement Squaak. Episode 2 discussed the general structure of PCT-based compilers. After this, we described each of the four default compilation stages: parse phase, parse tree to PAST, PAST to POST and POST to PIR. We also added a command line banner and command line prompt to the interactive language shell. In Episode 3, we introduced the full grammar of the Squaak language. After this, we started implementing the first bits, after which we were able to generate code for (simple) assignments. In Episode 4 we discussed the construction of Parrot Abstract Syntax Tree nodes in more detail, after which we implemented the if-statement and throw-statement. Episode 5 focused on variable declarations and variable scope. We implemented the necessary infrastructure to handle global and local variables correctly. In Episode 6 we continued the discussion of scope, but now in the context of subroutines. After this we implemented subroutine invocation. Episode 7 extended our grammar to handle complex expressions that allows us to use arithmetic and other operators. We discussed how to use PCT's built-in support for handling operator precedence. In the previous episode, Episode 8, we discussed the grammar and action methods for handling the aggregate data types of Squaak: arrays and hashes. We also touched on the topic of argument passing by reference and by value. If you followed the tutorial and did the exercises, your implementation should be complete. Although a lot of the implementation was discussed, some parts were left as the proverbial exercise to the reader. This is to stimulate you to get your hands dirty and figure out things for yourself, while the text contained enough hints (in my opinion) to



 Wrap-Up and Conclusion



157



solve the given problems. Sure enough, this approach requires you to spend more time and think for yourself, but I think you're reading all this stuff to learn something. The extra time spent is well worth it, in my opinion. Now it's time to see what we can do with this language. Squaak is more than just the average calculator example, which is often provided in beginner's discussions on parsers; it's a complete programming language.



What's Next? This is the last episode of the Parrot Compiler Tools tutorial. We showed how we implemented a complete language for the Parrot virtual machine in only a few hundred lines of source code. Surely, this must be the proof that the PCT really is an effective toolkit for implementing languages. At the moment of writing, the PCT still lacks efficient support for certain language constructs. Therefore, we focused on the parts that are easy to build with the PCT. Once the PCT is feature complete, there's bound to be another tutorial on advanced features. Think of object-oriented programming, closures, coroutines, and advanced control-flow such as return statements. Most of them can be done already, but are too complex for this tutorial's level.



The Game of Life You might have noticed that Squaak looks a bit like Lua, although it does differ in some points. This is not entirely accidental. In the distribution of the Lua source code, there's an example called "life.lua", which implements Conway's "Game of Life". This is a nice demonstration program, and it's easy to port it to Squaak. Its implementation is shown below. Run it, and enjoy! ## John Conway's Game of Life ## Implementation based on life.lua, found in Lua's distribution. ## var width



= 40



# width of "board"



var height



= 20



# height of "board"



var generation



= 1



# generation couner



var numgenerations = 50



# how often should we evolve?



## initialize board to all zeroes sub initboard(board) for var y = 0, height do for var x = 0, width do board[y][x] = 0 end end end ## spawn new life in board, at position (left, top), ## the life data is stored in shapedata, and shape width and ## height are specified. sub spawn(board, left, top, shapew, shapeh, shapedata) for var y = 0, shapeh - 1 do for var x = 0, shapew - 1 do board[top + y][left + x] = shapedata[y * shapew + x] end end end



 Wrap-Up and Conclusion



158



## calculate the next generation. sub evolve(thisgen, nextgen) var ym1 = height - 1 var y



= height



var yp1 = 1 var yi



= height



while yi > 0 do var xm1 = width-1 var x



= width



var xp1 = 1 var xi



= width



while xi > 0 do var sum = thisgen[ym1][xm1] + thisgen[ym1][x] + thisgen[ym1][xp1] + thisgen[y][xm1] + thisgen[y][xp1] + thisgen[yp1][xm1] + thisgen[yp1][x] + thisgen[yp1][xp1]



nextgen[y][x] = sum==2 and thisgen[y][x] or sum==3 xm1 = x x



= xp1



xp1 = xp1 + 1 xi



= xi - 1



end



ym1 = y y



= yp1



yp1 = yp1 + 1 yi



= yi - 1



end end ## display thisgen to stdout. sub display(thisgen) var line = "" for var y = 0, height do for var x = 0, width do if thisgen[y][x] == 0 then



 Wrap-Up and Conclusion



159 line = line .. "-" else line = line .. "O" end



end line = line .. "\n" end print(line, "\nLife - generation: ", generation) end



## main program sub main() var heart



= [1,0,1,1,0,1,1,1,1]



var glider



= [0,0,1,1,0,1,0,1,1]



var explode = [0,1,0,1,1,1,1,0,1,0,1,0] var thisgen = [] initboard(thisgen) var nextgen = [] initboard(nextgen) spawn(thisgen,3,5,3,3,heart) spawn(thisgen,5,4,3,3,glider) spawn(thisgen,25,10,3,4,explode) while generation <= numgenerations do evolve(thisgen, nextgen) display(thisgen) generation = generation + 1 ## prevent switching nextgen and thisgen around, ## just call evolve with arguments switched. evolve(nextgen, thisgen) display(nextgen) generation = generation + 1 end end ## start here. main()



Note the use of a subroutine "print". Check out the file src/builtins/say.pir, and rename the sub "say" (which was generated by the language shell creation script) to "print".



 Wrap-Up and Conclusion



Solution If you don't feel like doing exercises or just want to see what it looks like without doing any trouble, here's what it looks like (this is life generation 9). -----------------------------------------------------------------------------------------------------------------------------O--------------------------------------OO------------------------------------OO--O--------------------------------------OO-----------------------------------OOOOO------------------OOO------------------------------------O---O----------------------------------O-----O--------------------------------O---O---O-------------------------------O--O-O--O-------------------------------O---O---O--------------------------------O-----O----------------------------------O---O------------------------------------OOO--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Life - generation: 9



But really, it doesn't compare to seeing this program run on Parrot :-)



Exercises Squaak was designed to be a simple language, offering enough features to get some work done, but at the same time keeping it simple. Of course, after reading this tutorial, You are an expert too ;-) If you feel like adding more features, here are some suggestions. • Implement prefix and post fix increment/decrement operators, allowing you to write "gen eration++" instead of "generation = generation + 1". • Implement augmenting assign operators, such as "+=" and friends. • Extend the grammar to allo w multiple variable declarations in one statement, allowing you to write "var x = 1, y, z =3". Of course, the initialization part should still be optional. How do you make sure that the identifier and initialization expression are kept together? • Implement a mechanism (such as an "import" sta tement) to include or load an other Squaak file, so Squaak programs can be split into multiple files. The PCT does not have any support for this, so you'll need to write a bit of PIR to do this. • Improve the for-statement, to allow for a negativ e step. Note that the loop condition becomes more complex when doing so. Note that these are suggestions, and I did not implement them myself, so I won't have a solution for you at the end.



160



 Wrap-Up and Conclusion



Final words and Acknowledgments By now, you should have got a good impression of the PCT and you should be able to work on other languages targeting Parrot. Currently, work has been done on ECMAScript, Python, Ruby and of course Perl 6. Most of them are not complete yet (hint, hint). I hope you enjoyed reading this tutorial and learned enough to feel confident about working on other (existing) languages targeting Parrot. The Perl 6 implementation can still use more contributors! Many thanks to all who read this tutorial and provided me with hints, tips and feedback! Thank You for reading this!



161



 162



Resources and Licensing Resources Resources These blog posts are released into the public domain and can be adapted directly: • http://www.parrotblog.org/2008/03/targeting-parrot-vm.html • http://www.parrotblog.org/2008/03/episode-2-poking-in-compiler-guts.html • http://www.parrotblog.org/2008/03/episode-3-squaak-details-and-first.html • http://www.parrotblog.org/2008/03/episode-4-past-nodes-and-more.html • http://www.parrotblog.org/2008/03/episode-5-variable-declaration-and.html • http://www.parrotblog.org/2008/03/episode-6-scope-and-subroutines.html • http://www.parrotblog.org/2008/03/episode-7-operators-and-precedence.html • http://www.parrotblog.org/2008/03/episode-8-hashtables-and-arrays.html Other Resources • Parrot Documentation • Parrot Design Docs



[1]



[2]



• Parrot Project Home



[3]



References [1] http:// www.parrotcode.org/docs/ [2] http:// www.parrotcode.org/docs/pdd/ [3] http:// www.parrotcode.org



 Licensing



163



Licensing Licensing The text of this book is released under the following license: Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License , Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License."



Parrot, it's source code, and the tools required to build it are released under the terms of the Artistic License 2.0.



Compatibility with Artistic 2.0 Parrot and most of it's documentation library is released under the terms of the Artistic 2.0 license. For maximum compatibility, and for the ability to reuse content developed here on Wikibooks in other places in the Parrot ecosystem, some users have opted to have their contributions to this book explicitly dual-licensed GFDL and Artistic 2.0: • Whiteknight • Andrew Whitworth Contributions to this book by users who are listed above may be reused under the Artistic 2.0 license where necessary. This is not a condition to editing this book, and edits made by users who have not listed themselves above are to be considered GFDL-only.



 Article Sources and Contributors



Article Sources and Contributors Wikibooks:Collections Preface Source : http://en.wikibooks.org/w/index.php?oldid=2347851 Contributors : Adrignola, Jomegat, Magesha, Martin Kraus, Mike.lifeguard, RobinH, Whiteknight IntroductionSource: http://en.wikibooks.org/w/index.php?oldid=1381261 Contributors : Jkeenan, Whiteknight, Yosri IntroductionSource: http://en.wikibooks.org/w/index.php?oldid=1381261 Contributors : Jkeenan, Whiteknight, Yosri Building ParrotSource: http://en.wikibooks.org/w/index.php?oldid=1688198 Contributors : Jkeenan, KenJackson.US, Whiteknight, Yosri Running ParrotSource: http://en.wikibooks.org/w/index.php?oldid=1373737 Contributors : Dallas1278, Whiteknight, Yosri Parrot ProgrammingSource: http://en.wikibooks.org/w/index.php?oldid=2176469 Contributors : Ayardley, Jkeenan, KenJackson.US, ReiniUrban, Van der Hoorn, Whiteknight, Yosri, 2



anonymous edits Parrot Assembly Language Source : http://en.wikibooks.org/w/index.php?oldid=1227227 Contributors : Whiteknight, Yosri Parrot Intermediate Representation Source: http://en.wikibooks.org/w/index.php?oldid=2286724 Contributors : Whiteknight, Yosri, 4 anonymous edits Parrot Magic CookiesSource: http://en.wikibooks.org/w/index.php?oldid=1597840 Contributors : Adrignola, Jrtayloriv, Whiteknight, Yosri, 1 anonymous edits Multithreading and Concurrency Source: http://en.wikibooks.org/w/index.php?oldid=1275298 Contributors: Whiteknight, Yosri, 4 anonymous edits Exception HandlingSource: http://en.wikibooks.org/w/index.php?oldid=1238614 Contributors : Whiteknight, 1 anonymous edits Classes and ObjectsSource: http://en.wikibooks.org/w/index.php?oldid=2346012 Contributors : Mastersrp, Whiteknight, 1 anonymous edits The Parrot DebuggerSource: http://en.wikibooks.org/w/index.php?oldid=1266967 Contributors : Whiteknight, 1 anonymous edits Parrot Compiler ToolsSource: http://en.wikibooks.org/w/index.php?oldid=2146153 Contributors : Cinchent, Jrtayloriv, Whiteknight, 2 anonymous edits Parrot Grammar EngineSource: http://en.wikibooks.org/w/index.php?oldid=2146156 Contributors : Afbach, Cinchent, Furrykef, P anic2k4, Whiteknight, 4 anonymous edits Not Quite PerlSource: http://en.wikibooks.org/w/index.php?oldid=1705277 Contributors : Jrtayloriv, Whiteknight, 7 anonymous edits Optables and Expressions Source: http://en.wikibooks.org/w/index.php?oldid=1935729 Contributors : Jrtayloriv, Whiteknight, 3 anonymous edits Advanced PGESource: http://en.wikibooks.org/w/index.php?oldid=2146157 Contributors : Cinchent, Whiteknight, 1 anonymous edits Building A CompilerSource: http://en.wikibooks.org/w/index.php?oldid=1828126 Contributors : Tedkat, Whiteknight, 2 anonymous edits HLL InteroperationSource : http://en.wikibooks.org/w/index.php?oldid=1274012 Contributors : Whiteknight Parrot InternalsSource: http://en.wikibooks.org/w/index.php?oldid=1275313 Contributors : Whiteknight, 1 anonymous edits IMCC and PIRCSource: http://en.wikibooks.org/w/index.php?oldid=2249684 Contributors : Whiteknight, 2 anonymous edits Run Core Source : http://en.wikibooks.org/w/index.php?oldid=1258809 Contributors : Whiteknight, 1 anonymous edits Memory and Garbage Collection Source : http://en.wikibooks.org/w/index.php?oldid=1459282 Contributors : Van der Hoorn, Whiteknight, 2 anonymous edits PMC System Source: http://en.wikibooks.org/w/index.php?oldid=2372318 Contributors : DavidCary, Whiteknight, 1 anonymous edits String SystemSource: http://en.wikibooks.org/w/index.php?oldid=1274912 Contributors : Whiteknight Exception SubsystemSource: http://en.wikibooks.org/w/index.php?oldid=1443486 Contributors : Van der Hoorn, Whiteknight IO Subsystem Source: http://en.wikibooks.org/w/index.php?oldid=1238629 Contributors : Whiteknight, 1 anonymous edits JIT and NCI Source: http://en.wikibooks.org/w/index.php?oldid=1238630 Contributors : Whiteknight, 1 anonymous edits Parrot EmbeddingSource: http://en.wikibooks.org/w/index.php?oldid=1238631 Contributors : Whiteknight, 1 anonymous edits Extensions Source: http://en.wikibooks.org/w/index.php?oldid=1266974 Contributors : Whiteknight, 1 anonymous edits Packfiles Source : http://en.wikibooks.org/w/index.php?oldid=1274888 Contributors : Whiteknight PIR ReferenceSource: http://en.wikibooks.org/w/index.php?oldid=1238633 Contributors : Whiteknight, 1 anonymous edits PASM ReferenceSource: http://en.wikibooks.org/w/index.php?oldid=2020376 Contributors: Benabik, Whiteknight, 1 anonymous edits PAST Node ReferenceSource: http://en.wikibooks.org/w/index.php?oldid=1238635 Contributors : Whiteknight, 1 anonymous edits Languages on ParrotSource: http://en.wikibooks.org/w/index.php?oldid=2140991 Contributors: Whiteknight, 6 anonymous edits HLLCompiler ClassSource: http://en.wikibooks.org/w/index.php?oldid=1377590 Contributors : Whiteknight, 1 anonymous edits Command Line OptionsSource: http://en.wikibooks.org/w/index.php?oldid=1238638 Contributors : Whiteknight, 1 anonymous edits Built-In PMCsSource: http://en.wikibooks.org/w/index.php?oldid=2372324 Contributors : DavidCary, Whiteknight, 1 anonymous edits Bytecode File FormatSource : http://en.wikibooks.org/w/index.php?oldid=1274885 Contributors : Whiteknight, 1 anonymous edits VTABLE ListSource: http://en.wikibooks.org/w/index.php?oldid=1356142 Contributors : Whiteknight Squaak TutorialSource: http://en.wikibooks.org/w/index.php?oldid=2210477 Contributors : MC Scared of Bees, Whiteknight, 1 anonymous edits IntroductionSource: http://en.wikibooks.org/w/index.php?oldid=1957002 Contributors : Whiteknight, Wtachi, 7 anonymous edits Poking in Compiler Guts Source : http://en.wikibooks.org/w/index.php?oldid=1756857 Contributors : Adrignola, Jrtayloriv, Whiteknight, Wtachi Squaak Details and First Steps Source : http://en.wikibooks.org/w/index.php?oldid=2080516 Contributors : Amoe, Jrtayloriv, Whiteknight, 1 anonymous edits PAST Nodes and More Statements Source : http://en.wikibooks.org/w/index.php?oldid=1635846 Contributors : Jrtayloriv, Whiteknight, 1 anonymous edits Variable Declaration and Scope Source: http://en.wikibooks.org/w/index.php?oldid=1635849 Contributors : Jrtayloriv, Whiteknight, 2 anonymous edits Scope and SubroutinesSource: http://en.wikibooks.org/w/index.php?oldid=1635672 Contributors : Jrtayloriv, Whiteknight



164



 Article Sources and Contributors Operators and Precedence Source : http://en.wikibooks.org/w/index.php?oldid=1180380 Contributors : Whiteknight Hash Tables and ArraysSource: http://en.wikibooks.org/w/index.php?oldid=1635438 Contributors : Whiteknight, 1 anonymous edits Wrap-Up and ConclusionSource : http://en.wikibooks.org/w/index.php?oldid=1180383 Contributors : Whiteknight Resources Source: http://en.wikibooks.org/w/index.php?oldid=1147623 Contributors : Whiteknight Licensing Source: http://en.wikibooks.org/w/index.php?oldid=1278706 Contributors : Whiteknight



165



 Image Sources, Licenses and Contributors



Image Sources, Licenses and Contributors Image:Wikibooks-logo-en-noslogan.svg Source: http://en.wikibooks.org/w/index.php?title=File:Wikibooks-logo-en-noslogan.svg License: logo Contributors : User:Bastique, User:Ramac et al. File:Heckert GNU white.svg Source: http://en.wikibooks.org/w/index.php?title=File:Heckert_GNU_white.svg License: Creative Commons Attribution-Sharealike 2.0 Contributors : Aurelio A.



Heckert 



166



 License



License Creative Commons Attribution-Share Alike 3.0 //creativecommons.org/licenses/by-sa/3.0/



167