================================================================================ WICCA v2.0 (alpha) Copyright (C) 2006 Columbia University. ================================================================================ Created: 03/25/05 Updated: 05/26/06 Overview ======== Wicca transforms software programs to enable Aspect-Oriented Programming (AOP) with the goal of improving modularity and evolution, and reducing tedious and error-prone programming. Wicca can transform source code or byte code, at compile-time or runtime (yes, that's right, Wicca can transform the program while it's running). Wicca can also transform a program without modifying it via breakpoint weaving. Weaving Scenarios ================= Wicca is a hybrid weaver that supports 6 different weaving scenarios (see table below). A design goal of Wicca is that all weavers will be functionally equivalent and interchangeable. Weave Time Artifact Woven Component Responsible Supported? ---------- -------------- --------------------- ---------- Compile time Byte code Phx.Morph YES (pereader.exe) Source code wcs YES (wcs.exe) Load time Byte code Phx.Morph YES Breakpoint Phx.Morph YES Source code wcs NO Runtime Byte code Phx.Morph YES Breakpoint Phx.Morph YES Source code wcs NO Note: We will refer to compile-time weaving as "static weaving" and load-time and runtime weaving as "dynamic weaving". Phx.Morph --------- The static weaving scenarios are supported by two standalone tools: Phx.Morph and wcs. Static byte-code weaving is supported by Phx.Morph. Phx.Morph is built on top of Phoenix, Microsoft's back-end compiler infrastructure. To run Phx.Morph from the command-line, you specify it as a plugin for the Microsoft Phoenix pereader assembly rewriting tool. Aspects are specified as regular assemblies with custom [Advice] attributes. See \Morpher\README.TXT for more info about Phx.Morph. wcs --- wcs is a C# source weaver and compiler, similar to AspectJ. It can be used standalone via the wcs.exe command-line tool or programmatically via the Wicca.Wcs API. (Wicca currently does not support dynamic source weaving so the API is unused.) Aspects are specified as regular C# source files with custom [Advice] attributes. wcs supports a small extension to C# to enable statement annotations. Here's an example: public SimpleDraw() { [Log(Sev.Info, "Creating a line")] Shape s = new Line(new Point(1, 9), new Point(9, 1)); } wcs uses the standard C# compiler so before compiling it must convert the statement annotation to legal C#: [Statement(19, "LogAttribute", Sev.Info, "Creating a line")] public SimpleDraw() { Shape s = new Line(new Point(1, 9), new Point(9, 1)); } wcs and Phx.Morph both recognize this encoding and support advising annotated statements. The Microsoft Managed Debugger ------------------------------ The Wicca dynamic AOP system is implemented as a plug-in for the Microsoft Managed Debugger (mdbg). Mdbg takes care of launching the client process, attaching a debugger to it, managing communication between the debugger and the process, and providing an interactive command-line shell to the user. Dynamic Byte Code Weaving ------------------------- Phx.Morph also exposes an API to support load-time and runtime weaving ("dynamic weaving"). For dynamic byte code weaving, Phx.Morph weaves the assembly just like in the static weaving case and produces a woven temporary assembly. Wicca diffs the woven assembly with the currently running program using the AssemblyDiff component and updates it as necessary using the Microsoft Debugger Edit-and-Continue API. Dynamic Software Updating ------------------------- Wicca's diff-and-update mechanism is a general-purpose Dynamic Software Updating (DSU) mechanism that supports (almost) arbitrary updates to the running program via the Microsoft Debugger Edit-and-Continue (EnC) API. The diff process produces a byte code delta file and a metadata delta file which the EnC API consumes. Wicca supports three DSU scenarios: 1. Dynamic AOP - Phx.Morph automatically creates the woven assembly 2. Assembly Patch - The !patch command is used to supply a new assembly 3. Delta File Patch - The !patch command is used to supply delta files Breakpoint Weaving ------------------ For dynamic breakpoint weaving, Phx.Morph weaves the assembly just like in the static weaving case but instead of injecting advice calls, it calls a callback function that Wicca implements. When Wicca gets the callback it sets breakpoints in the running program using the Microsoft Debugger Breakpoint API. When a breakpoint is hit, Wicca obtains the context parameters using the Microsoft Debugger API and then executes the advice method method *in the process space of the running program* using the Microsoft Debugger Func-Eval API. General Caveats --------------- While the standalone Phx.Morph plugin is fairly mature, the dynamic Wicca AOP system is only an alpha release. Here are some of the major issues: - Wicca/Phx.Morph chokes when binary weaving some assemblies. In these cases, use the breakpoint weaver. - Not all of AspectJ's AOP functionality is supported - Dynamic weaving is inherently slow because it disables JIT optimizations and enables Edit-and-Continue support. Moreover, binary weaving is nonoptimized and breakpoint weaving is significantly slower than byte code weaving (as expected, but could still be optimized). - Weavers are almost but not quite functionally equivalent. The breakpoint weaver cannot pass some context, including annotation context. The byte code weaver may succeed when used statically but fail when used dynamically (probably due to a dynamic update failure). - The source weaver (wcs) has not been fully tested and is very flaky. - The dynamic software updating mechanism has the same limitations as the Edit-and-Continue API. We can only add private members (fields, properties, and methods) and replace method bodies. - Its currently not possible to disable aspects at runtime A more detailed issue list can be found below. ================================================================================ Requirements ------------ o Microsoft .NET 2.0 o Microsoft Visual Studio 2005 * Required to build Wicca and run regression test suite o Microsoft Phoenix RDK * Download: http://research.microsoft.com/phoenix o Perl * Download: http://www.activestate.com * Required to run regression test suite Installation ------------ 1. Install Visual Studio 2005 2. Add the VS 2005 environment variables from \Common7\Tools\vsvars32.bat to your Environment 3. Install Phoenix RDK (uninstall previous versions first). 4. Make sure the RDKRoot environment variable is set to the Phoenix RDK root directory 5. Install Wicca package 6. Build wicca using the Microsoft Build Tool: C:\PROJECTS\Wicca> msbuild.exe Wicca.sln You may see some warnings but you should see no errors. 7. Run the regression test suite: C:\PROJECTS\Wicca> dotest.bat This will build all the Wicca projects and build and run all the examples in the regression test suite. You should see no failed tests in the summary at the end. ================================================================================ Change History ============== 2.0 --- Note: Wicca v2 is a complete rewrite of v1. It uses the slower out-of- process Microsoft Debugger API instead of the faster in-process Microsoft Profiler API. This rewrite was required to support .NET 2.0. However, there are substantial benefits to using the Debugger API including: + More powerful DSU capabilities - We can now add fields and methods (although this hasn't been tested!) + Breakpoint weaving Unfortunately, we have not migrated the Edit-and-Continue and dynamic source weaving support to v2 yet. + Static byte-code weaving + Static source code weaving + Dynamic byte-code weaving + Dynamic breakpoint weaving + More powerful source code weaving + More powerful dynamic software updating + Integrated debugger with limited DSU debugging support 1.0 --- + Very limited support for dynamic source code weaving via the CSharpParser and Microsoft Profiler API + Multi-Language Edit-and-Continue (C#, JavaScript, VB.NET) ================================================================================ Package Contents ================ Common\ Common libraries and shared source files Common\bin\ Target directory for all Wicca C++ libraries Common\AssemblyHelper\ Assembly Diff COM component (C++) Common\ILXLog\ Logging COM component (C++) Common\ILXCrashFinder\ Stack Walker COM component (C++) Wicca ----- Wicca\ Wicca root directory Wicca\README.TXT This file Wicca\Wicca.sln Main solution file that includes the 17 projects that comprise Wicca. Run "msbuild Wicca.sln" to build the entire solution. Wicca\dotest.bat Runs the entire regression test suite Wicca\bin\ Target directory for all Wicca C# executables and libraries Wicca\Docs\ Wicca documentation Wicca\examples\ Dynamic and static weaving regression tests Wicca\examples\dotest.bat Runs the dynamic and static weaving regression tests Wicca\mdbg\ Microsoft Managed Debugger v2.0 (slightly modified). This directory contains several subprojects. Wicca\wicca\ Wicca plugin for the Microsoft Managed Debugger Phx.Morph --------- Wicca\Morpher\ Phx.Morph byte code weaver root directory Wicca\Morpher\MorphPlugin\ Phx.Morph plugin for the standalone Microsoft Phoenix Portable Executable rewriter tool (pereader.exe) Wicca\Morpher\Phx.Aop\ Shared AOP library Wicca\Morpher\Phx.Morph\ Phx.Morph core library Wicca\Morpher\examples\ Static weaving regression tests Wicca\Morpher\scripts\ Shared batch files and perl scripts for building and testing Wicca\Morpher\README.TXT Phx.Morph-specific readme Wicca\Morpher\dotest.bat Runs the static weaving regression tests wcs --- Wicca\wcs\ Standalone wcs.exe source code weaver and compiler Wicca\Wicca.wcs\ Source code weaver API ================================================================================ FAQ --- ================================================================================ KNOWN ISSUES ============ General ------- o Some call joinpoints have no filename or linenumber o If mdbg prompts the user (e.g., due to an error), dotest hangs Binary Weaving -------------- o Binary weaving causes the line number to be incremented in the woven executable. This causes the advice to be applied multiple times incorrectly. o Rewriting Goblin.Engine.dll without changes results in Goblin failing to run o dotest.Logging.bat binary fails with Unhandled Exception: System.MissingMethodException: Method not found: 'Void LogA ttribute..ctor(LogAspect, System.String)'. at SimpleDraw..ctor() at SimpleDraw.Main() unless I explicitly do a System.Type forceLoad2 = typeof(LogAspect); o Around/proceed currently doesn't work o Some pointcuts not supported: initialization, preinitialization, staticinitialization, handler, adviceexecution, cflow, cflowbelow, args, and if o This, args, and target only work for execution joinpoints o Can't import or replace methods with multiple return statements or exception handlers o If the client assembly initializes fields inline we get a PEVERIFY error: class Person { string name = "joe"; o Sometimes a certain aspect ordering or advice method implementation will cause BackPatchIR to fail for instance advice. Breakpoint Weaving ------------------ o Func-eval throws an EvalException event if the advice method throws an exception o Breakpoint weaving doesn't expose Annotation context o Breakpoint weaver requires aspect assembly to be loaded by the client app before the weaver can invoke advice methods. Have BP weaver call Reflection.Load() to load the aspect assembly on demand. o Breakpoint weaving still creates a morphed exe o Because BP Weaver sets line BPs, it can't fail if there are multiple statements on the same line. For example, assume we have a set() pointcut: Person p = new Person(); p.Name = "Marc"; The advice method would be called before p was instantiated so Target will be NULL. BP Weaving will also fail if there are multiple expressions on the same line: p.a = p.b + p.c; Dynamic Software Updating ------------------------- (These are inherited from the Microsoft Debugger Edit-and-Continue API limitations) o Can only add private members (fields, methods, properties) and replace method bodies o Cannot add members to a value type o Cannot modify a generic type o Cannot remove locals from a function o Cannot change a function's signature o Cannot make certain changes to the active function o Cannot change the base class of a type o Cannot modify mscorlib o Cannot modify NGEN'd assemblies o EnC does not pitch stale versions of the method o EnC requires super-non-optimized code-gen. EnC added fields are very nonoptimized: they're added to a link list off the sync-block, making them slow to access o EnC requires running under a debugger. The debugger can not attach (*) or detach. (*= not 100% true in general. Code loaded before attach can not be EnCed) Attributes ---------- o Attributes we instantiate cannot have tokens in the blob. For example, if the attribute ctor takes a Type argument, this will blow up because the type is not defined in the EmittedAssembly. o Statement annotation arguments must be a "constant expression, typeof expression or array creation expression" ================================================================================ TODO ---- o Support adding events o Support adding attributes o Support adding classes o Support thisJoinPoint and thisJoinpointStatic o Support aspect precedence o Support named pointcuts o Use msbuild to build Phx.Aop, Phx.Morph, and MorphPlugin from the command-line o Should be able to specify a base class for context parameters that are references o mdbg should allow the user to apply aspects or unweave at anytime. Currently, it waits for the hosted app to reach a stop state. o Statement annotations need to apply at arbitrary statements, not just calls o Should detect the following pointcut expression error: "[PrintAttr] call(* *.*(..))" should be "call([PrintAttr] * *.*(..))" o For unweaving, need to set the current path back to the original path o Add AppendBoolArg, AppendIntArg, etc. to MethodEditor.cs and AppendAttributeParam o Verify that we don't load assemblies using Reflection unless needed o Is there a way we can have a component in the client app listen and respond to breakpoints? o A possible implementation of Around advice in BP Weaver would be to set a bp on foo(). When the bp fires we call the advice method and then CHANGE THE IP to point to just after foo() is called. We also have to make sure that the stack is left in a consistent state (e.g., return values are popped). o Support aspect assemblies in wcs o Detect unsupported updates o Support Edit-and-Continue o Consolidate walking and parsing code for Attribute blobs ================================================================================ Main contact ------------ Marc Eaddy (eaddy@cs.columbia.edu) Credits ------- Marc Eaddy (Columbia University) Boriana Ditcheva (Columbia University) Rajesh Ramakrishnan (Columbia University) Adam Vartanian (Columbia University) Weiping Hu (Microsoft) Acknowledgements ---------------- We thank Mike Stall, Jonathan Keljo, and Rick Byers of the Microsoft CLR Team for there work on the Managed Debugger and help with the Profiler and Debugger APIs. We thank Paddy McDonald of the Microsoft Phoenix Team for his work on updating the PDB and Julian Burger, formerly of the Phoenix Team, for his initial Phoenix-based byte code weaver. ================================================================================