Expronicon

Stable Dev Build Status Coverage Downstream

Collective tools for metaprogramming on Julia Expr, the meta programming standard library for MLStyle.

Meta programming in general can be decomposed into three steps:

  1. analyse the given expression
  2. transform the given expression to:
    • another target expression
    • a convenient intermediate representation (IR) that is easy to further manipulate/analyse
  3. generate the target code, the target code can be:
    • most commonly the Julia Expr
    • another language's abstract/concrete syntax tree
    • a lower-level IR

The package Expronicon is thus written to assist you writing meta programs in the above manner.

Builtin Syntax Types

One of the most important tool in Expronicon is the syntax types, these are types describing a specific Julia syntax, e.g JLFunction describes Julia's function definition syntax. The syntax type uses a canonical data structure to represent various syntax that has the same semantic, which is convenient when one wants to manipulate, generate such objects.

They will allow you to:

  1. easily analysis a given Julia Expr by converting it to the syntax type
  2. easily manipulate a given Julia Expr with data structure designed for easier manipulation
  3. easily generate a Julia Expr using codegen_ast.

Let's take JLFunction as our example again, in Julia, function can be declared in many different ways:

You can define a function using the short form

f(x) = x

or you can declare the same function using the function keyword

function f(x)
    return x
end

If we look at their expression object, we will find they actually have quite different expression structure:

ex1 = @expr f(x) = x
:(f(x) = begin
          #= index.md:70 =#
          x
      end)
ex2 = @expr function f(x)
    return x
end
:(function f(x)
      #= index.md:74 =#
      #= index.md:75 =#
      return x
  end)

here we use a convenient tool to obtain the Julia expression object provided by Expronicon, the @expr macro.

Now if we convert them to the JLFunction type

jl1 = JLFunction(ex1)
f(x) = x  #= index.md:70 =#
jl2 = JLFunction(ex2)
function f(x)
    #= index.md:74 =#
    #= index.md:75 =#
    return x
end

we can see they have the same structure under the representation of JLFunction.

dump(jl1)
JLFunction
  head: Symbol =
  name: Symbol f
  args: Array{Any}((1,))
    1: Symbol x
  kwargs: Nothing nothing
  rettype: Nothing nothing
  whereparams: Nothing nothing
  body: Expr
    head: Symbol block
    args: Array{Any}((2,))
      1: LineNumberNode
        line: Int64 70
        file: Symbol index.md
      2: Symbol x
  line: Nothing nothing
  doc: Nothing nothing
dump(jl2)
JLFunction
  head: Symbol function
  name: Symbol f
  args: Array{Any}((1,))
    1: Symbol x
  kwargs: Nothing nothing
  rettype: Nothing nothing
  whereparams: Nothing nothing
  body: Expr
    head: Symbol block
    args: Array{Any}((3,))
      1: LineNumberNode
        line: Int64 74
        file: Symbol index.md
      2: LineNumberNode
        line: Int64 75
        file: Symbol index.md
      3: Expr
        head: Symbol return
        args: Array{Any}((1,))
          1: Symbol x
  line: Nothing nothing
  doc: Nothing nothing

we can easily access to some important information of this function by accessing the fields

julia> jl1.name:f
julia> jl1.args1-element Vector{Any}: :x
julia> jl1.bodyquote #= index.md:70 =# x end

This is the same for other syntax types, e.g we can get the corresponding syntax type instance of a struct definition

def = @expr JLStruct struct Foo{T} <: AbstractType
    x::Int
    y::T
end
struct Foo{T} <: AbstractType
    #= index.md:117 =#
    x::Int
    #= index.md:118 =#
    y::T
end

we again use @expr for convenience, however you can also just convert the expression to JLStruct manually

ex = quote
    struct Foo{T} <: AbstractType
        x::Int
        y::T
    end
end
def = JLStruct(ex.args[2])
struct Foo{T} <: AbstractType
    #= index.md:129 =#
    x::Int
    #= index.md:130 =#
    y::T
end

once you have the corresponding JLStruct object, you can access many useful information directly

julia> def.name:Foo
julia> def.typevars1-element Vector{Any}: :T
julia> def.supertype:AbstractType
julia> typeof(def.fields[1])JLField
julia> def.fields[1].name:x
julia> def.fields[1].type:Int

Some syntax types are defined for easy manipulation such as JLIfElse, Julia's representation of if ... elseif ... else ... end statement is a recursive tree in Expr, which sometimes is not very convenient to manipulate or analysis, for example, it is not easy to access all the conditions in a long ifelse statement

ex = @expr if x > 100
    x + 1
elseif x > 90
    x + 2
elseif x > 80
    x + 3
else
    error("some error msg")
end
:(if x > 100
      #= index.md:155 =#
      x + 1
  elseif #= index.md:156 =# x > 90
      #= index.md:157 =#
      x + 2
  elseif #= index.md:158 =# x > 80
      #= index.md:159 =#
      x + 3
  else
      #= index.md:161 =#
      error("some error msg")
  end)

we can find each condition

julia> ex.args[1]:(x > 100)
julia> ex.args[3].args[1]quote #= index.md:156 =# x > 90 end
julia> ex.args[3].args[3].args[1]quote #= index.md:158 =# x > 80 end
julia> ex.args[3].args[3].args[3]quote #= index.md:161 =# error("some error msg") end

imagine how would you construct such expression from scratch, or how would you access all the conditions. Thus JLIfElse allows you to access/manipulate ifelse statements directly as a dict-like object

julia> jl = JLIfElse(ex);
julia> jlif x > 100 x + 1 #= index.md:155 =# elseif x > 90 #= index.md:156 =# x + 2 #= index.md:157 =# elseif x > 80 #= index.md:158 =# x + 3 #= index.md:159 =# else #= index.md:161 =# error("some error msg") end
julia> jl.otherwisequote #= index.md:161 =# error("some error msg") end

you can access to each condition and its action using the condition as your key

julia> jl[:(x > 100)]quote
    #= index.md:155 =#
    x + 1
end

similarly, we can easily construct a JLIfElse

jl = JLIfElse()
jl[:(x > 100)] = :(x + 1)
jl[:(x > 80)] = :(x + 2)
jl.otherwise = :(error("some error msg"))
jl
if x > 100
    x + 1
elseif x > 80
    x + 2
else
    error("some error msg")
end

now let's generate back to Expr so that we can give Julia back some executable expression

julia> codegen_ast(jl):(if x > 100
      x + 1
  elseif x > 80
      x + 2
  else
      error("some error msg")
  end)

You can find available syntax types in Syntax Types

Pattern Matching

Since Expronicon serves as the meta programming stdlib for MLStyle, you can also use the syntax types along with MLStyle, e.g

using MLStyle
using Expronicon

f = @λ begin
   JLFunction(;name=:foo, args) => (args, )
   JLFunction(;name=:boo, args) => (args, )
   _ => nothing
end

ex_foo = @expr function foo(x::Int, y::T) where {T <: Real}
    x + y
end

ex_boo = @expr function foo(x::Int)
    x
end

then we can check if our match function gives the right result

julia> f(ex_foo)(Any[:(x::Int), :(y::T)],)
julia> f(ex_boo)(Any[:(x::Int)],)

You can use any syntax types builtin as your expression template to match using MLStyle. If you define your own syntax type, you can also support pattern matching via @syntax_pattern.

Expronicon.@syntax_patternMacro
@syntax_pattern <syntax type> <syntax checker>

Bundle a syntax type as MLStyle pattern.

Tip

this is not available in ExproniconLite since it depends on MLStyle's pattern matching functionality.

Example

struct MyFunction
    ex :: Expr
    test_field :: Any
end

is_xxx(ex::Expr) = Meta.isexpr(ex, :function)
is_xxx(_) = false

julia> MyFunction(ex::Expr) = MyFunction(ex, :aaa)
julia> @syntax_pattern(MyFunction, is_xxx)

julia> @match :(function f() end) begin
        MyFunction(;test_field) => test_field
    end
:aaa
source

Analysis Functions

Expronicon provides a lot common analysis functions, you can find the list of them in Analysis. you can use them to check if the expression satisfy certain property, e.g you can check if a given object is a struct definition via is_struct, or check if a given function definition supports keyword arguments via is_kw_function.

Transform Functions

You can find the list of them in Transform.

Transform functions usually takes an expression and returns an expression e.g sometimes you only want the name symbol of your function arguments

def = @expr JLFunction function foo(x::Int, y::Real=2)
end
julia> def.args2-element Vector{Any}:
 :(x::Int)
 :($(Expr(:kw, :(y::Real), 2)))
julia> name_only.(def.args)2-element Vector{Symbol}: :x :y

Code Generation Functions

The code generation functions help you generate other target expressions, e.g codegen_ast generates the Julia AST object Expr. All the syntax type can use codegen_ast to generate the corresponding Expr, there are also some other functions start with name codegen in CodeGen you may find useful.

Pretty Printing

Sometimes, when you define your own intermediate representation, you may want to pretty print your expression with colors and indents. Expronicon also provide some tools for this in Printings.

Common Gotchas

Use & operator inside the pattern if you are referring a specific value, e.g

stmt = Expr(:call, GlobalRef(Base, :sin), QuoteNode(1))
@match stmt begin
    Expr(:call, GlobalRef(Core, name), args...) => true
    _ => false
end # true

without &, @match may treat Core as a variable, thus the first pattern matches and return true, which is incorrect, if we add &, we have the expected behaviour

@match stmt begin
    Expr(:call, GlobalRef(&Core, name), args...) => true
    _ => false
end # false