Lrstar parser lexer generator

Moo!Moo is a highly-optimised tokenizer/lexer generator. Use it to tokenize your strings, before parsing 'em with a parser like nearley or whatever else you're into.FastConvenientuses Regular Expressionstracks Line Numbershandles Keywordssupports Statescustom Errorsis even Iterablehas no dependencies4KB minified + gzippedMoo!Is it fast?Yup! Flying-cows-and-singed-steak fast.Moo is the fastest JS tokenizer around. It's ~2–10x faster than most other tokenizers; it's a couple orders of magnitude faster than some of the slower ones.Define your tokens using regular expressions. Moo will compile 'em down to a single RegExp for performance. It uses the new ES6 sticky flag where possible to make things faster; otherwise it falls back to an almost-as-efficient workaround. (For more than you ever wanted to know about this, read adventures in the land of substrings and RegExps.)You might be able to go faster still by writing your lexer by hand rather than using RegExps, but that's icky.Oh, and it avoids parsing RegExps by itself. Because that would be horrible.UsageFirst, you need to do the needful: $ npm install moo, or whatever will ship this code to your computer. Alternatively, grab the moo.js file by itself and slap it into your web page via a tag; moo is completely standalone.Then you can start roasting your very own lexer/tokenizer: const moo = require('moo') let lexer = moo.compile({ WS: /[ \t]+/, comment: /\/\/.*?$/, number: /0|[1-9][0-9]*/, string: /"(?:\\["\\]|[^\n"\\])*"/, lparen: '(', rparen: ')', keyword: ['while', 'if', 'else', 'moo', 'cows'], NL: { match: /\n/, lineBreaks: true }, })And now throw some text at it: lexer.reset('while (10) cows\nmoo') lexer.next()

README - pyLRp python LR(1) parser generatorAuthor: Sebastian Riese The sample files show how to use the parser generator. But are out ofdate at the moment. More documentation will follow. The parsergenerator is written in python3, but it can generate both python2 andpython3 parsers.The generated parsers are standalone modules.usage: pyLRp.py [-h] [-o OFILE] [-l] [-L] [-g] [--print-lextable] [-D] [-d] [-f] [-q] [-T] [-3 | -2] infileA pure python LALR(1)/LR(1) parser generator and lexer generator.positional arguments: infile The parser specification to processoptional arguments: -h, --help show this help message and exit -o OFILE, --output-file OFILE Set the output file to OFILE [default: derived from infile] -l, --line-tracking Enable line tracking in the generated parser -L, --lalr Generate a LALR(1) parser instead of a LR(1) parser -g, --print-graph Print the LR state graph to stdout --print-lextable Print the lextables to stdout -D, --not-deduplicate Compress tables by reusing identical lines -d, --debug Write debug information to the generated file -f, --fast Fast run: generates larger and possibly slower parsers, but takes less time -q, --quiet Print less info -T, --trace Generate a parser that prints out a trace of its state -3, --python3 Generate python3 compatible parser [default] -2, --python2 Generate python2 compatible parserContributors------------Jonas Wielicki (Python 3 support, testing)Testing-------To run the TestSuite run[pyLR1]$ python3 -m unittest discover -s test -vLicense-------Copyright 2009, 2010, 2012 Sebastian RiesePermission is hereby granted, free of charge, to any person obtaining a copyof this software and associated documentation files (the "Software"), to dealin the Software without restriction, including without limitation the rightsto use, copy, modify, merge, publish, distribute, sublicense, and/or sellcopies of the Software, and to permit persons to whom the Software isfurnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included inall copies or substantial portions of the Software.THE

Method name, parameter list, and throws clause like so:methodDeclarator : Identifier '(' formalParameterList? ')' dims? ;And so Java8BaseListener has a method enterMethodDeclarator which will be invoked each time this pattern is encountered.So, let’s override enterMethodDeclarator, pull out the Identifier, and perform our check:public class UppercaseMethodListener extends Java8BaseListener { private List errors = new ArrayList(); // ... getter for errors @Override public void enterMethodDeclarator(Java8Parser.MethodDeclaratorContext ctx) { TerminalNode node = ctx.Identifier(); String methodName = node.getText(); if (Character.isUpperCase(methodName.charAt(0))) { String error = String.format("Method %s is uppercased!", methodName); errors.add(error); } }}5.4. TestingNow, let’s do some testing. First, we construct the lexer:String javaClassContent = "public class SampleClass { void DoSomething(){} }";Java8Lexer java8Lexer = new Java8Lexer(CharStreams.fromString(javaClassContent));Then, we instantiate the parser:CommonTokenStream tokens = new CommonTokenStream(lexer);Java8Parser parser = new Java8Parser(tokens);ParseTree tree = parser.compilationUnit();And then, the walker and the listener:ParseTreeWalker walker = new ParseTreeWalker();UppercaseMethodListener listener= new UppercaseMethodListener();Lastly, we tell ANTLR to walk through our sample class:walker.walk(listener, tree);assertThat(listener.getErrors().size(), is(1));assertThat(listener.getErrors().get(0), is("Method DoSomething is uppercased!"));6. Building Our GrammarNow, let’s try something just a little bit more complex, like parsing log files:2018-May-05 14:20:18 INFO some error occurred2018-May-05 14:20:19 INFO yet another error2018-May-05 14:20:20 INFO some method started2018-May-05 14:20:21 DEBUG another method started2018-May-05 14:20:21 DEBUG entering awesome method2018-May-05 14:20:24 ERROR Bad thing happenedBecause we have a custom log format, we’re going to first need to create our own grammar.6.1. Prepare a Grammar FileFirst, let’s see if we can create a mental map of what each log line looks like in our file. Or if we go one more level deep, we might say: := …And so on. It’s important to consider this so we can decide at what level of granularity we want to parse the text.A grammar file is basically a set of lexer and parser rules. Simply put, lexer rules describe the syntax of the grammar while parser rules describe the semantics.Let’s start by defining fragments which are reusable building blocks for lexer rules.fragment DIGIT : [0-9];fragment TWODIGIT : DIGIT DIGIT;fragment LETTER : [A-Za-z];Next, let’s define the remainings lexer rules:DATE : TWODIGIT TWODIGIT '-' LETTER LETTER LETTER '-' TWODIGIT;TIME : TWODIGIT ':' TWODIGIT ':' TWODIGIT;TEXT : LETTER+ ;CRLF : '\r'? '\n' |