From 74798ffd814bb5dca6569e09dd4bea9d9c4dcf83 Mon Sep 17 00:00:00 2001 From: Mireille Raad Date: Fri, 6 Nov 2020 10:28:52 -0500 Subject: [PATCH] add code highlight, fixes #9 --- _includes/head.html | 5 +- _layouts/post.html | 2 +- ...2020-03-12-thing-explainer-parser.markdown | 29 +- assets/monokai.css | 65 +++++ assets/style.css | 260 +++++++++--------- 5 files changed, 219 insertions(+), 142 deletions(-) create mode 100644 assets/monokai.css diff --git a/_includes/head.html b/_includes/head.html index 7d732ed5d..e328366a0 100644 --- a/_includes/head.html +++ b/_includes/head.html @@ -6,9 +6,12 @@ - + + + + {%- seo -%} diff --git a/_layouts/post.html b/_layouts/post.html index 7bd10b5a4..a6da06fc6 100644 --- a/_layouts/post.html +++ b/_layouts/post.html @@ -4,7 +4,7 @@
-
+
{%- assign date_format = site.minima.date_format | default: "%b %-d, %Y" -%} {{ page.date | date: date_format }} diff --git a/_posts/2020-03-12-thing-explainer-parser.markdown b/_posts/2020-03-12-thing-explainer-parser.markdown index ec94bf3a1..fda23c015 100644 --- a/_posts/2020-03-12-thing-explainer-parser.markdown +++ b/_posts/2020-03-12-thing-explainer-parser.markdown @@ -32,13 +32,12 @@ This list of steps introduces some new concepts like: tokens and abstract syntax ### Step 1: parsing source code into tokens The fastest way to understand what tokens are, is to see them. Conveniently, Python comes with a [tokenizer](https://docs.python.org/3/library/tokenize.html). Here is what happen if I run the tokenizer on the function that I created earlier. -`$ python -m tokenize file.py` - +`$ python -m tokenize file.py` `file.py` has the function that I used in the previous example. -``` +```python def add(x,y): return x+y ``` @@ -51,7 +50,7 @@ A picture IS worth a thousand word 😛 Those are the tokens. They are the basic If you want to sound fancy: - The tokens are the basic "lexical components" -- The parsing process is called "Lexical Analysis" +- The parsing process is called "lexical analysis" - The thing that does the process is a "lexer" Here is the link to the RustPython lexer. @@ -71,20 +70,20 @@ In the previous step, if you add random stuff to your function and tokenize it, So don't hate on the whole interpreter when you get error messages! or at least don't hate on the tokenizer! -To determine if the tokens are valid syntax, first you need a definition of what a valid syntax is. Python has a defined "grammar" or set of rules. The official reference is on [this link](https://docs.python.org/3/reference/grammar.html). There, you will find a machine readable file. You may read a book to know the rules of python, but words are too "fluffy", the machine needs a very strict set of rules encoded in a file. [This video](https://www.youtube.com/watch?v=KGMFvy2d5OI) explains the notation and the Python grammar. +To determine if the tokens are valid syntax, first you need a definition of what a valid syntax is. Python has a defined "grammar" or set of rules. The official reference is on [this link](https://docs.python.org/3/reference/grammar.html). There, you will find a machine readable file. You may read a book to know the rules of python, but words are too "fluffy", an algorithm that verifies if rules are followed needs a very strict set of rules encoded in a file. [This video](https://www.youtube.com/watch?v=KGMFvy2d5OI) explains the Python grammar and the file's notation. As the presenter puts it, this is the spirit of the beast (python) and it is only ~10KB 😭 (compare that to the size of python books you had to read!) So, we have the rules or grammar of a programming language in a machine encoded format... now we need to write something that verifies that those rules were followed... This sounds like something that other people could use and like something that should exist as an open source project! 🤔 Sure enough, there is a whole Rust framework called `LALRPOP`. It takes the tokens generated by the lexer, verifies the syntax and turns the tokens into an AST (Abstract Syntax Tree). More information and a tutorial can be found in the [LALRPOP book](https://lalrpop.github.io/lalrpop/README.html). -RustPython does one nice extra thing on top of `LALRPOP `. It masks the errors and provides you with safer, nicer errors. You can see the code for this in `RustPython/parser/src/error.rs` +RustPython does one nice extra thing on top of `LALRPOP`. It masks the errors and provides you with safer, nicer errors. You can see the code for this in `RustPython/parser/src/error.rs` **Using RustPython to generate an AST** You can do: -``` +```rust use rustpython_parser::{parser, ast}; let python_source = "print('Hello world')"; let python_ast = parser::parse_expression(python_source).unwrap(); @@ -92,13 +91,13 @@ You can do: ## Recap 🥴 🥵 -As a recap, when you write a line of python code and "run it", here is what happens: +As a recap, when you write a line of python code and "run it", here is what the RustPython interpreter does: -**your code** (in `file.py` or interactive shell) - ⭣ parse the line of source code into tokens - ⭣ determine if the tokens are valid syntax - ⭣ create an Abstract Syntax Tree (AST) - ⭣ compile the AST into bytecode - **bytecode** (in `__pycache__/file.pyc` or in memory) +**INPUT: your code** (in `file.py` or interactive shell) +⬇️ parse the line of source code into tokens +⬇️ determine if the tokens are valid syntax +⬇️ create an Abstract Syntax Tree (AST) +⬇️ compile the AST into bytecode +**OUTPUT: bytecode** (in `__pycache__/file.pyc` or in memory) - The compiler is under **`RustPython/compiler`** we'll dive into the details in a next post. In the meantime, check out the parser source code in [RustPython/parser/](https://github.com/RustPython/RustPython/tree/master/parser). + The compiler is under **`RustPython/compiler`**. Keep an eye on the blog for a future post about the details or the compiler. In the meantime, check out the parser source code in [RustPython/parser/](https://github.com/RustPython/RustPython/tree/master/parser). diff --git a/assets/monokai.css b/assets/monokai.css new file mode 100644 index 000000000..e020d51f6 --- /dev/null +++ b/assets/monokai.css @@ -0,0 +1,65 @@ +.highlight pre { background-color: #272822; } +.highlight .hll { background-color: #272822; } +.highlight .c { color: #75715e } /* Comment */ +.highlight .err { color: #960050; background-color: #1e0010 } /* Error */ +.highlight .k { color: #66d9ef } /* Keyword */ +.highlight .l { color: #ae81ff } /* Literal */ +.highlight .n { color: #f8f8f2 } /* Name */ +.highlight .o { color: #f92672 } /* Operator */ +.highlight .p { color: #f8f8f2 } /* Punctuation */ +.highlight .cm { color: #75715e } /* Comment.Multiline */ +.highlight .cp { color: #75715e } /* Comment.Preproc */ +.highlight .c1 { color: #75715e } /* Comment.Single */ +.highlight .cs { color: #75715e } /* Comment.Special */ +.highlight .ge { font-style: italic } /* Generic.Emph */ +.highlight .gs { font-weight: bold } /* Generic.Strong */ +.highlight .kc { color: #66d9ef } /* Keyword.Constant */ +.highlight .kd { color: #66d9ef } /* Keyword.Declaration */ +.highlight .kn { color: #f92672 } /* Keyword.Namespace */ +.highlight .kp { color: #66d9ef } /* Keyword.Pseudo */ +.highlight .kr { color: #66d9ef } /* Keyword.Reserved */ +.highlight .kt { color: #66d9ef } /* Keyword.Type */ +.highlight .ld { color: #e6db74 } /* Literal.Date */ +.highlight .m { color: #ae81ff } /* Literal.Number */ +.highlight .s { color: #e6db74 } /* Literal.String */ +.highlight .na { color: #a6e22e } /* Name.Attribute */ +.highlight .nb { color: #f8f8f2 } /* Name.Builtin */ +.highlight .nc { color: #a6e22e } /* Name.Class */ +.highlight .no { color: #66d9ef } /* Name.Constant */ +.highlight .nd { color: #a6e22e } /* Name.Decorator */ +.highlight .ni { color: #f8f8f2 } /* Name.Entity */ +.highlight .ne { color: #a6e22e } /* Name.Exception */ +.highlight .nf { color: #a6e22e } /* Name.Function */ +.highlight .nl { color: #f8f8f2 } /* Name.Label */ +.highlight .nn { color: #f8f8f2 } /* Name.Namespace */ +.highlight .nx { color: #a6e22e } /* Name.Other */ +.highlight .py { color: #f8f8f2 } /* Name.Property */ +.highlight .nt { color: #f92672 } /* Name.Tag */ +.highlight .nv { color: #f8f8f2 } /* Name.Variable */ +.highlight .ow { color: #f92672 } /* Operator.Word */ +.highlight .w { color: #f8f8f2 } /* Text.Whitespace */ +.highlight .mf { color: #ae81ff } /* Literal.Number.Float */ +.highlight .mh { color: #ae81ff } /* Literal.Number.Hex */ +.highlight .mi { color: #ae81ff } /* Literal.Number.Integer */ +.highlight .mo { color: #ae81ff } /* Literal.Number.Oct */ +.highlight .sb { color: #e6db74 } /* Literal.String.Backtick */ +.highlight .sc { color: #e6db74 } /* Literal.String.Char */ +.highlight .sd { color: #e6db74 } /* Literal.String.Doc */ +.highlight .s2 { color: #e6db74 } /* Literal.String.Double */ +.highlight .se { color: #ae81ff } /* Literal.String.Escape */ +.highlight .sh { color: #e6db74 } /* Literal.String.Heredoc */ +.highlight .si { color: #e6db74 } /* Literal.String.Interpol */ +.highlight .sx { color: #e6db74 } /* Literal.String.Other */ +.highlight .sr { color: #e6db74 } /* Literal.String.Regex */ +.highlight .s1 { color: #e6db74 } /* Literal.String.Single */ +.highlight .ss { color: #e6db74 } /* Literal.String.Symbol */ +.highlight .bp { color: #f8f8f2 } /* Name.Builtin.Pseudo */ +.highlight .vc { color: #f8f8f2 } /* Name.Variable.Class */ +.highlight .vg { color: #f8f8f2 } /* Name.Variable.Global */ +.highlight .vi { color: #f8f8f2 } /* Name.Variable.Instance */ +.highlight .il { color: #ae81ff } /* Literal.Number.Integer.Long */ + +.highlight .gh { } /* Generic Heading & Diff Header */ +.highlight .gu { color: #75715e; } /* Generic.Subheading & Diff Unified/Comment? */ +.highlight .gd { color: #f92672; } /* Generic.Deleted & Diff Deleted */ +.highlight .gi { color: #a6e22e; } /* Generic.Inserted & Diff Inserted */ diff --git a/assets/style.css b/assets/style.css index 1f18b4c23..a74625a8c 100644 --- a/assets/style.css +++ b/assets/style.css @@ -1,32 +1,32 @@ /* common html elements */ body { - font-family: 'Fira Sans', sans-serif; - margin: 0em; + font-family: "Fira Sans", sans-serif; + margin: 0em; } section { - padding: 2em; + padding: 2em; } a { - color: #000; + color: #000; } p, li { - line-height: 1.75em; - font-size: 1.1em; + line-height: 1.75em; + font-size: 1.1em; } hr { - border: 0px; - border-top: 1px solid #000; + border: 0px; + border-top: 1px solid #000; } hr.dashed { - border: 0px; - border-top: 1px dashed #ccc; + border: 0px; + border-top: 1px dashed #ccc; } h1, @@ -35,322 +35,332 @@ h3, h4, h5, h6 { - font-family: 'Sen', sans-serif; + font-family: "Sen", sans-serif; } .font-secondary { - font-family: 'Sen', sans-serif; + font-family: "Sen", sans-serif; } .text-small { - font-size: 0.9rem; + font-size: 0.9rem; } .text-muted { - color: darkslategray; + color: darkslategray; } .text-uppercase { - text-transform: uppercase; + text-transform: uppercase; } h1 { - font-size: 1.5em; - margin-top: 2em; + font-size: 1.5em; + margin-top: 2em; } mark.post-date, mark.highlight { - background-color: #F74C00; - color: #fff; - padding: 5px 15px 5px 15px; + background-color: #f74c00; + color: #fff; + padding: 5px 15px 5px 15px; } -.code, -code { - background-color: #F6F8FA; - letter-spacing: 2px; - line-height: 1.75em; - font-size: 1.1em; - padding: 5px; +/* code blocks */ + +.highlight { + font-size: 1rem; + padding: 5px; +} + +pre { + white-space: pre-wrap; /* css-3 */ + white-space: -moz-pre-wrap; /* Mozilla, since 1999 */ + white-space: -pre-wrap; /* Opera 4-6 */ + white-space: -o-pre-wrap; /* Opera 7 */ + word-wrap: break-word; /* Internet Explorer 5.5+ */ +} + +code.highlighter-rouge { + background-color: #F6F8FA; + color: #000; + letter-spacing: 2px; + line-height: 1.75em; + font-size: 1.1em; + padding: 5px; } /* width */ .w-100 { - width: 100%; + width: 100%; } .w-80 { - width: 80%; + width: 80%; } /* colors */ .bg-light { - background-color: #F6F8FA; + background-color: #f6f8fa; } .bg-rust { - background-color: #F74C00; + background-color: #f74c00; } .text-white { - color: #fff; + color: #fff; } - /* alignment */ .d-flex-space-between { - justify-content: space-between; + justify-content: space-between; } .m-auto { - margin: auto; + margin: auto; } .d-table { - display: table; + display: table; } .text-center { - text-align: center; + text-align: center; } .justify-center { - justify-content: center; - align-items: center; + justify-content: center; + align-items: center; } .float-left { - float: left; + float: left; } /* margins and padding */ .padding-small { - padding: 5px; + padding: 5px; } .mb-1 { - margin-bottom: 1em; + margin-bottom: 1em; } .mt-1 { - margin-top: 1em; + margin-top: 1em; } .mt-2 { - margin-top: 2em; + margin-top: 2em; } .mt-4 { - margin-top: 4em; + margin-top: 4em; } - .pl-1 { - padding-left: 1em; + padding-left: 1em; } .pl-2 { - padding-left: 2em; + padding-left: 2em; } .pr-2 { - padding-right: 2em; + padding-right: 2em; } .p-2 { - padding: 2em; + padding: 2em; } .m-2 { - margin: 2em; + margin: 2em; } .pr-1 { - padding-right: 1em; + padding-right: 1em; } /* lists */ ul.list-inline { - list-style: none; - display: inline-block; - padding: 0; + list-style: none; + display: inline-block; + padding: 0; } .list-unstyled { - list-style: none; - margin: 0; - padding: 0; + list-style: none; + margin: 0; + padding: 0; } .list-contributors { - list-style: none; - padding: 0; + list-style: none; + padding: 0; } .list-contributors li { - border: 1px solid #000; - display: inline-block; - padding: 2px 10px 2px 10px; - margin-right: 1em; - margin-top: 1em; + border: 1px solid #000; + display: inline-block; + padding: 2px 10px 2px 10px; + margin-right: 1em; + margin-top: 1em; } .post-link { - font-size: 1.2em; + font-size: 1.2em; } .img-fluid { - width: 100%; + width: 100%; } .blog-content p > img { - max-width: 100%; + max-width: 100%; } .post-date a { - background-color: #000; - color: #fff; + background-color: #000; + color: #fff; } .border { - border: 1px solid darkgray; + border: 1px solid darkgray; } .border-bottom { - border-bottom: 1px solid #000; + border-bottom: 1px solid #000; } - /* custom classes */ .title { - font-family: 'Sen', sans-serif; - font-size: 3em; + font-family: "Sen", sans-serif; + font-size: 3em; } .section-title { - font-family: 'Sen', sans-serif; - font-size: 1.5em; + font-family: "Sen", sans-serif; + font-size: 1.5em; } .logo { - height: 80px; + height: 80px; } - .goal { - line-height: 1.5em; + line-height: 1.5em; } .blog-intro { - background-color: #F6F8FA; - padding-left: 2em; - padding-right: 2em; - padding-top: 1em; - padding-bottom: 1em; + background-color: #f6f8fa; + padding-left: 2em; + padding-right: 2em; + padding-top: 1em; + padding-bottom: 1em; } /* test results */ .results-test { - margin-top: 50px; - color: #fff; - font-size: 1.1rem; - background-color: #F74C00; - word-wrap: anywhere; + margin-top: 50px; + color: #fff; + font-size: 1.1rem; + background-color: #f74c00; + word-wrap: anywhere; } .results-test-title { - font-family: 'Sen', sans-serif; - font-weight: bold; - text-transform: capitalize; + font-family: "Sen", sans-serif; + font-weight: bold; + text-transform: capitalize; } -.results-case-wrapper>div { - padding-left: 10px; - padding-right: 10px; +.results-case-wrapper > div { + padding-left: 10px; + padding-right: 10px; } .results-case { - background-color: white; - border-left: 1px solid #000; - border-right: 1px solid #000; - pointer-events: auto; + background-color: white; + border-left: 1px solid #000; + border-right: 1px solid #000; + pointer-events: auto; } .results-case-list-item { - background-color: white; - color: #000; + background-color: white; + color: #000; } /* flex table for results */ .tbl { - display: flex; - flex-direction: column; + display: flex; + flex-direction: column; } .row { - display: flex; + display: flex; } .cell { - flex: 4; - border-bottom: 1px solid lightslategray; - align-self: end; + flex: 4; + border-bottom: 1px solid lightslategray; + align-self: end; } .cell:nth-child(1) { - flex: 2; + flex: 2; } .cell:nth-child(2) { - flex: 2; + flex: 2; } .cell:nth-child(3) { - flex: 1; + flex: 1; } - .row .cell { - box-sizing: border-box; - padding: 5px; + box-sizing: border-box; + padding: 5px; } .case-result::after { - padding: 5px; + padding: 5px; } .passed > .case-result::after { - content: "🚀 👌"; + content: "🚀 👌"; } .skipped > .case-result::after { - content: "🤷‍♀️"; + content: "🤷‍♀️"; } .expected_failure > .case-result::after { - content: "😔"; + content: "😔"; } /* filters */ .filter { - cursor: pointer; - padding: 5px; - margin: 5px; + cursor: pointer; + padding: 5px; + margin: 5px; } .filter.active { - background-color: #FFD43B; + background-color: #ffd43b; } /* classes used in the results summary */ .cell_summary { - flex: 4; - border-bottom: 1px solid white; - align-self: end; + flex: 4; + border-bottom: 1px solid white; + align-self: end; }