RubyLit - This README is a program! ↑

The output of this README.md file is a program that turns documents into:

It's a <a href=“literate”>en.wikipedia.org/wiki/Literate_programming“>literate program</a> (wikipedia.org) and the concept comes from Donald Knuth.

In order to get the process rolling, there's a non-literate stage1.rb script that does the initial “tangling”. You'll see what that means in a moment.

To make this literate programming stuff work, all source is indented. That not only makes it easy to read in source form, but by chosing that method, I've also made this document a valid Markdown file, so the README will be properly formatted when you view it on the Web as HTML output via tools such as <a href=“RepoRatratfactor.com/repos/reporat/”>RepoRat>.

Here's how it starts. The program takes the root filename as a command line argument:

fname = ARGV[0]

I intially wanted to use the file extension “.lit”, but the real magic happened when I realized I could use “.md” and have the README itself be the program:

fname_input = "#{fname}.md"

Next, I “tangle” and “weave” the input document:

<<Tangle>>
<<Weave>>

“Tangle” creates the program and “Weave” creates the documentation. Those little <<bracket things>> are literate programming macros that include source code from other sections of the document (identified by Markdown subheadings).

Tangle ↑

What's neat about literate programming is that the source can be presented in any order, so you can explain it however you like. The tangling process puts it back into order so it can actually run.

The full literate programming concept as imagined by Knuth and implemented in his initial 'WEB' system not only allows you to include bits of code, but even lets you define parametric macros, so the literate document is actually a meta-language on top of the underlying programming language!

I've just implemented a crude “include” macro for this demonstration, but that alone gives me a ton of flexibility!

Here's how I've made that work. I've got a segments hash that will store all of the lines of a “segment” in the literate program. I have the program begin in a segment identified with the :start symbol:

myseg = :start
segments = {}
segments[myseg] = []

Then I loop through all of the lines in the source document and handle just two special cases. (All other lines are treated as the “document” part of the literate program and are completely ignored here!)

File.open(fname_input).each do |line|
    <<Handle segments>>
    <<Handle code lines>>
end

And after gathering the lines into segments, I recursively follow the includes to write out the final program to a file:

<<Write the program>>

Handle segments ↑

When I see a “## ” at the beginning of the line, I know it's a Markdown level 2 heading, which I'm using to indicate new code segments. They don't have to include code and even if they do, they don't have to be used.

Here you can see that I'm setting the current segment to the name of the heading and initializing a new array to store the code lines:

  if(line.start_with?('## '))
    myseg = line[3..].chomp
    segments[myseg] = []
  end

Handle code lines ↑

When I see a space at the beginning of the line, I know it's indented source code. This is extremely strict and not flexible and is just one example of the non-industrial nature of this demonstration program. :-)

  if(line.start_with?(' '))
    segments[myseg].push(line)
  end

Write the program ↑

Here's the fun part! I've got this recursive method called put_lines that takes an open destination file, the hash of named code segments (each segment being an array of lines), and a target segment to print.

I'm looking at each line to see if it's a <<macro thingy>> (include request). If it is, I recurse into the requested segment. Otherwise, I just output the current line to the file:

def put_lines(file, segments, sname)
  segments[sname].each do |line|

    if(m = /^\s*<<([^>]+)>>\s*$/.match(line))

      put_lines(file, segments, m[1])

    else

      file.puts line

    end

  end
end

To start the above recursive process, I open the “.rb” output file and request the :start segment:

File.open("#{fname}.rb", 'w') do |out|
  put_lines(out, segments, :start)
end

That's it!

Weave ↑

The “weave” part of the application is the documentation creation portion.

Since my scheme for this literate program is to encode it as pure Markdown, I could just rely on an external tool to create the HTML (in fact, that's probably how you're reading this README right now).

But since Ruby comes with a Markdown parser as part of it's Standard Library, I figured I might as well include it. The parser and generator are part of the RDoc (Ruby documentation) module:

require 'rdoc'

The markdown source is the literate program document (yeah, we read it in the Tangle process and we'll read it again for Weave):

data = File.read(fname_input)

Then some boilerplate. RDoc is like a mini-Pandoc in that it can take input and produce output in a bunch of different formats, and we pay for that flexibility with some complexity:

formatter = RDoc::Markup::ToHtml.new(RDoc::Options.new, nil)
html = RDoc::Markdown.parse(data).accept(formatter)

And then I just write that out to a “.html” file, bookended by start and end document tags:

File.open("#{fname}.html", 'w') do |out|
  out.puts("<html><body>")
  out.print(html)
  out.puts("</body></html>")
end

And that's it!

Running it! ↑

To turn this README into a program starts with “stage1”, which only includes the “tangle” part of the process (no documentation output):

$ ruby stage1.rb README
Found segment 'Tangle'
Found segment 'Handle segments'
Found segment 'Handle code lines'
Found segment 'Write the program'
Found segment 'Weave'
Found segment 'Comparing this document with its output'
Fetching segment 'start'
Fetching segment 'Tangle'
Fetching segment 'Handle segments'
Fetching segment 'Handle code lines'
Fetching segment 'Write the program'
Fetching segment 'Weave'

(As you can see, I also gave it some output to help me debug segment names.)

That produces README.rb, which is now the Ruby program we've described above, which can be used to process itself again:

ruby README.rb README

And running that again proves that we're fully bootstrapped. We're running the output of the README against the README:

ruby README.rb README

(Note that this part of the document you're reading right now has indented “code” blocks to show the command line and output. Those are not valid Ruby, so why is that okay? That's okay because they're never explicitly included in the program! The Ruby interpreter never sees them.)

Bootstrapping ↑

This repo includes two simple literate test programs (Markdown files) I used to get the intial “stage1” program working:

When stage1 was done, I copied it to use as the basis for the final document/program you're reading now. Then I followed the “Running it!” process exactly as shown above and it worked! :-)