Planet Ruby Gem of the Week

Updated Saturday, 09 September 2017 07:30
{}
Ruby Gem of the Week ( Feed )
Thursday, 23 April 2015
Week #17 - datapak gem - work with tabular data packages (.csv files w/ datapackage.json) using SQLite (w/ ActiveRecord)

Let’s say you want to share your data with the world or use the data that others share with the world. How to get started? A pragmatic way is to use tabular data packages.

What’s a tabular data package?

Tabular Data Package is a simple structure for publishing and sharing t

Let’s say you want to share your data with the world or use the data that others share with the world. How to get started? A pragmatic way is to use tabular data packages.

What’s a tabular data package?

Tabular Data Package is a simple structure for publishing and sharing tabular data with the following key features:

  • Data is stored in CSV (comma separated values) files
  • Metadata about the dataset both general (e.g. title, author) and the specific data files (e.g. schema) is stored in a single JSON file named datapackage.json which follows the Data Package format

(Source: Tabular Data Packages, Open Knowledge Foundation)

Here’s a minimal example of a tabular data package holding two files, that is, data.csv and datapackage.json:

data.csv:

Brewery,City,Name,Abv
Andechser Klosterbrauerei,Andechs,Doppelbock Dunkel,7%
Augustiner Bräu München,München,Edelstoff,5.6%
Bayerische Staatsbrauerei Weihenstephan,Freising,Hefe Weissbier,5.4%
Brauerei Spezial,Bamberg,Rauchbier Märzen,5.1%
Hacker-Pschorr Bräu,München,Münchner Dunkel,5.0%
Staatliches Hofbräuhaus München,München,Hofbräu Oktoberfestbier,6.3%
...

datapackage.json:

{
  "name": "beer",
  "resources": [
    {
      "path": "data.csv",
      "schema": {
        "fields": [ { "name": "Brewery",   "type": "string" },
                    { "name": "City",      "type": "string" },
                    { "name": "Name",      "type": "string" },
                    { "name": "Abv",       "type": "number" } ]
      }
    }
  ]
}

Where to find data packages?

For some “real world” examples see the Data Packages Listing at the Open Knowledge Foundation (OKFN) site for a start. Tabular data packages include:

Name Comments
country-codes Comprehensive country codes: ISO 3166, ITU, ISO 4217 currency codes and many more
language-codes ISO Language Codes (639-1 and 693-2)
currency-codes ISO 4217 Currency Codes
gdb Country, Regional and World GDP (Gross Domestic Product)
s-and-p-500-companies S&P 500 Companies with Financial Information
un-locode UN-LOCODE Codelist

and many more.

What’s the datapak gem?

Now the questions is how to work with tabular data packages in Ruby. Let’s try the datapak gem.

require 'datapak`

Datapak.import(
  's-and-p-500-companies',
  'gdb'
)

Using Datapak.import will:

1) download all data packages to the ./pak folder

2) (auto-)add all tables to an in-memory SQLite database using SQL create_table statements via ActiveRecord migrations e.g.

create_table :constituents_financials do |t|
  t.string :symbol            # Symbol         (string)
  t.string :name              # Name           (string)
  t.string :sector            # Sector         (string)
  t.float  :price             # Price          (number)
  t.float  :dividend_yield    # Dividend Yield (number)
  t.float  :price_earnings    # Price/Earnings (number)
  t.float  :earnings_share    # Earnings/Share (number)
  t.float  :book_value        # Book Value     (number)
  t.float  :_52_week_low      # 52 week low    (number)
  t.float  :_52_week_high     # 52 week high   (number)
  t.float  :market_cap        # Market Cap     (number)
  t.float  :ebitda            # EBITDA         (number)
  t.float  :price_sales       # Price/Sales    (number)
  t.float  :price_book        # Price/Book     (number)
  t.string :sec_filings       # SEC Filings    (string)
end

3) (auto-)import all records using SQL inserts e.g.

INSERT INTO constituents_financials
  (symbol,
   name,
   sector,
   price,
   dividend_yield,
   price_earnings,
   earnings_share,
   book_value,
   _52_week_low,
   _52_week_high,
   market_cap,
   ebitda,
   price_sales,
   price_book,
   sec_filings)
VALUES
  ('MMM',
   '3M Co',
   'Industrials',
   162.27,
   2.11,
   22.28,
   7.284,
   25.238,
   123.61,
   162.92,
   104.0,
   8.467,
   3.28,
   6.43,
   'http://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=MMM')

4) (auto-)add ActiveRecord models for all tables.

So what? Now you can use all the “magic” of ActiveRecord to query the datasets. Example:

puts "Constituent.count: #{Constituent.count}"

# SELECT COUNT(*) FROM "constituents"
# => 496

pp Constituent.first

# SELECT  "constituents".* FROM "constituents" ORDER BY "constituents"."id" ASC LIMIT 1
# => #<Constituent:0x9f8cb78
         id:     1,
         symbol: "MMM",
         name:   "3M Co",
         sector: "Industrials">

pp Constituent.find_by!( symbol: 'MMM' )

# SELECT  "constituents".*
         FROM "constituents"
         WHERE "constituents"."symbol" = "MMM"
         LIMIT 1
# => #<Constituent:0x9f8cb78
         id:     1,
         symbol: "MMM",
         name:   "3M Co",
         sector: "Industrials">

pp Constituent.find_by!( name: '3M Co' )

# SELECT  "constituents".*
          FROM "constituents"
          WHERE "constituents"."name" = "3M Co"
          LIMIT 1
# => #<Constituent:0x9f8cb78
         id:     1,
         symbol: "MMM",
         name:   "3M Co",
         sector: "Industrials">

pp Constituent.where( sector: 'Industrials' ).count

# SELECT COUNT(*) FROM "constituents"
         WHERE "constituents"."sector" = "Industrials"
# => 63

pp Constituent.where( sector: 'Industrials' ).all

# SELECT "constituents".*
         FROM "constituents"
         WHERE "constituents"."sector" = "Industrials"
# => [#<Constituent:0x9f8cb78
          id:     1,
          symbol: "MMM",
          name:   "3M Co",
          sector: "Industrials">,
      #<Constituent:0xa2a4180
          id:     8,
          symbol: "ADT",
          name:   "ADT Corp (The)",
          sector: "Industrials">,...]

and so on

How to dowload a data package (“by hand”)?

Use the Datapak::Downloader class to download a data package to your disk (by default data packages get stored in ./pak).

dl = Datapak::Downloader.new
dl.fetch( 'language-codes' )
dl.fetch( 's-and-p-500-companies' )
dl.fetch( 'un-locode`)

Will result in:

-- pak
   |-- language-codes
   |   |-- data
   |   |   |-- language-codes-3b2.csv
   |   |   |-- language-codes.csv
   |   |   `-- language-codes-full.csv
   |   `-- datapackage.json
   |-- s-and-p-500-companies
   |   |-- data
   |   |   |-- constituents.csv
   |   |   `-- constituents-financials.csv
   |   `-- datapackage.json
   `-- un-locode
       |-- data
       |   |-- code-list.csv
       |   |-- country-codes.csv
       |   |-- function-classifiers.csv
       |   |-- status-indicators.csv
       |   `-- subdivision-codes.csv
       `-- datapackage.json

How to add and import a data package (“by hand”)?

Use the Datapak::Pak class to read-in a data package and add and import into an SQL database.

pak = Datapak::Pak.new( './pak/un-locode/datapackage.json' )
pak.tables.each do |table|
  table.up!      # (auto-) add table  using SQL create_table via ActiveRecord migration
  table.import!  # import all records using SQL inserts
end

That’s it.

Bonus: How to connect to a different SQL database?

You can connect to any database supported by ActiveRecord. If you do NOT establish a connection in your script - the standard (default fallback) is using an in-memory SQLite3 database.

SQLite

For example, to create an SQLite3 database on disk - lets say datapak.db - use in your script (before the Datapak.import statement):

ActiveRecord::Base.establish_connection( adapter:  'sqlite3
                                         database: './datapak.db' )

PostgreSQL

For example, to connect to a PostgreSQL database use in your script (before the Datapak.import statement):

require 'pg'       ##  pull-in PostgreSQL (pg) machinery

ActiveRecord::Base.establish_connection( adapter:  'postgresql'
                                         username: 'ruby'",
                                         password: 'topsecret',
                                         database: 'database' )

Find Out More

datapak

Tabular Data Package

{}
Ruby Gem of the Week ( Feed )
Thursday, 16 April 2015
Week #16 - tilt gem - let's build (yet another) micro web framework in less than 33 lines of code

Do you think the Ruby on Rails web framework is a massive monster - more than 10,000+ lines of code just for ActionPack - not counting ActiveModel, ActiveRecord, ActiveJob, ActiveResource and what not? Or do you think that even the “

Do you think the Ruby on Rails web framework is a massive monster - more than 10,000+ lines of code just for ActionPack - not counting ActiveModel, ActiveRecord, ActiveJob, ActiveResource and what not? Or do you think that even the “classic” Sinatra web machinery clocking in at more than 1,000+ lines of code is more macro than micro?

Why not write a “real” micro web framework inspired by Konstantin Haase’s Almost Sinatra hack that ships as a 6(!) lines of code pastie:

almost_sinatra.rb:

%w.rack tilt date INT TERM..map{|l|trap(l){$r.stop}rescue require l};$u=Date;$z=($u.new.year + 145).abs;puts "== Almost Sinatra/No Version has taken the stage on #$z for development with backup from Webrick"
$n=Module.new{extend Rack;a,D,S,q=Rack::Builder.new,Object.method(:define_method),/@@ *([^\n]+)\n(((?!@@)[^\n]*\n)*)/m
%w[get post put delete].map{|m|D.(m){|u,&b|a.map(u){run->(e){[200,{"Content-Type"=>"text/html"},[a.instance_eval(&b)]]}}}}
Tilt.mappings.map{|k,v|D.(k){|n,*o|$t||=(h=$u._jisx0301("hash, please");File.read(caller[0][/^[^:]+/]).scan(S){|a,b|h[a]=b};h);v[0].new(*o){n=="#{n}"?n:$t[n.to_s]}.render(a,o[0].try(:[],:locals)||{})}}
%w[set enable disable configure helpers use register].map{|m|D.(m){|*_,&b|b.try :[]}};END{Rack::Handler.get("webrick").run(a,Port:$z){|s|$r=s}}
%w[params session].map{|m|D.(m){q.send m}};a.use Rack::Session::Cookie;a.use Rack::Lock;D.(:before){|&b|a.use Rack::Config,&b};before{|e|q=Rack::Request.new e;q.params.dup.map{|k,v|params[k.to_sym]=v}}}

Of course - the “magic” is possible because all the machinery and heavy-lifting gets outsourced to rack and to the lesser known tilt library.

What’s the tilt gem?

Let’s thank Ryan Tomayko, Magnus Holm and friends for creating the tilt gem offering a standard “generic” interface for template engines that you can use - surprise, surprise - for building your own web frameworks or static site generators, for example.

Let’s check-up what formats and template engines tilt includes out-of-the-box:

require 'tilt'

Tilt.mappings.each do |ext, engines|
  puts "#{ext.ljust(12)} : #{engines.inspect}"
end

Will result in:

str          : [Tilt::StringTemplate]
erb          : [Tilt::ErubisTemplate, Tilt::ERBTemplate]
rhtml        : [Tilt::ErubisTemplate, Tilt::ERBTemplate]
erubis       : [Tilt::ErubisTemplate]
etn          : [Tilt::EtanniTemplate]
etanni       : [Tilt::EtanniTemplate]
haml         : [Tilt::HamlTemplate]
sass         : [Tilt::SassTemplate]
scss         : [Tilt::ScssTemplate]
less         : [Tilt::LessTemplate]
rcsv         : [Tilt::CSVTemplate]
coffee       : [Tilt::CoffeeScriptTemplate]
nokogiri     : [Tilt::NokogiriTemplate]
builder      : [Tilt::BuilderTemplate]
mab          : [Tilt::MarkabyTemplate]
liquid       : [Tilt::LiquidTemplate]
radius       : [Tilt::RadiusTemplate]
markdown     : [Tilt::RedcarpetTemplate, Tilt::RedcarpetTemplate::Redcarpet2, Tilt::RedcarpetTemplate::Redcarpet1, Tilt::RDiscountTemplate, Tilt::BlueClothTemplate, Tilt::KramdownTemplate, Tilt::MarukuTemplate]
mkd          : [Tilt::RedcarpetTemplate, Tilt::RedcarpetTemplate::Redcarpet2, Tilt::RedcarpetTemplate::Redcarpet1, Tilt::RDiscountTemplate, Tilt::BlueClothTemplate, Tilt::KramdownTemplate, Tilt::MarukuTemplate]
md           : [Tilt::RedcarpetTemplate, Tilt::RedcarpetTemplate::Redcarpet2, Tilt::RedcarpetTemplate::Redcarpet1, Tilt::RDiscountTemplate, Tilt::BlueClothTemplate, Tilt::KramdownTemplate, Tilt::MarukuTemplate]
textile      : [Tilt::RedClothTemplate]
rdoc         : [Tilt::RDocTemplate]
wiki         : [Tilt::WikiClothTemplate, Tilt::CreoleTemplate]
creole       : [Tilt::CreoleTemplate]
mediawiki    : [Tilt::WikiClothTemplate]
mw           : [Tilt::WikiClothTemplate]
yajl         : [Tilt::YajlTemplate]
ad           : [Tilt::AsciidoctorTemplate]
adoc         : [Tilt::AsciidoctorTemplate]
asciidoc     : [Tilt::AsciidoctorTemplate]
html         : [Tilt::PlainTemplate]

Wow. That’s quite an offering. Let’s see tilt in action and let’s unroll the six somewhat obfuscated lines of Almost Sinatra to see its full inner beauty.

almost_sinatra_ii.rb:

# Note: to keep it simple; the inline template machinery e.g. S=/@@ *([^\n]+)\n(((?!@@)[^\n]*\n)*)/m
        and the before, configure 'n' friends blocks got dropped

## line 1
# %w.rack tilt date INT TERM..map{|l|trap(l){$r.stop}rescue require l};$u=Date;$z=($u.new.year + 145).abs;puts "== Almost Sinatra/No Version has taken the stage on #$z for development with backup from Webrick"

require 'rack'
require 'tilt'

trap( 'INT' )  { $server.stop }  # rename $r to $server
trap( 'TERM' ) { $server.stop }

$port = 4567              # rename $z to $port

puts "== Almost Sinatra II has taken the stage on #{$port} for development with backup from Webrick"


## line 2
# $n=Module.new{extend Rack;a,D,S,q=Rack::Builder.new,Object.method(:define_method),/@@ *([^\n]+)\n(((?!@@)[^\n]*\n)*)/m

$n = Module.new do
  app = Rack::Builder.new      # rename a to app
  req = nil                    # rename q to req

## line 3
# %w[get post put delete].map{|m|D.(m){|u,&b|a.map(u){run->(e){[200,{"Content-Type"=>"text/html"},[a.instance_eval(&b)]]}}}}

  ['get','post','put','delete'].each do |method|
    define_method method do |path, &block|
      app.map( path ) do
        run ->(env){ [200, {'Content-Type'=>'text/html'}, [app.instance_eval( &block )]]}
      end
    end
  end

## line 4
# Tilt.mappings.map{|k,v|D.(k){|n,*o|$t||=(h=$u._jisx0301("hash, please");File.read(caller[0][/^[^:]+/]).scan(S){|a,b|h[a]=b};h);v[0].new(*o){n=="#{n}"?n:$t[n.to_s]}.render(a,o[0].try(:[],:locals)||{})}}

  Tilt.mappings.each do |ext, engines|          # rename k to ext and v to engines
    define_method ext do |text, *args|          # rename n to text and o to args
      template = engines[0].new(*args) do
        text
      end
      locals = (args[0].respond_to?(:[]) ? args[0][:locals] : nil) || {}    # was o[0].try(:[],:locals)||{}
      template.render( app, locals )
    end
  end

## line 5
# %w[set enable disable configure helpers use register].map{|m|D.(m){|*_,&b|b.try :[]}};END{Rack::Handler.get("webrick").run(a,Port:$z){|s|$r=s}}

  # was END { ... }; change to run! method
  define_method 'run!' do
    Rack::Handler.get('webrick').run( app, Port:$port ) {|server| $server=server }
  end

## line 6
# %w[params session].map{|m|D.(m){q.send m}};a.use Rack::Session::Cookie;a.use Rack::Lock;D.(:before){|&b|a.use Rack::Config,&b};before{|e|q=Rack::Request.new e;q.params.dup.map{|k,v|params[k.to_sym]=v}}}

  ['params','session'].each do |method|
    define_method method do
      req.send method
    end
  end

  app.use Rack::Session::Cookie
  app.use Rack::Lock
  app.use Rack::Config do |env|
    req = Rack::Request.new( env )
  end
end # Module.new

All the tilt magic happens in line 4. Almost Sinatra will define a method for every format so you can use, for example:

markdown 'Strong emphasis, aka bold, with **asterisks** or __underscores__.'

or

erb "Hello <%= name %>!", locals: { name: params['name'] }

Let’s try the embedded ruby (erb) example by hand:

require 'tilt'

template =  Tilt::ErubisTemplate.new do        # Step 1: create a template
              "Hello <%= name %>!"
            end

puts template.render( self, name: 'World' )    # Step 2: render the template
                                               # There's no step 3 ;-)

Or to more closely reflect the Almost Sinatra setup:

require 'tilt'

text   = "Hello <%= name %>!"
params = { name: 'World' }
args   = [ { locals: { name: params[:name] } } ]

template =  Tilt::ErubisTemplate.new do
              text
            end

locals = args[0][:locals]
puts template.render( self, locals )

Resulting in:

Hello World!

The proof of the pudding. To wrap up let’s fire up Almost Sinatra II with an example app that renders a document (template) in markdown and another with embedded ruby (erb).

example.rb:

require_relative 'almost_sinatra_ii'

include $n    # include "anonymous" Almost Sinatra DSL module


get '/' do
  markdown <<EOS
## Welcome to Tilt

A Generic interface to multiple template engines

Try:
- [Say hello!](/hello?name=Tilt)

Powered by Almost Sinatra II (#{Time.now})
EOS
end


get '/hello' do
  erb "Hello <%= name %>!", locals: { name: params['name'] }
end


run!

Use

$ ruby ./example.rb

to startup the example app. Resulting in:

== Almost Sinatra II has taken the stage on 4567 for development with backup from Webrick
[2015-04-18 14:42:21] INFO  WEBrick 1.3.1
[2015-04-18 14:42:21] INFO  ruby 2.1.4 (2014-10-27) [i686-linux]
[2015-04-18 14:42:21] INFO  WEBrick::HTTPServer#start: pid=3955 port=4567

Open up your browser @ http://localhost:4567 or http://localhost:4567/hello?name=Tilt. Voila.

Learn More

Almost Sinatra

Tilt

{}
Ruby Gem of the Week ( Feed )
Thursday, 09 April 2015
Week #15 - beerdb gem - serving a Guinness Irish Stout or a Bamberg Aecht Schlenkerla Rauchbier Märzen as JSON w/ Ruby
What’s the beerdb gem?

The beerdb gem offers a ready-to-use database schema (in SQL) and models such as - surprise, surprise - Beer, Brewery, Brand and friends (using the ActiveRecord object-relational mapper machinery). Example:

What’s the beerdb gem?

The beerdb gem offers a ready-to-use database schema (in SQL) and models such as - surprise, surprise - Beer, Brewery, Brand and friends (using the ActiveRecord object-relational mapper machinery). Example:

Let’s try the brewery model:

by = Brewery.find_by( key: 'guinness' )

by.title
=> 'St. James's Gate Brewery / Guinness Brewery'

by.country.key
=> 'ie'

by.country.title
=> 'Ireland'

by.city.title
=> 'Dublin'

by.beers.first
=> 'Guinness', 4.2

...

Or let’s try the beer model:

b = Beer.find_by( key: 'guinness' )

b.title
=> 'Guinness'

b.abv    # that is, alcohol by volume (abv)
=> 4.2

b.tags
=> 'irish_dry_stout', 'dry_stout', 'stout'

b.brewery.title
=> 'St. James's Gate Brewery / Guinness Brewery'
...

What’s it good for? Good question. Let’s build an HTTP JSON service that serves up a Guinness Irish Stout or a Bamberg Aecht Schlenkerla Rauchbier Märzen as JSON? Example:

GET /beer/guinness

{
  "key": "guinness",
  "title": "Guinness",
  "synonyms": "Guinness Draught",
  "abv": "4.2",
  "srm": null,
  "og": null,
  "tags": ["irish_dry_stout","dry_stout","stout"],
  "brewery":
  {
    "key": "guinness",
    "title": "St. James's Gate Brewery / Guinness Brewery"
  },
  "country":
  {
    "key": "ie",
    "title": "Irland"
  }
}

Let’s use Sinatra - a micro webframework - that offers a mini language, that is, domain-specific language (DSL) that lets you define routes, that is, HTTP methods paired with an URL-machting pattern and much more. For example, you can code the GET /beer/guinness route in Sinatra as get '/beer/guinness'. To make it into a route for any beer lets replace the guinness beer key with a placeholder, thus, resulting in get '/beer/:key'. Let’s smoke it:

service.rb:


class BeerService < Sinatra::Base

  include BeerDb::Models     # lets (re)use the Beer, Brewery, etc. models

  get '/beer/:key' do |key|
    beer = Beer.find_by!( key: key )

    content_type :json       # set response to JavaScript Object Notation (JSON) content type 

    { key:      beer.key,
      title:    beer.title,
      synonyms: beer.synonyms,
      abv:      beer.abv,
      srm:      beer.srm,
      og:       beer.og  }.to_json
  end

end

That’s it. Ready to serve. Let’s boot-up the beer service with a web server (e.g. Thin) using a Rack handler. Example:

boot.rb:

require 'sinatra/base'    # note: sinatra will pull in web server machinery (e.g. rack, thin, etc.)
require 'beerdb/models'   # note: beerdb will pull in database access machinery (e.g. activerecord, etc.)

# database setup 'n' config
ActiveRecord::Base.establish_connection( adapter:  'sqlite3', database: './beer.db' )


require './service'

Rack::Handler::Thin.run BeerService.new, :Port => 9292

Try:

$ ruby ./boot.rb

Resulting in:

Thin web server (v1.6.3 codename Protein Powder)
Maximum connections set to 1024
Listening on 0.0.0.0:9292, CTRL+C to stop

Open up your browser and try http://localhost:9292/beer/guinness. Voila. Enjoy the Guinness responsibly.

Bonus: Let’s add brewery details and more

Let’s add brewery details to the beer service and lets add a new GET /brewery route. Example:

get '/beer/:key' do |key|
  beer = Beer.find_by!( key: key )

  brewery = {}
  if beer.brewery.present?
     brewery = { key:   beer.brewery.key,
                 title: beer.brewery.title }
  end

  tags = []
  if beer.tags.present?
     beer.tags.each { |tag| tags << tag.key }
  end

  country = {
    key:   beer.country.key,
    title: beer.country.title
  }

  content_type :json
    
  { key:      beer.key,
    title:    beer.title,
    synonyms: beer.synonyms,
    abv:      beer.abv,
    srm:      beer.srm,
    og:       beer.og,
    tags:     tags,
    brewery: brewery,
    country: country }.to_json
end


get '/brewery/:key' do |key|

  brewery = Brewery.find_by!( key: key )

  beers = []
  brewery.beers.each do |b|
    beers << { key: b.key, title: b.title }
  end

  tags = []
  if brewery.tags.present?
     brewery.tags.each { |tag| tags << tag.key }
  end

  country = {
    key:   brewery.country.key,
    title: brewery.country.title
  }

  content_type :json

  { key:      brewery.key,
    title:    brewery.title,
    synonyms: brewery.synonyms,
    since:    brewery.since,
    address:  brewery.address,
    web:      brewery.web,
    tags:     tags,
    beers:    beers,
    country:  country}.to_json
end

Bonus II: Let’s use the json helper from sinatra contrib

If you think the code looks to enterprisey (that is, too complex):

get '/beer/:key' do |key|
  beer = Beer.find_by!( key: key )

  content_type :json       # set response to JavaScript Object Notation (JSON) content type 

  { key:      beer.key,
    title:    beer.title,
    synonyms: beer.synonyms,
    abv:      beer.abv,
    srm:      beer.srm,
    og:       beer.og  }.to_json
end

Let’s make it simpler using the json helper from the sinatra contrib package. In your boot script (e.g. boot.rb) add:

require 'sinatra/json'

And than you can use:

get '/beer/:key' do |key|
  beer = Beer.find_by!( key: key )

  json key:      beer.key,
       title:    beer.title,
       synonyms: beer.synonyms,
       abv:      beer.abv,
       srm:      beer.srm,
       og:       beer.og
end

The json helper will automatically set the json content type and call the #to_json method on the passed in data hash. More magic. Less configuration (code).

Find Out More

beerdb

beer.db (datasets)

{}
Ruby Gem of the Week ( Feed )
Thursday, 02 April 2015
Week #14 - html-proofer gem - auto-proofread (check and validate) your hypertext (HTML) pages

Let’s say you have a project site and want to check that all your hypertext (HTML) pages open and close all tags according to the specs, that all internal (e.g. href="#datasets") and external (e.g. href="openbeer.github.io") links are working (no 404’s not found errors et

Let’s say you have a project site and want to check that all your hypertext (HTML) pages open and close all tags according to the specs, that all internal (e.g. href="#datasets") and external (e.g. href="http://openbeer.github.io") links are working (no 404’s not found errors etc.), that all images include an alternative text (alt="Database Schema") attribute and so on.

What’s the html-proofer gem?

Let’s thank Garen Torikian and friends who have bundled up all these checks and more in a ready-to-use gem, that is, html-proofer.

Let’s try it on the Open Mundi (world.db) project site:

_site
|-- i
|   |-- sqlitestudio.png
|   |-- worlddb-models-place.png
|   `-- worlddb-models.png
|-- build.html
|-- index.html
`-- style.css

You can run html-proofer using a script in Ruby:

proof.rb:

require 'html/proofer'

HTML::Proofer.new('./site').run

Or as an alternative use the htmlproof command-line tool. Try:

$ htmlproof --help

Resulting in:

htmlproof 2.1.0 -- Runs the HTML-Proofer suite on the files in PATH. For more details, see the README.

Usage:

  htmlproof PATH [options]

Options:
      --as-links     Assumes that `PATH` is a comma-separated array of links to check.
      --alt-ignore image1,[image2,...]  Comma-separated list of Strings or RegExps containing `img`s whose missing `alt` tags are safe to ignore
      --checks-to-ignore check1,[check2,...]   An array of Strings indicating which checks you'd like to not perform.
      --check-external-hash  Checks whether external hashes exist (even if the website exists). This slows the checker down (default: `false`).
      --directory-index-file  Sets the file to look for when a link refers to a directory. (default: `index.html`)
      --disable-external  Disables the external link checker (default: `false`)
      --error-sort SORT  Defines the sort order for error output. Can be `path`, `desc`, or `status` (default: `path`).
      --ext EXT      The extension of your HTML files (default: `.html`)
      --file-ignore file1,[file2,...]  Comma-separated list of Strings or RegExps containing file paths that are safe to ignore
      --href-ignore link1,[link2,...]  Comma-separated list of Strings or RegExps containing `href`s that are safe to ignore.
      --href-swap re:string,[re:string,...]  Comma-separated list of key-value pairs of `RegExp:String`. Transforms links matching `RegExp` into `String`
      --only-4xx     Only reports errors for links that fall within the 4x status code range.
      --check-favicon  Enables the favicon checker (default: `false`).
      --check-html   Enables HTML validation errors from Nokogiri (default: `false`).
      --verbose      Enables more verbose logging.
  -h, --help         Show this message
  -v, --version      Print the name and version
  -t, --trace        Show the full backtrace when an error occurs

Quite a lineup. Let’s try the “standard” checks with verbose logging turned on:

$ htmlproof --verbose ./_site

Resulting in:

Running ["ImageCheck", "ScriptCheck", "LinkCheck"] checks on ./_site on *.html... 

Checking imagecheck on ./_site/index.html ...
Checking scriptcheck on ./_site/index.html ...
Checking imagecheck on ./_site/build.html ...
Checking linkcheck on ./_site/index.html ...
Checking scriptcheck on ./_site/build.html ...
Checking linkcheck on ./_site/build.html ...
Checking 25 external links...
Running requests for all 25 external URLs...
Received a 200 for http://openbeer.github.io/ in ./_site/build.html
Received a 200 for http://openfootball.github.io/ in ./_site/build.html
Received a 200 for http://openmundi.github.io/ in ./_site/build.html
Received a 200 for http://opensport.github.io/ in ./_site/build.html
Received a 200 for https://github.com/openmundi/openmundi.github.io in ./_site/build.html
...
Received a 200 for https://github.com/openmundi/world.db/blob/master/north-america/countries.txt in ./_site/index.html
Received a 200 for https://github.com/opensport in ./_site/index.html
Received a 200 for https://github.com/mledoze/countries in ./_site/index.html
Received a 200 for https://github.com/opensport/formula1.db in ./_site/index.html
Received a 200 for https://github.com/opensport/ski.db in ./_site/index.html
Received a 200 for https://github.com/ewheeler/current-countries-of-earth in ./_site/index.html
Received a 200 for https://github.com/geraldb/world.db.admin in ./_site/index.html
Received a 200 for https://github.com/geraldb/world.db.ruby in ./_site/build.html
Received a 404 for https://github.com/openmundi/world.db/blob/master/europe/at-austria/cities.txt in ./_site/index.html
Received a 200 for http://groups.google.com/group/openmundi in ./_site/build.html ./_site/build.html
Received a 200 for https://raw.github.com/openmundi/openmundi.github.io/master/i/sqlitestudio.png in ./_site/build.html
Received a 200 for http://worlddb.herokuapp.com/ in ./_site/index.html
Ran on 2 files!

- ./_site/build.html
  *  image https://raw.github.com/openmundi/openmundi.github.io/master/i/sqlitestudio.png does not have an alt attribute (line 73)
- ./_site/index.html
  *  External link https://github.com/openmundi/world.db/blob/master/europe/at-austria/cities.txt failed: 404 No error
  *  image i/worlddb-models-place.png does not have an alt attribute (line 74)
  *  image i/worlddb-models.png does not have an alt attribute (line 70)

htmlproof 2.1.0 | Error:  HTML-Proofer found 4 failures!

Bingo! The proof reader found four erros in the two pages (that is, build.html and index.html). The link to at-autstria/cities.txt is broken (404s) and all three images miss the required alt attribute.

To wrap up lets try two more options, that is, --check-favicon that checks that every page includes a favicon and --check-html that checks that all hypertext markup tags and attributes are in order using the HTML parser from the Nokogiri gem.

$ htmlproof --verbose --check-favicon --check-html  ./_site

Resulting in:

Checking htmlcheck on ./_site/build.html ...
Checking faviconcheck on ./_site/build.html ...
Checking htmlcheck on ./_site/index.html ...
Checking faviconcheck on ./_site/index.html ...
Ran on 2 files!

- ./_site/build.html
  *  no favicon specified
- ./_site/index.html
  *  no favicon specified

htmlproof 2.1.0 | Error:  HTML-Proofer found 2 failures!

Oh well that’s right. The site has no favicon - but at least all the hypertext tags are lined-up properly (that is, well-formed without any validation errors).

Bonus: Configuring Your Travis Builds w/ Jekyll & HTML Proofreader

Note: If you use a continuous build service (such as Tarvis) you can set it up to run the HTML proof reading tests on every check-in, for example.

.travis.yml:

language: ruby
rvm:
- 2.1
script: jekyll build && htmlproof ./_site

# branch whitelist
branches:
  only:
  - gh-pages      # test the gh-pages branch

env:
  global:
  - NOKOGIRI_USE_SYSTEM_LIBRARIES=true   # speeds up installation of html-proofer

Find Out More

{}
Ruby Gem of the Week ( Feed )
Thursday, 26 March 2015
Week #13 - props gem - yet another config (INI) reader in Ruby

Let’s say you’re looking for an easy configuration file format for your tool or service. Why not use the Windows-inspired INI format as an (easier and simpler) alternative to the popular JSON, YAML and TOML formats? Example:

ruby.conf:

title = Planet Ruby

[ru

Let’s say you’re looking for an easy configuration file format for your tool or service. Why not use the Windows-inspired INI format as an (easier and simpler) alternative to the popular JSON, YAML and TOML formats? Example:

ruby.conf:

title = Planet Ruby

[rubylang]
  title = Ruby Lang News
  link  = http://www.ruby-lang.org/en/news
  feed  = http://www.ruby-lang.org/en/feeds/news.rss

[rubyonrails]
  title = Ruby on Rails News
  link  = http://weblog.rubyonrails.org
  feed  = http://weblog.rubyonrails.org/feed/atom.xml

[viennarb]
  title = Vienna.rb News
  link  = http://vienna-rb.at
  feed  = http://vienna-rb.at/atom.xml

Now all that’s missing is a gem that reads in the Windows-inspired INI format and returns a plain “old” Ruby hash.

What’s the props gem?

Using the props gem let’s you read in configuration settings in the Windows-inspired INI format. Use the - surprise, surprise - INI class to get started. Example:

require 'pp'
require 'props'

hash = INI.load_file( './ruby.conf' )

puts "hash:"
pp hash

Will print:

hash:
{"title"=>"Planet Ruby",
 "rubylang"=>
  {"title"=>"Ruby Lang News",
   "link"=>"http://www.ruby-lang.org/en/news",
   "feed"=>"http://www.ruby-lang.org/en/feeds/news.rss"},
 "rubyonrails"=>
  {"title"=>"Ruby on Rails News",
   "link"=>"http://weblog.rubyonrails.org",
   "feed"=>"http://weblog.rubyonrails.org/feed/atom.xml"},
 "viennarb"=>
  {"title"=>"Vienna.rb News",
   "link"=>"http://vienna-rb.at",
   "feed"=>"http://vienna-rb.at/atom.xml"}}

To access any settings works like any “standard” hash because it’s just a “standard” hash. Use hash['rubylang'] and hash['rubylang']['title'] and so on. Example:

puts "rubylang:"
pp hash['rubylang']
# => {"title"=>"Ruby Lang News",
      "link"=>"http://www.ruby-lang.org/en/news",
      "feed"=>"http://www.ruby-lang.org/en/feeds/news.rss"}

puts "viennarb:"
pp hash['viennarb']
# => {"title"=>"Vienna.rb News",
      "link"=>"http://vienna-rb.at",
      "feed"=>"http://vienna-rb.at/atom.xml"}

puts "rubylang/title: #{hash['rubylang']['title']}"
# => rubylang/title: Ruby Lang News

puts "viennarb/title: #{hash['viennarb']['title']}"
# => viennarb/title: Vienna.rb News

Manage Settings Hierachies (e.g. Work, Home, Defaults, etc.) with the ‘Props’ class

If you prefer you can wrap the hash in a Props class. Example:

props = Props.new( hash, 'ruby.ini' )

puts "rubylang:"
pp props.get('rubylang')

puts "viennarb:"
pp props.get('viennarb')

puts "rubylang/title: #{props.get_from_section('rubylang','title')}"
# => rubylang/title: Ruby Lang News

puts "viennarb/title: #{props.get_from_section('viennarb','title')}"
# => viennarb/title: Vienna.rb News

What’s the point you might ask? Using the Props class you can linkup hash settings into a lookup hierachy. Example:

default_props = Props.new( defaults_hash, 'DEFAULTS' )
home_props    = Props.new( home_hash,     '~/ruby.conf', default_props )
props         = Props.new( work_hash,     './ruby.conf', home_props)

Now if you try props.get('viennarb'), for example, the getter will first look in the working (current) folder settings hash and than in the home folder settings hash and finally if still not found in the defaults hash. For a “real-world” example lets wrap-up with the config reader snippet from the markdown gem using the Props class. Example:

class Config

  DEFAULTS = { 'libs' => [ 'kramdown' ],
               'extnames' => [
                 '.markdown',
                 '.m',
                 '.mark',
                 '.mkdn',
                 '.md',
                 '.mdown',
                 '.markdn',
                 '.txt',
                 '.text' ],
                'redcarpet' => {
                 'extensions' => [
                    'no_intra_emphasis',
                    'fenced_code_blocks',
                    'tables',
                    'strikethrough' ] }
             }

  def initialize
    @props = @props_default = Props.new( DEFAULTS, 'DEFAULTS' )

    ## check for user settings in home folder

    props_home_file = File.join( Env.home, 'markdown.conf' )
    if File.exists?( props_home_file )
      puts "Loading settings from '#{props_home_file}'..."
      @props = @props_home = Props.load_file( props_home_file, @props )
    end

    ## check for user settings in working (current) folder

    props_work_file = File.join( '.', 'markdown.conf' )
    if File.exists?( props_work_file )
      puts "Loading settings from '#{props_work_file}'..."
      @props = @props_work = Props.load_file( props_work_file, @props )
    end
  end

  def markdown_extnames
    @props.get( 'extnames' )
  end

  ...

end

Find Out More

{}
Ruby Gem of the Week ( Feed )
Thursday, 19 March 2015
Week #12 - logutils gem - yet another (lightweight, simple) logging library in Ruby

Sooner or later you will debug your code and add some print statements to help you along. Example:

puts "[debug] value: #{value}"
puts "[debug] value.class.name: #{value.class.name}

A “better” way is to use a logger that lets you turn off and on your debug messag

Sooner or later you will debug your code and add some print statements to help you along. Example:

puts "[debug] value: #{value}"
puts "[debug] value.class.name: #{value.class.name}

A “better” way is to use a logger that lets you turn off and on your debug messages as needed and why not a add some more message classes for errors or warnings, for example?

The “classic” message class hierarchy reads:

  • [OFF]
  • FATAL
  • ERROR
  • WARN
  • INFO
  • DEBUG
  • [ALL]

What’s the logutils gem?

Using the logutils gem - yet another logging library - you can print all messages and more. Start by getting a logger e.g.:

logger = LogUtils::Logger.new

And now you can use the “standard” methods such as #debug, #info, #warn, etc.

logger.debug "msg"
logger.info "another msg"
logger.warn "another msg"
logger.error "another msg"
logger.fatal "another msg"

To turn on/off debug messages or any others in the hierarchy use the level attribute. Example:

logger.level = :info

Will print only message of class info and above and, thus, turn off debug messages.

Logging Mixin

What else? For your convenience you can include the logging machinery in a Ruby class with a single line using the Logging mixin e.g.

include LogUtils::Logging

This will add/mixin the logger attribute reader e.g.

def logger
  @logger ||= Logger[ self ]
end

And you’re all setup to start logging. Example:

class SampleClass

 include Logging
    
  def initialize
    logger.info 'Hello, SampleClass!'
  end
end

Bonus: Log to the database using LogDb

To log to the database use an addon, that is, the logutils-activerecord gem. Example:

require 'logutils'
require 'logutils/activerecord'   # Note: will also require 'active_record'
    
include LogUtils    # lets you use Logger instead of LogUtils::Logger

logger = Logger[ 'Test' ]
logger.info 'Hello, LogUtils!'

LOG_DB_CONFIG = {
  adapter:   'sqlite3',
  database:  ':memory:'
}

ActiveRecord::Base.establish_connection( LOG_DB_CONFIG )

LogDb.create   # create logs table
LogDb.setup    # setup handler that saves logs in the database

logger.info '¡Hola! LogUtils'
logger.warn 'Servus LogUtils!'

To create the logs table in the database use:

LogDb.create

Just a shortcut for:

create_table :logs do |t|
  t.string  :msg,   null: false
  t.string  :level, null: false  # e.g. fatal, error, warn, info, debug
  t.string  :app
  t.string  :tag
  t.integer :pid
  t.integer :tid, limit: 8
  t.string  :ts
  t.timestamps
end

To start logging to the database add a log event handler that stores log messages in the database. Use:

LogDb.setup

To dump all log messages to the console you can use the log model. Example:

LogDb::Model::Log.order('created_at DESC').each do |log|
  puts "[#{log.level}] #{log.msg}"
end

Will print:

[info] ¡Hola! LogUtils
[warn] Servus LogUtils!

That’s it. Happy logging.

Find Out More

logutils

logutils-activerecord

{}
Ruby Gem of the Week ( Feed )
Thursday, 12 March 2015
Week #11 - worldlite gem - lightweight public domain country data (all data included as good ol' ruby code)

Ever wondered how many countries are in the world? 222? 204? 196? 193? It all depends - of course - what’s your definition of country is. The United Nations lists 193 countries (as members), the World Football Association (that is, FIFA) lists 209 countries (as members) plus 13 more countries (as

Ever wondered how many countries are in the world? 222? 204? 196? 193? It all depends - of course - what’s your definition of country is. The United Nations lists 193 countries (as members), the World Football Association (that is, FIFA) lists 209 countries (as members) plus 13 more countries (as members of continental federations only), or the Olympics Committee lists 204 countries or, the ISO 3166 country code standard lists 196 countries, for example, and so on. Now lets say you need a country list for all FIFA members for the World Cup Qualifiers 2018 in Russia in Ruby?

What’s the worldlite gem?

Let’s welcome the worldlite gem that includes “lightweight” public domain country data - that is, all data gets included as good ol’ ruby code - no database, no dependencies, no copyright. For example, the class defition for Austria reads:

module WorldLite

  AT = Country.new do |c|

    c.name   = 'Austria'
    c.key    = 'at'
    c.alpah2 = 'AT'   # iso two-letter country code (ISO 3166-1 alpha-2)
    c.alpha3 = 'AUT'  # iso three-letter country code (ISO 3166-1 alpha-3)
    c.fifa   = 'AUT'  # football country code (fifa = Fédération Int'le de Football Association)
    c.ioc    = 'AUT'  # olympics country code (ioc = International Olympic Committee)
    c.net    = 'at'   # internet top level domain
    c.motor  = 'A'    # motor vehicle license plate code
    c.num3   = '040'  # iso numeric three-digits code as string (ISO 3166-1 numeric)
    c.num    = 40     # iso numeric code as number

    c.continent_name =  'Europe'

    c.un     = true     # United Nations member?  -- 193 member countries
    c.eu     = true     # European Union member?  -- 27 member countries
    c.euro   = true     # Euro Zone member?       -- 17 member countries

    c.wikipedia = 'Austria'   # e.g. see en.wikipedia.org/wiki/Austria
    c.wikidata  = 40          # e.g. see wikidata.org/wiki/Q40
    c.factbook  = 'au'        # e.g. see www.cia.gov/.../the-world-factbook/geos/au.html
    
  end

  WORLD      << AT
  WORLD_UN   << AT
  WORLD_ISO  << AT
  WORLD_FIFA << AT

  EUROPE       << AT
  EUROPE_EU    << AT
  EUROPE_EURO  << AT

end  # module WorldLite

And you can use it in Ruby like:

require 'worldlite'

include WorldLite

WORLD.size       # => 245
WORLD_UN.size    # => 193
WORLD_G20.size   # => 20
WORLD_FIFA.size  # => 243
WORLD_WTO.size   # => 157
WORLD_OECD.size  # => 34

EUROPE.size      # => 51
EUROPE_UEFA.size # => 54
EUROPE_EU.size   # => 27
EUROPE_EURO.size # => 17

AT.class.name     # => 'WorldLite::Country'
AT.name           # => 'Austria'
AT.continent_name # => 'Europe'
AT.alpha3         # => 'AUT'
AT.slug           # => 'austria'
AT.un?            # => true
AT.fifa?          # => true
AT.g20?           # => false
AT.eu?            # => true
AT.euro?          # => true

AT.wikpedia       # => 'Austria'
AT.wikidata       # => 40
AT.wikpedia_url   # => 'http://en.wikipedia.org/wiki/Austria'
AT.wikidata_url   # => 'http://www.wikidata.org/wiki/Q40'
AT.factbook       # => 'au'
AT.factbook_url   # => 'http://www.cia.gov/.../the-world-factbook/geos/au.html'

That’s it.

Bonus Question

Now you might wonder - where’s the country data coming from and who’s going to keep it up-to-date?

The answer is recursive. First, the good ol’ ruby code gets auto-generated with a ruby script using the world.db. The country template, for example, uses embedded Ruby (erb) and - surprise, surprise - looks like:

module WorldLite

  c = Country.new
  c.name   = <%= fmt_str( country.name ) %>
  c.key    = <%= fmt_str( country.key )  %>
  c.alpha3 = <%= fmt_str( country.iso3 ) %>
  c.fifa   = <%= fmt_str( country.fifa ) %>
  c.net    = <%= fmt_str( country.net  ) %>
  
  c.continent_name =  <%= fmt_str( country.continent.name ) %>

  c.un     = <%= fmt_bool( has_tag?( country, 'un' )) %>
  c.eu     = <%= fmt_bool( has_tag?( country, 'eu' )) %>
  c.euro   = <%= fmt_bool( has_tag?( country, 'euro' )) %>
  
  WORLD      << <%= country.key.upcase %>

<% if has_tag?( country, 'un' ) %>
  WORLD_UN   << <%= country.key.upcase %>
<% end %>
<% if country.iso3 %>
  WORLD_ISO  << <%= country.key.upcase %>
<% end %>
<% if country.fifa %>
  WORLD_FIFA << <%= country.key.upcase %>
<% end %>
...

The world.db itself gets build from plain text datasets that get hosted on GitHub and, thus, updating the world.db datasets works like updating anything on GitHub. Update it in your brower like-a-wiki or use a pull request and so on.

Find Out More

worldlite

world.db (datasets)

{}
Ruby Gem of the Week ( Feed )
Thursday, 05 March 2015
Week #10 - annotate gem - annotate your ActiveRecord models with comments about your table structure

Magic. ActiveRecord models can be as simple as:

class Beer < ActiveRecord::Base
end

or

class Brewery < ActiveRecord::Base
end

Some may find that’s a little too much magic. Where’s the code? What attribute can you use?

Magic. ActiveRecord models can be as simple as:

class Beer < ActiveRecord::Base
end

or

class Brewery < ActiveRecord::Base
end

Some may find that’s a little too much magic. Where’s the code? What attribute can you use?

By default ActiveRecord models require no information on the database tables wrapped (it all works - thanks to convention over configuration, that is, the class Beer (singular noun), for example, gets mapped to the table beers (plural noun) and Brewery to breweries and so on.

Best of both worlds. Less code is great and it’s easy to update the model - just update the table - there are no out-of-date setter and getters duplicated in the model, for example. If you want the best of both worlds - you can always add the table columns to your models as comments. Example:

# == Schema Information
#
# Table name: beers
#
#  id         :integer          not null, primary key
#  key        :string(255)      not null
#  title      :string(255)      not null
#  synonyms   :string(255)
#  web        :string(255)
#  since      :integer
#  seasonal   :boolean          default(FALSE), not null
#  limited    :boolean          default(FALSE), not null
#  kcal       :decimal
#  abv        :decimal
#  og         :decimal
#  srm        :integer
#  ibu        :integer
#  brewery_id :integer
#  brand_id   :integer
#  grade      :integer          default(4), not null
#  txt        :string(255)
#  txt_auto   :boolean          default(FALSE), not null
#  country_id :integer          not null
#  region_id  :integer
#  city_id    :integer
#  created_at :datetime
#  updated_at :datetime
#

class Beer < ActiveRecord::Base
end

or

# == Schema Information
#
# Table name: breweries
#
#  id          :integer          not null, primary key
#  key         :string(255)      not null
#  title       :string(255)      not null
#  synonyms    :string(255)
#  address     :string(255)
#  since       :integer
#  closed      :integer
#  brewpub     :boolean          default(FALSE), not null
#  web         :string(255)
#  wikipedia   :string(255)
#  country_id  :integer          not null
#  region_id   :integer
#  city_id     :integer
#  created_at  :datetime
#  updated_at  :datetime
#

class Brewery < ActiveRecord::Base
end

That looks like a lot of work if you type it in by hand. If all the schema information is already stored in the database - why not automate the annotation procedure with a script in Ruby?

What’s the annotate gem?

Let’s thank Dave Thomas and friends who created the first annotate-models script back in 2006 as a Rails plugin and let’s thank Cuong Tran and friends who continue the tradition with a modern up-to-date annotate gem.

Not just for Rails. Out-of-the-box the annotate gem includes a command line tool named - surprise, surprise - annotate. Let’s try it:

$ annotate -h

Will result in:

Please run annotate from the root of the project.

The annotate tool requires a Rakefile or Gemfile in the current working folder. Let’s create an empty Rakefile. Example:

Rakefile:

# beer.db Models Annotate Example

Now try:

$ annotate -h

Will result in:

Usage: annotate [options] [model_file]*
    -d, --delete                     Remove annotations from all model files or the routes.rb file
    -p, --position [before|after]    Place the annotations at the top (before) or the bottom (after) of the model/test/fixture/factory/routes file(s)
        --pc, --position-in-class [before|after]
                                     Place the annotations at the top (before) or the bottom (after) of the model file
        --pf, --position-in-factory [before|after]
                                     Place the annotations at the top (before) or the bottom (after) of any factory files
        --px, --position-in-fixture [before|after]
                                     Place the annotations at the top (before) or the bottom (after) of any fixture files
        --pt, --position-in-test [before|after]
                                     Place the annotations at the top (before) or the bottom (after) of any test files
        --pr, --position-in-routes [before|after]
                                     Place the annotations at the top (before) or the bottom (after) of the routes.rb file
    -r, --routes                     Annotate routes.rb with the output of 'rake routes'
    -v, --version                    Show the current version of this gem
    -m, --show-migration             Include the migration version number in the annotation
    -i, --show-indexes               List the table's database indexes in the annotation
    -s, --simple-indexes             Concat the column's related indexes in the annotation
        --model-dir dir              Annotate model files stored in dir rather than app/models
        --ignore-model-subdirects    Ignore subdirectories of the models directory
        --sort                       Sort columns alphabetically, rather than in creation order
    -R, --require path               Additional file to require before loading models, may be used multiple times
    -e [tests,fixtures,factories],   Do not annotate fixtures, test files, and/or factories
        --exclude
    -f [bare|rdoc|markdown],         Render Schema Infomation as plain/RDoc/Markdown
        --format
        --force                      Force new annotations even if there are no changes.
        --timestamp                  Include timestamp in (routes) annotation
        --trace                      If unable to annotate a file, print the full stack trace, not just the exception message.
    -I, --ignore-columns REGEX       don't annotate columns that match a given REGEX (i.e., `annotate -I '^(id|updated_at|created_at)'`

Looking good. Let’s try to annotate the standalone beer.db models, that is, Beer, Brand, and Brewery. Create a new /lib folder and add:

beer.rb:

class Beer < ActiveRecord::Base
end

brand.rb:

class Brand < ActiveRecord::Base
end

brewery.rb:

class Brewery < ActiveRecord::Base
end

And to wrap-up add the required setup code for an in-memory SQLite datebase to the empty Rakefile:

Rakefile:

# beer.db Models Annotate Example

def setup_in_memory_db
  require 'beerdb'
  
  ActiveRecord::Base.establish_connection(
      adapter:  'sqlite3',
      database: ':memory:'
  )
  
  BeerDb.create_all
end


setup_in_memory_db()

That’s it. Get ready to annotate the models. Try:

$ annotate --model-dir lib

Resulting in:

Annotated (3): Beer, Brand, Brewery

Open up the beer.rb, brand.rb or brewery.rb scripts and Voila! All the table schema information is now included. To update the table schema information simply rerun annotate.

Find Out More

{}
Ruby Gem of the Week ( Feed )
Thursday, 26 February 2015
Week #9 - state_machine(s) gem - model processes and work flows with finite state machines (FSM) and automata theory
What’s a state machine? What’s a finite state machine (FSM)?

A state machine or finite state machine (FSM) lets you model a traffic light, a shopping cart checkout procedure, a conference proposal process, a credit card transaction and many more “real world” tasks using 1) states, 2) events, and

What’s a state machine? What’s a finite state machine (FSM)?

A state machine or finite state machine (FSM) lets you model a traffic light, a shopping cart checkout procedure, a conference proposal process, a credit card transaction and many more “real world” tasks using 1) states, 2) events, and 3) transitions. Or to quote the Wikipedia:

A finite-state machine (FSM) or finite-state automaton (plural: automata), or simply a state machine, is a mathematical model of computation used to design both computer programs and sequential logic circuits. It is conceived as an abstract machine that can be in one of a finite number of states. The machine is in only one state at a time; the state it is in at any given time is called the current state. It can change from one state to another when initiated by a triggering event or condition; this is called a transition. A particular FSM is defined by a list of its states, and the triggering condition for each transition.

Example: The State of the Football Match

Let’s start defining a state machine for modelling football matches in Ruby.

class Match

  state_machine :initial => :scheduled do
    event :start do
      transition :scheduled => :playing
    end
    event :finish do
      transition :playing => :finished
    end
  end

end

That you can use like:

m = Match.new
m.state        # => :scheduled
m.scheduled?   # => true
m.playing?     # => false
m.finished?    # => false
m.can_finish?  # => false
m.can_start?   # => true

m.start        # transition from :scheduled to :playing by calling event method e.g. start
m.state        # => :playing
m.playing?     # => true
m.can_start?   # => false
m.can_finish?  # => true

m.finish
m.state        # => Trivia Quiz: a) :scheduled, b) :playing, c) :finished

The state is a variable that holds - surprise, surprise - the state of the model. All possible states are strings (symbols) e.g. :scheduled, :playing, :finished.

Events are methods (e.g. Match#start, Match#finish) that change the state of the model (that is, transition from one state to another e.g. from :playing to :finished).

Why use a state machine? Because it lets you do more with less code thanks to automata theory packed-up into ready-to-use gems for use in your models in practice.

What’s the state_machine(s) gem?

Let’s thank Aaron Pfeifer and friends - who started the state_machine gem and Abdelkader Boudih and friends - who continue with the state_machines gem that lets you create state machines for attributes on any class in Ruby.

Let’s continue the football match example - yes, it’s the state_machine gem in action - and let’s add more states. Football matches in the real-world can get canceled, postponed, awarded points if canceled or can get rescheduled or matches can get scheduled for a matchday but not yet timed for a fixed day and time and so on and on. A picture says more than a thousand words:

Let’s define the state machine in Ruby using the state_machine mini language:

class Match

  state_machine :initial => :scheduled do
    event :time do
      transition [:scheduled,:postponed] => :timed
    end
    event :postpone do
      transition :timed => :postponed
    end
    event :start do
      transition :timed => :playing
    end
    event :cancel do
      transition [:timed,:postponed,:playing] => :canceled
    end
    event :reschedule do
      transition :canceled => :scheduled
    end
    event :finish do
      transition :playing => :finished
    end
    event :award do
      transition :canceled => :awarded
    end
  end

end

And let’s try it out:

m = Match.new
m.state        # => :scheduled
m.scheduled?   # => true
m.timed?       # => false
m.can_start?   # => false
m.time
m.timed?       # => true
m.can_start?   # => true
m.start
m.playing?     # => true
m.finished?    # => false
m.can_cancel?  # => true
...

What else can state_machine do for you? Let’s store the state in a database table with ActiveRecord. First let’s create a table with the required column - surprise, surprise - called state:

create_table :matches do |t|
  t.string :state
end

And derive the Match class from ActiveRecord::Base:

class Match < ActiveRecord::Base
  ## same as above
end

That’s it. Let’s test drive the script with an in-memory SQLite database:

require 'active_record'
require 'state_machine'

ActiveRecord::Base.establish_connection(
  adapter:  'sqlite3',
  database: ':memory:'
)

ActiveRecord::Schema.define do
  create_table :matches do |t|
    t.string :state
  end
end


class Match < ActiveRecord::Base
  ## same as above
end

m = Match.create!   # creates the record in the database table using SQL
                    #  e.g INSERT INTO matches (state) VALUES ("scheduled")
m.state             # => :scheduled
m.time              # gets auto-saved (and auto-commited)
                    #  e.g. UPDATE matches SET state = "timed" WHERE matches.id = 1
m.timed?            # => true
m.finished?         # => false
m.start             # again gets auto-saved (and auto-commited)
                    #  e.g. UPDATE matches SET state = "playing" WHERE matches.id = 1
m.state             # => Trivia Quiz: a) :scheduled, b) :playing, c) :finished
...

That’s just the beginning. More state_machine highlights include:

  • Validations
  • Hooks e.g. before/after/around/failure transition callbacks
  • State predicates
  • Attribute-based event transitions
  • Multiple state machines on a single class and namespaced state machines
  • State values of any data type (use integer instead of strings/symbols etc.)
  • GraphViz visualization creator
  • And much more

Bonus - Order Processing Example by Markus Prinz

Thanks to Markus Prinz - who talked about “Ghost in the State Machine” at the last Vienna.rb meetup and lets us share the order state machine example:

require 'state_machine'

class Order
  state_machine :state, :initial => :new do
    around_transition do |order, transition, block|
      puts "Starting transition from #{transition.from}"
      block.call
      puts "Finished transition to #{transition.to}"
    end

    event :submit do
      transition :new => :payment_waiting
    end

    event :payment_received do
      transition :payment_waiting => :waiting_for_processing
    end

    event :payment_failed do
      transition :payment_waiting => :payment_failed
    end

    event :retry_payment do
      transition :payment_failed => :payment_waiting
    end

    event :process do
      transition :waiting_for_processing => :waiting_for_shipping
    end

    event :ship do
      transition :waiting_for_shipping => :shipped
    end
    after_transition :on => :ship, :do => :notify_customer

    # Multiple transitions, first matching is taken
    event :cancel do
      transition :waiting_for_processing => :canceled
      transition :waiting_for_shipping => :canceled
      transition :shipped => :waiting_for_return
    end

    event :return_shipment_received do
      transition :waiting_for_return => :canceled
    end
  end

  def notify_customer
    puts "Your package has been shipped! :shipit:"
  end
end

if __FILE__ == $0
  order_fsm = Order.new
  order_fsm.submit!
  order_fsm.payment_received!
  order_fsm.process!
  order_fsm.ship!

  puts "\n\n"

  print "Received payment twice (soft): "
  order_fsm.payment_received
  puts order_fsm.state
  puts "Received payment twice:"
  order_fsm.payment_received!
end

(Source: Ghost in the State Machine, Talk Slides)

Find Out More

state_machines (ActiveRecord 4.1+)

state_machine (ActiveRecord up-to 4.0.x)

{}
Ruby Gem of the Week ( Feed )
Thursday, 19 February 2015
Week #8 - rails-erd gem - generate entity-relationship diagrams (ERD) for your activerecord models

Let’s say you have defined your database schema (tables) with ActiveRecord in Ruby. Example:

create_table :breweries do |t|
  t.string  :key,    null: false
  t.string  :title,  null: false
  t.string  :address
  t.string  :web
end

create_table :beers do |t|
  t.references :brewer

Let’s say you have defined your database schema (tables) with ActiveRecord in Ruby. Example:

create_table :breweries do |t|
  t.string  :key,    null: false
  t.string  :title,  null: false
  t.string  :address
  t.string  :web
end

create_table :beers do |t|
  t.references :brewery
  t.string  :key,     null: false
  t.string  :title,   null: false
  t.text    :comments
end

And your models with classes in Ruby and assocations with class macros such as belongs_to, has_many, and so on:

class Beer < ActiveRecord::Base
  belongs_to :brewery
end

class Brewery < ActiveRecord::Base
  has_many   :beers
end

How can you auto-generate an entity-relationship diagram? For example:

The good news. The ActiveRecord machinery already has everything built-in for a minimal (quick ‘n’ dirty) do-it-yourself version.

Step 1: “Discover” all models

Use ActiveRecord::Base.descendants that gets you an array with all loaded (known) models at runtime to find (discover) all models of your app. Example:

models = ActiveRecord::Base.descendants

puts " #{model.size} models:"

models.each do |model|
  puts "  #{model.name}"
end

Will print for our simple example schema:

 2 models:
     Beer
     Brewery

Step 2: Get all “meta” info - all column definitions and associations

Now lets print out all columns with its name and SQL type plus all associations (defined with the “classic” belongs_to, has_many, etc. macros):

models.each do |model|
  puts "#{model.name}"
  puts '  columns:'
  model.columns.each do |column|
    puts "    #{column.name} #{column.sql_type}"
  end

  puts '  assocs:'
  model.reflect_on_all_associations.each do |assoc|
    puts "    #{assoc.macro} #{assoc.name}"
  end
end

Results in:

Beer
  columns:
    id         integer
    brewery_id integer
    key        varchar(255)
    title      varchar(255)
    comments   text
  assocs:
    belongs_to brewery
Brewery
  columns:
    id         integer
    key        varchar(255)
    title      varchar(255)
    address    varchar(255)
    web        varchar(255)
  assocs:
    has_many beers

Step 3: Turn the text describing your models and assocations into a diagram

Now all that’s left is turning the text into a diagram. Again the good news - tools and services abound - let’s start with the yuml.me service. Use:

[note: A simple beer.db diagram with yuml.me  {bg:wheat}]

[Brewery|key;title;address;web] -> [Beer|key;title;comments]

that gets turned into:

Now why not find a gem that alreay has all the code packed up for easy (re)use with more examples and a getting started guide and much more?

What’s the rails-erd gem?

Let’s thank Rolf Timmermans, Kerri Miller, and friends who have created the rails-erd gem that lets you easily auto-generate entity-relationship diagrams (ERD) from your ActiveRecord models.

Not just for Rails. Although the gem includes rails in its name it works great with “plain vanilla” ActiveRecord models without requiring the Rails machinery. Let’s try it using the beer.db ActiveRecord models and schema bundled-up for easy (re)use in the beerdb-models gem.

require 'beerdb/models'            # use $ gem install beerdb

## Let's create an in-memory SQLite database

DB_CONFIG = {
  adapter: 'sqlite3',
  database: ':memory:'
}

ActiveRecord::Base.establish_connection( DB_CONFIG )

BeerDb.create_all   ## create tables (e.g. breweries, beers, etc.)

## Now hand over to rails-erd

require 'rails_erd/diagram'

class YumlDiagram < RailsERD::Diagram

  setup do
    @edges = []
  end

  each_relationship do |relationship|
    line = if relationship.indirect? then "-.-" else "-" end
    
    arrow = case 
    when relationship.one_to_one?   then "1#{line}1>"
    when relationship.one_to_many?  then "1#{line}*>"
    when relationship.many_to_many? then "*#{line}*>"
    end

    @edges << "[#{relationship.source}] #{arrow} [#{relationship.destination}]"
  end

  save do
    puts @edges.join("\n")
  end
end

YumlDiagram.create

will result in (simplified):

[Country] 1-*> [State]
[State] 1-*> [City]
[City] 1-*> [Brewery]
[Brewery] 1-*> [Beer]
[Brewery] 1-*> [Brand]
[Brand] 1-*> [Beer]

And turned into a diagram:

Note: Instead of using the all-in-one YumlDiagram.create convenience method you can walk through step-by-step. Example:

## Get all meta-info

domain  = RailsERD::Domain.generate

pp domain.entities        ## dump all entities (models)
pp domain.relationships   ## dump all relationships (assocs)

## Generate diagram

diagram = YumlDiagram.new( domain )

diagram.generate   ## step 1 - generate
diagram.save       ## step 2 - save

What’s Graphviz and the DOT language?

Note, by default the rails-erd uses the Graphviz class to build your diagrams using the graphviz machinery (and its DOT language).

Graphviz (short for Graph Visualization Software) is a free open source package by AT&T Labs Research for drawing graphs specified in DOT language scripts started more than fifteen years ago. Example:

digraph example
{  
  Brewery [shape=box, style=filled, color=blue]
  Beer [shape=box, color=navy]

  Country -> State -> City -> Brewery
  Brewery -> Beer
  Brewery -> Brand
  Brand   -> Beer
}

Change the YumlDiagram.create method to RailsERD::Diagram::Graphviz.create and you will get a GraphViz-generated diagram as a PDF document, PNG pixel graphic, SVG vector graphic or whatever filetype you desire. That’s it.

Find Out More

{}
Ruby Gem of the Week ( Feed )
Thursday, 12 February 2015
Week #7 - gli gem - git-like interfaces for awesome command-line tools

What’s OptionParser?

Ruby ships with a built-in class, that is, OptionParser that lets you define and parse options for your command line tool. Let say you’re building a command line tool for the open beer database and as options you want to offer a switch to turn

What’s OptionParser?

Ruby ships with a built-in class, that is, OptionParser that lets you define and parse options for your command line tool. Let say you’re building a command line tool for the open beer database and as options you want to offer a switch to turn on debug messages e.g. -v or --verbose and another switch to change the database name from the default beer.db to lets say pivo.db using -n pivo.db or --dbname=pivo.db.

A minimal version with the built-in OptionParser looks like:

require 'optparse'

config = { name: 'beer.db' }

parser = OptionParser.new do |opts|
  opts.banner = "Usage: beerdb [OPTS]"

  opts.on("-v", "--verbose", "Show debug messages") do |verbose|
    config[:verbose] = verbose
  end

  opts.on("-n", "--dbname=NAME", "Database name (default: beer.db)") do |name|
    config[:name] = name
  end

  opts.on("-h", "--help", "Prints this help") do
    puts opts
    exit
  end
end

parser.parse!(ARGV)

p config
p ARGV

Try

$ ruby beerdb.rb --help

Resulting in:

Usage: beerdb [OPTS]
    -v, --verbose          Show debug messages
    -n, --dbname=NAME      Database name (default: beer.db)
    -h, --help             Prints this help

or try:

$ ruby beerdb.rb --verbose --dbname=pivo   # or
$ ruby beerdb.rb -vnpivo

Resulting in:

{:name=>"pivo", :verbose=>true}
[]

Note: The OptionParser#parse! method modifies ARGV, that is, removes all command-line options (such as --verbose and --dbname=pivo) from ARGV, thus, the argument vector ends up empty []).

The OptionParser works great if all you need is a couple of options for your little command-line tool. Now imagine building a command-line tool like git - the stupid content tracker - that offers thousands of options. Let’s try:

$ git help

Resulting in:

usage: git [--version] [--exec-path[=GIT_EXEC_PATH]] [--html-path]
           [-p|--paginate|--no-pager] [--no-replace-objects]
           [--bare] [--git-dir=GIT_DIR] [--work-tree=GIT_WORK_TREE]
           [--help] COMMAND [ARGS]

The most commonly used git commands are:
   add        Add file contents to the index
   bisect     Find by binary search the change that introduced a bug
   branch     List, create, or delete branches
   checkout   Checkout a branch or paths to the working tree
   clone      Clone a repository into a new directory
   commit     Record changes to the repository
   diff       Show changes between commits, commit and working tree, etc
   fetch      Download objects and refs from another repository
   grep       Print lines matching a pattern
   init       Create an empty git repository or reinitialize an existing one
   log        Show commit logs
   merge      Join two or more development histories together
   mv         Move or rename a file, a directory, or a symlink
   pull       Fetch from and merge with another repository or a local branch
   push       Update remote refs along with associated objects
   rebase     Forward-port local commits to the updated upstream head
   reset      Reset current HEAD to the specified state
   rm         Remove files from the working tree and from the index
   show       Show various types of objects
   status     Show the working tree status
   tag        Create, list, delete or verify a tag object signed with GPG

Git not only offers options but also offers commands and options for commands and commands for commands and so on. For example, to see the options and commands for the remote command try:

$ git help remote

Resulting in:

NAME
       git-remote - manage set of tracked repositories

SYNOPSIS
       git remote [-v | --verbose]
       git remote add [-t <branch>] [-m <master>] [-f] [--mirror] <name> <url>
       git remote rename <old> <new>
       git remote rm <name>
       git remote set-head <name> (-a | -d | <branch>)
       git remote set-url [--push] <name> <newurl> [<oldurl>]
       git remote set-url --add [--push] <name> <newurl>
       git remote set-url --delete [--push] <name> <url>
       git remote [-v | --verbose] show [-n] <name>
       git remote prune [-n | --dry-run] <name>
       git remote [-v | --verbose] update [-p | --prune] [group | remote]...

DESCRIPTION
       Manage the set of repositories ("remotes") whose branches you track.

OPTIONS
       -v, --verbose
           Be a little more verbose and show remote url after name. NOTE: This
           must be placed between remote and subcommand.

COMMANDS
       With no arguments, shows a list of existing remotes. Several
       subcommands are available to perform operations on the remotes.

       add
           Adds a remote named <name> for the repository at <url>. The command
           git fetch <name> can then be used to create and update
           remote-tracking branches <name>/<branch>.

           With -f option, git fetch <name> is run immediately after the
           remote information is set up.

           With -t <branch> option, instead of the default glob refspec for
           the remote to track all branches under $GIT_DIR/remotes/<name>/, a
           refspec to track only <branch> is created. You can give more than
           one -t <branch> to track multiple branches without grabbing all
           branches.
           ...

If you feel adventurous you might build your own git-like command parser ontop of the built-in option parser. Example:

include 'optparse'

config = {}

parser = OptionParser.new do |opts|
  opts.banner = "Usage: beerdb [GLOBAL_OPTS] COMMAND [OPTS]"

  opts.on("-v", "--verbose", "Show debug messages") do |verbose|
    config[:verbose] = verbose
  end

  opts.on("-n", "--dbname=NAME", "Database name (default: beer.db)") do |name|
    config[:name] = name
  end
end

parser.parse!(ARGV)

command = ARGV.shift

case command do
when 'new':
  # do something
when 'build'
  # do something 
when 'serve'
  # do something 
else
  # print help
end

While a start - it’s missing options for commands or help messages or nested command and on and on.

What’s the gli (Git-Like Interfaces) gem?

Let’s thank David Bryant Copeland - the author of the gli gem - who has done all the work and has packed up (yet another) command parser built ontop of OptionParser in an easy-to-(re)use package offering it’s very own mini-language (domain-specific language) to let you define your commands (or even commands of commands of commands) in plain old Ruby.

Example:

program_desc 'beer.db command line tool'
version      '2.1.1'

### global options

desc 'Database path'
arg_name 'PATH'
default_value opts.db_path
flag [:d, :dbpath]

desc 'Database name'
arg_name 'NAME'
default_value opts.db_name
flag [:n, :dbname]

desc 'Show debug messages'
switch [:verbose], negatable: false

### commands

desc "Build DB w/ quick starter Datafile templates"
arg_name 'NAME'           # optional setup profile name
command [:new,:n] do |c|
  c.action do |g,o,args|

    # do something here

  end
end # command setup


desc "Build DB (download/create/read); use ./Datafile - zips get downloaded to ./tmp"
command [:build,:b] do |c|
  c.action do |g,o,args|

    # do something here

  end 
end # command build

...

If you run the “real-world” beerdb command-line tool built using the git-like interfaces (gli) machinery:

$ beerdb help

results in:

NAME
    beerdb - beer.db command line tool

SYNOPSIS
    beerdb [global options] command [command options] [arguments...]

GLOBAL OPTIONS
    -d, --dbpath=PATH - Database path (default: .)
    -n, --dbname=NAME - Database name (default: beer.db)
    -q, --quiet       - Only show warnings, errors and fatal messages
    --verbose         - Show debug messages
    --version         - Display the program version
    --help            - Show this message

COMMANDS
    build, b      - Build DB (download/create/read); use ./Datafile - zips get
                    downloaded to ./tmp
    create        - Create DB schema
    download, dl  - Download datasets; use ./Datafile - zips get downloaded to
                    ./tmp
    help          - Shows a list of commands or help for one command
    load, l       - Load beer fixtures
    logs          - Show logs
    new, n        - Build DB w/ quick starter Datafile templates
    props         - Show props
    read, r       - Read datasets; use ./Datafile - zips required in ./tmp
    serve, server - Start web service (HTTP JSON API)
    setup, s      - Create DB schema 'n' load all world and beer data
    stats         - Show stats
    update, up, u - Update all beer data

and

$ beerdb help serve 

results in:

NAME
    serve - Start web service (HTTP JSON API)

SYNOPSIS
    beerdb [global options] serve [command options]

COMMAND OPTIONS
    -p, --port=PORT - Port to listen on (default: 6666)
    -h, --host=HOST - Host to bind to (default: 127.0.0.1)

Got interested? David Bryant Copeland again has you covered and documented the gli gem, has written a tutorial titled “Introduction to GLI” and even an book titled - surprise, surprise - “Build Awesome Command-Line Applications in Ruby”.

Bonus: Quick Starter Code Templates with gli init

To get you started quicker the gli gem ships with its own awesome command-line tool built with gli. Try:

$ gli help

resulting in:

NAME
    gli - create scaffolding for a GLI-powered application

SYNOPSIS
    gli [global options] command [command options] [arguments...]

GLOBAL OPTIONS
    -n             - Dry run; dont change the disk
    -r, --root=arg - Root dir of project (default: .)
    -v             - Be verbose
    --version      - Display the program version
    --help         - Show this message

COMMANDS
    help           - Shows a list of commands or help for one command
    init, scaffold - Create a new GLI-based project

Now run gli init beerdb or using the alias gli scaffold beerdb and you will get ready-to-use and read-to-run code:

Creating dir ./beerdb/lib...
Creating dir ./beerdb/bin...
Creating dir ./beerdb/test...
Created ./beerdb/bin/beerdb
Created ./beerdb/README.rdoc
Created ./beerdb/beerdb.rdoc
Created ./beerdb/beerdb.gemspec
Created ./beerdb/test/default_test.rb
Created ./beerdb/test/test_helper.rb
Created ./beerdb/Rakefile
Created ./beerdb/Gemfile
Created ./beerdb/features
Created ./beerdb/lib/beerdb/version.rb
Created ./beerdb/lib/beerdb.rb

Find Out More

{}
Ruby Gem of the Week ( Feed )
Thursday, 05 February 2015
Week #6 - schemadoc gem - auto-generate your database schema docs for tables, columns, etc.
What’s schemadoc?

The schemadoc gem includes a ready-to-use command line tool named - surprise, surprise - schemadoc that lets you auto-generate your database schema documentation for tables, columns, and more.

Overview. The schemadoc tool connects to your database (e.g. SQLite, PostgreSQ

What’s schemadoc?

The schemadoc gem includes a ready-to-use command line tool named - surprise, surprise - schemadoc that lets you auto-generate your database schema documentation for tables, columns, and more.

Overview. The schemadoc tool connects to your database (e.g. SQLite, PostgreSQL, etc.) and writes out the schema info in database.json

{
  "schemas": [
    {
      "name": "football",
      "tables": [
        {
          "name": "alltime_standing_entries",
          "columns": [
            {
              "name": "id",
              "type": "integer",
              "default": null,
              "null": false
            },
            {
              "name": "alltime_standing_id",
              "type": "integer",
              "default": null,
              "null": false
            },
            {
              "name": "team_id",
              "type": "integer",
              "default": null,
              "null": false
            },
...

and also builds an A-Z symbols index stored in symbols.json.

{
    "name": "A",
    "tables": [
      "alltime_standing_entries",
      "alltime_standings",
      "assocs",
      "assocs_assocs"
    ],
    "columns": [
      {
        "name": "abbr",
        "tables": [
          "regions"
        ]
      },
      {
        "name": "address",
        "tables": [
          "grounds",
          "teams"
        ]
      },
...

Drop the JSON documents in the _data/ folder for your static site theme (template pack) and let Jekyll (or GitHub Pages) do the rest.

Examples in the real world. See the football.db or beer.db for live examples.

Getting Started w/ schemadoc

Let’s document the football.db SQLite version in three steps:

  • Step 1: Let’s create the football.db
  • Step 2: Let’s write out the schema info in JSON
  • Step 3: Let’s generate a static schema documentation site

Step 1: Let’s create the football.db

First let’s create the football.db itself. Pull in the sportdb-models gem and use the built-in “auto-migrate” method SportDb.create_all that will create all database tables. Example:

mkfootball.rb:

require 'logger'
require 'sportdb/models'      # use $ gem install sportdb-models

DB_CONFIG = {
  adapter: 'sqlite3',
  database: './football.db'
}

ActiveRecord::Base.logger = Logger.new( STDOUT )
ActiveRecord::Base.establish_connection( DB_CONFIG )

SportDb.create_all

puts 'Done.'

Run the script:

$ ruby ./makfootball.rb

Now you’ve got an empty football.db with many many tables. Let’s document the database schema(ta).

Step 2: Let’s write out the schema info in JSON

The schemadoc command line tool requires a configuration file, that is, /schemadoc.yml that lists the connection settings and the schemas (such as football, world, and the works.) Example:

schemadoc.yml:

## connection spec

database:
  adapter:  sqlite3
  database: ./football.db


## main tables

football:
  name: Football

## world tables

world:
  name: World
  tables:
    - continents
    - countries
    - regions
    - cities
    - places
    - names
    - langs
    - usages
    
## works tables

works:
  name: The Works
  tables:
     - logs
     - props
     - tags
     - taggings

Now run the schemadoc tool:

$ ./schemadoc

and you will end-up with two JSON files, that is, database.json and symbols.json.

Step 3: Let’s generate a static schema documentation site

Get a copy of the book-templates/schema static site theme and drop (copy) the two JSON files, that is, database.json and symbols.json into the _data/ folder. Change the site settings in _config.yml and run:

$ ./jekyll build

That’s it. Open up in your browser the ./_site/index.html page. Enjoy your databasse schema documentation.

Find Out More

{}
Ruby Gem of the Week ( Feed )
Thursday, 29 January 2015
Week #5 - feedparser gem - web feed parser and normalizers (for RSS 2.0, Atom, n friends)
What’s a Web Feed?

A web feed (or news feed) is a simple document/data format that 1) lets you publish a list of status updates, blog postings, articles, pictures, cartoons, recordings, etc. and that 2) lets others subscribe to your updates.

Example:

<feed>
  <ti

What’s a Web Feed?

A web feed (or news feed) is a simple document/data format that 1) lets you publish a list of status updates, blog postings, articles, pictures, cartoons, recordings, etc. and that 2) lets others subscribe to your updates.

Example:

<feed>
  <title>beer.db - Open Beer, Brewery n Brewpubs Data News and Updates</title>
  <link href="http://openbeer.github.io/" />
  <updated>2015-01-04T09:25:14+00:00</updated>
  <entry>
    <title>Beer-A-Day - Open Data Eternal Page-A-Day Calender</title>
    <link href="http://openbeer.github.io/2014/12/12/beer-a-day.html" />
    <updated>2014-12-12T00:00:00+00:00</updated>
    <summary>As an example of using the beer.db...</summary>
  </entry>
  <entry>
    <title>New beer.db Build System - Welcome ./Datafile e.g. $beerdb new at</title>
    <link href="http://openbeer.github.io/2014/12/01/new-build-system-w-datafile.html" />
    <updated>2014-12-01T00:00:00+00:00</updated>
    <summary>The beerdb command line tool now includes a new build system...</summary>
  ...
</feed>

Ruby ships with a standard library that lets you read web feeds in the “classic” Really Simple Syndication (RSS 1.0/2.0) flavors or in the “modern” Atom Publishing format.

Task I: Reading Web Feeds in the Really Simple Syndication (RSS) Format

Let’s read in a “classic” web feed in the Really Simple Syndication (RSS) format from the RubyFlow site and print out the latest headlines:

require 'rss'
require 'open-uri'

feed  = RSS::Parser.parse( 'http://www.rubyflow.com/rss' )

puts "==  #{feed.channel.title} =="

feed.items.each do |item|
  puts "- #{item.title}"
  puts "  (#{item.link})"
  puts
end

Prints:

==  RubyFlow ==

- The All New RubyFlow for 2015
  (http://www.rubyflow.com/p/8hqxpd-the-all-new-rubyflow-for-2015)

- Jekyll Quick Reference (Cheat Sheet) for Static Site Generator
  (http://www.rubyflow.com/p/k8l2im-jekyll-quick-reference-cheat-sheet-for-static-site-generator)

- Timeago date filter for Liquid templates
  (http://www.rubyflow.com/p/12092-timeago-date-filter-for-liquid-templates)

To include a post’s summary use item.description or to include a post’s full-text use content_encoded. Example:

puts item.description
puts item.content_encoded

That’s it.

Task II: Reading Web Feeds in the Atom Publishing Format

Next let’s read a “modern” web feed in the Atom Publishing format from the beer.db - Open Beer Data site and print out the latest headlines:

feed = RSS::Parser.parse( 'http://openbeer.github.io/atom.xml' )

puts "== #{feed.title.content} =="

feed.entries.each do |entry|
  puts "- #{entry.title.content}"
  puts "  (#{entry.link.href})"
  puts
end

And use entry.content.content to get a post’s full-text. That’s it.

Note, that the standard library maps the different web feed flavors 1:1 to Ruby fields reflecting the different formats e.g. the feed title stored in feed.channel.title in RSS 2.0 becomes feed.title.content in Atom and the feed item’s full-text stored in feed.item.content_encoded in RSS 2.0 becomes feed.entry.content.content in Atom (Yes, it’s content.content). Welcome to the wonderful world of web feed formats.

What’s the feedparser gem?

The feedparser gem lets you read in web feeds in the RSS 2.0 or Atom Publishing formats (using the built-in libraries) and normalizes the feed and item fields e.g. item.content gets mapped to item.content_encoded in RSS 2.0 and to entry.content.content in Atom and so on.

Feed Struct

Mappings

Title ‘n’ Summary

Feed Struct RSS 2.0 Notes Atom Notes
feed.title title plain vanilla text title plain vanilla text
feed.summary description plain vanilla text subtitle? plain vanilla text

Note: The Feed parser will remove all HTML tags and attributes from the title (RSS 2.0+Atom), description (RSS 2.0) and subtitle (Atom) content and will unescape HTML entities e.g. &amp; becomes & and so on - always resulting in plain vanilla text.

Dates

Feed Struct RSS 2.0 Notes Atom Notes
feed.updated lastBuildDate? RFC-822 format updated ISO 801 format
feed.published pubDate? RFC-822 format -  

RFC-822 date format e.g. Wed, 14 Jan 2015 19:48:57 +0100

ISO-801 date format e.g. 2015-01-14T19:48:57Z

Note: Mappings use question mark (?) for optional elements (otherwise assume required elements).

class Feed
  attr_accessor :format   # e.g. atom|rss 2.0|etc.
  attr_accessor :title    # note: always plain vanilla text
  attr_accessor :url

  attr_accessor :items

  attr_accessor :summary   # note: is description in RSS 2.0 and subtitle in Atom; always plain vanilla text

  attr_accessor :updated     # note: is lastBuildDate in RSS 2.0
  attr_accessor :published   # note: is pubDate in RSS 2.0; not available in Atom

  attr_accessor :generator
  attr_accessor :generator_version  # e.g. @version (atom)
  attr_accessor :generator_uri      # e.g. @uri     (atom)
end

Item Struct

Title ‘n’ Summary

Feed Struct RSS 2.0 Notes Atom Notes
item.title title plain vanilla text title plain vanilla text
item.summary description plain vanilla text summary? plain vanilla text
item.content content? html content? html

Note: The Feed parser will remove all HTML tags and attributes from the title (RSS 2.0+Atom), description (RSS 2.0) and summary (Atom) content and will unescape HTML entities e.g. &amp; becomes & and so on - always resulting in plain vanilla text.

Note: In plain vanilla RSS 2.0 there’s no difference between (full) content and summary - everything is wrapped in a description element; however, best practice is using the content “module” from RSS 1.0 inside RSS 2.0. If there’s no content module present the feed parser will “clone” the description and use one version for item.summary and the clone for item.content.

Dates

Item Struct RSS 2.0 Notes Atom Notes
item.updated pubDate? RFC-822 format updated ISO 801 format
item.published - RFC-822 format published? ISO 801 format

Note: In plain vanilla RSS 2.0 there’s only one pubDate for items, thus, it’s not possible to differentiate between published and updated dates for items; note - the item.pubDate will get mapped to item.updated. To set the published date in RSS 2.0 use the dublin core module e.g dc:created, for example.

class Item
  attr_accessor :title   # note: always plain vanilla text
  attr_accessor :url

  attr_accessor :content
  attr_accessor :content_type  # optional for now (text|html|etc.)

  attr_accessor :summary

  attr_accessor :updated    # note: is pubDate in RSS 2.0 and updated in Atom
  attr_accessor :published  # note: is published in Atom; not available in RSS 2.0 (use dc:created)

  attr_accessor :guid
end

Reading Web Feeds in All Formats

require 'open-uri'
require 'feedparser'

url = 'http://openbeer.github.io/atom.xml'
xml = open( url ).read

feed = FeedParser::Parser.parse( xml )
puts "== #{feed.title} =="

feed.items.each do |item|
  puts "- #{item.content}"
  puts "  (#{item.url})"
  puts
end

Prints:

== beer.db - Open Beer, Brewery n Brewpubs Data News and Updates ==

- Beer-A-Day - Open Data Eternal Page-A-Day Calender
  (http://openbeer.github.io/2014/12/12/beer-a-day.html)

- New beer.db Build System - Welcome ./Datafile e.g. $beerdb new be
  (http://openbeer.github.io/2014/12/01/new-build-system-w-datafile.html)
 
- New Repo /maps - Free Full-Screen Interactive Beer Maps w/ Brewery Listings
  (http://openbeer.github.io/2014/11/11/new-repo-maps.html)

Now change the feed url to:

url = 'http://www.rubyflow.com/rss'

and everything will work without changes for both formats (that is, RSS and Atom). That’s it.

Bonus: Planet Feed Reader in 20 Lines of Ruby

planet.rb:

require 'open-uri'
require 'feedparser'
require 'erb'
  
# step 1) read a list of web feeds

FEED_URLS = [
  'http://vienna-rb.at/atom.xml',
  'http://rubyflow.com/rss',
  'http://openfootball.github.io/atom.xml',
  'http://openbeer.github.io/atom.xml'
]

items = []

FEED_URLS.each do |url|
  feed = FeedParser::Parser.parse( open( url ).read )
  items += feed.items
end

# step 2) mix up all postings in a new page

FEED_ITEM_TEMPLATE = <<EOS
<% items.each do |item| %>
  <div class="item">
    <h2><a href="<%= item.url %>"><%= item.title %></a></h2>
    <div><%= item.content %></div>
  </div>
<% end %>
EOS

puts ERB.new( FEED_ITEM_TEMPLATE ).result

Run the script:

$ ruby planet.rb      

Prints:

<div class="item">
  <h2><a href="http://vienna-rb.at/blog/2015/01/28/picks/">Picks / what the vienna.rb team thinks is worth sharing this week</a></h2>
  <div>
   <h3>01/28 Picks!</h3>
   <p>In a series on this website we'll entertain YOU with our picks...
 ...

Find Out More

{}
Ruby Gem of the Week ( Feed )
Thursday, 22 January 2015
Week #4 - kramdown gem - turn easy-to-read and easy-to-write wiki-style plain text in markdown into hypertext
What’s Markdown?

Markdown is an easy-to-write and easy-to-read wiki-style markup language that lets you author web pages in plain text. Example:

## What's `football.db`?

A free and open public domain football database & schema
for use in any (programming) language (e.g. uses 

What’s Markdown?

Markdown is an easy-to-write and easy-to-read wiki-style markup language that lets you author web pages in plain text. Example:

## What's `football.db`?

A free and open public domain football database & schema
for use in any (programming) language (e.g. uses plain datasets). Example:

    ### Teams
    
    barcelona, Barcelona|FC Barcelona|Fútbol Club Barcelona, BAR
    madrid,    Real Madrid|Real Madrid CF,                   RMD
    ...

becomes

<h2>What's <code>football.db</code>?</h2>

<p>A free and open public domain football database &amp; schema
for use in any (programming) language (e.g. uses plain datasets). Example:</p>

<pre><code>
### Teams

barcelona, Barcelona|FC Barcelona|Fútbol Club Barcelona, BAR
madrid,    Real Madrid|Real Madrid CF,                   RMD
...
</code></pre>

What’s kramdown?

kramdown is a gem that that lets you convert Markdown (.md, .mkdwn, .markdown) to Hypertext (.html). Thanks to Thomas Leitner from Vienna, Austria for polishing the gem - more than 30+ releases - leading to today’s version 1.5.0.

Usage

Converting your wiki-style plain text markup to hypertext using kramdown is as easy as:

require 'kramdown'

Kramdown::Document.new( '¡Barça, Barça, Baaarça!' ).to_html

# => "<p>¡Barça, Barça, Baaarça!</p>\n"

Let’s create another static site generator in 5 minutes

First lets create a new ruby script and let’s add the kramdown gem plus the find standard Ruby library module to help us with finding files. Example:

./sitegen.rb:

require 'kramdown'
require 'find'

Next let’s make a _site output folder.

SITE_PATH = './_site'

Dir.mkdir( SITE_PATH ) unless File.exist?( SITE_PATH )

Now let’s add a markdown converter helper method.

def markdown( text )
  Kramdown::Document.new( text ).to_html
end

Finally, to wrap up let’s loop over all files in the current working folder, that is, . and generate hypertext (.html) documents in the ./site folder from the markdown (.md) source documents.

Find.find('.') do |path|
  if File.extname(path) == '.md'              # e.g. ./index.md => .md
    basename = File.basename(path, '.md')     # e.g. ./index.md => index

    File.open( "#{SITE_PATH}/#{basename}.html", 'w') do |file|
      file.write markdown( File.read( path ) )
    end
  end
end

That’s it. Create a new markdown file. Example:

./index.md:

## What's `football.db`?

A free and open public domain football database & schema
for use in any (programming) language (e.g. uses plain datasets). Example:

    ### Teams
    
    barcelona, Barcelona|FC Barcelona|Fútbol Club Barcelona, BAR
    madrid,    Real Madrid|Real Madrid CF,                   RMD
    ...

Run the static site generation machinery e.g. type:

$ ruby ./sitegen.rb

That’s it. Open up the web page ./_site/index.html in your browser.

Bonus: kramdown Command Line Tool

Note: The kramdown gem ships with a command line tool named - suprise, suprise - kramdown. Type in your shell:

$ kramdown -h

resulting in:

Command line options:

    -i, --input ARG
          Specify the input format: kramdown (default), html, GFM or markdown
    -o, --output ARG
          Specify one or more output formats separated by commas: html (default), kramdown, latex, pdf or remove_html_tags
    -v, --version
          Show the version of kramdown
    -h, --help
          Show the help

kramdown options:

        --template ARG
          The name of an ERB template file that should be used to wrap the output
          or the ERB template itself.
          
          This is used to wrap the output in an environment so that the output can
          be used as a stand-alone document. For example, an HTML template would
          provide the needed header and body tags so that the whole output is a
          valid HTML file. If no template is specified, the output will be just
          the converted text.
          
          When resolving the template file, the given template name is used first.
          If such a file is not found, the converter extension (the same as the
          converter name) is appended. If the file still cannot be found, the
          templates name is interpreted as a template name that is provided by
          kramdown (without the converter extension). If the file is still not
          found, the template name is checked if it starts with 'string://' and if
          it does, this prefix is removed and the rest is used as template
          content.
          
          kramdown provides a default template named 'document' for each converter.
          
          Default: ''
          Used by: all converters

        --[no-]auto-ids
          Use automatic header ID generation
          
          If this option is `true`, ID values for all headers are automatically
          generated if no ID is explicitly specified.
          
          Default: true
          Used by: HTML/Latex converter

        --[no-]auto-id-stripping
          Strip all formatting from header text for automatic ID generation
          
          If this option is `true`, only the text elements of a header are used
          for generating the ID later (in contrast to just using the raw header
          text line).
          
          This option will be removed in version 2.0 because this will be the
          default then.
          
          Default: false
          Used by: kramdown parser

        --auto-id-prefix ARG
          Prefix used for automatically generated header IDs
          
          This option can be used to set a prefix for the automatically generated
          header IDs so that there is no conflict when rendering multiple kramdown
          documents into one output file separately. The prefix should only
          contain characters that are valid in an ID!
          
          Default: ''
          Used by: HTML/Latex converter

        --[no-]transliterated-header-ids
          Transliterate the header text before generating the ID
          
          Only ASCII characters are used in headers IDs. This is not good for
          languages with many non-ASCII characters. By enabling this option
          the header text is transliterated to ASCII as good as possible so that
          the resulting header ID is more useful.
          
          The stringex library needs to be installed for this feature to work!
          
          Default: false
          Used by: HTML/Latex converter

        --[no-]parse-block-html
          Process kramdown syntax in block HTML tags
          
          If this option is `true`, the kramdown parser processes the content of
          block HTML tags as text containing block-level elements. Since this is
          not wanted normally, the default is `false`. It is normally better to
          selectively enable kramdown processing via the markdown attribute.
          
          Default: false
          Used by: kramdown parser

        --[no-]parse-span-html
          Process kramdown syntax in span HTML tags
          
          If this option is `true`, the kramdown parser processes the content of
          span HTML tags as text containing span-level elements.
          
          Default: true
          Used by: kramdown parser

        --[no-]html-to-native
          Convert HTML elements to native elements
          
          If this option is `true`, the parser converts HTML elements to native
          elements. For example, when parsing `<em>hallo</em>` the emphasis tag
          would normally be converted to an `:html` element with tag type `:em`.
          If `html_to_native` is `true`, then the emphasis would be converted to a
          native `:em` element.
          
          This is useful for converters that cannot deal with HTML elements.
          
          Default: false
          Used by: kramdown parser

        --link-defs ARG
          Pre-defines link definitions
          
          This option can be used to pre-define link definitions. The value needs
          to be a Hash where the keys are the link identifiers and the values are
          two element Arrays with the link URL and the link title.
          
          If the value is a String, it has to contain a valid YAML hash and the
          hash has to follow the above guidelines.
          
          Default: {}
          Used by: kramdown parser

        --footnote-nr ARG
          The number of the first footnote
          
          This option can be used to specify the number that is used for the first
          footnote.
          
          Default: 1
          Used by: HTML converter

        --entity-output ARG
          Defines how entities are output
          
          The possible values are :as_input (entities are output in the same
          form as found in the input), :numeric (entities are output in numeric
          form), :symbolic (entities are output in symbolic form if possible) or
          :as_char (entities are output as characters if possible, only available
          on Ruby 1.9).
          
          Default: :as_char
          Used by: HTML converter, kramdown converter

        --toc-levels ARG
          Defines the levels that are used for the table of contents
          
          The individual levels can be specified by separating them with commas
          (e.g. 1,2,3) or by using the range syntax (e.g. 1..3). Only the
          specified levels are used for the table of contents.
          
          Default: 1..6
          Used by: HTML/Latex converter

        --line-width ARG
          Defines the line width to be used when outputting a document
          
          Default: 72
          Used by: kramdown converter

        --latex-headers ARG
          Defines the LaTeX commands for different header levels
          
          The commands for the header levels one to six can be specified by
          separating them with commas.
          
          Default: section,subsection,subsubsection,paragraph,subparagraph,subparagraph
          Used by: Latex converter

        --smart-quotes ARG
          Defines the HTML entity names or code points for smart quote output
          
          The entities identified by entity name or code point that should be
          used for, in order, a left single quote, a right single quote, a left
          double and a right double quote are specified by separating them with
          commas.
          
          Default: lsquo,rsquo,ldquo,rdquo
          Used by: HTML/Latex converter

        --[no-]remove-block-html-tags
          Remove block HTML tags
          
          If this option is `true`, the RemoveHtmlTags converter removes
          block HTML tags.
          
          Default: true
          Used by: RemoveHtmlTags converter

        --[no-]remove-span-html-tags
          Remove span HTML tags
          
          If this option is `true`, the RemoveHtmlTags converter removes
          span HTML tags.
          
          Default: false
          Used by: RemoveHtmlTags converter

        --header-offset ARG
          Sets the output offset for headers
          
          If this option is c (may also be negative) then a header with level n
          will be output as a header with level c+n. If c+n is lower than 1,
          level 1 will be used. If c+n is greater than 6, level 6 will be used.
          
          Default: 0
          Used by: HTML converter, Kramdown converter, Latex converter

        --[no-]hard-wrap
          Interprets line breaks literally
          
          Insert HTML `<br />` tags inside paragraphs where the original Markdown
          document had newlines (by default, Markdown ignores these newlines).
          
          Default: true
          Used by: GFM parser

        --syntax-highlighter ARG
          Set the syntax highlighter
          
          Specifies the syntax highlighter that should be used for highlighting
          code blocks and spans. If this option is set to +nil+, no syntax
          highlighting is done.
          
          Options for the syntax highlighter can be set with the
          syntax_highlighter_opts configuration option.
          
          Default: coderay
          Used by: HTML converter

        --syntax-highlighter-opts ARG
          Set the syntax highlighter options
          
          Specifies options for the syntax highlighter set via the
          syntax_highlighter configuration option.
          
          The value needs to be a hash with key-value pairs that are understood by
          the used syntax highlighter.
          
          Default: {}
          Used by: HTML converter

        --math-engine ARG
          Set the math engine
          
          Specifies the math engine that should be used for converting math
          blocks/spans. If this option is set to +nil+, no math engine is used and
          the math blocks/spans are output as is.
          
          Options for the selected math engine can be set with the
          math_engine_opts configuration option.
          
          Default: mathjax
          Used by: HTML converter

        --math-engine-opts ARG
          Set the math engine options
          
          Specifies options for the math engine set via the math_engine
          configuration option.
          
          The value needs to be a hash with key-value pairs that are understood by
          the used math engine.
          
          Default: {}
          Used by: HTML converter

Yes, it’s well documented. Welcome to the wonders of kramdown.

Find Out More

{}
Ruby Gem of the Week ( Feed )
Thursday, 15 January 2015
Week #3 - slideshow gem - a free web alternative to PowerPoint and Keynote in Ruby
What’s Slide Show (S9)?

A Ruby gem that lets you create slide shows and author slides in plain text using a wiki-style markup language that’s easy-to-write and easy-to-read. The Slide Show (S9) project also collects and welcomes themes and ships “out-of-the-gem” with built-in support for “loss-fr

What’s Slide Show (S9)?

A Ruby gem that lets you create slide shows and author slides in plain text using a wiki-style markup language that’s easy-to-write and easy-to-read. The Slide Show (S9) project also collects and welcomes themes and ships “out-of-the-gem” with built-in support for “loss-free” gradient vector graphics themes.

Getting Started in 1-2-3 Easy Steps

  • Step 1: Author your slides in plain text using a wiki-style markup language
  • Step 2: Build your slide show using the slideshow gem
  • Step 3: Open up your slide show (web page) in your browser and hit the space bar to flip through your slides
  • That’s it. Showtime!

Step 1: Author your slides in plain text using a wiki-style markup language

Slide Show lets you author your slides using a wiki-style markup language that’s easy-to-write and easy-to-read. Let’s create some slides about best practices for web services using REST:

# Web Services REST-Style: Universal Identifiers, Formats & Protocols

Agenda

- What's REST?
- Universal Identifiers, Formats & Protocols
- The Holy REST Trinity - Noun, Verbs, Types
- REST Design Principles 
- Architecture Astronaut REST Speak


# Representational State Transfer (REST) - Meaningless Acronym? Wordplay?

rest - n. - peace, ease, or refreshment resulting from the insight that the web works

No matter what vendors tell you - no need to "Light Up the Web" - relax - built on
an *open architecture using universal identifiers, formats & protocols and __evolving__
open standards* - no need to reinvent the wheel and sign-up for single-vendor offerings.

### Broad Definition

- Best Practices for Designing Web Services for a Unified Human and Programable Web

### Narrow Definition

- Alternative to BigCo Web Services (SOAP, WS-*) and RPC-Style Web Services (XML-RPC)

Use # headings to start a new slide in Markdown. That’s it.

Step 2: Build your slide show using the slideshow gem

Run slideshow to build your slide show. The slideshow gem expects the name of your slide show source document (e.g. rest) and will build a web page (e.g. rest.html that is an all-in-one-page handout and a live slide show all at once.

$ slideshow build rest

=> Preparing slideshow 'rest.html'...
=> Done.

Step 3: Open up your slide show in your browser

Open up your slide show rest.html in your browser (Firefox, Chrome, Opera, Safari, and others) and hit F11 to switch into full screen projection and hit the space bar or the right arrow, down arrow or page down key to flip through your slides.

That’s it. Voila.

Bonus: Try some different templates/theme packs

Find Out More

{}
Ruby Gem of the Week ( Feed )
Thursday, 08 January 2015
Week #2 - hoe gem - build, package and publish gems with hoe rake tasks
What’s Hoe?

Hoe is a gem that bundles ready-to-use rake tasks to help you build, package and publish your own gems. Thanks to Ryan Davis and friends (from Seattle.rb) for polishing the gem all those years - more than 100+ releases - leading to today’s version 3.13.0.

Let’s create a bare

What’s Hoe?

Hoe is a gem that bundles ready-to-use rake tasks to help you build, package and publish your own gems. Thanks to Ryan Davis and friends (from Seattle.rb) for polishing the gem all those years - more than 100+ releases - leading to today’s version 3.13.0.

Let’s create a bare bones gem (hellohoe) and publish it on RubyGems.org.

Set up your gem by adding a build script, readme, change log and manifest

To use Hoe together with your own code use the following structure:

/hellohoe
  + README.txt        - Description in plain text 
  + History.txt       - Version change log in plain text
  + Manifest.txt      - List of all files to include in plain text
  + Rakefile          - Build script (requires your name and email)  
  + /lib
     + hellohoe.rb    - Ruby code to bundle up into gem here

Note: You can grab all files from this post from the hellohoe GitHub repo.

Let’s look at hellohoe.rb:

class HelloHoe
  VERSION = '0.1.0'
   
  # your code here
end

Hoe requires a VERSION string in your Ruby code that you can reference in your build script. Let’s look at the build script, that is, Rakefile next:

require 'hoe'                # pull in the hoe machinery (that is, ready-to-use rake tasks)
require './lib/hellohoe.rb'

Hoe.spec 'hellohoe' do
  self.version = HelloHoe::VERSION

  self.author  = '[Your Name Here]'
  self.email   = '[Your Email Here]'
  
  # or use shortcut
  # self.developer( '[Your Name Here]', '[Your Email Here]' )
end

As a minimum Hoe requires you to set the author and email fields in the gemspec. As a shortcut you can use the developer method to set it all at once.

Next Hoe requires a readme in plain text stored in README.txt:

= hellohoe

* https://github.com/geraldb/hellohoe

== DESCRIPTION:

Sample on how to use Hoe Rake tasks to build, package and publish gems.

== LICENSE:

The hellohoe sources are dedicated to the public domain. 

Hoe will use the link from the first section, that is, github.com/geraldb/hellohoe to auto-fill the homepage field in the gemspec and will use the description to auto-fill the summary field and the description in the gemspec.

Next Hoe requires a version change log in plain text stored in History.txt:

=== 0.1.0 / 2015-01-08

* Everything is new. First release.

Hoe will you use the change log to auto-fill the changes field in the gemspec and use the change log for emails and announcements.

Finally, Hoe requires a manifest - a list of all files to include in plain text stored in Manifest.txt:

History.txt
Manifest.txt
README.txt
Rakefile
lib/hellohoe.rb

Now you’re all set to use Hoe’s rake tasks to build, package and publish gems and more. You can list all tasks issuing rake -T. Resulting in:

rake announce              # publish   # Announce your release.
rake audit                 # test      # Run ZenTest against the package.
rake check_extra_deps      # deps      # Install missing dependencies.
rake check_manifest        # debug     # Verify the manifest.
rake clean                 # clean     # Clean up all the extras.
rake clobber_docs          # publish   # Remove RDoc files
rake clobber_package       # package   # Remove package products
rake config_hoe            # debug     # Create a fresh ~/.hoerc file.
rake dcov                  # publish   # Generate rdoc coverage report
rake debug_email           # publish   # Generate email announcement file.
rake debug_gem             # debug     # Show information about the gem.
rake default               # test      # Run the default task(s).
rake deps:email            # deps      # Print a contact list for gems dependent on this gem
rake deps:fetch            # deps      # Fetch all the dependent gems of this gem into tarballs
rake deps:list             # deps      # List all the dependent gems of this gem
rake docs                  # publish   # Generate rdoc
rake gem                   # package   # Build the gem file hellohoe-0.1.gem
rake generate_key          # signing   # Generate a key for signing your gems.
rake install_gem           # package   # Install the package as a gem.
rake install_plugins       # deps      # Install missing plugins.
rake newb                  # newb      # Install deps, generate docs, run tests/specs.
rake package               # package   # Build all the packages
rake post_blog             # publish   # Post announcement to blog.
rake publish_docs          # publish   # Publish RDoc to wherever you want.
rake release               # package   # Package and upload; Requires VERSION=x.y.z (optional PRE=a.1)
rake release_sanity        # package   # Sanity checks for release
rake release_to_gemcutter  # gemcutter # Push gem to gemcutter.
rake repackage             # package   # Force a rebuild of the package files
rake ridocs                # publish   # Generate ri locally for testing.

Using debug_gem, gem, package, install_gem tasks

Let’s try some Hoe tasks. Run rake debug_gem to show the gemspec Hoe generates from your build script settings, readme, change log and manifest. Next, let’s build the gem. Run rake gem. Resulting in:

mkdir -p pkg
  Successfully built RubyGem
  Name: hellohoe
  Version: 0.1.0
  File: hellohoe-0.1.0.gem
mv hellohoe-0.1.0.gem pkg/hellohoe-0.1.0.gem

Hoe will place your gem in the pkg folder. If you run rake package Hoe will bundle up all your sources in a tar’ed and gzipped package (e.g. pkg/hellohoe-0.1.0.tgz).

Next, let’s test drive the gem. Run rake install_gem to install the gem and try it in the Ruby console:

$ irb
>> require 'hellohoe'
=> true
>> HelloHoe::VERSION
=> "0.1.0"

Checking and updating your manifest with check_manifest

Hoe includes a check_manifest task that lets you check the manifest against your files and see if any files are missing or need to get added.

If you run the task the first time you need to create a ~/.hoerc setting file first that includes a regex (regular expression) pattern that excludes files from the manifest check. To create a new ~/.hoerc file run rake config_hoe. Resulting in a file such as:

---
exclude: !ruby/regexp /tmp$|CVS|TAGS|\.(svn|git|DS_Store)/
signing_key_file: ~/.gem/gem-private_key.pem
signing_cert_file: ~/.gem/gem-public_cert.pem
publish_on_announce: true
blogs:
- user: user
  password: password
  url: url
  blog_id: blog_id
  extra_headers:
    mt_convert_breaks: markdown

Now let’s try rake check_manifest. If everything is in order (no files missing or waiting to get added). You will see:

rm -r doc
rm -r pkg
rm Manifest.tmp

Let’s create a new Todo.txt file and let’s retry rake check_manifest. Now you will see a diff:

@@ -2,4 +2,5 @@
 Manifest.txt
 README.txt
 Rakefile
+Todo.txt
 lib/hellohoe.rb

Using the release task to upload (push) your gem to RubyGems.org

Next, let’s upload (push) the gem to RubyGems.org using the release task.

Before you can upload to RubyGems.org you will need to setup an account and save your RubyGems.org API key on your computer. Issue the command to store your RubyGems.org API key on your computer (only needed the first time):

$ curl -u carlos https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentials

Now you’re ready to use hoe to upload (push) your gem to RubyGems.org without requiring to enter a user and password. Run the command rake release and pass along the required release version. Example:

$ rake release VERSION=0.1.0

Check your RubyGems.org project page (e.g. rubygems.org/gems/hellohoe) if it all worked. Note, that it will take a minute or more until your uploaded gem gets added to the public RubyGems index. You can check if your gem is available using the list command with the -r (remote) switch. Example:

$ gem list hellohoe -r

*** REMOTE GEMS ***

hellohoe (0.1.0)

That’s it.

Bonus Tip by Ryan Davis: Quick Starter Templates with sow

Ryan Davis writes: The easiest way to get started with hoe is to use its included command-line tool sow:

$ sow hellohoe

That will create a new directory hellohoe with a skeletal project inside. You need to edit the Rakefile with developer information in order to meet the minimum requirements of a working gemspec. You should also go fix all the things it points out as being labeled with FIX in the README.txt file.

(Source: Hoe PDF Booklet; 6 Pages)

Find Out More

Hoe References

Gems References

{}
Ruby Gem of the Week ( Feed )
Thursday, 01 January 2015
Week #1 - factbook gem - turn the world factbook into open structured data e.g JSON
What’s the World Factbook?

The World Factbook [1][2] published by the Central Intelligence Agency (CIA) offers free country profiles in the public domain (that is, no copyright(s), no rights reserved).

  • [1] The World Factbook
  • [2] Wikipedia Article: The World Factbook

    What’s the World Factbook?

    The World Factbook [1][2] published by the Central Intelligence Agency (CIA) offers free country profiles in the public domain (that is, no copyright(s), no rights reserved).

    What’s the factbook gem?

    The factbook gem ships with scripts for the world factbook that let you turn web pages into open structured data e.g JSON and friends. Example:

    To get the country profile page for Austria as a hash (that is, structured data e.g. nested key/values) use:

    page = Factbook::Page.new( 'at' )   # at is the country code for Austria
    pp page.data                        # pretty print hash
    

    To save the hash to disk as JSON use:

    page = Factbook::Page.new( 'at' )
    File.open( 'at.json', 'w') do |f|
      f.write page.to_json( pretty: true )
    end
    

    Resulting in:

    {
      "intro": {
        "background": {
          "text": "Once the center of power for the large Austro-Hungarian Empire,
                   Austria was reduced to a small republic after its defeat in World War ..."
        }
      },
      "geo": {
        "location": {
          "text": "Central Europe, north of Italy and Slovenia"
        },
        "geographic_coordinates": {
          "text": "47 20 N, 13 20 E"
        },
        "map_references": {
          "text": "Europe"
        },
        "area": {
          "total": "83,871 sq km",
          "land": "82,445 sq km",
          "water": "1,426 sq km"
        },
        "area_comparative": {
          "text": "slightly smaller than Maine"
        },
        "land_boundaries": {
          "total": "2,562 km",
          "border_countries": "Czech Republic 362 km, Germany 784 km, Hungary 366 km, Italy 430 km, Liechtenstein 35 km, Slovakia 91 km, Slovenia 330 km, Switzerland 164 km"
        },
        "coastline": {
          "text": "0 km (landlocked)"
        },
        "maritime_claims": {
          "text": "none (landlocked)"
        },
        "climate": {
          "text": "temperate; continental, cloudy; cold winters with frequent rain and some snow in lowlands and snow in mountains; moderate summers with occasional showers"
        },
        "terrain": {
          "text": "in the west and south mostly mountains (Alps); along the eastern and northern margins mostly flat or gently sloping"
        },
        "elevation_extremes": {
          "lowest_point": "Neusiedler See 115 m",
          "highest_point": "Grossglockner 3,798 m"
        },
        "natural_resources": {
          "text": "oil, coal, lignite, timber, iron ore, copper, zinc, antimony, magnesite, tungsten, graphite, salt, hydropower"
        },
        ...
    

    Note: The World Factbook includes 267 entries - 195 sovereign countries, 2 others, 58 dependencies, 6 miscellaneous, 5 oceans, and 1 world profile.

    That’s it.

    Bonus: Ready-To-Use Public Domain Factbook Datasets

    openmundi/factbook.json - open (public domain) factbook country profiles in JSON for all the world’s countries (using internet domain names for country codes e.g. Austria is at.json not au.json, Germany is de.json not gm.json and so on)

    Find Out More


pluto.models/1.4.0, feed.parser/1.0.0, feed.filter/1.1.1 - Ruby/2.0.0 (2014-11-13/x86_64-linux) on Rails/4.2.0 (production)