Gregory Trubetskoy

Blockchain Proof-of-Work is a Decentralized Clock

Tue, 23 Jan 2018 11:41:00 +0000

This is an explanation of the key function on Proof-of-Work in the Bitcoin blockchain. It focuses on the one feature of Proof-of-Work that is essential and shows that other features often talked about such as security are secondary side-effects, useful, but not essential.

This explanation rests on illustrating a few interesting properties of how Proof-of-Work is used in the blockchain that are not immediately obvious and sometimes are rather counter-intuitive, for example how participants collectively solve a problem without ever communicating.

The Bitcoin Blockchain PostgresSQL Schema

Fri, 15 Dec 2017 08:28:00 +0000

In a previous post I wrote some initial thoughts on storing the blockchain in Postgres. It’s been a couple of months and I’ve made some progress on the import project. This post documents the latest incarnation of the SQL schema used to store the blockchain as well as thoughts on why it was decided to be this way.

Blockchain Data Structure Overview

The Bitcoin blockchain consists of blocks. A block is a set of transactions. A block also contains some block-specific information, such as the nonce for the Proof-Of-Work validating the block.

Blockchain in PostgreSQL Part 2

Fri, 20 Oct 2017 08:05:00 +0000

Update: there is now a better write up of the PostgreSQL schema. This post was rather half-baked as much was still not understood when I wrote it.

In a previous post I described a simplistic schema to store the Bitcoin blockchain in PostgreSQL. In this post I’m investigating pushing the envelope with a bit of C programming.

The Missing Functionality

Postgres cannot do certain things required to fully handle transactions. The missing functionality is (at least):

Bitcoin Transaction Hash in Pure PostgreSQL

Tue, 10 Oct 2017 17:54:00 +0000

Update: hacked together this, more details to follow later…

In theory, Postgres should be able to verify transactions and blocks, as well as do a lot of other things that are currently only done by full nodes. For this to be performant, it will most likely require an extension written in C, but I’m curious how far we can get with bare bones Postgres.

More importantly, would that actually be useful? A node is really just a database, a very efficient one for a very specific purpose, but would leveraging the full power of Postgres be somehow more beneficial than just running Bitcoin-Qt or btcd, for example?

Electricity cost of 1 Bitcoin (Sep 2017)

Thu, 28 Sep 2017 16:38:00 +0000

How much does it cost in electricity to mine a Bitcoin?

As of Sep 28, 2017, according to blockchain.info the hashrate is: 9,214,860,125 GH/s.

These days it seems that the best miner available for sale is the AntMiner S9. It is actually over a year old, and there are faster and more energy efficient ASICs now, e.g. BitFury, but it is very hard to get any information on those, so we will just use the S9 information.

Bitcoin: USD Value

Mon, 25 Sep 2017 08:50:00 +0000

In part 1 I explained how money has always been a global ledger and Bitcoin is just a different implementation of one. The million dollar question remains, what should Bitcoin be worth in a currency we’re more familiar with, such as USD?

Asset Pricing

To illustrate the dilemma we’re faced with, lets look at three types of assets and how we price them.

Stocks: price is determined by the present and future profits, which is relatively straight forward.

Bitcoin: Better Ink than Gold?

Fri, 22 Sep 2017 07:52:00 +0000

The fundamental question about Bitcoin is not whether it is sound from the cryptography standpoint. The question is: what is it?

Money is Debt Ink

To define Bitcoin we need to look back at the history of money. The earliest money was in the form of things that were scarce and impossible to falsify, something like specific kinds of sea shells. Everyone knew that stuff could be traded for these tokens.

Tgres Status - July 4th 2017

Tue, 04 Jul 2017 08:28:00 +0000

It’s been a while since I’ve written on Tgres, here’s a little update, Independence Day edition.

Current Status

The current status is that Tgres is looking more and more like a finished product. It still needs time and especially user testing (the ball is in your court, dear reader), because only time reveals the weirdest of bugs and validates stability. I would not ditch your current stack just yet, but at this point you’d be remiss not having given Tgres a spin.

Building a Go Web App in 2017

Thu, 27 Apr 2017 15:00:00 +0000

Update: part 2 is here, enjoy. And part 3. And part 4.

A few weeks ago I started building yet another web-based app, in Go. Being mostly a back-end developer, I don’t have to write web apps very often, and every time I do, it seems like a great challenge. I often wish someone would write a guide to web development for people who do not have all day to get into the intricacies of great design and just need to build a functional site that works without too much fuss.

Building a Go Web App - Part 2

Thu, 27 Apr 2017 14:00:00 +0000

This is a continuation of part 1. (There is also part 3 and part 4).

So our app is going to have two major parts to it: client and server. (What year is this?). The server side is going to be in Go, and the client side in JS. Let’s talk about the server side first.

The Go (Server) Side

The server side of our application is going to be responsible for initially serving up all the necessary JavaScript and supporting files if any, aka static assets and data in the form of JSON. That’s it, just two functions: (1) static assets and (2) JSON.

Building a Go Web App - Part 3

Thu, 27 Apr 2017 13:00:00 +0000

This is part 3. See part 1 and part 2.

The previous two posts got us to a point where we had a Go app which was able to serve a tiny bit of HTML. This post will talk about the client side, which, alas, is mostly JavaScript, not Go.

JavaScript in 2017

This is what gave me the most grief. I don’t really know how to categorize the mess that present day JavaScript is, nor do I really know what to attribute it to, and trying to rationalize it would make for a great, but entirely different blog post. So I’m just going to accept this as the reality we cannot change and move on to how to best work with it.

Building a Go Web App - Part 4

Thu, 27 Apr 2017 09:13:00 +0000

This is part 4. See part 1, part 2 and part 3.

In this part I will try to briefly go over the missing pieces in our very simplistic Go Web App.

HTTP Handler Wrappers

I tiny rant: I do not like the word “middleware”. The concept of a wrapper has been around since the dawn of computing, there is no need to invent new words for it.

Having that out of the way, let’s say we need to require authentication for a certain URL. This is what our index handler presently looks like:

Tgres 0.10.0b - Time Series with Go and PostgreSQL

Wed, 22 Mar 2017 13:52:00 +0000

After nearly two years of hacking, I am tagging this version of Tgres as beta. It is functional and stable enough for people to try out and not feel like they are wasting their time. There is still a lot that could and should be improved, but at this point the most important thing is to get more people to check it out.

What is Tgres?

Tgres is a Go program which can receive time series data via Graphite, Statsd protocols or an http pixel, store it in PostgreSQL, and provide Graphite-like access to the data in a way that is compatible with tools such as Grafana. You could think of it as a drop-in Graphite/Statsd replacement, though I’d rather avoid direct comparison, because the key feature of Tgres is that data is stored in PostgreSQL.

Tgres Load Testing Follow Up

Tue, 28 Feb 2017 21:40:00 +0000

To follow up on the previous post, after a bunch of tweaking, here is Tgres (commit) receiving over 150,000 data points per second across 500,000 time series without any signs of the queue size or any other resource blowing up.

This is both Tgres and Postgres running on the same i2.2xlarge EC2 instance (8 cores, 64GB, SSD).

At this point I think there’s been enough load testing and optimization, and I am going to get back to crossing the t’s and dotting the i’s so that we can release the first version of Tgres.

PostgreSQL vs Whisper, which is Faster?

Thu, 23 Feb 2017 09:49:00 +0000

Note: there is an update to this post.

TL;DR

On a 8 CPU / 16 GB EC2 instance, Tgres can process 150,000 data points per second across 300,000 series (Postgres running on the same machine). With some tweaks we were able to get the number of series to half a million, flushing ~60K data points per second.

Now the long version…

If you were to ask me whether Tgres could outperform Graphite, just a couple of months ago my answer would have been “No”. Tgres uses Postgres to store time series data, while Graphite stores data by writing to files directly, the overhead of the relational database just seemed too great.

Storing Time Series in PostgreSQL - Optimize for Write

Sat, 21 Jan 2017 09:33:00 +0000

Continuing on the previous write up on how time series data can be stored in Postgres efficiently, here is another approach, this time providing for extreme write performance.

The “horizontal” data structure in the last article requires an SQL statement for every data point update. If you cache data points long enough, you might be able to collect a bunch for a series and write them out at once for a slight performance advantage. But there is no way to update multiple series with a single statement, it’s always at least one update per series. With a large number of series, this can become a performance bottleneck. Can we do better?

Simple Tgres Part II - A High Rate Counter

Wed, 28 Dec 2016 17:06:00 +0000

Continuing on the the previous post on simple use of Tgres components, let’s try to count something that goes by really fast.

This time let’s start out with creating a memory-based SerDe. This means that all our data is in memory and there is no database backing our series.

package main

import (
    "fmt"
    "net/http"
    "time"

    "github.com/tgres/tgres/dsl"
    h "github.com/tgres/tgres/http"
    "github.com/tgres/tgres/receiver"
    "github.com/tgres/tgres/rrd"
    "github.com/tgres/tgres/serde"
)

func main() {

    step := 1 * time.Second // 1 second resolution
    span := 600 * step      // spanning 10 minutes

    // In-memory SerDe
    ms := serde.NewMemSerDe()

    // Create a receiver of our data points backed by the above
    // memory SerDe
    rcvr := receiver.New(ms, &receiver.SimpleDSFinder{&rrd.DSSpec{
        Step: step,
        RRAs: []rrd.RRASpec{
            rrd.RRASpec{Function: rrd.WMEAN,
                Step: step,
                Span: span,
            },
        }}})
    rcvr.Start()

Now let’s create a goroutine which creates data points as fast as it can, the difference from the previous blog post is that we are using QueueGauge(), which is a paced metric, meaning that it flushes to the time series only periodically (once per second by default) so as to not overwhelm the I/O and or network (even though in this case it doesn’t really matter since we’re using a memory-based SerDe anyway).

Why is there no Formal Definition of Time Series?

Fri, 23 Dec 2016 09:13:00 +0000

If you’re reading this, chances are you may have searched for definition of “Time Series”. And, like me, you were probably disappointed by what you’ve found.

The most popular “definition” I come across amongst our fellow programmer folk is that it’s “data points with timestamps”. Or something like that. And you can make charts from it. And that’s about it, alas.

The word time suggests that is has something to do with time. At first it seems reasonable, I bite. The word series is a little more peculiar. A mathematician would argue that a series is a sum of a sequence. Most people though think “series” and “sequence” are the same thing, and that’s fine. But it’s a clue that time series is not a scientific term, because it would have been called time sequence most likely.

Simple Time Series App with Tgres

Wed, 21 Dec 2016 19:55:00 +0000

Did you know you can use Tgres components in your code without PostgreSQL, and in just a dozen lines of code instrument your program with a time series. This example shows a complete server emulating Graphite API which you can use with Grafana (or any other tool).

In this example we will be using three Tgres packages like so (in addition to a few standard ones, I’m skipping them here for brevity - complete source code gist):

Storing Time Series in PostgreSQL (Continued)

Fri, 16 Dec 2016 19:35:00 +0000

Edit: there is now a part iii in this series of articles.

I have previously written how time series can be stored in PostgreSQL efficiently using arrays.

As a continuation of that article, I shall attempt to describe in detail the inner workings of an SQL view that Tgres uses to make an array of numbers appear as a regular table (link to code).

In short, I will explain how incomprehensible data like this:

=> select * from ts;
 rra_id | n |           dp
--------+---+------------------------
      1 | 0 | {64,67,70,71,72,69,67}
      1 | 1 | {65,60,58,59,62,68,70}
      1 | 2 | {71,72,77,70,71,73,75}
      1 | 3 | {79,82,90,69,75,80,81}

… can be transformed in an SQL view to appear as so:

Parsing Table Names from SQL

Mon, 14 Nov 2016 14:40:00 +0000

Sometimes it is useful to extract table names from an SQL statement, for example if you are trying to figure out dependencies for your Hive or BigQuery (or whatever) tables.

It is actually a lot simpler than it seems and you don’t need to write your own SQL parser or find one out there. In SQL table names always follow the FROM and JOIN keywords. So all you have to do is split the statemement into tokens, and scan the list for any mention of FROM or JOIN and grab the next token.

Load testing Tgres

Tue, 08 Nov 2016 14:41:00 +0000

Edit: There is an update to this story.

So I finally got around to some load testing of Tgres. Load testing is mysterious, it never goes the way you think it would, and what you learn is completely unexpcted.

Given that I presently don’t have any spare big iron at my disposal and my “servers” are my macbook and an old thinkpad, all I really was after is making sure that Tgres is “good enough” whatever that means. And I think it is.

Golang receiver vs function argument

Thu, 22 Sep 2016 08:15:00 +0000

What is the difference between a Go receiver (as in “method receiver”) and a function argument? Consider these two bits of code:

func (d *duck) quack() { // receiver
     // do something
}

versus

func quack(d *duck) { // funciton argument
    // do something
}

The “do something” part above would work exactly the same regardless of how you declare the function. Which begs the question, which should you use?

In the object-oriented world we were used to objects doing things, and in that context d.quack() may seem more intuitive or familiar than quack(d) because it “reads better”. After all, one could argue that the former is a duck quacking, but the latter reads like you’re quacking a duck, and what does that even mean? I have learned that you should not think this way in the Go universe, and here is why.

How Data Points Build Up

Thu, 04 Aug 2016 10:35:00 +0000

This silly SVG animation (animation not my strong suit) demonstrates what happens when multiple Tgres data points arrive within the same step (i.e. smallest time interval for this series, also known as PDP, primary data point).

You browser does not support SVG objects?

Explanation

Let’s say we have a series with a step of 100 seconds. We receive the following data points, all within the 100 second interval of a single step:

Introducing Tgres - A Time Series DB on top of PostgreSQL

Fri, 29 Jul 2016 10:12:00 +0000

Tgres is a metrics collection and storage server, aka a time series database. I’m not very comfortable with referring to it as a database, because at least in case of Tgres, the database is actually PostgreSQL. But also “database” to me is in the same category as “operating system” or “compiler”, a thing so advanced that only few can claim to be it without appearing pretentious. But for the sake of tautology avoidance, I might occasionally refer to Tgres as a TS database.

Deploying a Golang app to AWS ECS with Terraform

Tue, 19 Apr 2016 10:53:00 +0000

I’ve put together a basic example of a “Hello World” Go program which runs in Amazon AWS Elastic Compute Service (ECS), which allows running applications in Docker containers and has the ability to scale on demand.

I initially wanted to write about the components of this system and the tools you use to deploy your application, but soon realized that this would make for an extremely long post, as the number of components required for a simple “Hello World” is mind boggling. However problematic it may seem, it’s par for the course, this is what takes to run an application in our cloudy times.

Holt-Winters Forecasting for Dummies - Part III

Wed, 17 Feb 2016 10:39:00 +0000

If you haven’t read Part I and Part II you probably should, or the following will be hard to make sense of.

In Part I we’ve learned how to forceast one point, in Part II we’ve learned how to forecast two points. In this part we’ll learn how to forecast many points.

More Terminology

Season

If a series appears to be repetitive at regular intervals, such an interval is referred to as a season, and the series is said to be seasonal. Seasonality is required for the Holt-Winters method to work, non-seasonal series (e.g. stock prices) cannot be forecasted using this method (would be nice though if they could be).

Holt-Winters Forecasting for Dummies - Part II

Tue, 16 Feb 2016 13:05:00 +0000

If you haven’t read Part I you probably should, or the following will be hard to make sense of.

All the forecasting methods we covered so far, including single exponential smoothing, were only good at predicting a single point. We can do better than that, but first we need to be introduced to a couple of new terms.

More terminology

Level

Expected value has another name, which, again varies depending on who wrote the text book: baseline, intercept (as in Y-intercept) or level. We will stick with “level” here.

Holt-Winters Forecasting for Dummies (or Developers) - Part I

Fri, 29 Jan 2016 15:36:00 +0000

This three part write up [Part II Part III] is my attempt at a down-to-earth explanation (and Python code) of the Holt-Winters method for those of us who while hypothetically might be quite good at math, still try to avoid it at every opportunity. I had to dive into this subject while tinkering on tgres (which features a Golang implementation). And having found it somewhat complex (and yet so brilliantly simple), figured that it’d be good to share this knowledge, and in the process, to hopefully solidify it in my head as well.

Storing Time Series in PostgreSQL efficiently

Wed, 23 Sep 2015 22:01:00 +0000

With the latest advances in PostgreSQL (and other db’s), a relational database begins to look like a very viable TS storage platform. In this write up I attempt to show how to store TS in PostgreSQL. (2016-12-17 Update: there is a part 2 of this article.)

A TS is a series of [timestamp, measurement] pairs, where measurement is typically a floating point number. These pairs (aka “data points”) usually arrive at a high and steady rate. As time goes on, detailed data usually becomes less interesting and is often consolidated into larger time intervals until ultimately it is expired.

Time Series Accuracy - Graphite vs RRDTool

Mon, 04 May 2015 17:40:00 +0000

Back in my ISP days, we used data stored in RRDs to bill our customers. I wouldn’t try this with Graphite. In this write up I try to explain why it is so by comparing the method of recording time series used by Graphite, with the one used by RRDTool.

Graphite uses Whisper to store data, which in the FAQ is portrayed as a better alternative to RRDTool, but this is potentially misleading, because the flexibility afforded by the design of Whisper comes at the price of inaccuracy.

On Time Series

Sat, 28 Mar 2015 15:40:00 +0000

Is it even a thing?

Time Series is on its way to becoming a buzzword in the Information Technology circles. This has to do with the looming Internet of Things which shall cause the Great Reversal of Internet whereby upstream flow of data produced by said Things is expected to exceed the downstream flow. Much of this data is expected to be of the Time Series kind.

This, of course, is a money-making opportunity of the Big Data proportions all over again, and I predict we’re going to see a lot of Time Series support of various shapes and forms appearing in all manners of (mostly commercial) software.

How InfluxDB Stores Data

Fri, 20 Mar 2015 15:52:00 +0000

A nice, reliable, horizontally scalable database that is designed specifically to tackle the problem of Time Series data (and does not require you to stand up a Hadoop cluster) is very much missing from the Open Source Universe right now.

InfluxDB might be able to fill this gap, it certainly aims to.

I was curious about how it structures and stores data and since there wasn’t much documentation on the subject and I ended up just reading the code, I figured I’d write this up. I only looked at the new (currently 0.9.0 in RC stage) version, the previous versions are significantly different.

Ruby, HiveServer2 and Kerberos

Tue, 19 Aug 2014 08:03:00 +0000

Recently I found myself needing to connect to HiveServer2 with Kerberos authentication enabled from a Ruby app. As it turned out rbhive gem we were using did not have support for Kerberos authentication. So I had to roll my own.

This post is to document the experience of figuring out the details of a SASL/GSSAPI connection before it is lost forever in my neurons and synapses.

First, the terminology. The authentication system that Hadoop uses is Kerberos. Note that Kerberos is not a network protocol. It describes the method by which authentication happens, but not the format of how to send Kerberos tickets and what not over the wire. For that, you need SASL and GSSAPI.

Graceful restart in Golang

Tue, 03 Jun 2014 12:49:00 +0000

Update (Apr 2015): Florian von Bock has turned what is described in this article into a nice Go package called endless.

If you have a Golang HTTP service, chances are, you will need to restart it on occasion to upgrade the binary or change some configuration. And if you (like me) have been taking graceful restart for granted because the webserver took care of it, you may find this recipe very handy because with Golang you need to roll your own.

mod_python performance part 2: high(er) concurrency

Thu, 07 Nov 2013 17:51:00 +0000

Tl;dr

As is evident from the table below, mod_python 3.5 (in pre-release testing as of this writing) is currently the fastest tool when it comes to running Python in your web server, and second-fastest as a WSGI container.

Server	Version	Req/s	% of httpd static	Notes
nxweb static file	3.2.0-dev	512,767	347.1 %	"memcache":false. (626,270 if true)
nginx static file	1.0.15	430,135	291.1 %	stock CentOS 6.3 rpm
httpd static file	2.4.4, mpm_event	147,746	100.0 %
mod_python handler	3.5, Python 2.7.5	125,139	84.7 %
uWSGI	1.9.18.2	119,175	80.7 %	-p 16 --threads 1
mod_python wsgi	3.5, Python 2.7.5	87,304	59.1 %
mod_wsgi	3.4	76,251	51.6 %	embedded mode
nxweb wsgi	3.2.0-dev, Python 2.7.5	15,141	10.2 %	posibly misconfigured?

The point of this test

I wanted to see how mod_python compares to other tools of similar purpose on high-end hardware and with relatively high concurrency. As I’ve written before you’d be foolish to base your platform decision on these numbers because speed in this case matters very little. So the point of this is just make sure that mod_python is in the ballpark with the rest and that there isn’t anything seriously wrong with it. And surprisingly, mod_python is actually pretty fast, fastest, even, though in its own category (a raw mod_python handler).

Separate Request and Response or a single Request object?

Wed, 30 Oct 2013 13:18:00 +0000

Are you in favor of a single request object, or two separate objects: request and response? Could it be that the two options are not contradictory or even mutually exclusive?

I thouhgt I always was in favor of a single request object which I expressed on the Web-SIG mailing list thread dating back to October 2003 (ten years ago!). But it is only now that I’ve come to realize that both proponents of a single object and two separate objects were correct, they were just talking about different things.

My thoughts on WSGI

Sat, 26 Oct 2013 11:24:00 +0000

I’m not very fond of it. Here is why.

CGI Origins

WSGI is based on CGI, as the “GI” (Gateway Interface) suggests right there in the name.

CGI solved a very important problem using the very limited tools at hand available at the time. Though CGI wasn’t a standard, it was ubiquitous in the early days of the WWW, despite its inherent slowness and other limitations. It became popular because it worked with any language, was easy to turn on and provided such a thick wall of isolation that admins could turn it on for their users without too much concern for problems caused by user-generated CGI scripts.

mod_python: the long story

Fri, 25 Oct 2013 20:05:00 +0000

This story started back in 1996. I was in my early twenties, working as a programmer at a small company specializing in on-line reporting of certain pharmaceutical data.

There was a web-based application (which was extremely cool considering how long ago this was), but unfortunately it was written in Visual Basic by a contractor and I was determined to do something about it. As was very fashionable at the time, I was very pro Open Source, had Linux running on my home 386 and had recently heard Guido’s talk at the DC Linux user group presenting his new language he called Python. Python seemed like a perfect alternative to the VB monstrosity.

mod_python performance and why it matters not.

Thu, 10 Oct 2013 13:04:00 +0000

TL;DR: mod_python is faster than you think.

Tonight I thought I’d spend some time looking into how the new mod_python fares against other frameworks of similar purpose. In this article I am going to show the results of my findings, and then I will explain why it really does not matter.

I am particularly interested in the following:

a pure mod_python handler, because this is as fast as mod_python gets.
a mod_python wsgi app, because WSGI is so popular these days.
mod_wsgi, because it too runs under Apache and is written entirely in C.
uWSGI, because it claims to be super fast.
Apache serving a static file (as a point of reference).

The Test

I am testing this on a CentOS instance running inside VirtualBox on an early 2011 MacBook Pro. The VirtualBox has 2 CPU’s and 6GB of RAM allocated to it. Granted this configuration can’t possibly be very performant [if there is such a word], but it should be enough to compare.

Running a WSGI app on Apache should not be this hard

Wed, 25 Sep 2013 20:08:00 +0000

If I have a Django app in /home/grisha/mysite, then all I should need to do to run it under Apache is:

$ mod_python create /home/grisha/mysite_httpd \
    --listen=8888 \
    --pythonpath=/home/grisha/mysite \
    --pythonhandler=mod_python.wsgi \
    --pythonoption="mod_python.wsgi.application mysite.wsgi::application"

$ mod_python start /home/grisha/mysite_httpd/conf/httpd.conf

That’s all. There should be no need to become root, tweak various configurations, place files in the right place, check permissions, none of that.

Well… With mod_python 3.4.0 (alpha) that’s exactly how it is…

The Next Smallest Step Problem

Sat, 10 Aug 2013 15:10:00 +0000

“A journey of a thousand miles begins with a single step”

Most of my journeys never begin, or cannot continue because of that one single step, be it first or next. Because it is hard, at times excruciatingly so.

Here I speak of software, but this applies to many other aspects of my life just as well.

I recon it’s because I do not think in steps. I think of a destination. I imagine the end-result. I can picture it with clarity and in great detail. I know where I need to be. But what is the next step to get there? And it doesn’t help that where I travel, there are no signs.

Hacking on mod_python (again)

Fri, 02 Aug 2013 16:19:00 +0000

Nearly eight years after my last commit to Mod_python I’ve decided to spend some time hacking on it again.

Five years without active development and thirteen since its first release, it still seems to me an entirely useful and viable tool. The code is exceptionally clean, the documentation is amazing, and the test suite is awesome. Which is a real testament to the noble efforts of all the people who contributed to its development.

json2avro

Fri, 21 Jun 2013 14:17:00 +0000

As you embark on converting vast quantities of JSON to Avro, you soon discover that things are not as simple as they seem. Here is how it might happen.

A quick Google search eventually leads you to the avro-tools jar, and you find yourself attempting to convert some JSON, such as:

{"first":"John", "middle":"X", "last":"Doe"}
{"first":"Jane", "last":"Doe"}

Having read Avro documentation and being the clever being that you are, you start out with:

java -jar ~/src/avro/java/avro-tools-1.7.4.jar fromjson input.json --schema \
 '{"type":"record","name":"whatever",
   "fields":[{"name":"first", "type":"string"},
             {"name":"middle","type":"string"},
             {"name":"last","type":"string"}]}' > output.avro
Exception in thread "main" org.apache.avro.AvroTypeException: Expected field name not found: middle
        at org.apache.avro.io.JsonDecoder.doAction(JsonDecoder.java:477)
        at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
        ...

A brief moment of disappointment is followed by the bliss of enlightment: Duh, the “middle” element needs a default! And so you try again, this time having tacked on a default to the definition of “middle”, so it looks like {"name":"middle","type":"string","default":""}:

Avro performance

Tue, 11 Jun 2013 00:17:00 +0000

Here are some un-scientific results on how Avro performs with various codecs, as well as vs JSON-lzo files in Hive and Impala. This testing was done using a 100 million row table that was generated using random two strings and an integer.

| Format    | Codec          | Data Size     | Hive count(1) time | Impala count(1) time
|-----------|----------------|---------------|--------------------|----------------------
| JSON      | null           | 686,769,821   | not tested         | N/A                  
| JSON      | LZO            | 285,558,314   | 79s                | N/A                  
| JSON      | Deflate (gzip) | 175,878,038   | not tested         | N/A                  
| Avro      | null           | 301,710,126   | 40s                | .4s                  
| Avro      | Snappy         | 260,450,980   | 38s                | .9s                  
| Avro      | Deflate (gzip) | 156,550,144   | 64s                | 2.8s

So the winner appears to be Avro/Snappy or uncompressed Avro.

Apache Avro

Thu, 06 Jun 2013 22:53:00 +0000

Short version

Avro is better than Json for storing table data
Avro supports schema resolution so that the schema can evolve over time
Hive supports Avro and schema resolution nicely
Impala (1.0) can read Avro tables, but does not support schema resolution
Mixing compression codecs in the same table works in both Hive and Impala

The TL;DR version

Introduction

If you’re logging data into Hadoop to be analyzed, chances are you’re using JSON. JSON is great because it’s easy to generate in most any language, it’s human-readable, it’s universally supported and infinitely flexible.

Simple Solution to Password Reuse

Fri, 31 May 2013 17:12:00 +0000

Here's a KISS solution to all your password reuse problems. It requires remembering only *one* strong password, lets you have a virtually limitless number of passwords, and, most importantly, does NOT store anything anywhere or transfer anything over the network (100% browser-side Javascript).

Compiling Impala from Github

Fri, 31 May 2013 10:48:00 +0000

Apparently Impala has two versions of source code, one internal to Cloudera, the other available on Github. I’m presuming that code gets released to Github once undergone some level of internal scrutiny, but I’ve not seen any documentation on how one could tie publically available code to the official Impala (binary) release, currently 1.0.

Anyway, I tried compiling the github code last night, and here are the steps that worked for me.

SQLite DB stored in a Redis hash

Wed, 29 May 2013 17:08:00 +0000

In a recent post I explained how a relational database could be backed by a key-value store by virtue of B-Trees. This sounded great in theory, but I wanted to see that it actually works. And so last night I wrote a commit to Thredis, which does exactly that.

If you’re not familiar with Thredis - it’s something I hacked together last Christmas. Thredis started out as threaded Redis, but eventually evolved into SQLite + Redis. Thredis uses a separate file to save SQLite data. But with this patch it’s no longer necessary - a SQLite DB is entirely stored in a Redis Hash object.

Checking out Cloudera Impala

Thu, 23 May 2013 12:43:00 +0000

I’ve decided to check out Impala last week and here’s some notes on how that went.

First thoughts

I was very impressed with how easy it was to install, even considering our unusual set up (see below). In my simple ad-hoc tests Impala performed orders of magnitude faster than Hive. So far it seems solid down to the little details, like the shell prompt with a fully functional libreadline and column headers nicely formatted.

Relational database on top of key-value store explained (or why B-trees are cool)

Sat, 11 May 2013 11:36:00 +0000

This post attempts to explain how a relational database can be implemented atop a key/value store, a subject that I’ve long found rather mysterious.

Every once in a while I would come across a mention that a relational database can be implemented using a key/value store (aka dictionary, hash table or hash map - for brevity I’ll be using map from here on).

Whenever I thought about it, it just didn’t make sense. A relational database needs to store rows in order, and that’s one feature that maps do not provide. Imagine we have a table keyed by employee id stored in a map and we need to traverse it by id in ascending order. A hypothetical keys() method would return us a list of ids ordered randomly. It’s impossible to iterate over a hash map in order. So how would a relational database work then?

MapJoin: a simple way to speed up your Hive queries

Fri, 19 Apr 2013 11:15:00 +0000

Mapjoin is a little-known feature of Hive. It allows a table to be loaded into memory so that a (very fast) join could be performed entirely within a mapper without having to use a Map/Reduce step. If your queries frequently rely on small table joins (e.g. cities or countries, etc.) you might see a very substantial speed-up from using mapjoins.

There are two ways to enable it. First is by using a hint, which looks like /*+ MAPJOIN(aliasname), MAPJOIN(anothertable) */. This C-style comment should be placed immediately following the SELECT. It directs Hive to load aliasname (which is a table or alias of the query) into memory.

Linus on understanding pointers

Tue, 02 Apr 2013 10:52:00 +0000

A while back Linus Torvalds answered questions on Slashdot.

One of the answers was on the subject of understanding of pointers:

At the opposite end of the spectrum, I actually wish more people understood the really core low-level kind of coding. Not big, complex stuff like the lockless name lookup, but simply good use of pointers-to-pointers etc. For example, I’ve seen too many people who delete a singly-linked list entry by keeping track of the “prev” entry, and then to delete the entry, doing something like

Storm Notes

Mon, 01 Apr 2013 17:05:00 +0000

Some random thoughts on having tinkered with Storm over the past few weeks.

It took me some time to understand what Storm is, and I am still not clear I have found a perfect use for it. (This is not a criticism of Storm, the point is that the concepts it introduces are new, somewhat diffuclt and will need some time so sync in). The best way to get the basic understanding of Storm concepts is to watch Nathan Marz’s excellent presentation.

on prioritization - important vs urgent

Wed, 27 Mar 2013 10:13:00 +0000

Every item on a TODO list can be classified as urgent, important or neither. We should act on important items first, urgent second and ignore the rest.

Sometimes an item lands on our TODO list described as extremely urgent without any explanation of importance. In this case the important item (and thus to be done first) becomes determining the importance of the extremely urgent item in question, even if it means delaying it.

On keeping lots of integers in memory

Mon, 25 Mar 2013 22:09:00 +0000

Once upon a time (over a year ago) I found myself needing to store large numbers of integers in memory. The goal was to store a graph of all our purchasers and items purchased, so that we could quickly identify like-minded purchasers based on common purchases and make real-time recommendations of the form “people like you also bought”. This approach is commonly known as collaborative filtering, and exactly how we did it would be a subject of some future post (perhaps).

Testing out Octopress

Mon, 25 Mar 2013 11:46:00 +0000

Thanks to Octopress and Github I blog….

And I Tweet, too?

Thanks to Octopress and Github pages I now have a blog: grisha.org
— Grisha Trubetskoy (@humblehack) March 26, 2013

A link to Gregory Trubetskoy videos or a particular video