Object/Relational Mappings (ORM’s) are in the wrong place in the architecture.

An application should have minimal impedance mismatch with the persistence of its own data. External or ancillary systems should bear the cost of mapping between paradigms. If you are writing an application with the concepts modelled in an object-oriented fashion, then the state of those objects should be stored in an object-oriented fashion. Mapping from the object-oriented structure to a relational structure should be done where and when it’s needed. If you want to access application data in a relational way for reports, do the mapping for the report.

There has been a low uptake of OODBMS’s. Why is outside the scope of this post. However, there are other, older (albeit feeling newer) data storage techniques available that more readily persist an object graph than a relational database. A flurry of open-source implementations have come along. You may have heard of MongoDB, neo4j, Redis, Hadoop, CouchDB, Memcached, and Raven DB (a native .NET document database). These are all under active development, and almost all have been proven on many large commercial projects (Raven was being developed by Oren while he was at speakerconf 2010!).

For an introductory look at evaluating the landscape, check out Sarah Mei’s RailsConf presentation slides.

One of the projects I’ve worked on at Hashrocket involved a large MongoDB database. The reasons for the choice for MongoDB were clear:

• the way the users handled most of the data more naturally fitted an embedded document model;
• the scale and speed required were far greater than relational databases could achieve; and
• the need for ACID transactions was almost nil

Additionally, the data was mostly read, rather than written (although MongoDB writes blazingly fast too). I’m hoping to address the equivocations in those reasons in a subsequent post.

MongoDB uses a binary form of JSON. As JSON is a serialization format for object state, it’s a good candidate for persisting an object graph. JavaScript, remember, uses composition rather than inheritance, so there is also a tendency to “embed” document instances within other document instances. So, for MongoDB, most of the object graph is stored a tree structure. Cyclic graphs and associated objects (those that aren’t aggregated with composition) are handled with references to identifiers.

As a consequence of using Rails and MongoDB, the team looked to various ways of manipulating the JSON structures from Ruby: directly through the driver, MongoMapper, MongoDoc, and Mongoid. The rationales behind these libraries and their different and interesting approaches are the subject of additional blogs (hopefully jointly with Nunemaker, Hill, and Jordan respectively). The needs and makeup of the project we were on fell into the lap of mongoid to solve.

DBAs want to interact directly with the database. Because of this “new-fangled” approach to data storage that has been around since before SQL, some are unwilling or unable to learn the query language to interact directly with the database. So, to make their world a less scary place and let them retire without the stress of intellectual endeavour, I built a simple, declarative language where the mapping from the tree structure of MongoDB to the tuple space in an RDBMS can be scripted.

Durran Jordan (author of Mongoid) and I paired on “squealer” (and on emptying bottles) in our evenings while he was stationed in the Chicago office of Hashrocket. The name came from “SQL” (which some of us still pronounce “squeal” rather than “sequel”) and the facetious notion that it turned the data into a pig.

We decided to focus on mySQL as the target and MongoDB as the source as these are the technologies we were using. Upcoming targets will include PostgreSQL, SQL Server, Oracle. Upcoming sources will include Redis and CouchDB.

UPDATE: squealer 2.2 now supports PostgreSQL!

Matt Yoho provided multiple insights into implementing the Ruby DSL, and Bernerd Schaefer helped me reduce the DSL syntax and provide sensible defaults (as well as the righteous progress bar). Bernerd then struck on the idea of reflecting on the Mongoid declarations in the Rails domain model classes to generate an initial squealer script and SQL DDL to build the target SQL database. He then set about writing it overnight and we tweaked it over lunch. We called this tool “skewer”, because that’s what you do to something to make it squeal . It’s part of the squealer RubyGem.

Next time: how to use skewer and squealer.