Virtual Surreality

Object/Relational Mappings (ORM’s) are in the wrong place in the architecture.

An application should have minimal impedance mismatch with the persistence of its own data. External or ancillary systems should bear the cost of mapping between paradigms. If you are writing an application with the concepts modelled in an object-oriented fashion, then the state of those objects should be stored in an object-oriented fashion. Mapping from the object-oriented structure to a relational structure should be done where and when it’s needed. If you want to access application data in a relational way for reports, do the mapping for the report.

There has been a low uptake of OODBMS’s. Why is outside the scope of this post. However, there are other, older (albeit feeling newer) data storage techniques available that more readily persist an object graph than a relational database. A flurry of open-source implementations have come along. You may have heard of MongoDB, neo4j, Redis, Hadoop, CouchDB, Memcached, and Raven DB (a native .NET document database). These are all under active development, and almost all have been proven on many large commercial projects (Raven was being developed by Oren while he was at speakerconf 2010!).

For an introductory look at evaluating the landscape, check out Sarah Mei’s RailsConf presentation slides.

One of the projects I’ve worked on at Hashrocket involved a large MongoDB database. The reasons for the choice for MongoDB were clear:

• the way the users handled most of the data more naturally fitted an embedded document model;
• the scale and speed required were far greater than relational databases could achieve; and
• the need for ACID transactions was almost nil

Additionally, the data was mostly read, rather than written (although MongoDB writes blazingly fast too). I’m hoping to address the equivocations in those reasons in a subsequent post.

MongoDB uses a binary form of JSON. As JSON is a serialization format for object state, it’s a good candidate for persisting an object graph. JavaScript, remember, uses composition rather than inheritance, so there is also a tendency to “embed” document instances within other document instances. So, for MongoDB, most of the object graph is stored a tree structure. Cyclic graphs and associated objects (those that aren’t aggregated with composition) are handled with references to identifiers.

As a consequence of using Rails and MongoDB, the team looked to various ways of manipulating the JSON structures from Ruby: directly through the driver, MongoMapper, MongoDoc, and Mongoid. The rationales behind these libraries and their different and interesting approaches are the subject of additional blogs (hopefully jointly with Nunemaker, Hill, and Jordan respectively). The needs and makeup of the project we were on fell into the lap of mongoid to solve.

DBAs want to interact directly with the database. Because of this “new-fangled” approach to data storage that has been around since before SQL, some are unwilling or unable to learn the query language to interact directly with the database. So, to make their world a less scary place and let them retire without the stress of intellectual endeavour, I built a simple, declarative language where the mapping from the tree structure of MongoDB to the tuple space in an RDBMS can be scripted.

Durran Jordan (author of Mongoid) and I paired on “squealer” (and on emptying bottles) in our evenings while he was stationed in the Chicago office of Hashrocket. The name came from “SQL” (which some of us still pronounce “squeal” rather than “sequel”) and the facetious notion that it turned the data into a pig.

We decided to focus on mySQL as the target and MongoDB as the source as these are the technologies we were using. Upcoming targets will include PostgreSQL, SQL Server, Oracle. Upcoming sources will include Redis and CouchDB.

UPDATE: squealer 2.2 now supports PostgreSQL!

Matt Yoho provided multiple insights into implementing the Ruby DSL, and Bernerd Schaefer helped me reduce the DSL syntax and provide sensible defaults (as well as the righteous progress bar). Bernerd then struck on the idea of reflecting on the Mongoid declarations in the Rails domain model classes to generate an initial squealer script and SQL DDL to build the target SQL database. He then set about writing it overnight and we tweaked it over lunch. We called this tool “skewer”, because that’s what you do to something to make it squeal . It’s part of the squealer RubyGem.

Next time: how to use skewer and squealer.

OK so I’ve been moving from Sydney to Chicago with lots of time at a client in Calgary. No posts, but some coming.

In the meantime, based on a note from my colleague Ian Cartwright on an article about Microsoft embracing AMQP, I wrote a little poem, for I am in a very poetic mood at present. I’ll tell you why next week!

There is a message queue open standard /
almost all implementations are free /
it’s the killer app of the internet /
and what’s its name? Lo! It’s SMTP!

In response to Dr Nick’s technological feat, Ajey suggested I take on the much more challenging task of running OS X on a PC.

Result!!

Also, Mingle 2.0 is released! Another fantastic effort from the ThoughtWorks Studios crew.

Software intelligence has come of age!

My colleagues at ThoughtWorks Studios have cracked the automation of the software development process. Compuware, IBM and Borland have been trying for years. MDA is the useless love-child of CASE and UML. It takes true genius to solve this problem:

http://studios.thoughtworks.com/mingle-hidden/introducing-mingle-proj-o-matic

My erstwhile colleague, Jon Tirsen, now at Google in Sydney, has also announced a truly amazing feat in blending statistical analysis, complex hypercube maths, and their super clever search engine. I’m proud that Jon, although Swedish, has embraced the Aussie vernacular and called the product “gDay” (which is supposed to be “Google Day” but we all know where he keeps his surfboard). I’m also happy that they put Graham’s number to good use in this truly extraordinary achievement.

http://google.com.au/gday

My CTO, Rebecca Parsons, announced the publication of the ThoughtWorks Anthology a few days ago:

I am thrilled to announce that the ThoughtWorks Anthology is now ON SALE!

http://www.pragprog.com/titles/twa

There are essays by Roy and Michael Robinson, Martin, Neal Ford, Tiffany Lentz, Stelios Pantazopoulos, Ian Robinson, Erik Doernenburg, Kristan Vingrys and James Bull, as well as ex-TWers Dave Farley, Jeff Bay and Julian Simpson. (And of course your’s truly made her own contribution). Mike Aguilar wrote the introduction.

Some comments that appear in the book:

Jim Fischer writes, “The anthology provides a peek into the diversity of views and perspectives held by a company that dares to take on the long-accepted tenet of the IT industry that custom software is too hard and too costly.”

Big Dave Thomas writes, “Software is in many ways a team sport, and the leaders shape the software culture. Often successful organizations don’t take the time to document them, and hence others don’t benefit from them. This interesting collection of personal essays gives a glimpse into the culture of ThoughtWorks through some of its leaders.”

I’m really excited that this project has come to fruition and I hope you all enjoy what you see.

Rebecca

I was lucky enough to review the content earlier this year. It’s a keeper.

UPDATE: Some press… http://www.sys-con.com/read/541124.htm

http://money.cnn.com/2008/03/14/technology/kirkpatrick_thoughtworks.fortune/index.htm

Nuff said (although journalists never quite get it.)

After a meeting of the office of the CTO, most of us stayed around in our San Francisco office for a few days to do some podcasts and to participate in a Code Jam for Inveneo, a not-for-profit who provide computers and connectivity to developing countries (especially their schools, hospitals, and poorer villages).

They install a server in, say, a hospital with a few lower-powered, custom desktops (almost iMac in configuration). These, as well as the servers, can run off solar panels for power.

We were presented with a worthwhile problem with a number of interesting constraints:

• Low-power server running Ubuntu, with two small SATA hard disks in a Linux software RAID-1 array
• VMWare images of the servers for testing
• Python, bash, mdadm, and beep as our “programming languages”

When a RAID array fails, we need to alert any (if any) humans who are near the server. This can be interesting as the only things nearby might be the tree it is mounted in with a long-range WiFi, or the goat who uses it as a heat source at night. This means that any alert should be sufficiently frequent and annoying for the locals to contact someone who can let the support technician know. The conflict is that it also might be the nurses in their office at the hospital who have work to do and don’t want to be disturbed.

The solution was to use the PC speaker to beep. We can control the pitch and duration of the beep. Some combinations sounded too much like an ECG machine so that was no good. In the end, we chose a simple rising scale that would sound odd in any environment (except, perhaps, in a Mike Oldfield recording). This is repeated by default every 30 minutes.

We also had to send an email to the support technician. This doesn’t work when the server doesn’t actually have any connectivity (as some are used only as a local communications hub), or when connectivity is unreliable. Even then, many of the technicians are hours or even days away from the servers.

As many of the technicians aren’t particularly technical, we also had help by identifying which of the two disks had failed and allow them to simply change the one labelled “Disk 1″ or “Disk 2″. Serial numbers are good for this but VM hard disks don’t have serial numbers (I think that’s a feature request to both VMWare and Parallels).

We had Jeff Wishnie from Inveneo as the customer, Anda Abramovici as IM, Jonny Leroy as BA, Paul Hammant and Chad Wathington as QA, and the star developer crew of Drew Olson, Sammy Zahabi, Ola Bini, Erik Doernenburg and your’s truly. We quickly learned the following:

• The skills we needed (and had, just a little rusty) were more along the lines of Unix devices, shell, ASCII control characters, and simulation
• Python sucks (a bad tradesman blames his tools? perhaps – but it still sucks)

Anyway, we got most of what we wanted done in the time, and given the context, more than we anticipated. But we all would have liked to get a lot more done and would have if we were using our tools of choice (which are chosen for very good reasons).

Nonetheless, we’re doing it for the kids and it was great!

What a buzz. Super Agile. Super Fun. Go Inveneo, you rock!!

If you’re complaining to someone about how wet it is, or cold, or hot, or how much interest rates have gone up, or your shares in Acme corp have gone down, or how your neighbour turns the music up too loud and you can’t sleep – just have a reality check for a few hours and read the 2007 Because I Am A Girl report from Plan organisation.

Some atrocious statistics to prepare your stomach lining:

• Pregnancy is a leading cause of death for young women aged 15 to 19 worldwide
• About 140 million girls have undergone female genital cutting and two million are subjected to it every year
• In South Africa, 32% of reported child rapes were carried out by a teacher
• In India, 60% to 70% of adolescent girls are anaemic
• In Britain, among 16-24 year olds, twice as many young women as young men are overweight
• One in 10 births worldwide is to a mother who is still a child herself
• 70% of the 1.5 billion people living on $1 a day or less are female
• Britain’s Equal Opportunities Commission between 2003 and 2005 showed that there is still a gender pay gap in the UK, of 18% for women working full time and 41% for those working part-time
• 80% of the 600,000 to 800,000 victims of trafficking per year are girls and women

So, along with the UN, Plan have an action plan that includes nations enforcing their own laws (including the international ones they subscribe to).

Try every day to make it better. It’s in your own back yard, wherever you live.

Naming a framework is like naming a child – you need to be careful because people will find ways to make fun of the name, especially when it becomes surreal in truth. You’re a happy countenance and before you guano it, a tiny dipthong comes along and you’re excrement.

That’s good advice, or my name isn’t Richard “worm pants” Fiddlesbottom.

I’m visiting San Francisco and stayed with old friends of mine for the first couple of days, so I offered to cook for the family.

Mostly optional, mostly whatever ratios look good. But more cumin than anything in this one, and try try try for the orange blossom water – it really makes the dish.

The Lamb
5lbs (2.5kg) lamb shoulder
dried apricots
garlic cloves
1 lemon (1/8th’d)
fresh rosemary

The sauce
2 cans peeled tomato
1/3 bottle ?? orange blossom water
chopped fresh coriander leaves and stalks (cilantro)
diced brown onion
halved button or brown mushrooms
1 sliced red capsicum (bell pepper) – halve the slices
1 diced carrot
garlic chives

The spices
chilli flakes
cumin powder
coriander powder
nutmeg powder
cinnamon powder
2 cinnamon sticks
5 star anise
3 bay leaves
sumac powder
paprika
saffron
turmeric powder
garlic powder
salt
pepper

The method
Pre-heat oven to 280F, including tagine / casserole pot

stuff shoulder, bind with cooking twine
coat lamb in spice mix (use a little olive oil), rub into flesh
seal lamb in hot skillet / thick-based fry pan
place in tagine or thick-based stewing / casserole pot
de-glaze skillet with tomato
add other sauce ingredients to skillet
pour sauce over lamb
cook at 280F for 1.5 hours (low shelf)

add mushrooms, garlic chives, capsicum, carrots
cook for 30 mins (low shelf)

The couscous salad
1 roughly chopped large flat mushroom
coriander leaves (cilantro)
juice 2 limes
1/2 red capsicum (bell pepper)
1 finely diced small red onion
pine nuts

lightly toast pine nuts on medium heat in fry pan, set aside
lightly saute onion and capsicum, remove from heat, add mushroom

The couscous
1.5 cups water
2 cups couscous
1 knob butter or tsp olive oil
salt
pepper

boil water, remove from heat, add couscous and butter/oil, stir thoroughly
cover for 2 mins
stir in salad

The greens
blanch green beans for 3 mins
blanch baby spinach for 1 min
drain

The plating
remove twine from lamb
carve into thick slices
remove star anise, cinnamon stick, and bay leaves (through colander if desired)
if desired, to strained sauce, add yoghurt and cornflour to thicken (add some water or some sauce to cornflour to make a runny paste before adding to main pot)
serve couscous with lamb and sauce, greens on side

The vino
White: Lightly wooded Chardonnay / Chablis or Pinot Gris
Red: Shiraz Viognier or Pinot Noir

Goddamn it tastes gooooood.