A developer’s status update: Test driven deadlock

No worries, I do value testing. But test driven? It depends.

The last few weeks I’ve been working off and on building a crawler. The thing triggers a series of scheduled tasks that could run in parallel, generating (possibly) tasks (that are consequently scheduled again in their own task-specific queue) on its own and so on. The end goal is structured copies of external resources (read: webpages). But I’ve been stuck close to the start for quite a while, setting up the base architecture using the test driven development approach. And I’m failing, it’s going too slow :(

Small victories

Like many developers, this is not my first parser/crawler. But this time I wanted to make a GoodParser™. Make it more extensible, and flexible and foremost robust, building in fault-tolerance from start. But I’m not getting near the end result and it is frustrating.

The TDD-school says make small victories, and oh yes, I had my sheer number of victories already. But the end is nowhere in sight. I’m testing the movement of one step forward: did we move one step forward? Yes! But did we move closer to the target? It’s too hard to see in this dark…

BDD instead?

You might say TDD is good for unit test type of tests, try Behaviour Driven Development (BDD) instead. But my ultimate behavioural ‘test’ would be: When call this, it returns a JSON representation of the structural data on site X. And it would force me the simplest way to implement that… I’ve been there too many times.

Go, explore!

I’ve chosen an architecture that I’d wanted to explore.

Realising that I’m still exploring I will skip the test driven approach for now. If the dust caused by my digging settles down I’ll freeze some of it in tests. Will it be bug-free from start? I guess not, but that’s where TDD has never failed me: analyse the stack trace, write the test that replicates the bug and fix it.