May 29, 2014 · Events

Let's Test 2014 in review

As I sit in the departure lounge at Stockholm's Arlanda airport, already feeling a sense of loss after four great days, I decided to write a summary of my experiences at Let's Test 2014.

The following post doesn't contain much in the way of detail from any individual session; it's overly long already! The conference slides should be made available online in due course; I can provide Evernote links to my boring plain-text notes, but for awesome visual summaries, look no further than the beautiful sketchnote tweets from Zeger Van Hese and Ruud Cox.

Instead, here you'll find a summary (often in tweet form) of the week as I experienced it. Other people's experiences were likely different. Everyone's experiences were likely memorable.

Sunday: Day 0

As my plane descended into Stockholm, there was one thing that stood out as I looked out of the cabin window - Sweden is a nation which respects its forests. The entire country is awash with trees. England is well known for its fields, but I've never seen anything quite on the scale of the wooded areas of Scandinavia. The breathtaking scenery was a running theme throughout my time in the country.

I arrived at the Runö conference venue just in time for dinner, where I had time to catch up with some old friends, and put names to some faces that I'd only previously known from Twitter. It was a slightly surreal experience, as these were people that I'd never met before, but whom I felt like I already knew extremely well.

During dinner, our first "testing challenge" of the conference was announced:

When I arrived in the test lab, the activity was already in full-swing. There were groups huddled around five buttons; each button was connected to its own array of five LED lights. The groups were taking fairly similar approaches, quickly noticing that different numbers of button-presses were producing different combinations of LED-blinks. However, none of the combinations seemed particularly rational or connected; for instance, pressing the button 3 times produced a much longer sequence of lights than any other.

Interesting avenues were explored by other groups; it was observed that there was a number on the bottom of each device, and that each of the five lights was on a different length cord. Rob Sabourin's team was developing some interesting theories about transposing the lights into a seven-segment display, with some possible patterns emerging and then being dismissed.

However, our minds were refocused when we were told that all of the groups were "making the same mistake". We quickly realised that we'd all been treating our results in isolation; I (and several others) had been comparing data across the groups, for instance to check whether all the buttons were behaving the same, but we hadn't tried to pool the data.

The breakthrough came when we reviewed the significance of the different cord lengths. With the devices having numbers on the bottom, we arranged the devices in numerical order, trailing the lights in order of cord length, and... hey presto. Each blink resulted in a letter/symbol being drawn.

Some of the letters/symbols were harder to understand than others, and there was at least one which we still failed to comprehend, but the most important realisation was that we would never have made even the slightest dent into translating these messages without working together.

By the time we'd finished the exercise it was almost 11pm and the end of a long day, so I retired (relatively) early to bed. At this point I would use a flowery phrase such as "the sun set on the first day", except... it didn't:

Monday: Day 1

The first day, naturally, began with breakfast. As with all of the other meals (all of which were included in the conference price), it was an eclectic mix of regional specialities: a far more interesting mix than the generic tea-and-toast offerings of UK hotels and events. Mealtimes also proved to be an excellent opportunity to meet fellow attendees; speakers, facilitators and delegates inter-mingled throughout, and I must've eaten with 60-70 different people during my time at LT2014.

The opening keynote by Tim Lister, entitled "Forty Years of Trying to Play Nicely With Others", was simultaneously engrossing and informative. Tim told heartwarming tales of his career to date, instilling us with lessons that he's learned throughout that time.

Among some of Tim's key takeaways were:

  • The days of the "lone tester" are over. Collaboration and trust are key if you want to achieve anything.
  • The best teams are like a chorus / orchestra rock band, not like a sports team. (You are not competing with your team-mates, you are working together to harness each other's strengths.)
  • If you don't enjoy it, don't do it. (If your work is not giving you innate pleasure: use your feet.)
  • Hard challenges are good, rewarding, intellectual activities.
  • It's exciting to be around people who are way ahead of you

Tim also gave a big welcome to all of the first-time Let's Test attendees, which seemed to consist of almost one-third of the crowd, describing us as "clan members". All signs indicate that this clan is only getting stronger.

The remainder of the day was spent with James Bach and Pradeep "Panda" Soundararajan, in their hands-on workshop "Review by Testing: Analyzing a Specification by Testing the Product". It was a highly hands-on class which, in-between the slide decks and group discussions, focused around the testing of three pieces of software. It's exactly the kind of practical testing exercise that I thrive on, and it wasn't long before James said to me "Oh, you're that Neil" (our paths have crossed a few times in recent months).

Although most of my learning was directly related to analysing the particular workshop tasks - which I won't spoil here - there were some useful points for a wider audience, which I interpreted as follows:

  • If a spec exists, it's reasonable to suggest amendments to clarify the document and/or bring it in-line with actual observed behaviour. (If the actual observed behaviour is what's desired!) - "Review-by-rewrite"
  • You'll have many questions when performing this sort of exercise. Most of these are ephemeral questions which you'll solve yourself with a little more reading/exploration, so it's important (for your credibility and sanity!) that you surface questions once you are sure they're not answered in the specification.
  • Your final test output will be built on a mountain of broken/fallible test ideas. That's OK. Consider your early exploratory test activities as "rehearsals" for the final performance.
  • Given an immeasurable number of test execution options, for any given test, consider the relative value vs importance that you need to deliver. ("Should I do a $10 test? $100? $1000?)
  • Beware of encouraging the introduction of unnecessary detail into a specification. James gave an example of a spec which specified that a dialog must load in 300ms ("would it be the end of the world if it was 301ms?") when the developer really meant "the application isn't annoyingly slow" - they'd added the detail because they believed that testers needed measurable metrics.
  • When looking at an incomplete specification, we coined the term "Specification Interpolation" to describe the process of filling-in the gaps using our experiences with comparable products. (For instance, if cut-copy-paste isn't explicitly defined in the spec for a word processor, it's a good bet that it should work in much the same way as Notepad.)

After a long day of practical activities, it was a nice change of pace to view the "Lightning Talks" session. I'd considered doing a talk myself (and had even sketched-out a couple of topics in the Heathrow departure lounge) but I wimped out of it, on the basis that my talks were too big to cover in 3 minutes. Hopefully I'll give these talks at UK meetups some time soon, to give me the confidence to speak to a larger group next year.

The talks were lively, varied, and occasionally heated. Each person spoke for 3 minutes, followed by another 3 minutes of follow-up questions. This meant that the room guided the discussion, in a forum-style format which was familiar to me from events such as UKTMF. It was an invigorating experience, and one at which I plan to talk next year.

After digesting the talks and dinner, it was time for a first visit to the Test Lab. James Lyndsay is renowned for his intriguing black-box testing machines, many of which can be found online on his Workroom Productions website. In the company of Ru Cindrea, the lab visitors were treated not only to some of these classic exercises, but also some physical machines, and a pair of colour-matching robots. We tested late into the night, trying to deduce the behaviours of these mysterious devices.

I turned-in about midnight, but discovered that many people stayed in the lab and lounge area until gone 3am. I discovered that a lot of the best discussions were happening after-hours, and resolved to participate until the very end on Tuesday.

And no, it still didn't get dark.

Tuesday: Day 2

We began with a very different session; a 9am facilitator dance-off (featuring Cha-Cha Slide and Gangnam Style) should've been a clue that we were about to witness something quite unique...

Steve Smith's experiential keynote, "Whose Ideas Are In and Whose Ideas Are Out" was the most fun that I had in the entire day, or perhaps in many days.

We were split into teams of 12, with each team given an A2 sheet of flipchart paper. We had 5 minute to prepare, after which we had to stand as many people on the paper as possible, with 1 point for every person who wasn't touching bare floor. The position had to be held for 15 seconds.

Before the task began, we were allowed to proffer questions to Steve; I asked whether we were permitted to tear-up the paper. I was told that this would be OK, and quickly formulated a strategy. I shared the idea with my team: we split the paper into 12 individual foot-sized chunks, meaning that everybody could stand on 1 foot for 15 seconds within their own personal space, overcoming the need to squeeze 12 bodies into the area of the paper.

With it being my idea, I found myself leading the exercise a lot more than I intended. There was team consensus that this was a good idea, although there was some doubtful mumbles when I started to rip-up the paper; we were committed now, there was no turning back.

It proved to be an effective technique, with all of our team surviving the first round. We were then told that there would be a 2nd and 3rd round, during which each piece of paper would be folded in half; this played into our hands, as we didn't experience the massive reduction in floor-space which the other teams faced. (In fact, at least one team adapted to our strategy in round 2.) We modified our technique slightly in these later rounds, moving closer together so that we could use each other for balance. We had two fallen comrades in round 2, but this made us stronger in round 3, as those of us with large feet were able to utilise the extra scraps of paper from our eliminated colleagues. It was enough to propel our team into the silver medal position, just a couple of points behind the winners.

Following the activity, we discussed the activity as a group, specifically referring to ideas which we'd chosen, ideas we'd rejected, and how we thought they'd worked. I found myself acting as scribe; again I hadn't particularly intended to, I guess I was just settling into the role :)

Finally, each team was given 60 seconds on-stage, with a spokesperson describing their approach. I was nominated as our spokesperson, as the originator of the idea - they agreed that if the idea had failed miserably, I would've had to carry the brunt of that too!

The 60 seconds flew by, but (as Steve subsequently pointed out) I can now say that I gave a keynote speech at Let's Test, which is a lofty claim!

I was already physically and emotionally exhausted, but it was only 11am and we had a whole day of sessions ahead of us. We kicked off with Alessandra Moreira's presentation "The DIY Guide to Raising the Testing Bar", which (like Tim Lister's talk) was primarily an autobiographical talk interspersed with some interesting discussion points.

Here's a brief bulleted summary:

  • Alessandra talked of her "accidental tester" status, her roots in scripted testing, and her ISTQB qualifications. They're common traits these days - Joel Montvelisky recently wrote an article Are We All Accidental Testers? along similar lines.
  • Her career took a turning point after acquiring a mentor, with her timeline showing a sudden shift from "nothing much" to a subsequent increase in conferences, meetups, blogs and published articles.
  • There was discussion (in the presentation and the subsequent open forum) about what it means to "be an expert". Softer terms were suggested, such as "have experience" or "have expertise"; and the expertise might mean knowing the best person for a task (even if it's not you).
  • We talked about what we can do to shape our careers: how to gain reputation, participate in active learning, and spread this to work colleagues and local communities.
  • I asked whether this session was perhaps "preaching to the choir", as we had all chosen to attend the conference, and in particular to attend this session! Although I was using the statement to lead into a further point, there were useful follow-ups from James Bach, Scott Barber and Dawn Haynes, saying that "sometimes the choir needs the message" and that there is value in creating shared experiences/bonds.

There was some discussion about the Weekend Testing movement, which has active chapters in America, India and Aus/NZ, but which isn't currently running in Europe:

With Twitter being as awesome as it is, I was quickly put in touch with the major figures who'd previously been involved in running Weekend Testing in Europe, and (post-conference) I'm hoping that I can spark a revivial, and maybe even lead it (if people are interested).

The next session that I chose was Carsten Feilberg's seminar on "Getting Issues Sorted". We began by each sketching a difficult scenario that we were experiencing in our workplace, drawing a digaram of impacted parties (internal and external) and their relationships/hierarchies. We then role-played several of these scenarios, to help to bring them to life and allow us to visualise where bottlenecks might be occurring.

Although my scenario wasn't role-played, and I can't reproduce the photo online in case it's business-sensitive, I found it rewarding to sketch-out my own situation. It illustrated some things that I already knew (that we have complex inter-relationships between multiple product teams) and some things which I perhaps hadn't previously given enough credence (these teams report-up to common product leaders). I'm confident that I could use the information from this session to better understand business concerns in the future.

The programme ended with a session from Christin Wiedemann and Martin Hynie, "Can playing games actually make you a better tester? Exploring the science behind games". With so many of the Let's Test extra-curricular activities involving testing games, such as the ever-popular Set and Zendo, it seemed prudent to look into whether there is proven benefit to them. We saw evidence from several studies, including MRI scans performed by Christin and Martin in their own empirical research, demonstrating that (in all analysed cognitive activities) the brain patterns of testers, and games-players, were considerably "cooler" than those of people outside the field, or those who did not play games.

In the evening, we had a unique opportunity to view a webinar recording in progress. Rob Sabourin's online course, Generating Great Testing Ideas, was being broadcast live from Runö, and we were invited to sit-in to watch it, and ask questions as we saw fit. Given that online attendees had each paid several hundred dollars to participate in the course, it was another example of just how much valuable content the Let's Test organisers were looking to squeeze in.

Once the webinar had finished (at 10pm - some eleven hours after the morning's experiential keynote), I fully intended to head to the test lab and participate in 'Games Night'. However, I found myself coming over extremely cold and shivery, and instead found myself under a duvet wearing a fleece and attempting to regulate my temperature. I hoped that it was just the exertions of a long day (well, 3 long days) and further hoped that I didn't miss out on too much in the wee small hours.

Wednesday: Day 3

Thankfully, it seemed that nine hours of sleep was enough to shake my horrible feelings of the night before. That said, I didn't rise until after breakfast, although there were lots of similar-looking faces in the lobby when I emerged at 9am; I think the drinkers may have felt worse than I did... (I know of several who were kicked-out at about 4am!)

My first session of the day was Fiona Charles' workshop "We can't know everything - Promoting healthy uncertainty on software projects". This was a really well-structured, substantial session where (in groups) we surfaced some of the everyday uncertainties that we face, discussed how we could communicate this uncertainty without sowing panic, and considered some ways in which we could become comfortable with a level of uncertainty.

The shared experiences of this session were typical of the very best of Let's Test. We were able to gain confidence from seeing that so many people (in different countries/organisations) are battling the same issues, and to appreciate that uncertainty is a way of life - it's all about how you come to accept it and embrace it.

Here were some of my group's examples of times when we expected to feel uncertainty:

We talked about how we could communicate that uncertainty to stakeholders in a positive manner, and how we can learn to deal with uncertainty in a more rational fashion:

We also had a brief interlude from Eddy Bruin, introducing us to the Orders of Ignorance:

As with the other days, most of the time during breaks/lunches was spent comparing notes from the other workshops. There's so many excellent sessions at Let's Test; it's not hard to pick a good one, the bigger challenge is to make peace with the ones that you'll miss out on.

In this particular break, I caught-up with some of the people who were in Alan Richardson's workshop "Why you should learn skills that have no real application in life". After discussing some of the skills that were on-show, I was persuaded (thanks to the nearby breakfast buffet) to attempt the trick where you can blow a hard-boiled egg out of its shell. Sadly, the egg I chose was quite soft-boiled. Needless to say, mess ensued.

The final track session that I opted for was by Anna Royzman: "Quality Leader: The Changing Role of Software Testers". I expected this to closely reflect what I'm seeing in my own organisation, which indeed it did. We discussed whether the required skills for a tester are now very different from the days before agile (spoilers: they are). We talked about the rise of testing as a facilitative role, and exchanged examples of our own experiences. In pairs, we rated ourselves Top Trumps-style on a variety of different skills, and (where one of us outscored the other) exchanged our tips and advice on how to improve those areas. It was the sort of rich, personal discussion which is often neglected in less community-led conferences.

With our luggage in hand and a heavy heart, we proceeded to the main hall for Jon Bach's closing keynote: "A Critical Look at Best Practices". Again it was something of an experimental keynote, as its content was brainstormed and developed by students in Jon's workshop at Let's Test. (Of course, the slides and discussion gained their inimitable Bach coat of paint.)

The workshop group came up with 64 ideas (when asked to imagine they were "an elder council" requested to lay down the best practices for testing), which they voted-down to a top ten, and then elaborated on them. There was wisdom in the list, but that's because the list wasn't really practices; they mostly reflected desirable skills/qualities of testers which could be developed.

During the talk, we gave consideration as to whether any practice (in any walk of life) could truly be called "best"; can't we always think of a situation where the "best" practice is inappropriate? This led to a lively open discussion, which (while aided by microphoned facilitators, Stephen and Richard) gained a life of its own thanks to those who could project their voices without need of a mic!

We each had to write a letter, for the Let's Test team to post to us in 3 months, featuring our thoughts on where we wanted to take our careers next. I wrote it in a hurry but I'm sure that it included the following:

  • Give talks at my local meetups (Nottingham, Cambridge)
  • ...possibly using my lightning talk material that I didn't use
  • ...or feeding-back my Let's Test learnings to a larger group
  • Help to revive (and facilitate) the Weekend Testing group in Europe
  • Blog more
  • Be happy

And with that, my time in Runö was over, and (after exchanging many goodbye hugs and handshakes) it was into an airport-bound taxi. During the journey I chatted with my fellow taxi-mates, Bolette and Morten, who told me how they wanted to recreate the atmosphere of Let's Test at their own event, CopenhagenContext, which is running for a second time in February 2015. It's great to see so many people spreading the context-driven message in their own neighbourhoods, and the conversation allowed us to begin reflecting and absorbing what we'd experienced during the past four days.

So, in conclusion...

Well done if you made it this far. I won't keep you here much longer.

My sales pitch is also the same as I gave to my wife, to explain why she didn't want to be here: "Imagine an entire resort filled with people like me." In truth, she would have found some wonderful activities to relax/recuperate, and several people had families in attendance, but my central point remained true: There is no escape. This is a non-stop event, for testers, by testers.

It was like being in a hotel where every guest was your friend. Remember that time when you were on holiday, and you randomly bumped-into that person you knew, a thousand miles from home? Now imagine that repeated continuously for the duration of the conference. Tim Lister's reference to "the clan" of Let's Test was absolutely accurate, as I spent several days in close family bonds which, less than 6 hours later, I'm already missing.

More than that, it was a real all-star cast from the world of testing. In front of me is Jerry Weinberg's Perfect Software book, which contains recommendations from Michael Bolton and Fiona Charles on the cover. Just hours ago I was having a face-to-face chat with both Michael and Fiona. To quote Emma Keaveny from her TestBash 99-second talk, there was some serious "testing royalty" at the event.

Every session (and even the breaks, meals and evenings outside of these sessions) was tailored to include audience participation; there was a well-thought-through system for asking questions, involving colour-coded cards (I played several green 108 cards, and a couple of yellow 108s for good measure). The structure gave me more confidence in making my voice heard, and (as Alessandra Moreira's talk reinforced) I learned that I shouldn't assume I don't have anything interesting to say, or (as Tim Lister espoused) I shouldn't keep my head down just because there are more experienced/knowledgeable people in the room. I also found myself taking a leadership role in several activities, something which is often outside my comfort zone.

The more long-lasting effects of Let's Test will take a while to realise themselves, as my main learnings will impact upon my analysis and decisions when faced with particular challenges. I feel much better-equipped to deal with such situations in the future, both emotionally and as a professional tester.

I spoke to many people from different countries, industries and backgrounds during my time at Let's Test. I'd estimate that I had meaningful conversations with at least 50 people, and brief encounters with another 50. I discovered that my name (and... sigh... my "brand") is increasing, as I was surprised by the number of people who knew who I was, and who I worked for. (I even had one conversation with 3 people where everybody knew of Towers Watson, and another where I was asked specific technical questions about our software portfolio!)

I've skimmed-through the post to see whose names I haven't mentioned, and here are a few that I've ommitted. To those who still aren't featured on the list below: I humbly apologise. It's late, I've had precious little sleep, and {insert excuse 3!} - I will repay you in the future, I promise.

  • Sandro Ibig - the first of the Twitterati to greet me on Sunday, we spent a lot of time together, particularly outside sessions, and a great three hours in the Test Lab collaborating to crack the code of the black box machine.
  • Graham Maxfield - another person who seemed to be everywhere I went, reflecting the fact that we seem to be facing similar challenges in our working lives. One of the most common-sense, matter-of-fact personalities that I encountered; Graham, you're not active on Twitter, but I hope you'll keep in touch. (Or, y'know, get active on Twitter. I did it. It's not that painful.)
  • Alexander Simic - Alexander and I had previously worked together for several years, and re-met online during January's Rapid Testing Intensive Online course. Alexander is now one of the most prominent members of our community, and he was always there to make me feel welcome during Let's Test. We had a particularly meaningful in-depth chat about the state of our respective careers.
  • Members of the UK community who I'd met before, including Richard Bradshaw, Stephen Blower, Amy Phillips, Alan Richardson, James Lyndsay.
  • Some UK members that I was meeting for the first time: Bill Matthews, Christopher Chant, Benjamin Kelly, ChengVoon Tong.
  • Well-known members of the wider community that I'd never met before: Pradeep Soundararajan, Kristoffer Nordstrom, Dawn Haynes, Helena Jeret-Mäe, Scott Barber, Jon Bach, Michael Bolton, Rob Sabourin, Zeger Van Hese, Geir Gulbrandsen, Ruud Cox
  • ...and some that I had met before (and whose reputation precedes them): Huib Schoots, James Bach
  • Lots of people who, for my sins, I hadn't crossed paths with previously. (This list is particularly fuzzy, as it mostly relies on whether I could find your Twitter account and add you to my "people I met at Let's Test" list): Erik Davis, Eddy Bruin, Viktor Johansson, Arjen Verweij, Chris Blain, Lars Sjödahl, David Greenlees, Meike Mertsch, Martin Nilsson

Again: If I neglected to mention you, I am truly sorry. Next year I'll devise a better system for remembering everybody. (Hint: A printed delegate list would be a massive help.)

Yes, of course there'll be a next year. I'm already looking forward to Let's Test 2015, but (just as excitingly) I'm looking forward to the next year of my life, for the experiences which will fuel my thoughts when we next all meet. Until next time...

  • LinkedIn
  • Tumblr
  • Reddit
  • Google+
  • Pinterest
  • Pocket
Comments powered by Disqus