Category

agile

SimCity BuildIt – A Lean Software Training Ground

Confession time. I’m addicted to SimCity BuildIt.

Ok, with that out of the way I want to talk about how this game is making me more conscious of lean software development principles. The gameplay is pretty simple – you construct roads and buildings, then produce raw materials and manufacture goods to complete missions and upgrade buildings.

Simple. But there are some challenges. One challenge is that the time it takes to produce of materials and goods varies by the type. This complicates my life because I don’t know what I’ll need in order to complete a mission ahead of time. So, if a mission comes up which requires 5 watches, for example, I’m in trouble. A watch is built from a chemical, a glass, and a couple plastic materials. Chemicals take 2 hours to make, glass takes 5 hours, plastic takes 15 minutes. The watch itself takes a couple hours and I can only produce 1 watch at a time.

If I go full-blown lean, I’ll do this just-in-time and the first watch will take about 7 hours to build, then each additional one will take 6 hours (I can produce up to 45 materials in parallel, so the material production time is equal to the longest part, in this case the glass).

Here’s the problem. If the mission only lasts 3 hours I won’t be able to complete it. What’s a mayor to do?

I have the ability to carry a limited set of inventory, so I could pre-produce some of these more ‘expensive’ items and hang on to them for when they’re needed. The drawback is that this takes up shelf space and I have no idea when or how many items I’ll need. If I fully stock the shelves but end up needing another item, I’ll have to get rid of some of my fully produced items.

I can purchase fully produced items from other cities. I don’t have to carry the inventory, but I pay a premium in this case and there may not be any available when I need it.

GET TO THE POINT, NICK

Ok, I hear ya. So what does any of this have to do with software development?

One of my all time favorite software development books is Lean Software Development by Mary and Tom Poppendieck.

I learned that overproduction or producing things that nobody needs is waste. We should be building the smallest increment of an application, measuring the value it returns, and iterating on that. Spending a month adding a feature that nobody uses is a month of lost time that could have been used to produce something of value.

Along the same lines, producing something and letting it sit on a shelf provides no value. Small batch sizes can help with this. Often, we believe that we can’t release a software product until it’s “done”. So we build all the features we can think of, then push them all out at once. While we’re holding all of those completed features in inventory, we’re not realizing any value from them.

Another obvious, but seldom recognized, truth of software development is the effect of the theory of constraints. When building an application we’re only as fast as the slowest part of the process. If I need to produce cheese in SimCity, for example, it’s only a couple of hours – but to produce the raw materials is 5 hours. So what is a 2 hour process on it’s own is actually 7 hours end-to-end.

The theory of constraints shifts our focus to optimizing the slowest part of a system. Instead of compartmentalizing the development process into analysis, design, development, testing, release, etc, see the process as one piece. It doesn’t matter if we can test 20 things per week if we can only develop 4. Optimize the whole.

We have a lot of tools to visualize and optimize our processes. Value Stream Maps can show us where our time is actually spent from start to finish. Cumulative Flow Diagrams can visualize bottlenecks in our workflow. There’s nothing I can do to speed up the production of electrical components, but we have complete control over everything we do when building a software project. The hardest part can be identifying where to focus.

Well, my appliances are done being made. Time to go upgrade a building in my city!

Look Mom, No Hands! Test Driving Code Without A Mocking Framework

I love TDD. I haven’t found a more effective way to incrementally design and build software in the 15+ years that I’ve been doing this. I have formed and evolved a lot of opinions about how I approach TDD, though.

Recently, I wrote a post for EuroStar Software Testing titled Look Mom, No Hands! Test Driving Code Without A Mocking Framework

This is a topic that has been on my mind for a long time. It’s not intended to start a mocks vs stubs flamewar or anything like that. Instead, I wanted to walk through my progression of TDD practices over the years and share what I’ve learned.

Don’t get me wrong – test-driving with a mocking framework is better than not test-driving at all. I just prefer stubs.

Looking back at the test cases in the Booked source code which utilize PHPUnit’s mocking framework (yes, there are still a lot), I can see just how entangled the test code is with the implementation of the production code. The source for Booked changes frequently and it is covered by more than 1000 unit tests. New features are introduced and, occasionally, some of the unrelated tests fail.

They fail because there is too much specified in the mock setup. In order to validate the behavior of some area of the code, I have to set up unrelated mock expectations to get collaborating objects to return usable data. If I change the implementation of an object to no longer use that data, my test shouldn’t fail.

A couple of years ago I stopped using PHPUnit’s mock objects and I’ve seen the resiliency of my unit test suite increase. I’ve also seen my development speed and design quality improve. Instead of ambiguous mock expectations scattered throughout the tests, I’ve built up a library of stub objects which have logical default behavior.

When test-driving increments of functionality, I’m able to concentrate on the behavior that I need to implement rather than getting distracted with test setup and management.

More focus. Better design. Higher quality. No mocks.

We Built the Wrong Thing – From Ambiguity to Stability

Let’s set the scene. You’re out to lunch with your team celebrating a successful launch of a new feature. Your product owner interruptions the conversation to relay an email from a disappointed stakeholder.

From: Stakeholder, Mary (mary.stakeholder@nickkorbel.com)
Sent: Thursday, April 19, 2018 11:51 AM
To: Owner, Product (product.owner@nickkorbel.com)
Subject: Can we talk?

Thank you for all of your work, but this doesn’t do what I thought it would. Can we talk?

– Mary

The discussion around the table quietly shifts to how nobody ever knows what they want. “We followed all the agile best practices”, a senior developer frustratedly quips, “How did we build the wrong thing?”

What went wrong?

When you get back to the office, you huddle up with Mary and pull up the acceptance criteria.

Story: Adding events to a calendar
As a user
I want to enter events into a calendar
So that everyone knows when people are available
Scenarios:
Given I've entered an event into the calendar
When I view the calendar
Then I can see that event

“This is what you asked for, right?”

Mary replies, “Yes – but it’s not what I wanted.”

“What do you mean?”

“Look at this. I want to set up a 3 day training session, but I only have one date picker. And every new event is the same color, so it’s really hard to see who is booked when. And I have no way to know when a new event is created. And…”

“Oh.” you interrupt. “We didn’t know you wanted that. You had all of those meetings with our PO. Why didn’t you ask?”

Mary, now frustrated with the amount of time seemingly wasted, responds “I thought we were all on the same page!”

Specification by Example

Is this a familiar story? Even using the de-facto acceptance criteria format so popular in agile, it’s very easy to build ambiguous expectations. Ambiguity leads to disappointed customers and frustrated developers.

Years ago, I read Gojko Adzic’s Specification by Example and it changed the way I view user stories. I cannot possibly do justice to all of the incredible advice and ideas from the book in the blog post, but I’ll try to summarize.

Instead of a PO or BA working with customers to capture the stories and later reviewing those stories with developers, Gojko recommends running specification workshops. We follow a simple workflow for this:

Derive scope from goals > Specify collaboratively > Illustrate requirements using examples > Refine specifications > Frequently validate the application against the specifications

Deriving scope from goals is probably the biggest change a team will need to make. Instead of being presented a set of acceptance criteria, the team is presented with a goal. For example, stating a goal to know people’s availability instead of the scope to build a calendar.

Working with the stakeholders, the team collaboratively identifies the acceptance criteria. Maybe a calendar is what is built. Maybe it’s a simple list. Maybe it’s a search. The point is that we start with the goal in mind, and collectively identify the scope. This eliminates the translation layer between stakeholder to product owner to development team.

Ambiguity— —

The next couple steps are iterative. We extract real-world examples from the scenarios, and illustrate the acceptance criteria using those examples.

Instead of

Given I've picked a date
When I book that date
Then that date is booked

We have something like

Given Mary has selected 10:00 am on April 18th, 2018
When she completes the booking
Then the calendar indicates that Mary is unavailable on April 18th 2018 between 10:00 am and 10:30 am

It’s only a slight change, but it has massive effects. Using real examples leads to real questions. What if Mary is already busy at that time? What kind of indication should we show? Is the default event length 30 minutes? Can that be changed?

Ambiguity— —

And here’s where it gets fun

Most teams write automated end-to-end tests for their applications, but a lot of the time these tests are defined and written after the functionality is built. We end up simply validating that what we built works how we built it. Even if the tests are built based on more traditional acceptance criteria, the person writing the test has to make some assumptions about how to make the application behave in the way that meets the criteria.

If we have a Cucumber feature file that looks like this:

Story: Adding events to a calendar
As a user
I want to enter events into a calendar
So that everyone knows when people are available
Scenarios:
Given I've entered an event into the calendar
When I view the calendar
Then I can see that event

The person implementing the tests has no choice but to make up some dates to pick and the validation will likely be something generic.

When writing automated acceptance tests based on real-world examples, the tests can match the acceptance criteria 1:1. Not only does this enhance the clarity of how to test the application, it also brings gaps in the shared understanding of a story to light early.

Story: Adding events to a calendar
As an event organizer
I want to be able to indicate any events I'm participating in
So that everyone knows when I am available
Scenarios:
Given Mary has selected 10:00 am on April 18th, 2018
When she completes the booking
Then the calendar indicates that Mary is unavailable on April 18th 2018 between 10:00 am and 10:30 am

Ambiguity— —

Automating the Acceptance Criteria

One common frustration of test automation is maintenance and fragility. Features change and evolve over time. When tests are driven from an interpretation of the specifications rather than the the actual specifications, maintenance becomes a challenge. It’s difficult to trace a specification change to an associated test (or set of tests). So minor changes in specifications tend to have major impacts to tests.

If the specifications are automated, instead of translated into automated tests, you know exactly what test is affected. In changing the specification, you are forced to change the test and underlying code. You can make micro changes and receive instant feedback that the application still works.

Stability++

No silver bullets

This isn’t an overnight change. Like most things, it takes deliberate practice. Practice facilitating specification discussions with non-technical people. Practice with finding the right type and number of examples.

The return on this investment can be huge. Specification workshops often lead to significant reduction in scope because technical people and business people are speaking the same language and understand the problem in the same way.

The resulting specifications are free of ambiguity, so everyone has a shared understanding of the exact behaviors they should expect from the application. Validating the application against the specifications in an automated way ensures the application is always working the way everyone understands and expects.

Eliminating the specification ambiguity builds a shared understanding between everyone involved, which leads to long term application stability. And that’s good for everyone.

Have you tried this?

I’m interested in hearing from readers about their experiences. Have you tried this or something similar? How did it go?

How Transaction Costs Influence Story Size

Of all the aspects of the INVEST principle, I think the attribute that software developers have the most influence over is S. Small

Small stories have huge benefits. We can control what gets built with much more granularity.  Less scope means fewer requirements for the team to understand. Less for the end user to understand. Less code to for the developer to change. Less code to test. Less code to release and maintain.

Less risk.

Ron Jeffries recently tweeted a short video on story slicing that triggered a question in my mind about why engineers resist small stories. As I thought about it more, connections formed to Don Reinertsen’s work in  Managing the Design Factory and Principles of Product Development Flow. There he describes the “transaction cost” of an activity. The transaction cost is the cost/time of everything needed to execute an activity.

If we put this in the context of releasing software, this may include the time to build, test, and deploy an application. If those tasks are manual and time consuming, human nature is to perform these tasks less often.

Transaction costs follow a U-curve, where performing an activity may be prohibitively expensive if the size of the activity is too small or too big.  For example, think about a story for user account registration. It would be costly to create a story for each word in the form text. Likewise, the cost of a story would be huge if it includes creating the form, validating it, sending notifications, account activation, error handling, and so on.

So, back to Ron’s video and push for thinly sliced stories. I posed the question about what to do when developers resist small stories. To dig into that, we need to understand why developers resist small stories.

A common argument that I hear against small stories is that it’s not efficient to work that way. We developers are always looking to maximize the work not done 🙂

This can be a problem, because the true work for a story includes everything to get it production ready. Writing each story. Developing each story. Code reviewing each story. Testing each story.

That is the transaction cost of a story.

Driving down those costs drives down the resistance from the developers. There are lots of ways to reduce these costs. We can pair program to eliminate the out-of-band code review. Use TDD and automated acceptance testing to build quality in from the start. Create an automated build and deploy pipeline to continuously deliver stories.

As we reduce the overhead of each story, we can slice stories thinner and thinner.

But wait, there’s more!

As Bill Caputo rightfully pointed out, big “efficient” stories include a lot more work than may be necessary. Thinking back about an account registration story, we may put account registration and account activation into the same story. That builds in an assumption that we have to activate accounts – which we may not.

Not splitting stories means we may efficiently build something we don’t even need. Peter Drucker famously said –

There is nothing so useless as doing efficiently that which should not be done at all.

Addendum – Be careful what you measure

Another reason developers may resist breaking things down into small deliverables is how they’re measured.

In most scrum worlds, a team is being measured based on velocity. Small stories are fewer story points, thus lower velocity.

Uh oh, let’s make bigger stories and drive higher velocity.

To address this problem, Ron suggested giving “credit” for stories completed rather than story points completed. This encourages splitting large stories into thin stories with little pieces of the overall functionality.

What Do DevOps, Test Automation, and Test Metrics Have in Common?

This is a guest post by Limor Wainstein

Agile testing is a method of testing software that follows the twelve principles of Agile software development. An important objective in Agile is frequently releasing high-quality software regularly, and automating tests is one of the main practices that achieves this aim through faster testing efforts.

DevOps is a term that Adam Jacob (CTO of Chef Software) defines as “a word we will use to describe the operational side of the transition to enterprises being software led”. In other words, DevOps is an engineering practice that aims at unifying software development (Dev) and software operation (Ops). DevOps strongly advocates automation and monitoring at all stages of the software project.

DevOps is not a separate concept to Agile, rather, it extends Agile to also include Operations in its cross-functional team. In a DevOps organization, different parts of the team that were previously siloed collaborate as one, with one objective to deliver software fully to the customer.

Agile and DevOps both utilize the value of automation. But there must be a way to measure automation, its progress, and how effective it is in achieving the aims of Agile and DevOps—this is where test metrics become useful.
In this article, you’ll find out about different types of test automation, how automating tests can help your company transition to DevOps, and some relevant test metrics that Agile teams and organizations transitioning to DevOps can benefit from to improve productivity and achieve their aims.

Types of Test Automation

Test automation means using tools that programmatically execute software tests, report outcomes, and compare those outcomes with predicted values. However, there are different types of automation that aim to automate different things.

Automated Unit Tests

Unit tests are coded verifications of the smallest testable parts of an application. Automating unit tests corresponds with one of the main Agile objectives—rapid feedback on software quality. You must aim to automate all possible unit tests.

Automated Functional Tests

Functional tests verify whether applications do what the user needs them to do by testing a slice of functionality of the whole system in each test. Automating functional tests is useful because it saves time—typical testing tools can mimic the actions of a human, and then check for expected results, saving valuable time and improving productivity.

Automated Integration Tests

Integration tests combine individual software modules (units) and test how they work together. Again, by automating integration tests, you get tests that are repeatable and run quickly, increasing the chances of finding defects as early as possible, when they are cheaper to fix.

Test Automation and DevOps

DevOps is a culture that aims to reduce overall application deployment time by uniting development and operations in software-led enterprises. Automation is at the heart of the DevOps movement—reduced deployment time and more frequent software releases means increased testing volume. Without automation, teams must run large numbers of test cases manually, which slows down deployment and ensures the DevOps movement does not succeed in its aims.

One potential pitfall that can hamper the transition to DevOps is not having the required automation knowledge. After all, test automation is technically complex. Acquiring the knowledge to effectively automate tests takes time and costs money. You can either hire expert consultants to get you up and running with automation, hire qualified automation engineers, or retrain current testing staff. Whichever option you choose, it’s clear that automating is essential for implementing a DevOps culture in your development teams.

Enter Test Metrics

What Are Test Automation Metrics?

Implementing automation blindly without measuring it and improving on automated processes is a waste of time. This is where test metrics provide invaluable feedback on you automated testing efforts—test automation metrics are simply measurements of automated tests.

Test automation metrics allow you to gauge the ROI from automated tests, get feedback on test efficiency in finding defects, and a host of other valuable insights.

How Can Test Automation Metrics Help DevOps?

By measuring testing duration, you find out whether current automation efforts are decreasing development cycles and accelerating the time-to- market for software. If tests don’t run quicker with automation than manual tests, then there are clearly technical issues with the automation efforts—perhaps the wrong tests are being automated.

How to Measure Test Automation

Some examples of test metrics used to measure test automation are:

  • Total test duration—a straightforward and useful metric that tracks whether automation is achieving the shared Agile and DevOps aim of faster software testing through increased automation.
  • Requirements coverage—a helpful metric to track what features are tested, and how many tests are aligned with a user story or requirement. This metric provides insight on the maturity of test automation in your company.
  • Irrelevant results—this measurement highlights test failures resulting from changes to the software or problems with the testing environment. In other words, you get insight on the factors that reduce the efficiency of automation from an economic standpoint. Irrelevant results are often compared with useful results, which are test results corresponding to a simple test pass or test failure caused by a defect.

Closing Thoughts

The DevOps movement extends Agile and aims to modernize software development with faster releases of high-quality software. Testing is a common bottleneck in the development cycle, which can hamper any attempt to integrate a DevOps culture.

Test automation is the link that gets software testing up to the speed of development, helping to achieve the aims of Agile and DevOps.

However, there must be a way to track all attempts to automate tests, since test automation is, in itself, an expensive investment and a technical challenge. Test metrics provide valuable feedback to help improve automation and ensure positive ROI.

About Limor

Limor is a technical writer and editor at Agile SEO, a boutique digital marketing agency focused on technology and SaaS markets. She has over 10 years’ experience writing technical articles and documentation for various audiences, including technical on-site content, software documentation, and dev guides. She specializes in big data analytics, computer/network security, middleware, software development and APIs.

When will you be home?

What time will you be home today? It’s a simple question. If you’re like most people you go to work and come home about the same time every day. You should be an expert in estimating when you’ll arrive at home. I bet you cannot predict the time that you’ll get home tonight, though. I bet you’d be even less accurate if I asked you to predict what time you’ll be home six months from now.

Sometimes I feel like this is what we ask software teams to do when we ask them for estimates. Software developers build software every day. We should be able to predict how long it will take to build new software – but we can’t reliably do it! There are ways to improve our estimates, though.

Let’s jump back to the commute example. If your commute is short then you’ll be accurate most of the time. Sometimes you’ll have a meeting that runs late. Maybe it’s Friday and your calendar is clear so you take off early. You can be way off from time to time, but pretty often you’ll be pretty darn accurate. This is why we want to break deliverables into small pieces. Small pieces have less uncertainty.

Things get less predictable the further out you go. Going from a half-mile commute to a mile commute will increase the variability. One mile to twenty miles increases that variability by orders of magnitude. There are simply more unknowns as the size of your commute grows. Traffic, accidents, weather – there are a lot of variables that can affect the actual time.

Big features and deliverables have this same problem. There are simply too many unknowns to be accurate. Undocumented limitations of libraries, complex algorithms, changing requirements, for example. And asking software developers to commit to anything with that level of variance isn’t fair.

But it’s done anyway and leads to all kinds of dysfunctions. Unrealistic estimates based on little knowledge are treated as promises. Broken promises lead to distrust. Distrust leads to infighting, incessant status reporting, and pissed off developers.

So lets stop asking for estimates that are months away. I don’t know even know what time I’ll be home tonight 🙂

TDD: Back to Hand Rolled Stubs

I’m unashamedly an Agile practitioner and self-proclaimed enthusiast. It’s not a perfect way to build software, but I haven’t found anything better. And until I do, this is the approach I’m taking.

Building Quality In

One of the core principles of Agile is the focus of building quality in from the start, not asserting quality after the product is built. At first this is counter-intuitive, right? How can you ensure the quality of something which is in the process of being built? This is where TDD saves the day. Write your tests first – set the expectations of what the code is supposed to do – then write the production code.

When I rewrote Booked for the 2.0 release I did it all test-first. In fact, there are thousands of assertions covering the majority of the code base. This is super important for a dynamic language like PHP where it is very easy to shoot yourself in the foot. Lots of tests = high confidence that my change didn’t break something. Of course, unit tests in a statically typed language are also critical.

The Unit Testing Approach

I use PHPUnit in Booked and it works great. Aside from providing your typical unit testing functionality, PHPUnit also includes test doubles. Mocks and stubs are great for substituting behavior and isolating components in your tests. It’s a technique I rely on heavily and it leads to loosely coupled code and more focused tests.

Martin Fowler has a great writeup comparing mocks, stubs and other substitution patterns.

Death By Over-Specification

When I first starting practicing TDD there weren’t a lot of mocking frameworks or libraries available. We wrote our stubs and mocks by hand and used a plain assertion framework (nUnit in the .NET space). It takes some time to write this code – you’re building new objects to stand in for existing objects. The interfaces need to match, the return types need to match, and in some cases the return object needs to be set up in a certain way. It can be a decent amount of work. My laziness is what led me to mocking frameworks.

Mocking libraries gained a lot of traction around 2006/2007. Mocks, and mocking frameworks in particular, work by setting up expectations and then returning appropriate responses when you exercise the code. This awesome because you can quickly substitute behavior of specific methods without building up a whole object.

Great! Look how fast I’m going!

So Super Fast

One problem with mocks is that they want to know everything about your code. For example, given you pass parameters x, y, z to this function then return this specific result. This is great when you care about how you execute a function – and there are valid reasons to care about that. This is not great when all you want to do is return a canned response and move on. One of the most frequent arguments I hear against unit testing and TDD is that the tests are fragile and difficult to maintain. Over-specification is generally the real cause of these problems.

An Example

Here’s a test pulled straight from the Booked test suite.

public function testBindsDefaultScheduleByMonthWhenNothingSelected()
{
	$this->page = $this->getMock('ICalendarPage');
	$this->repository = $this->getMock('IReservationViewRepository');
	$this->scheduleRepository = $this->getMock('IScheduleRepository');
	$this->calendarFactory = $this->getMock('ICalendarFactory');
	$this->resourceService = $this->getMock('IResourceService');
	$this->subscriptionService = $this->getMock('ICalendarSubscriptionService');
	$this->privacyFilter = new FakePrivacyFilter();

	$this->presenter = new CalendarPresenter(
		$this->page,
		$this->calendarFactory,
		$this->repository,
		$this->scheduleRepository,
		$this->resourceService,
		$this->subscriptionService,
		$this->privacyFilter);

	$showInaccessible = true;
	$this->fakeConfig->SetSectionKey(ConfigSection::SCHEDULE, ConfigKeys::SCHEDULE_SHOW_INACCESSIBLE_RESOURCES, 'true');

	$userId = $this->fakeUser->UserId;
	$defaultScheduleId = 10;
	$userTimezone = "America/New_York";

	$calendarType = CalendarTypes::Month;

	$requestedDay = 4;
	$requestedMonth = 3;
	$requestedYear = 2011;

	$month = new CalendarMonth($requestedMonth, $requestedYear, $userTimezone);

	$startDate = Date::Parse('2011-01-01', 'UTC');
	$endDate = Date::Parse('2011-01-02', 'UTC');
	$summary = 'foo summary';
	$resourceId = 3;
	$fname = 'fname';
	$lname = 'lname';
	$referenceNumber = 'refnum';
	$resourceName = 'resource name';

	$res = new ReservationItemView($referenceNumber, $startDate, $endDate, 'resource name', $resourceId, 1, null, null, $summary, null, $fname, $lname, $userId);

	$r1 = new FakeBookableResource(1, 'dude1');
	$r2 = new FakeBookableResource($resourceId, $resourceName);

	$reservations = array($res);
	$resources = array($r1, $r2);
	/** @var Schedule[] $schedules */
	$schedules = array(new Schedule(1, null, false, 2, null), new Schedule($defaultScheduleId, null, true, 3, null),);

	$this->scheduleRepository
			->expects($this->atLeastOnce())
			->method('GetAll')
			->will($this->returnValue($schedules));

	$this->resourceService
			->expects($this->atLeastOnce())
			->method('GetAllResources')
			->with($this->equalTo($showInaccessible), $this->equalTo($this->fakeUser))
			->will($this->returnValue($resources));

	$this->resourceService
			->expects($this->atLeastOnce())
			->method('GetResourceGroups')
			->with($this->equalTo(null), $this->equalTo($this->fakeUser))
			->will($this->returnValue(new ResourceGroupTree()));

	$this->page
			->expects($this->atLeastOnce())
			->method('GetScheduleId')
			->will($this->returnValue(null));

	$this->page
			->expects($this->atLeastOnce())
			->method('GetResourceId')
			->will($this->returnValue(null));

	$this->repository
			->expects($this->atLeastOnce())
			->method('GetReservationList')
			->with($this->equalTo($month->FirstDay()),
				   $this->equalTo($month->LastDay()->AddDays(1)),
				   $this->equalTo(null), $this->equalTo(null),
				   $this->equalTo(null), $this->equalTo(null))
			->will($this->returnValue($reservations));

	$this->page
			->expects($this->atLeastOnce())
			->method('GetCalendarType')
			->will($this->returnValue($calendarType));

	$this->page
			->expects($this->atLeastOnce())
			->method('GetDay')
			->will($this->returnValue($requestedDay));

	$this->page
			->expects($this->atLeastOnce())
			->method('GetMonth')
			->will($this->returnValue($requestedMonth));

	$this->page
			->expects($this->atLeastOnce())
			->method('GetYear')
			->will($this->returnValue($requestedYear));

	$this->page
			->expects($this->atLeastOnce())
			->method('SetFirstDay')
			->with($this->equalTo($schedules[1]->GetWeekdayStart()));

	$this->calendarFactory
			->expects($this->atLeastOnce())
			->method('Create')
			->with($this->equalTo($calendarType),
				   $this->equalTo($requestedYear), $this->equalTo($requestedMonth), $this->equalTo($requestedDay),
				   $this->equalTo($userTimezone))
			->will($this->returnValue($month));

	$this->page->expects($this->atLeastOnce())->method('BindCalendar')->with($this->equalTo($month));

	$details = new CalendarSubscriptionDetails(true);
	$this->subscriptionService->expects($this->once())->method('ForSchedule')->with($this->equalTo($defaultScheduleId))->will($this->returnValue($details));

	$this->page->expects($this->atLeastOnce())->method('BindSubscription')->with($this->equalTo($details));

	$calendarFilters = new CalendarFilters($schedules, $resources, null, null, new ResourceGroupTree());
	$this->page->expects($this->atLeastOnce())->method('BindFilters')->with($this->equalTo($calendarFilters));

	$this->presenter->PageLoad($this->fakeUser, $userTimezone);

	$actualReservations = $month->Reservations();

	$expectedReservations = CalendarReservation::FromScheduleReservationList($reservations,
																			 $resources,
																			 $this->fakeUser,
																			 $this->privacyFilter);

	$this->assertEquals($expectedReservations, $actualReservations);
}

This is 75 lines of mock setup code! We’re expecting specific parameters and have mock objects being returned all over. Let’s take a look at this same test with stubs.

public function testBindsDefaultScheduleByMonthWhenNothingSelected()
{
	$this->page = new StubCalendarPage();
	$this->repository = new StubReservationViewRepository();
	$this->scheduleRepository = new StubScheduleRepository();
	$this->calendarFactory = new StubCalendarFactory();
	$this->resourceService = new StubResourceService();
	$this->subscriptionService = new StubCalendarSubscriptionService();
	$this->privacyFilter = new FakePrivacyFilter();

	$this->presenter = new CalendarPresenter(
		$this->page,
		$this->calendarFactory,
		$this->repository,
		$this->scheduleRepository,
		$this->resourceService,
		$this->subscriptionService,
		$this->privacyFilter);

	$showInaccessible = true;
	$this->fakeConfig->SetSectionKey(ConfigSection::SCHEDULE, ConfigKeys::SCHEDULE_SHOW_INACCESSIBLE_RESOURCES, 'true');

	$userId = $this->fakeUser->UserId;
	$defaultScheduleId = 10;
	$userTimezone = "America/New_York";

	$calendarType = CalendarTypes::Month;

	$requestedDay = 4;
	$requestedMonth = 3;
	$requestedYear = 2011;

	$month = new CalendarMonth($requestedMonth, $requestedYear, $userTimezone);

	$startDate = Date::Parse('2011-01-01', 'UTC');
	$endDate = Date::Parse('2011-01-02', 'UTC');
	$summary = 'foo summary';
	$resourceId = 3;
	$fname = 'fname';
	$lname = 'lname';
	$referenceNumber = 'refnum';
	$resourceName = 'resource name';

	$res = new ReservationItemView($referenceNumber, $startDate, $endDate, 'resource name', $resourceId, 1, null, null, $summary, null, $fname, $lname, $userId);

	$r1 = new FakeBookableResource(1, 'dude1');
	$r2 = new FakeBookableResource($resourceId, $resourceName);

	$reservations = array($res);
	$resources = array($r1, $r2);
	/** @var Schedule[] $schedules */
	$schedules = array(new Schedule(1, null, false, 2, null), new Schedule($defaultScheduleId, null, true, 3, null),);

	$this->scheduleRepository->SetSchedules($schedules);

	$this->resourceService->SetResources($resources);

	$this->page->SetScheduleId(null);

	$this->page->SetResourceId(null);

	$this->repository->SetReservaions($reservations);

	$this->page->SetCalendarType($calendarType);

	$this->page->SetDate($requestedDay, $requestedMonth, $requestedYear);

	$this->page->SetFirstDayStart($schedules[1]->GetWeekdayStart());

	$this->calendarFactory->SetCalendarMonth($month);

	$details = new CalendarSubscriptionDetails(true);
	$this->subscriptionService->SetDetails($details);

	$this->presenter->PageLoad($this->fakeUser, $userTimezone);

	$actualReservations = $month->Reservations();

	$expectedReservations = CalendarReservation::FromScheduleReservationList($reservations,
																			 $resources,
																			 $this->fakeUser,
																			 $this->privacyFilter);

	$this->assertEquals($expectedReservations, $actualReservations);
	$this->assertEquals($month->FirstDay(), $this->repository->_SearchStart);
	$this->assertEquals($month->LastDay()->AddDays(1), $this->repository->_SearchEnd);
	$this->assertEquals($calendarType, $this->calendarFactory->_CalendarType);
	$this->assertEquals($requestedYear, $this->calendarFactory->_CreateYear);
	$this->assertEquals($requestedMonth, $this->calendarFactory->_CreateMonth);
	$this->assertEquals($requestedDay, $this->calendarFactory->_CreateDay);
	$this->assertEquals($userTimezone, $this->calendarFactory->_CreateTimezone);
	$this->assertEquals($month, $this->page->_BoundMonth);
	$this->assertEquals($details, $this->page->_BoundSubscription);
	$calendarFilters = new CalendarFilters($schedules, $resources, null, null, new ResourceGroupTree());
	$this->assertEquals($calendarFilters, $this->page->_BoundFlters);
	$this->assertEquals($this->subscriptionService, $this->subscriptionService->_ScheduleId);
}

This is still a huge test (that’s a problem for another post). Here the stub setup code gets cut down to about 15 lines. What I also love is that I don’t have expectations set in the mock specification. Any assertions around parameters have been moved down with the rest of the assertions. This is wonderful for a few reasons.

1 – This follows the Arrange-Act-Assert model that helps us in writing clean tests.
2 – Most of the time I don’t care about what parameters I pass to a method.
3 – Mocks will typically not return anything if you don’t pass the right value to a mocked-out function. When you pass the wrong value null is returned, causing a null reference exception in your code. This is very frustrating and difficult to trace, especially when you changed the parameters intentionally.

I don’t care which method is called (most of the time). Often times I don’t care about the data that we’re feeding to a collaborating object, either. I just want to test this specific object’s functionality and assume that everything else is tested elsewhere and works as it should.

The Stub Library

One of the biggest ancillary benefits I’ve found to hand writing my stubs is that I’ve built up a pretty solid library of reusable objects. I have stub objects ready to go for different pages, for reservations, data access and so on. I can pull in an existing stub and continue on with my test. I don’t need to set up the same stub expectation or response over and over. I don’t even have to think about how this collaborating object works. And if I apply an automated refactor to a method signature I don’t have to worry about breaking a ton of tests – the refactoring would apply to the stub object, as well.

Looking Forward

I’ve come full circle on stubs since my early days of TDD. I used to view them as a productivity killer, but my opinion now is that they actually save me time. Sure, there’s a little upfront investment, but the return on that investment is high. Who knows, maybe I’ll venture back into the land of mocks again one day. Until then, I’m happy with my well-tested, easy to maintain production code base.

Software Development is Like…

Software development is complex and difficult and unlike most other professions that I know of. This makes it tough to explain to colleagues in other parts of the organization, clients, and (all too often) management. We’ll typically try to find something well understood to use as a basis for comparison. Something to help explain the idiosyncrasies that come along with building software and being a software developer.

Software development is just like manufacturing, right?

I’ve heard dozens of comparisons of software development to other professions or activities. My least favorite – although seemingly most popular – is manufacturing. Sure. In both cases we’re building something based on specifications. That’s pretty much where the similarities begin and end. Manufacturing succeeds when you repetitively build the same thing with as little variation as possible. As a software team, if we build the same thing repeatedly then we’re doing something wrong. Software development is about solving a unique problem.

A different metaphor

Metaphors, by definition, are not perfect. Even so, I’d like to try a different one on for size. Lately I’ve been thinking that software development is not unlike drawing. Hang with me for a second… I’ll explain. Now, I’m not a drawer or a painter or a police sketch artist. But imagine having to draw or paint a scene from the real world. But in this case you have to recreate the scene from someone’s description. This is pretty much what we do every day in software. We bring people’s ideas to life, only we’re doing it on a computer screen rather than a canvas.

Keep playing along – this gets fun

Let’s extend this metaphor a bit to other challenges that we face in the software development world. How many times in your career has a requirement been misunderstood or missed altogether? Gee, I bet that would happen if I were trying to draw your words. Little details, even big details, can easily get lost in translation. “Yes, I said there was a bridge there, but it was iron – not brick.”

Or how about the situation where the user doesn’t exactly know what they want? “Can you paint a yellow-ish, blue-ish but definitely not green-ish house? Oh, now that I see it maybe green is right.” But sometimes your artistic influence leads them in another direction. “I had pictured this as a midday scene, but I love the way you have amber the sunset reflecting off the water. Let’s go with that.”

And now for one of my all time favorite lines of though. This being that we can add more people to the team and deliver software faster. Any software developer knows this is true to a point. But once that point is crossed, adding people will actually slow the team down. Also, the smaller the project and the closer to being “done” the harder it is to have more hands on the code. I’d image this is true when painting, as well. It may very well be possible to have three artists working on the same painting at the same time. The larger the painting, the more individual components, the more ways to split up the work. Can you imagine thirty artists working on the same piece at the same time? I expect this wouldn’t work so well.

But software, like drawing, is soft!

Across the industry there are folks who seem to forget that software is supposed to be “soft”. We’re not welding a steel frame together. We’re typing words into an editor and almost magically creating something that the user can interact with on the screen. Changing direction, even late in a project, is supposed to be easy. When we make it difficult all we’re doing is reinforcing that notion that software development is similar to manufacturing. Do yourself a favor and be nimble in your process and your architecture. There’s no need to redraw the whole city just to change the brick bridge to an iron bridge.

Avoiding the Temptation of Record and Playback Acceptance Tests

I’m sold hook, line and sinker on the value of automated testing. Unit tests, integration tests, acceptance tests – I love em all. I joined a new company a few months back (more on that in a later post) and I’ve had the opportunity to get back into some acceptance testing using Selenium/WebDriver.

The allure of record and playback

Selenium makes it trivial to create web tests using the open source IDE. Hit record, click around, then play it back to verify that your web application works as expected. It’s so easy, in fact, that you barely have to think about what you’re actually testing. To be fair, if you’re doing this then you’re doing more automated testing than a lot of teams have. And it may be enough for simple smoke tests. There are a number of reasons why I avoid this, though.

The problems with record and playback

Ok, I admit that it’s incredibly fast to get a test up and running with record and playback. Unfortunately, writing tests is only half the battle. Your tests evolve along with your codebase. Take a look at the selectors that the IDE uses to locate elements to click and interact with. If you were to come back to this test in a month would you understand it’s intent? If you add a new element – or worse a whole new feature – to your application can you easily find a seam in your tests to inject the new verification? It is unexpectedly difficult to maintain tests that were set up through a UI recorder.

An alternative approach

I don’t always test through the UI, but when I do, I prefer Selenium WebDriver. WebDriver is an open source API for writing browser based tests. In my current role I’m using the Python driver (along with some some awesome other tools that I’ll get into soon), but there is support for a handful of other languages. Using the driver API, we can write tests with all of the power of your programming language of choice. We can perform intelligent element extractions and assertions. Tests become readable and maintainable.

We now have to think about what we’re testing, which is just as valuable as all of the technical benefits. If we have well written stories and specifications we can turn those specifications into acceptance test scenarios.

Imagine a story in the e-commerce space where the customer receives an automatic discount if they put at least $200 of products into their cart. This would be nearly impossible to understand as a record and playback test. But we can easily make this scenario crystal clear when writing the tests. Not only that, but we can clearly illustrate the edge case expectations, such as when the cart contains exactly $200 of products. Heck, we can use a variable and have a single place to make a change when the product owner decides that they want to bump the amount to $250.

Well layered tests will set you free

Using the WebDriver API is a huge leap forward over record and playback. But if we just have a long list of element selections, invocations and assertions, we’re not that much better off than we were. Your tests should be treated with the same respect as your production code. That means putting proper abstractions in place, naming things clearly, staying DRY and SOLID.

One of the most useful abstractions when testing your web application is the Page Object. The idea is to keep a clean separation of tests and HTML/page structure. This ends up making tests even more readable and maintainable by providing an API for your pages. No more hunting through test files when you make changes to your page’s DOM – everything is in a single place.

A well stocked tool belt

You can using WebDriver from your normal unit testing framework and it will work brilliantly. But there are many incredible supporting tools for acceptance testing and behavior driven development for all major languages. In Python, I’m using Behave and loving it – .NET teams can use SpecFlow or StoryQ, in Java there’s JBehave.

Behave uses the Gherkin natural language format. This allows us to describe the desired behavior of the application in a format that is easy to read and write by both business and technical folks. We then provide implementations for each of the steps in each scenario, using the API provided by our page objects to drive the tests and get values used in our assertions. Running the tests produces output that reads just like our stories which we can use to show our stakeholders exactly how the application behaves.

Try doing that with record and playback!