SOA: An enabler for Continuous Delivery and innovation

Building on my experience at Westfield Labs, this presentation was delivered to Sydney CTO Summit and explores how implementing a Service Oriented Architecture allowed Westfield Labs to embrace a Lean, Agile approach to Product delivery.

The presentation covers how an SOA assisted with:

  • Management of backlog, particularly bugs
  • Managing build times
  • Cross-functional teams
  • Faster iterations
  • The ‘QA Paradox’ of better quality through reducing testers

Data is Money

When I use Google Search, I don’t pay anything. I get a great service but I don’t pay them cash for it.

There’s a popular expression now: “if you’re not paying for it you are not the customer. You are the product.”  I like this expression, because it captures the way I feel about the Google and Facebook (GAF) business model. But while I like it, I no longer agree with it, because it’s not an accurate picture of my relationship with GAF. I am a customer, and I do pay for their services. I just happen to pay in data, instead of dollars.

Data Is Money

Data is now a currency. With data I can buy thousands of apps on the Apple App Store. I can search the web, the world’s academic journals, and millions of photos of cats doing funny things. I can send and receive email. It’s the business model of the Internet, and it has its limitations, but nevertheless it is here.

Data is money. In exchange for creating, and then transferring data to GAF, they give me a web service. Both companies specialise in aggregating all that data and selling it for dollars to advertisers. In effect, there is a data to dollars exchange rate (and, you’ll note, a dollar to data one too).

Data is Money. Not like money, or as good as money. It is money.

Anthony, Brazen Thoughts, 2012

It is money because it is a medium of exchange, a unit of account, and a store of value. Currently, organisations that have a lot of it either “mine” it for information that can be used to design better products or services, or package it directly for sale in another form of currency (eg dollars).

Where it might get interesting is when we start asking to be paid in data directly, instead of in dollars first. There’s an example of a musician, asking to be paid in data, instead of the measly fractions of cents she gets as a cut from iTunes. It’s not that big a stretch to imagine a supermarket where I pay for my groceries in personal data (making Woolworths an advertising platform, as well as a supermarket. They’re halfway there already).

This doesn’t necessarily lead to a world where “everyone is an advertiser” however. The advertising business model exists because we haven’t yet thought of any other way to convert data to dollars, which we want to do, because we need dollars for food. But if we had even one farmer who was willing to supply food in exchange for data…

Now, all we need is a trusted record of exchange of data. I wonder if anyone is working on that?

First Data Bank

Here’s an idea I’d like to see: a data bank.

You “deposit” your data in a bank. You can withdraw it, which means it’s deleted. You can add to it at any time. You can deposit any kind of data you want, and transfer it to other accounts if you choose. The data is yours, in the same sense that money in a bank is “yours”.

The bank “loans” data to borrowers, under strict terms (in essence, the bank doesn’t need to physically transfer anything, or even give direct access to the data… but I digress). The borrowers have that data on loan, and must pay “interest.” The interest takes the form of the insights that they gain from analysing that data. The insights flow back to the data bank, and ultimately the data depositors.

This is a very different business model to GAF.

Money is Data

Money, by the way, is data. This is where I actually started, but I decided to lead this blog post with the conclusion rather than the introduction.

Money is an act of collective imagination. A mass, mutual suspension of disbelief. Money has value because we all believe it has value. This is easy for us, because our government says it’s true, and everyone is acting as if it is. Fairly catastrophic things happen to societies when people stop believing that their money still holds value (or that it will in the future). We call these catastrophic things hyper-inflation, and the collapse of civilisation.

While we have physical manifestations that represent money (coins, notes, bearer bonds, etc), most of our money these days exists purely as data recorded on bank computers. I rarely think about it, but I go about my day secure in the belief that the money in my account is “real.” But it’s not physically real. There’s no vault, no physical ledger, no gold or cash. It’s just flipped bits on a platter in a private cloud.

To access our money, we often use “money avatars”, such as credit & debit cards, gift cards, or cheques. They are avatars in the sense that they are physical manifestations that represent something imaginary, an intangible value. The bank note is not the thing, it just represents the thing. Its value is based on a promise we all beleive will be kept. The item itself is near-worthless paper.

Modern avatars pop up in other places too. Service avatars are physical manifestations of intangible service value. My iPhone is a service avatar. The true value of the iPhone is in the intangible apps. My Kindle is a service avatar (“The Kindle is not a product. It is a service” – Bezos).

Maybe I’m just a data avatar. :)

TL;DR: Conclusion

Money is data, but more interestingly, data is money. In exactly the same way as a fiat currency, data has an irrational but reliable intangible value and is used in exchange for services.

Could be really interesting times ahead.

Postscript

 

Continue reading

My 6 Coursera (& MOOC) Study Tips

Calculus I at The Ohio State UniversitySince I’ve been really quite slack in contributing to this blog I thought I should try and re-boot.

I’ve studied a few courses now on Coursera and completed about half of them. So from my rich well of both success & failure, here are Hisso’s Tips for Actually Completing Courses Online (if you’re just mucking about, these tips won’t apply):

  1. Study one course at a time, no matter how tempting it might be to add “just one more” because it sounds really interesting. As cheap as it is to sign up, the extra courses only serve to distract and overwhelm.
  2. Use the estimated hours. If the course convener has said the course will need 8-10 hours per week they mean it. I’ve found those effort estimates pretty reliable. You might look at the video lectures and think it’s only 1-2 hours a week — but the lecturers have run this before and know that the exercises, peer-assessments and reading actually make up the bulk of the time commitment. So go with their estimate.
  3. Study what’s interesting to you, not your employer. Because presumably you’re having to do this study in your own time, so you’re going to need to rely on intrinsic motivation to finish.
  4. Keep up with the lectures and quizzes. Once you start to fall behind it can be almost impossible to catch-up, because the cumulative study time required will easily exceed your spare time available. Which leads me to…
  5. Schedule in the Study Time. Ideally this is a regular time (I schedule 7-8 am weekdays) which not only helps keep you on track but also gives you a good idea of the maximum course load you can take on. If you know you only have 5 hours a week then you know not to bother trying to complete an 8-10 hour a week course (without scheduling in additional time).
  6. Find a Friend. If you can find a friend or colleague who will do the course at the same time then your chances of completing are greatly improved.

So there you go. Guaranteed Coursera Course Completion or your money back (har har).

Continue reading

Google is not Crazy

Google is not Crazy

Last month, the New York Times ran an article which raised some questions about Google’s recent acquisitions and how they fit in with its larger strategy. Some analysts see Google’s acquisition of Nest in a positive light, however, the article noted, not everyone is convinced:

Colin Gillis of BGC Partners is more sceptical. “Do you trust Google’s management as visionaries?” he asked. The analyst questioned the Nest purchase. Making thermostats does not fit in with Google’s core advertising, he said. Neither does robotics.

In my view, this is not necessarily correct. Here’s why.

Google’s Vision vs Business Model

Google defines it’s mission as: “[…] to organize the world’s information and make it universally accessible and useful.” Their pursuit of this mission is clearly seen in Google Search, but also its academic publications search, e-mail, social, scanned books, and other services. It’s perhaps harder to see that mission in driverless cars and “thermostats” (assuming that’s what Nest’s business really is). Nevertheless, I think the mission is still there. But today I want to talk about Google’s business model. In terms as simple as its mission, what would you say Google’s Business Model is? Here’s my take:

Sell precisely targeted advertising.

The mission and the business model are not the same. That’s OK, they shouldn’t be the same. The role of the business model is to support the mission. Google’s business model is deceptively simple. Yes, they sell advertising. However for Google (and Facebook, and maybe Twitter), the point is that the advertising is precisely targeted, because of Google’s access to consumer intent through its search engine, Gmail, Google+ and other services.

Limits to Google’s Business Model

There is, however, a limitation to this business model. It is this: Most of what we do in our lives we do “offline” to Google. Most people, most of the time, buy products in physical stores. We drive cars, catch trains, visit friends & family and we do these things without necessarily letting Google know about it. If your business model was to sell precisely targeted advertising, and you realised that most consumer activity was actually happening without you knowing about it, what would you do?

Intermediation Model 2.0

What Google wants to do is intermediate our lives. A lot has been written about dis-intermediation as being the defining feature of the current change sweeping the business world. But it would be more accurately described as re-intermediation. In the past, newspapers aggregated consumer eyeballs and then sold those to advertisers. Then they got dis-intermediated (and unbundled, and out manoeuvred, and et hoc genus omne). But that hasn’t meant that you get your news directly from journalists. Rather, you get it from the new aggregators and intermediaries: Google, Facebook, and Twitter.

Similarly, Amazon disrupted the book retailing market, and with its Kindle service, is now in the position to cut out the publishers completely. But again, it hasn’t meant that you buy your books directly from authors. You buy them from the new intermediaries: Amazon, Apple, and Google. Google is in the intermediation business. By learning everything it possibly can about us, it’s able to sell very precisely targeted advertising, and effectively mediate consumer access to service providers. And it can make a lot of money doing it.[1]

Google is not “Predicting the Future.”

Let’s revisit the analyst’s reservations about Google:

“Do you trust Google’s management as visionaries?” […] Making thermostats does not fit in with Google’s core advertising [business]. Neither does robotics.

Except, yes they do. They fit because Google’s advertising is precisely targeted based on what it knows about us. What do you think a home thermostat connected to the Internet could tell Google about the people who lived there?

What correlations might Google discover between thermostat settings and, say, disposable income? What happens when the Nest product suite branches out to gather more than just temperature data? What about noise levels? Movement? Air quality? Could a Nest sensor infer the emotional state of a household based on voice intonations? Could it infer what people are watching or listening to based on background noise? Imagine having Shazam running all the time, only it identified more than just music but also news stories, movies, TV shows… Might that be interesting to an “advertising” company?

What could a self-driving car tell Google about where people went, and how often they went there? Would the car see interesting events on its travels? What would it hear people inside talk about? Do you think it might want to talk back? What would it say? Anything here interesting to an “advertising” company?

What about robots generally? Who will the robots work for and what will they do? If we delegated service consumption to an automated system that worked for us, would a company that wants to mediate your consumption of services want to know about it? Robotics is key for Google because it offers the potential for Google to break free of the digital interface straight-jacket that is mobile and desktop computing. The robot will be the new service interface. Does anyone really doubt that owning a slice of the new service interface wouldn’t make any company salivate?

Summary

TL;DR?

  • Google is not in the “advertising” business as traditionally described. It is in the business of service intermediation. They are one of the new service intermediaries.
  • The service intermediation business requires advertising that is precisely targeted for its advertiser customers to get real value. This targeting requires large volumes of data about the target consumers. Google is missing data about our “real-world” interactions, and sensor companies like Nest can fix that for them.
  • The (automated) service interfaces of the future are not the screens and keyboards we know today. “Online services” is going to break out of the constraints of mobile and desktop computing and into the physical world via “robots” and other automated services that can manipulate their physical environment.

Google will be able to learn more, and then mediate our service acquisition, in more places and more often. So no, Mr Analyst who asked a rhetorical question that I will answer anyway: No I don’t think Google’s founders are visionary. I think they are lucky to be where they are, and they’re following up that advantage with some very canny business development.

Continue reading

We are the business

An ex-manager of mine once pointed out that we need to stop talking about “the business.” Doing so gives leverage to those claiming to represent “the business” and limits the influence of the engineering team. His was more a political observation than a call to change our mind state, but ever since then I have noticed how commonplace it is for colleagues to make vague assurances that “the business requested it,” or “the business want it like that.”

When someone uses the term “the business” they invoke shadowy high priests with absolute knowledge — but they could be referring to a clueless HIPPO, an opinionated sales or marketing exec or equally to any end user of a piece of software. The engineer should have every right to question those requirements and to request exactly whom “the business” refers to in this scenario.

At Westfield Labs we are very fortunate to work in a ‘digital’ department which combines product, design, end-users and engineering resources as equal collaborators. Most recently we have moved to truly cross-functional teams where the only direction given to us by sponsors and stakeholders is high-level: e.g. bring us more customers and more conversions through focusing on streams x and y. Sure, the product team provide the ultimate direction from a product perspective but only after close consultation and collaboration with all other relevant parties.

In this scenario it is not hard for an engineer to think of oneself as part of the business and I positively encourage my team to stop using the term ‘the business’ to refer to others as it implies that we are not an equal partner. To take it further, I actively encourage my engineers to question and understand business requirements and to shout out if they don’t make sense.

Talking recently to an engineer from a UK online retailer, he noted that his company “think of themselves as a marketing company, not a software company” and see the engineering department as a necessary expense to realising their feature requests. In a business that is so dependent on the quality of the implementation and the iterative improvements upon that implementation, it is naive to think that engineers are not equal partners.

Obviously it’s not so easy when you are working in an agency (and let’s face it, sometimes you are working from a spec and clearly not ‘equal partners’) but any enterprise that wishes to succeed in the digital age will ultimately depend on the quality of its implementation – and the feedback from those that are doing the implementing. Otherwise it will be made irrelevant by a competitor that does.

So, I entreat software developers everywhere: let’s stop talking about “the business” and start talking about: customers, stakeholders, sponsors, sales team, marketers, product team, whatever. Make it clear that we consider ourselves part of ‘the business’.

New look

I just changed the theme. Not because I didn’t like the old Chunk theme but because it didn’t credit the author on the home page and I thought that was important now that hissohathair is contributing as well. It appears that editing templates on wordpress.com is not an option so welcome truly minimal, hopefully we will have a long and fruitful relationship.

While browsing the 300+ themes available on WordPress I was reminded of the elegance with which WordPress separates content from presentation and the experiences I have had moving from a CMS toward a set of APIs that serve ‘structured data’.

At westfield.com.au we threw out our Teamsite CMS a few years ago and moved away from the idea of a meta-data driven ‘universal CMS’ toward an API that serves structured data to our Presentation Layer(s). As any OO programmer will appreciate, this allows us to store the behaviour and business rules with the data – as opposed to treating data as an anonymous blob – and we’ve never looked back. We avoided the problem of an ever-growing database schema and ‘Business Layer’ by implementing an SOA but more about that another time.

We’ve been able to provide content parity across all devices by fully embracing responsive web design and we can do this because (like the WordPress themes) there is a complete separation of data from the presentation layer. As we start to move toward providing richer editorial-style content we are really starting to reap the benefits of this approach; we store data in it’s small component parts and then tag and curate this data. This allows us to create simple editorial pages such as the recently launched Curations but as we start to discover more about our users and their tastes through the power of big data we can start to tailor our content to their individual tastes and look to serve our content algorithmically rather than through creating pages as one would have in traditional publishing.

From my perspective, well constrained CMS’s like WordPress that solve a clearly defined and well understood requirement will continue to flourish but if you’re looking to fulfil a unique requirement you’re going to have to write a lot of bespoke code so why not embrace it and give yourself that freedom and control. Leave the off-the-shelf CMS for the brochure-ware sites and the big publishing companies that have the money and patience to customise them.

 

Iterating vs Dabbling

Iterating vs Dabbling

Agilists are often urging each other to “iterate” — that is to make small, incremental changes and test each change in the market as you go.

Iterating makes sense, especially when building complex systems. In complex systems, even small changes can have unintended or unexpected effects, and by moving incrementally we give ourselves the best chance of detecting early enough to minimise the cost of correction. We’re not just talking about software bugs in the new features here. “Cost of correction” includes addressing misjudged customer requirements, pivoting to reach a better market fit, and fixing regression errors.

We can also iterate without coding and this can make a lot of sense in the very early days of product development. Deming’s “Plan, Do, Study, Act” (PDSA) cycle is helpful even in early concept development. For example, we might have a few iterations of the basic concept development, talking to potential customers and then reflecting back our understanding of their needs, just to make sure we’ve really understood, and are talking about those needs in ways the potential customers recognise.

Iterating

So Iterating is Good and we’ve all got that religion. Excellent.

What we might not have is a common understanding of what “iterate” means. It’s obviously suggestive of a cycle, but are we all using the same steps in the cycle? I’ve specifically called out Deming above because it’s a genuine iterative cycle:

  • Plan: Decide, up front, what question you’re asking and how you’re going to get the answer. If you believe you’re not asking a question then what you’re about to do is probably not an “iteration” of anything so you can stop reading now.
  • Do: Execute the Plan. Measure and gather results.
  • Study: Analyse the data gathered above. Look at what we’ve learned, and if our original question is answered.[1]
  • Act: (Action Research calls this “reflect”) Finally, we integrate the learning generated from the Study above into our product, service, processes or plans.

Looking at this in the four steps above it might be easy to miss an essential feature: the cycle repeats. That is, you are not iterating if you only do it once.

Dabbling

Apple’s App Store is littered with dabblers.

Dabbling has some shallow similarities to iterating:

  • Small things are done, then released.
  • There’s an apparent attempt to gather feedback.

And then, after that, nothing happens.

There are some particular examples I have in mind but it’s unfair to call them out because there are so many others. On the App Store the telltale sign of a dabble is that there’s been only one or two releases and they were some time ago. There’s nothing really wrong with this kind of development. Perhaps the author just wanted to learn the process, and that they’ve taken things all the way through to a release on the App Store is actually a Really Good Thing from a learning point of view.

On the other hand you can also find apps that have had one or two releases and then been abandoned because they “haven’t been profitable.” It’s easy and relatively cheap to build new apps — but it’s difficult and expensive to build new businesses.

Even large and well-funded projects can catch a touch of the dabbles. Requirements churn, stories that fiddle with colours and fonts, and funding cliff-falls are all potential warning signs that you’ve stopped being serious and have started pretending:

  • Requirements Churn: Are you implementing the same basic functionality but you’re not sure why? Are your non-functional requirements changing rapidly? Are you contemplating switching to Yet Another Framework? Take a moment to check if these changes are going through a proper iteration cycle (PDSA).
  • Polishing the Poo: In past projects I look back at the number of “0.5 point” stories and realise how useless all that was. It added up to a lot of “points” but I suspect little business value was added. There was no measurement of effects in the “Do” step, and no attempt to Study the impact. We’ll simply never know if any of that work was worth it.
  • Budget/Funding Cliff-Falls: Looking back at research projects that I’ve managed I’ve decided one mistake I made was approving “build a prototype” projects with no funding or plan locked in to check it and develop it further. We certainly intended to do those things. But without a pre-approved budget to actually do them we lost momentum, staff, and potential customer enthusiasm.

Lessons Learned

There’s nothing wrong with dabbling, but dabble consciously. Understand when you’re doing it and why. If you’re iterating, iterate consciously. Call out the specific steps in the Deming/Shewhart cycle and confirm for yourself that the results of this iteration will be used in the next.

Now get back to work.
:)

Continue reading