Glass Wall Anti-Pattern

Glass Walls

Nigel has come up with a concept of a Glass Wall, and it really struck a chord with me. You are expending a lot of energy to reach a goal. You can see your goal clearly, and you can see how to get there - its only a few steps away. After a disproportionate amount of time and effort, you find yourself in a totally unintended place, and with no real idea why. You haven't achieved your goal, but you've expended all the time and energy you have available for that task. When someone asks you why you haven't just taken the few simple steps to reach your intended destination, you find that you can't answer. Its those Glass Walls. But what puts those Glass Walls in the way? Is is the Malignant Requirement and the Headless Chicken ganging up on you?

Financial Model For Architecture

The Question

Should Architecture have a budget of its own? If you give Architecture all the money and power, they tend to spend it all building ivory towers and inventing wonderful architectures that only live on Visio diagrams, while the rest of Technology is busy delivering 'tactical' solutions on a shoe-string. This means that no architectural considerations are taken into account when building solutions. If projects have all the money, then they want the quickest, cheapest solution - they don't care about building the architecture. It isn't looking good for architecture.

Creating a balance

The only way to ensure that projects are delivered quickly and cheaply, but also to architectural standards is to create a balance of power. Architecture can take advantage of economies of scale. They care about the infrastructure, the non-functionals - I see these in the horizontal plane. Projects care about delivering capabilities, the functionals - I see these in the vertical plane. Their primary concerns are orthogonal - they usually appear to be in conflict, but they are actually supporting each other. What we need is a way for architecture to be able to invest in the infrastructure in a way that supports the projects.

The Financial Model

No project wants to take on the financial burden of being the first one to use a new, expensive piece of hardware or software that every other project is going to come along and use for free. Therefore they will avoid using the most architecturally sound solution on the grounds of cost and time-to-market. I propose a financial model, where architecture has a virtual budget. They can invest in whatever infrastructure they want, but at the end of the year they have to break even.

An Example

They initially get into debt by investing in a centralised customer database, because they know that there is a project in the propositions pipeline that will need this capability. It costs them £80,000, and they have it up and running as the project enters its implementation phase. They only charge the project £20,000 because they expect to have at least 5 other projects that will need to use it. In addition, they are making the project more efficient because they don't have to take on all the work to deliver a solution to store customer data.

Reasons Why

This solution is basically speculation in the infrastructure market. Architecture is the only department that can take a long term view. It needs to be able to invest in the infrastructure, because nobody else has responsibility for delivering it. However, they need to have a reason to invest in the right things at the right time to support projects, otherwise they will do what they want without consideration for project delivery. If the financial model forces them to work hand-in-hand with the projects, and encourages the projects to consider the best architectural solutions, the balance is there to ensure that effective, architecturally sound systems are build in support of project delivery.

Estimating - The Wicks Kitchen

Estimating

This ties in with the propositions process and estimating for projects. The idea is that you should be able to do a quick estimate to get a ballpark figure to work with so that the project has a good idea of the magnitude of the work. This can feed back into the priorisation and cost-benefit analysis, plus budgeting. However, this estimate should not be used for detailed planning, as its accuracy is only +/-50%, as there is limited information available to base the estimate on. You don't want to spend £50,000 on people's time to perform a detailed breakdown of how long a project will take. It won't be accurate, and you've already spend £50k that could have been spent on building the thing. Once everything is approved then you should enter a second phase of more detailed estimation (Iteration, Story, Task) until you have a task by task breakdown. This should be estimated in Story Points, and the true cost of the project will only become apparent as you monitor your Velocity and translate Story Points into mandays. This is hard to explain to a project manager, but if you provide them with the statistics and accurate predictions as the project team settles down, they will appreciate the benefits that this approach gives them.

The Wicks Kitchen

Nigel likens this process to the one he experienced at Wicks recently when ordering a kitchen.

Headless Chicken Mode

What is Headless Chicken Mode?

Headless Chicken Mode is a state of mind, and can sometimes seem like some sort of highly contagious mental condition that can spread through the office like wildfire. Anybody from architect to junior developer can suffer from this condition, and it is usually brought on by some pressure from the business. The further up the chain of command this condition permiates, the bigger the problem is. Symptoms include a fixed, slightly glazed stare, a sheen of sweat on the forehead, and rapid walking around the office talking to the wrong people about the wrong things at the wrong level, and lots of enthusiastic discussion about completely inappropriate solutions. The business and non-technical managers love this sort of thing. They see people jumping into action, reacting in a positive way to their motivational techniques, lots of activity and enthusiasm.

The Business Idea

The business has this fantiastic idea, and have convinced all the senior managers that this solution would really benefit the company. They build up huge enthusiasm for the idea, get loads of budget, print special T-shirts, have promotional events, put adverts on the big screens. And to be fair, their business outcome they are aiming for makes absolute sense. Then two days before their go-live date, they remember that somebody has to actually enable the systems to support this idea. But how hard can it be?

Top Priority - Process Ignored

They don't need to go through the normal 'propositions' process, so carefully put together by Technology, because the CEO has decreed that this is the priority and will be delivered. Therefore they approach a team leader, and make sure that he is fully aware of the absolute urgency of their request. They usually present a technical solution, rather than a business outcome, and give the absolute deadline of, say, two days. The team leader really doesn't want to be seen to fail given the high profile of the project, and to even appear unco-operative wouldn't be a good career move, so he immediately makes two serious errors. He implicitly accepts both the deadline and the technical solution put forward by the business.

Maintaining Control

You have various controls at your disposal when you are running a project. You can control the scope of the project, so you can choose to deliver more or less features. You can control the quality of the technical solution, but this is a very dangerous control to use. You can control the amount of resource allocated to the project, but you have to be aware that they gain from this isn't linear, and can actually cost you efficiency if you try to ramp up too quickly. You can control the time allocated to deliver the project. Finally you can choose to fulfil the business outcomes with a non-technical solution if that makes sense - and this will usually be invaluable if you have a tight deadline, but over the long term will usually not be the most appropriate solution.

Losing Control

The team leader has just given away scope, time and manual alternatives to the business. He hasn't got time to introduce a big team effectively, so he can't really use resource as a control either. As for quality, he hasn't discussed SLA's (Service Level Agreements), so they default to production quality, 99% uptime. So he now has no control left at all.

Switching to Headless Chicken Mode

This is when Headless Chicken Mode strikes, and he runs round the office in a mild panic, asking random developers if he can use this database to get this bit of information, if they could go direct to the database without having to build a web service, if they could build a web page in a couple of hours this afternoon. He rarely gives context, so nobody has the opportunity to suggest an alternative, or highlight the problems with this approch. Then he tries to find a fast track to live, skipping testing.

The Mythical Next Release

Many people who should know better do this sort of thing, and they always believe that they have no choice, and that its ok to deliver this complete rubbish, because 'the next release' will sort out all of its shortcomings. However, the next release never happens, because the business never spend money on making the technical solution better, they will be investing in their next idea. If the app keeps falling over, it is up to Operations to fix it. If its got bugs in it, then development will provide bug fixes. They see no point in spending any more money when they already have their application in live, no matter that it is costing Operations, Dev and Testing a huge amount to operate, test and maintain. It's not their fault that we can't do our jobs properly. But Technology has now got high operating costs, and their ability to respond to new ideas is compromised.

Avoiding Headless Chicken Moments

So how should we deal with this situation? Well, the key thing to remember is that it should always be a negotiation between technology and the business, neither should be able to dictate things to the other. In addition, the business should never present a technical solution; they should present the business outcomes that they want to achieve, and negotiate with Technology to agree on the most appropriate way fulfil these.

Iterative Delivery

If the deadline is unrealistic, first try and work out what actually needs delivering for that deadline. Usually you only need to deliver a small subset of the solution for that date, and then you can iteratively add features on a 'just-in-time' basis. The next step is to look at what manual solutions you could put into place to deliver the same business outcome while you deliver the automated solution. Could you set up a call centre with people manually entering data? Could you send emails to customers, send a form through the post? Place an advert in a magazine? You don't always need to deliver the technical solution.

Is It Worth It?

Another good discussion point is cost benefit. If you can demonstrate that it will cost X to deliver the automated system, and the predicted benefit isn't several times higher than this, is it worth doing at all? When discussing things with business people, always bring it down to money - then you're talking the same language.

Conclusion

Once you are aware of the Headless Chicken Mode, it is easy to spot. The look of mild panic is the first sign. If you then overhear snippets of conversation, or see an email trail that indicates that ill-conceived solutions are being hacked together for an unrealistic deadline, you've got a 'HCM' problem.You have to put a stop to it, or your technical platform will be corroded to the point that you will spend your time fire-fighting rather than producing inductry-leading solutions that you are proud of.

The 80/20 Rule

The Pareto Principle

The 80/20 rule, or the Pareto Principle asserts that 20% of something is always responsible for 80% of the results. This is a rule that can be applied to almost anything, and if you apply it appropriately, it can save you from investing time and effort in areas that give you diminishing returns. I mention it here because you can apply the 80/20 rule very effectively to software development, and I use it every day to guide my decisions on where to focus my effort and how to approach project work.

Requirements

So why don't we always just invest 20% of the effort to get 80% of the benefit? In my experience, if you ask people what they want from a given system, they will always ask for everything. They will spend weeks or months coming up with the most comprehensive list of requirements you could imagine. They will, as usual, try to tell you how to implement the system too.

Prioritisation

You can deal with the sheer mass of requirements by converting them first to high level outcomes. These outcomes can be prioritised. Then comes a period of negotiation, where you try and agree on the most appropriate way to deliver each outcome. Once this is done, you can create high level stories that you can estimate. Telling someone that feature X will cost £100,000 soon makes them focus on what is really important. You can then use this additional information to further prioritise these high level stories and see what you can produce for a sensible amount of money, in an iterative manner, achieving real benefits within an acceptable timeframe.

The Malignant Requirement

So we're doing pretty well - we have got a prioritised list of the best value stories that will deliver the outcomes that the business want. We've made sure that the solutions are appropriate, and we understand the impact on the existing infrastructure. Now let's implement that 20% to gain the 80% benefit and move on. But we can't. Why not? Because there is always one Malignant Requirement that the customer will not let go of. This will usually be some really strange, specific requirement, based on some historical pain that they have suffered as a result of an old system or process. Or it could be the sort of requirement that seems really important if you've spent weeks over-analysing the problem domain in isolation.

An Example

They might have had a lot of problems with timeouts in the past because the old system had really long, synchronous call chains. They worked out that it cost the company £500,000 over the past few years, so they are quite happy to spend a huge amount of money on a requirement to dynamically set all timeouts across all technologies and consume real-time data; monitoring each call with respect to the timeout it has been given. The new system has short call chains and an asynchronous architecture, but this isn't taken into account. The fact that dynamic timeouts and real-time monitoring and alerting will make the system far more complex and more expensive to maintain isn't seen either. This one requirement will mean that you put in 80% of the effort, and see 20% of the benefit.

How to Supress the Malignant Requirement

This is a tough one. Trying to educate the customer in the technical considerations is one approach, but this won't work unless they are technical in the first place. Risk and cost-benefit analysis is probably the way to go, and it really helps if you can get an idea of the impact that it will have on future development and maintainance costs. If it will cost £250,000 more over the next 5 years, then this will offset the benefit of the requirement. However you do it, it is really worthwhile trying to supress this requirement, or you lose your ability to capitalise on the 80/20 rule.

Achieving 80/20

Most problems have been solved many times before. You're not the first person to want to monitor the behaviour of your live machines. You're not the first person to want to take orders over the web. These problems have been solved over and over. People write frameworks for this sort of thing, so if you can't find what you need in the opensource community or from a vendor, you need to question your requirements. If your problem domain is really specialised, and the solution that you are looking at really is the first time the industry has encountered this problem, then go ahead and spend your time and resources on solving it. However, if you are writing your own framework for logging, or for website navigation, then question what you're doing. Research on the web. Google, newsgroups, technology sites, find out what's out there. Top technical minds will have spent years solving the problem that you've encountered. They will have learned from previous releases and incorporated this learning into the current release. Bugs will have been fixed, performance optimised. Do you really want to spend years reproducing all this work?

Don't Bespoke the Solution

Another critical point to understand is that if the solution solves 80% of your problems, and its only taken 20% of the effort to implement it, don't go and bespoke the system to try and achieve the remaining 20%. It might seem to make sense to begin with, but it will really hurt you in the longer term. By using a product or solution developed and maintained by a third party, you should be able to benefit from upgrades and bugfixes really cheaply. You might also be able to use third party tools and utilities that integrate with this product to give you instant benefits. If you go for this final 20% to fulfil the Malignant Requirement, you will lose a significant proportion of the 80% benefit you had in the first place. I see it time and time again, where a company has totally bespoked a third party product and then has to maintain and develop it itself, and are unable to upgrade and subsequently fall out of support. The development and maintenance costs are huge, and the system has to be eventually ditched and re-written, incurring further costs.

Conclusion

You can apply the 80/20 rule to all the areas of technology where the problem has been encountered before, enabling you to reuse the solutions that are already out there. The benefit of the snowballs when you look at the costs future development, maintenance and system integration. You should seriously question any requirements that aren't satisfied by a mature, industry-accepted solution. This frees up resource to focus on those areas that are specific to your business, your particular problem domain, and pushing your technology forwards to better serve the business.