Web Technologies

Introduction

When you want to provide services to customers over the internet, you are faced with a confusing array of options. The landscape is changing rapidly, and you have to make the call on which technology is the most appropriate for you. What are the options? What are the pros and cons of each? How do you decide which one you want? Here's the Dummies Guide to internet technology.

Standard HTML Web Pages

Starting with the humble HTML web page, you would think that you were on safe ground. Then people start mentioning W3C standards, XHTML, accessibility, cross browser compatibility and stylesheets (CSS). Anything other than the simplest page demands that you use javascript. So what are these things, and why do you need them?

W3C standards

The World Wide Web Consortium (W3C) is an organisation that creates standards and guidelines for the web, and most browser vendors produce 'standards' browsers that follow these (Firefox, Opera, Safari). Microsoft's Internet Explorer is the notable exception, unfortunately, but IE is improving its standards support with each release. It makes absolute sense to follow these standards rather than support a particular vendor's browser to the detriment of all others. Standards support is getting better, so the browsers will support your web pages better as time progresses. When there is a browser bug, it isn't unusual for people to demand that you, as author of the web page, fix it. It is a misunderstanding of how the web works, and especially how it will work over the next few years. You don't support a browser, the browser supports standards, which you adhere to. If the browser has a bug, and there isn't an elegant way to work around the problem, then it is up to the browser vendor to fix it. "Don't we support IE any more?" is a question that highlights a fundamental misunderstanding of the internet.

XHTML

So you know web pages are written in HTML. Well, now we have XHTML, and the benefit of this over HTML is that its far better defined, and can be described by an XML schema (hence the X prefix). The upshot of this is that browsers find it far easier to interpret, and therefore next generation browsers should have far better, more consistent support for XHTML web pages. As a result, cross-browser compitbility should improve, lessening development and maintenance overheads that we currently incur ensuring that our web pages work on all browsers. It takes less processing to read XHTML, so it works far better on browsers that run on mobile phones or PDA's. Basically, if your site is XHTML, you are expanding your user base and future-proofing your website.

Cascading Style Sheets (CSS)

Cascading Style Sheets (CSS) allow you to seperate your web page's document structure and data from its presentation data. You create a central style sheet that gives you the ability to make global changes to the look and feel of your website. If you want all buttons to be red instead of blue, its on quick change to the CSS file. If you want to dynamically rebrand your site depending who is viewing it, you can do that too. Although it isn't the work of a moment to build CSS that works across all browsers, once it is in place, writing brand-complient web pages becomes a doddle, as you just have to write the correct document structure, and the page will automatically conform to brand guidelines. Investing in CSS, seperating out the presentation information, will save you when it comes to development and maintenance and rebranding.

Accessibility

Accessibility of web pages means writing them in such a way as to allow everyone to access them. People who have difficulty reading a monitor should be able to use a screen-reader to read your web pages and give a good user experience. Turning images off should not compromise the user experience. Using a different browser or device should not make the site inaccessible. And more controversially, turning off javascript should not deny the user access to the site.
Following web standards makes your site accessible, see the guidelines on the W3C site.
The basic motivation for making your website accessible is to make it available to the widest possible audience, but more crucially, not to discriminate against anyone. If you do discriminate against somebody by not making your website accessible, you leave your company open to legal action. Motivation enough, I think you'll agree...

Flash, Flex and Rich Internet Applications (RIA)

Adobe's (formerly Macromedia) Flash player allows you to provide a richer graphical experience using the FlashPlayer browser plugin, and also allows you to stream music and video. The term RIA (Rich Internet Application) refers to a browser based application that takes on more of the UI processing at the client side, therefore running something approaching a desktop application within the browser. However, most of the data and state information is still held at server side and is accessed via the internet. Adobe's Flex offering provides a toolkit to enable development of RIAs.

Smart Clients

If we take this transfer of responsibility to its natural conclusion, Smart Clients are applications that are installed on the user's desktop with access to local resources (memory, disk, database etc), and merely accesses data from the internet when necessary. This is where the industry is heading in the next few years.

Web Services

If you want to access data in a standards compliant way across the internet from your website, RIA or Smart Client, then you will be needing Web Services. Web Services are a standards complient way to offer services securely over the internet. The W3C have produced the WS-* standards, and various implementations of these include Microsoft's WSE2, WSE3 and Indigo and Apache's Java based Axis.

Model View Controller

What is MVC?

When developing a web application that has anything other than the simplest page navigation, it is extremely important to seperate the presentation, the business logic and the control of the page-flow. The best way to do this is by using the Model View Controller pattern (MVC).
The 'Model' is the part of the system that deals with the business logic, it is the logical representation of the problem domain. In OO this would give your abstrations such as Employee, Customer, Payslip etc.
The 'View' refers to the web page layout and presentation of the information to the user. In a web application, this is usually represented by markup.
The 'Controller' is the part of the web application that takes an event (usually a user action) and decides how the system should respond to it. It performs any updates that are required to the Model, and then decides which page (View) to display next.

MVC in J2ee

This pattern is implemented in the Apache Foundation's Struts project, an opensource Java framework. The specific pattern is called MVC II, which is a Front Controller implementation. A Front Controller is where the Controller, not the View, is the first to receive the user action and then decides what actions to take in response. Once it has performed any actions it needs to in response to the event, it decides which page to display. However, a particular event isn't tightly coupled with the page is displays, and to ensure that you can easily change the destination page, each page is responsible for getting any information it needs to display from the Model, and this information is not passed by the Controller to the View.

MVC in ASP.NET

In my opinion, the MVC II model is the best option for any enterprise scale web application. However, I tried to apply this to an ASP.NET project, and I found myself battling against language features every step of the way. The main issue is that ASP.NET by default employs the Page Controller model, where Postbacks are used to enable the page to deal with any events that result from user interaction with any of its controls. Therefore you often end up in the awkward position of handling the Postback in page X , and in response to that action, you actually want to be displaying page Y. It appears that, unless you want to take on the additional complexity of Microsoft's User Interface Process Application Block, it makes sense to accept that a lot of the language features demand that you use the Page Controller pattern, and try to make the best of it. I have read several Microsoft articles that claim that the ASP.NET page code-behind feature seperates the View (the markup) from the Controller logic (code-behind). This is not really true, as the code-behind contains a muddle of View logic and Controller logic. To ensure that you get an MVC seperation of concerns, the code-behind should contain only 'View' logic, and the Controller logic needs to be seperated out from this. As the ASP.NET model means that the events are handled in the code-behind, you end up with the View handling events. This is not really in keeping with the MVC model we wanted to implement.

Model View Presenter

As a result of these ASP.NET language features, I discovered that Model View Presenter (MVP) was the best pattern to employ when using ASP.NET, as it separates the behavior of a presentation from the view whilst allowing the View to receive user events. See Martin Fowler's definition, and a good example implementation by Jeremy D. Miller. Basically, the MVP pattern has the ASP.NET page (including code behind) as the View, its has the Model behind a Facade, and the Presenter object acts as the go-between that gets events from the view and performs the necessary interactions with the Model, and then informs the View with the results. MVP also aids the unit testing of the page logic, which is very cumbersome to attempt using NUnit when you are relying on ASP.NET runtime infrastructure to hook the page together.

Mocking Basics

Introduction

As an XP team that uses NUnit and Continuous Integration, we had the problem that our unit tests ended up being integration tests, and any error would propagate causing many tests to fail and making it hard to find where the original error occurred.
To solve this problem we used NMock to mock out any external dependencies that a class has, therefore making our unit tests insular. However, there are several patterns that we have to use to make this work effectively, and this is the topic addressed in this document.

NMock

NMock is a dynamic mock-object library for .NET. Its purpose is to provide a clean API for rapidly creating mock implementations of custom objects and verifying that objects interact with them correctly from unit tests.

What is a mock?

A mock:
takes on the interface of another object, allowing it to be substituted for a real one for testing purposes.
allows Expectations to be set up, specifying how the class under test is expected to interact with the mock.
fails the test if any of the expectations are violated.
can also act as a stub, allowing the test to specify objects to be returned from mocked methods.

Defining a Dynamic Mock

A dynamic mock sets up a dynamic proxy that implements a given interface, or overrides any virtual methods on a given object type. It is initialised as follows:
IMock mockObject = new DynamicMock( typeof( Interface ));
This mock object can take the place of the real object in unit tests and allow you to control the way that the mock behaves and verify the way the object you are testing interacts with it.

Setting Up a Mock

In order to verify the way that the object under test interacts with our mock we can set up expectations on our mock object.
mockObject.Expect("MethodOnMock", param1, param2, ...);
Here we tell the mock to expect a call to method MethodOnMock with the given parameters. An extension to this is the ability to return an object when a method is called.
mockObject.ExpectAndReturn("MethodOnMock", returnObject, param1, param2, ...);
Here we instruct our mock to return returnObject when MethodOnMock is called. There are several other expectations that you can set up, which we cover in the Appendix.

Verifying a Mock

Once you have run the scenario that should fulfil the expectations that you have set up on the mock object, you have to verify that this is the case.
mockObject.Verify();
If the expectations have not been fulfilled this will fail the test.

Passing a mock to a method

Pattern 1

Whenever the class that you are testing instantiates an external class, always acquire it through a virtual property or method; eg

public void Method() {
Interface object = ObjectInstance;
}

protected virtual Interface ObjectInstance {
get{ return new Object(); }
}

This means that whenever you interact with an external class, in the test you can override this property or method and return a mock object in its place. The simplest way of doing this is passing your mock into the constructor of your test extension and storing it for your overridden property or method to return. The unit test code will look something like this:

[TestFixture]
public class TestClass {

private ClassTestExtension _testObject;
private IMock _mockObject;

[SetUp]
public void SetUp() {
_mockObject = new DynamicMock( Interface );
_testObject = new ClassTestExtension((Interface)mockObject.MockInstance);
}

[Test]
public void SampleTest() {
_mockObject.Expect("Method");
_testObject.MethodUnderTest();
_mockObject.Verify();
}

private class ClassTestExtension {

Interface _mockObject;

public ClassTestExtension( Interface mockObject ) {
_mockObject = mockObject;
}

protected override Interface ObjectInstance {
get{ return _mockObject; }
}
}
}


Rules of thumb

If a class depends on any other class to perform any of its tasks then do the following:
In the NUnit test, derive from the class you are testing to give you control over the external classes it uses.
In the base implementation of the class you are testing, every time you need to use an external class, get it via a virtual property.
If you own the external classes, make it implement an interface to aid with mocking.
In the inherited test class, override these properties to return mock instances of these classes.
In the derived class, overload the constructor to provide one that takes the required mock objects and stores them as member variables so that they can be returned by the properties.
In the setup of the test, create a new instance of the derived class and all the mock objects, passing the mock objects into the constructor. Store all of these objects as member variables.
In each test set up the mock objects (ExpectAndReturn etc) to provide the scenario that you want.
Call Verify on your mock objects at the end of each test.

Propositions

The Proposals Stage

It is very difficult to get a business proposal from the idea stage to the start of an Agile project. Most business ideas go through some sort of process that decides whether or not the business thinks it is worth the expenditure. However, the way that the expenditure is calculated is usually by some pretty in-depth business and technical analysis. Usually quite bit of time and effort (and therefore expenditure) is spent on this stage in the projects life, and the outcomes from this are usually a high-level technical solution and a budget. The business uses this information the work out whether it is beneficial to go ahead with this proposal. Already we aren't working in an Agile way, as the first that the Agile team will see of a project will be at a point where the budget is set and the technical solution has been decided. In addition, a lot of money will already have been spent on the project, leaving less for the Agile team to spend creating the solution.

Business Outcomes

Recently, a collegue and I were approached by somebody who was part of the Analysis stage of a proposal. They thought they might want to do a project in an Agile way, and wanted to know how much budget would be required to implement a timesheets system. I said that it really depended on what they wanted, and asked them what their outcomes were. They replied with outcomes such as 'to reduce the time it takes to process the timesheets from 4 days to 1 day.'. This is a perfectly valid outcome, but it is a Business Outcome. It cites the benefits of the solution being implemented.

Solution Outcomes

Each Business Outcome has to be supported by one or more Solution Outcomes. Solution Outcomes identify what the solution has to achieve. An example of a Solution Outcome for the timesheets system would be:
The ability to electronically receive 400 timesheet submissions from employees who are on the corporate network, peaking at 10 submissions a minute.
To validate that the timesheet submission is actually from the employee in question.
The ability to process 400 timesheet submissions and collate them into central database within 5hrs.
To report progress on the import and report if there are any issues when processing the timesheets.

These outcomes are non-technical, and they don't identify how you should implement the solution. This retains the Agility to fulfil the outcomes in various ways.
A full and prioritised list of business and solution outcomes, with the relationships between them is a great outcome from the Analysis stage of the process. It allows the Agile team to approach the problems in an Outcome Orientated way to provide the best business value. It also gives them the freedom to choose the technical implementation, which really should be their call, in association with the Architecture team if required. Once you have a Solution Outcome, you can then make the decision of how to implement this outcome, on a value centric basis, and then the truth can write the stories in association with the XP/Agile Coach in the 'Getting Your Stories Straight'. From this a Planning Game ensues and then you can cut the first iteration.

Upfront estimation of Budget

There still remains the problem of providing an upfront budget estimation. You know least about a project at the start, so it is the worst time to make any decisions about cost. Also, you don't know how you are going to implement each outcome - that should be decided later. However, the business need a guide so that they can make the go/no-go call. In another article, I propose that instead of an absolute budget, the output from the Analysis stage should be a project size rating (1-5) and a confidence rating of its accuracy (%). If an estimated budget is really required for the Analysis stage, then maybe it should work the other way round: how much is it worth for the business to achieve these outcomes? Then ask an Agile team whether they think they can achieve the outcomes within the budget provided.

Pre-Conceptions

Even though the business was unclear on what they want the system to do, they had already decided on an implementation. In the timesheets example, they knew they wanted a web application, written in .NET, writing data into a SQLServer database, the schema of which was already decided. They were also deciding how much it was going to cost and what resource they would need to allocate to implement their chosen technologies. However, they didn't really know what they wanted the system to do, or the best value way of achieving this. They were worrying about the screen design of the web page and the implementation of the user authorisation feature. This is all the wrong way round. To fully enable an Agile/XP project, it is necessary for the Analysis stage to undertake thorough Business Analysis, and identify the scale and risk of the project. They have to accurately define the benefits they want out of the project, and the outcomes that the solution will have to provide to support these. Then if they decide to press on with the proposed project, the Outcome Management and technical implementation should be in the hands of the Agile team who will work in association with Architecture, Security, Brand, etc to ensure that the solution is fit for purpose.

Outcome Orientated Solutions

The Problem

When developers are faced with a problem, it is natural for them to immediately go into 'Solutions Mode' and try to work out how they would implement a system to provide a full solution to the problem. They quickly get into the technical detail, and tend to forget to look at the wider picture.

The Solution

It is important to be Outcome Orientated at this stage in the game and look at the simplest way to provide the outcomes required by the business, and not to assume that a full system implementation is the only way to go.

Cost Benefit

This is all about business value. If we can get a solution out there quickly and cheaply, doing the job and delivering value for the business, then the cost-benefit is far greater than if the full technical solution spends an extended time in development. We will have spent less money on development, and we will be deriving more business value by having the solution out there earlier. Once the initial solution is in place, the business can make the call whether to iteratively progress it towards a more complete technical product, based on cost, risk, performance etc.

Maintaining Quality

Getting a product out there quickly doesn't mean that we hack together some code as fast as we can. All code should be Test Driven, and of the highest quality, and we should never cut corners here to get to market more quickly. We should just look at different ways to deliver each outcome other than developing a full technical solution. For example, instead of a web application form integrating with all the back-end systems from day one, it could write out each application to a file, and a process could be defined where the entries in this file are processed manually. This could later be improved if the business considers that this is a good use for their budget.

Team Structure - F1 Style

What?

OK, it sounds a bit daft, and you might think that my interest in motor sport has addled my brain into thinking my job is in any way related to working in a Formula One team. But just bear with me, its not quite as stupid as it first sounds...

Team Principle

For a Formula One team, there is one simple objective, and that is to develop and hone a driver and car combination that can go faster than its competitors. Although the objective may be simple, the recipe for success is incredibly complex. There are so many elements that have to be brought together to achieve success, and so many different approaches that can be adopted that the primary aim can easily get lost in a mire of complexity. To have any hope of success, the team have to have a common vision and to have that vision instilled in them by someone that they respect, and who has the ability to motivate them. In an F1 team, this is the Team Principle.

Technical Director

The Team Principle has the vision and provides the team with motivation and focus, but doesn't necessarily have the necessary skills to implement that vision in a highly technical arena. Therefore every team has a Technical Director. This is a highly experienced Engineer, who also has excellent management and communication skills. It is his job to take overall responsibility for the design of the car, and to ensure that the various aspects of the car's design all work together to provide the best overall solution. This is an absolutely critical role, because the Technical Director is the one who fully understands the overall technical solution.

Engine Department

If you told your Engine Department to produce the optimal solution, then they would probably go and build a 6 litre, turbo-charged V12 which would produce 3000bhp. They would have figures that would prove beyond doubt that this is the perfect engine, taking into account power, torque and reliability. However, a car built around such an engine would be slow, as it wouldn't go round corners and it would be extremely heavy. Brilliant technical minds tend to forget simple things like this when left to their own devices.

Aerodynamics Department

The Aerodynamics department would build you a beautiful car with a drag co-efficient approaching zero and enough downforce to stick it to the ceiling. However, the driver might be lying down to such a degree that he couldn't drive the car effectively (Brabham '86) or he might not be able to fit in it at all (McLaren '95).

Optimised Overall Solution

You get the picture. Each area is so complex in its own right, that it needs experts in the field working in a dedicated team to provide solutions. However, they will always think that their area is the most important, and therefore optimise their solution to the detriment of the overall package. There needs to be one person who can make the decisions to ensure that the right compromises are reached so that the overall package fulfils the overriding objective: to make the car faster than its competitors.
An example of this sort of decision would be to demand that the engine can run at very high temperatures, so that the radiators that cool the engine can be smaller. Although this doesn't help the engine in any way, it allows the aerodynamics to be improved without compromising in any other area, making the car faster.

Developing IT Solutions

The problems faced by IT organisations are very similar, in that the aim is to provide an extremely complex solution to a fairly simple problem. To stop the complexity and specific areas of expertise taking over, the team (or teams) need a very clear vision to keep them focused. This visionary would probably be the CEO or CIO.
They also need someone to take that vision and understand how to turn it into a technical reality, albeit from a fairly high level. F1 teams call this person the Technical Director - but what is the equivalent role in IT? Each project team needs to be given technical direction and guidance to ensure that their contribution to the solution is consistent with the overall technical vision.

Architecture Gone AWOL

It is interesting to note that this role is rarely apparent in IT organisations. It should really be Architecture that have this sort of control over what is being built, but Architecture tends to sit in its Ivory Tower drawing diagrams and looking at the future direction of the company's technical infrastructure. It is rare that they think that they should have to get their hands dirty and get involved with creating today's solutions. I think this is a really important role, and one that must be filled by an extremely capable technical person who also has great management skills, not a people manager.

Technical Specialists

One difference in IT is that skills can be more transferable. The same person might be equally valuable in writing ASP.NET pages or Web Services. However, experience tells me that dividing your organisation into skills-based teams never works by itself.

How to Organise Your Organisation?

Skills-based teams don't really work. Organising your organisation into teams based on infrastructure (DB, Services, Website) isn't usually that effective either. Cross-skilled project teams seem to work best, but then you tend to get Project Orientated Architecture.

Balance

The solution that I think works the most effectively is a combination of these. You need to have pools of people with certain skills to ensure that you can develop all your systems effectively. These people don't need to sit together, or even work together, but you need to know who can do what and what training requirements they have. You need project teams to drive the business value into your technical solutions, and to provide communication, momentum, motivation and team spirit. These cross skilled teams can be assembled for a given project, and should ideally sit with each other and a business representative who is driving the project.

Governance

You also need to have governance of the various areas of your infrastructure, to ensure that you avoid Project Orientated Architecture. This needs an Architecture that is actively involved in today's projects, including a Technical Director and Architects responsible for each area of the system, providing governance and guidance. These need to balance each other; if one becomes too dominant then it won't work. If projects dominate, you will screw your architecture and future delivery capability. If Architecture dominate, you'll never deliver business value. If skills dominate, you will lose team spirit and motivation, communication will suffer and you will end up with sub-cultures and an us-and-them attitude.

Project Orientated Architecture

Business Requirements

Within most IT organisations, projects are driven by business requirements. For example, the business might want to increase cross-sales across the website. The requirements will come through as a 'Cross Sales' project and usually developers will be put under time pressure to come up with a technical solution.

Developer Reaction

This usually puts developers into 'solutions mode'. This means that they forget that the project is just one business requirement that the technical framework has to support, and instead see the delivering the project as the only thing that matters, and the technical solution will reflect this.

Project Orientated Design

There will be a 'Cross Sales' control in the website, there will be a 'Cross Sales' service in the service layer and there will be a 'Cross Sales' database schema. We will end up with a vertical slice of the system that is purely there to support the Cross Sales business requirement. This may or may not be the appropriate solution, but things start to go really wrong when the second project comes along.

The Next Project

This project is called 'New Offers'. The business want to display offers to customers while they are on the website. The developers come up with the technical solution with a 'New Offers' control in the website, there will be a 'New Offers' service in the service layer and there will be a 'New Offers' database schema. We have another separate solution to satisfy another business requirement, all the way through the stack. This generally continues until you end up with a system that has many, many components and is very complex, hard to maintain and develop.

Project Orientated Architecture makes for overly complex solutions

What Should Happen

What should happen is that the system is built with logical components that support the business requirement, but aren’t built specifically for that particular requirement. This can then be extended to support other business ideas without writing another separate solution.

How To Approach Project Requirements

I think the first rule of thumb is to avoid calling the technical solution after the business requirement. Think about what functionality is called for to satisfy each part of the business requirement. Using the Cross Sales example, the UI control needs to display content in various slots available on each page. Therefore the control is a 'Content Slots' Control. The fact that the content that is served up is to do with Cross Sales isn't really going to have any effect on the solution. The service might have the responsibility of choosing what content to serve to the UI based on the customer profile, so the a 'Targetted Content' service would make sense. The back-end logic can use whatever logic it wants to decide on the appropriate content to send back. In the case of the 'New Offers' project, the back-end code is the only part of the system that should change, and this can go off and work out what new offers should be displayed to the customer, along with any cross-sales adverts or other useful information. It can then decide on the various priorities of the content and deliver this back to the UI, which is simply displaying various pieces of content that it is given, and not caring in the slightest what business outcome it is fulfilling by delivering this content to the customer.

Advantages

This approach will give us a far simpler and more extensible architecture and development and maintenance costs will be far less through the life of the system.

Introduction

This blog focuses on everything to do with developing enterprise scale web applications using Agile, or more specifically XP. It will cover areas such as architecture, security, organisational structure, processes and website accessibility.