Sinking in a data storm? Ideas for investment companies

Posted on : 30-06-2013 | By : richard.gale | In : Data

Tags: , , , , , , , , , , , , , ,


All established organisations have oceans of data and only very basic ways to navigate a path through it

This data builds up over time through interaction with clients, suppliers and other organisations. It is usually stored in different ways on disconnected systems and documents

Trying to Identify what it means on a single system is a big enough challenge, trying to do this across a variety of applications is a much bigger problem with different meaning and interpretations of the fields and terms in the system

How can a company get a ‘360’ view of their client when they have different identifiers in various applications and there is no way of connecting them together. How can you measure the true value of your client when you can only see a small amount of the information you hold about them.

Many attempts have been made to join and integrate these data sets (through architected common data structures, data warehouses, messaging systems, business intelligence applications etc) but it has proved a very expensive and difficult problem to solve. These kind of projects take a long time to implement and the business has often moved on by the time they are ready. In addition early benefits are hard to find so these sorts of projects can often fall victim to termination if a round of cost cutting is required.

So what can be done? Three of the key problems are identification of value from data, duration & costs of data projects and ability to deal with a changing business landscape.

There is no silver bullet but we have been working with a number of Big Data firms and have found a key value from them is the ability to quickly load large volumes of data (both traditional database and unstructured documents, text, multi-media). This technology is relatively cheap and the hardware required is both generic and cheap and again can be easily sourced from cloud vendors.

Using a Hadoop based data store on Amazon cloud or a set of spare servers enables large amounts of data to be uploaded and made available for analysis.

So that can help with the first part, having disparate data in one place. So how to start extracting additional value from that data?

We have found a good way is to start asking questions of the data – “what is the total value of business client X does with my company?” or “what is our overall risk if this counterparty fails?” or “what is my cost of doing business with supplier A vs. supplier B?” if you start building question sets against the data and test & retest you can refine the questions, data and results and answers with higher levels of confidence start appearing. What often happens is that the answers create new questions and so answers etc.

There is nothing new about using data sets to enquire and test but the emerging Big Data technologies allow larger, more complex sets of data to be analysed and cheaper cloud ‘utility’ computing power makes the experimentation economically viable.

What is also good about this is that as the business grows and moves on – to new areas, systems or processes then loading the new data sets should be straightforward and fast. The questions can be re-run and results reappraised quickly and cheaply.

As we have discussed previously we think the most exciting areas within Big Data are the Data science and analytics – find which questions to ask and refining the results.

Visualisation of these results is another area where we see some exciting developments and we will be writing an article on this soon.



Google Eyephone takes on Apple iPhone – what’s next in wearable technology?

Posted on : 31-05-2013 | By : richard.gale | In : Innovation

Tags: , , , , , , ,


Phones are everywhere

Obvious statement but true – mobile & smart phones are the first true utility computing for individuals. The growth has been phenomenal and the integration to our lives has been huge and growing. They We now don’t have to plan how to get somewhere before we start, we don’t need to know how to get somewhere, we have virtually unlimited information on tap and local to our person. This was still a fantasy world just 10 years ago and the convergence of technology and our lives has only just begun.

Phones are limited

Phones are basically blocks of metal and plastic and you need to use at least one of your two hands and both of your eyes  to use it effectively. When you are at a desk this is not a major issue but when you are mobile driving or walking then it becomes a problem. Using phones for anything but calls whilst driving is frowned upon and often illegal, using phones whilst walking can be either annoying for other pavement users or dangerous when crossing streets etc.

The fundamentals of phones have not changed much in the last 20 years, from the original ‘bricks’ they shrank down to a size that was virtually unusable and now the trend is to grow them to a size which, again, is virtually unusable. The new 6″ screens being developed may mean you’ll need a bigger head to use the phone correctly.

There has been huge jump forwards in technology and the amount of functions that can be built into a device but the limitations are holding the thing in your hand and putting it to your ear to make a call.

The next big trend is wearable technology

When I was at college many years ago one of the lecturers specialised in human computer interaction. His classes were often the most entertaining of the course and I still remember him talking about ‘data gloves’ – used by NASA in training astronauts – they were gloves which provided feedback through sensors and feedback motors to provide sensation and feeling so enabling the virtual world to become reality. He did also make reference to  ‘data suits’ and hinted of the ‘fun’ that they could provide through full body sensors – but that’s another story…

Google Glass has been the story of the last few months. Wearable tech that you can use to navigate, find friends, things you want to do and generally give you an amazing amount of information about the world you are travelling through whilst allowing you to keep your hands and most of your vision free.

Whilst laughing off Google Glass Tim Cook has also hinted that Apple is heavily into wearable technology. There have been many rumours of the iWatch and Sony already has a watch paired with it’s Xperia Z smartphone.

We think this is one of the most exciting areas of personal technology growth. Combining clothing, wearable and jewellery and phones/computing must be the way forward to for the next stages of human computer convergence.

Why won’t wearable technology take off?

Overall we think it will but moving from a separate block of technology you can hide in your pocket to something which is either attached or worn by a person will introduce other variables. These have failed in the past – Bluetooth earpieces have not been and never will be fashionable, also trends in fashion may counter some of the aims of the technology companies.

Will people want to wear technology? – now most people under 40 don’t wear watches – if they do it is more likely to be as a statement  rather than to tell the time – their phone has the time and is in constant use – so they don’t need another timepiece. Again with glasses, there is a fashion element of ‘geek chic’ but I’m not sure Google Glass quite has that yet. Apple is perhaps the most fashionable technology brand they may be able overcome this.

What’s next?

As technology gets smaller and cheaper then it will become easier to build into more things. There are already head’s up displays in high end cars and this will become standard with Google streetview overlaying the road you are driving on with real and real-time directions. Clothing will have technology built in – maybe shoes could have Satnavs with sensors directing you left and right whilst you walk, hoodies with headphones, running tee shirts with heart monitors, also as projection is the next big thing for mobiles then you could have a 3D projection of the world around you from your coat as you walk, cycle, drive along the road or a film as you sit on a train or plane.

We are looking forward to the next period in the evolution of truly personal computing.



Big Data: Big Society or Big Brother?

Posted on : 26-01-2012 | By : richard.gale | In : Data

Tags: , , , , , , ,


  • In 1948 George Orwell wrote 1984 to highlight the risks to democracy and freedom through the misuse of information and control of society from the state using the ‘Big Brother’ all seeing eye to illustrate this.
  • In the UK David Cameron has stated that he wants to see a ‘Big Society’ where power and responsibility is transferred away from central government and that people are empowered to control and shape their own communities and environment.

Technological advancements often have positive and negative aspects. They can be great enablers to business and further the democratisation of society through wider access to and the ability to use information to increase knowledge. They can also be used to control and analyse information in less benign ways.

‘Big Data’ is a term which encompasses the increasing amounts and speed of data creation and mechanisms for analysing & using that data which we think is already creating new business and social models.

  • What is the story behind Big Data?
  • Why is it important for both technology & business?
  • What will be the impact?

Could it be the end of privacy where information on all locations, activities and interests are recorded and analysed or may it prove to be a breakthrough in advancement for people, society and organisations?

We at Broadgate have been thinking about the business implications of big data for a while now. We think this has great potential to extract, analyst and gain business value from the data and information already within organisations.

Micro loans data explosion
A data explosion visualised – Micro loans made using the Kiva software

What is Big Data?

The term Big Data describes amounts of data that are too big for conventional data management systems to handle. The volume, velocity and variety of data overwhelm databases and storage. The result is that either data is discarded or unable to be analysed and mined for value.

There has always been Big Data in the sense that data volumes have always exceeded the ability for systems to process it. The tool sets to store & analyse and make sense of the data generally lag behind the quantity and diversity of information sources.

Existing relational databases (such as Oracle & SQL-Server) are excellent at processing large terabyte level volumes of structured, transactional data but are not efficient or even capable of storing and making sense of petabyte and beyond with video, image and other loosely formatted information. The database vendor have recognised this and extended their traditional management systems to cater for Big Data, Microsoft’s next version due in March is focussing on dealing with these themes. Other technologies have also emerged to handle large quantities but converting that into useful information is still proving challenging.

At the same time the value of data is becoming more widely recognised. All organisations rely on access and interpretation of data but only recently the ability to trawl through and make sense of huge amounts of data is seen as a key competitive advantage. Amazon do this very well with their suggestions and hints comparing your purchases and searches with many others with similar patterns. Trading organisations are increasingly looking towards the ‘social internet’ as additional tools to gain insight into market sentiment and direction. Google are the masters at guiding searches towards accurate results based on data and previous results.

The amount of data being produced is increasing exponentially. Increasing use of social networks, unstructured and multimedia data are accelerating this growth month by month.

The scope of the internet is expanding too – the ‘internet of everything’ is here with data from mobile, telemetry, social & machine based sources all available to be exploited.

Challenges of Big Data

Why is Big Data a challenge? What are the fundamental aspects which cause so many issues for conventional storage and analytics tools? An example is weather forecasting. It is a complicated set of problems requiring vast amounts of data combined with complex algorithms to calculate future patterns. It is also a dynamic environment where multiple factors change constantly to produce a multitude of different possible outcomes. Weather forecasting has improved dramatically in the last few years, it is very now very good at predicting what is happening now… and in the short term future but becomes far less precise at longer term predictions.

– Volume

Data volumes in organisations are growing 40% year on year and that is data that is currently being stored and managed. Additional data such as CCTV outputs or SCADA sensors (used in telemetry for industrial systems) are generally stored only for a brief period before being overwritten as the cost of storing the volume of data is so high. To extract value the data has to be stored and to be retrievable in order to analyse and summarise the information.

– Velocity

The speed of data inputting and also outputting is increasing. As volumes are rising so too is the velocity. Over the last few years data has accelerated from generally slow and controlled transactions (such as emails, messages, trades etc. ) to high frequency trading transactions, messaging, mobile and video based information flows. These have pushed the limits of IT infrastructure to allow them to pass with minimal latency impacts without even considering the ability to store and utilise the information value in the data.

– Variety

Data has changed from being largely text & number based to include voice, images, music, video. Again the previous generation of relational databases have difficulty managing these data types and file based storage has challenges running analytical examination of them.

– Data Privacy

As more data is captured and analysed there is the potential to go beyond help organisations learn better about their clients’ buying habits and to discover more private information about them.  Mechanisms for controlling and policing this capability need to be built into Big Data plans.

Current Solutions

Big Data issues are have two main challenges. How to capture and store the data and then how to process it and analyse it for its information value.

– Capture

All the traditional Database manufacturers have Big Data offerings. These range from Hardware & Software combinations (such as Teradata and Oracle’s Big Data appliances) to tuned databases such as SAP’s Sybase IQ.

Storage is another key area where a combination of high capacity and high performance. This often includes utilising cheap commodity storage, including cloud based disk alongside SSD type solutions such as Violin memory to allow faster access to the datasets.

Hadoop from Apache has emerged as a contender to capture large volumes of unstructured data. It can deal with huge quantities and can break up the data into manageable sizes and distribute it across a large set of machines to allow efficient processing.

 – Process

Storing the data solves one half of the Big Data problem but mining or extracting the value out of it is the key to success and is building new business concepts and companies.

Search companies such as Google and Yahoo have utilised technology developed to process this data. Their success and revenue depends on the accuracy of their algorithms to find information people want and also to direct them to clients’ products and services.

Hadoop and NoSQL are tools that can process and analyse the volumes of data. They approach from different angles but essentially they remove the rigid structure of relational databases and can utilise distributed processing methods to handle the volumes and complexity of the data sets. From a ‘traditional’ business perspective these tools are in their infancy. As Big Data grows in importance we can foresee opportunities for companies to build more accessible methods and tools to access and process data using these frameworks. The RDBMS manufacturers have Hadoop, NoSQL interfaces and solutions and this will only gather pace as the land-grab for Big Data grows.

There will be increasing usage of sophisticated Business Intelligence tools such as Qliktech’s Qlikview that can manipulate the outputs and provide further analytical value & visualisation of these complex data sets.

What is the Future?

Getting value or business intelligence from these sets of Big Data is key for businesses now. Having the ability to trawl and analyse the data to identify patterns not seen before and build ‘what-if’ models on the data may bring to light new business opportunities or scientific breakthroughs. In the weather forecasting example above, very long term weather forecasting may help us start to comprehend changes in global temperatures and to then understand the likely impact of global warming.


Could Big Data be the end of privacy or the start of a new age of information empowerment… Well probably neither but also a bit of both. It is aiming to provide tools to deal with the increasing amounts of information that are available and there will undoubtedly be new businesses created from the managing and use of this valuable by-product of technology.

Broadgate have created the Broadgate Big Data Strategy to help organisations deal with these challenges. Please contact us if you would like us to help you?

New ways of visualising information are developing to help depict Big Data. This is an exciting topic and will grow in importance over the coming year. We are working with our associates to publish an article on this soon.


From CIO to CEO – Can clouds break glass ceilings?

Posted on : 24-11-2011 | By : richard.gale | In : Cloud

Tags: , , , , , ,


As technology becomes even more entwined in the fabric of organisations, the opportunities for technology executives will increase. Will a CIO’s potential promotion to CEO be as commonplace as the CFO or COO in the next few years? Historically, with a few technology industry exceptions, it is rare for CIOs of organisations to become CEOs. CEOs either come from the profit-making, client side of the business or the financial area. CIOs are generally seen as managing a silo and being a cost centre rather than being a part of business growth.

How can cloud computing change this? Cloud infrastructure has been hyped to be the answer to almost every technology issue and we think it does have great potential. However, will it change the make-up of a CIO and go as far as to alter the way businesses view them enough to take the chair at the top table?

Well, we think that the IT department in a significant number of organisations will transform radically over the next few years. Let’s take universal banks as an example. Why would a global bank build and operate £200m datacentres? It is nothing to do with their core business and has significant financial, personnel and regulatory complexities. They only do it because they have to. They have vast processing requirements and need to support a huge level of increasing demand. Furthermore, new technologies are always breaking through so the equipment, skills and services constantly have to be upgraded and renewed.

Cloud or Utility computing fundamentally changes this model. If computing is seen as another resource to be switched on and off as required (with an associated usage based charging model) then the basic questions to be answered are:

–          What is the cost of supply compared with others?

–          How reliable, safe and secure is the supply?

–          How flexible and appropriate are the providers?

Obviously it is unlikely that all of the major banks’ IT operations would be placed in the cloud, but it will become the exceptions that are not in the cloud rather than the default.

The traditional IT department would then shrink down to very specific IT functions that were not suited to be run elsewhere. Obviously business-focused change and delivery teams will be the core functions and will keep on growing. Another focus of the ‘IT Department 2016’ will be on management of the demand and supply of technology with vendors. Infrastructure IT will become a relationship management and negotiation function requiring people to change their skillset radically or a different set of resources altogether. The emphasis will be on finding the most appropriate execution venue with external suppliers for an application rather than building the disk or server farm to house it.

So how will this impact the CIO and their future career direction?

–          The move to utility computing will enable CIOs to focus more or real business value and change.

–          CIOs will have to be even more business-orientated, managing external suppliers and their internal customers.

–          CIOs are less likely to be dismissed as ‘techies’ as they will no longer manage large technology-led departments & datacentres.

–          They will be more involved in the strategic future of organisations as the commodity aspects fall away

The modern CIO is already on this road and the future will further embed IT into the backbone of firms.

However, being an essential part of the fabric of an organisation is not enough in itself to get the leading role. Other aspects which are common to CEOs are needed such as the ability to have an external focus, international or overseas experience and proven business experience and qualifications. The business sector also matters – technology & manufacturing organisations currently have many more CIO to CEO promotions than financial services, for instance. But there can be little doubt that the impact of cloud could play a small but important part in ensuring that more CIOs become CEOs in the future.


Clouds, Grids & Meters – The commoditisation of computing. Are there blue skies ahead?

Posted on : 07-05-2011 | By : richard.gale | In : Cloud, Data, Innovation

Tags: , , , , ,


In the last year or so Cloud technologies seem to be everywhere, every bank’s technology department is desperately building private clouds, and most vendors are re-badging  their services as Cloud products.

Cloud computing is advertised on television &  the Underground in London – even my mum knows about the Cloud.

Gartner’s Hype Cycle shows Cloud technologies are currently near the ‘Peak of Inflated Expectations’ and are ready to fall into the ‘Trough of disillusionment’.

Gartner Hype Cycle for Cloud Computing, 2010 (c) Gartner Inc

Cloud computing is and will be a significant part of the future of computing technology – basically because it makes sense to utilise only what you need when you need it – as you do with electricity, water and gas.

Without the hype Cloud will become a utility.  Why would anyone want to spend money on hardware, infrastructure, real-estate, support, licences and resources when there are computing utility companies out there ready to supply you with reliable, secure computing power on demand from their extensive, global grid.

We think a couple other references from the supply of utilities and the market place are crossing over to the computing world:



The location of a electricity meter within your home is a given. It provides both you and the supply company with common reference point the amount of power consumed.

In most IT organisations the usage processing power is more granular, it is generally bought by the server, the processor or maybe a proportion of the machine and charged whether the power is utilised or not. This is changing with the provision of first virtualised and now cloud based services but does metering make sense in this context?

A number of challenges need to be addressed in order for this to work. Your household supply is generally from one provider supplying all the power. The electricity company itself will purchase power from multiple suppliers and sources but you have one retailer.

With Cloud there are multiple vendors potentially both internal and external to your organisation that want to supply you directly with computing resources. It is likely the larger organisations would have more than one supplier for different purposes, pricing or risk mitigation purposes – the hybrid cloud.

There are opportunities here for service companies to provide the metering capability whilst managing the wholesale suppliers of computing – this model could result in better prices for clients as the metering company’s bulk purchasing power should drive the ‘cost per CPU tick’ down.

Smart metering is currently a hot topic in the utilities world and a convergence of computing, electricity & other utilities is likely in the near future.

­­­Virtual Marketplace and Exchanges


Amazon constantly innovates to provide new services and products which has kept it the number one online retailer for years. The success of the ideas varies but the introduction of a virtual marketplace where other vendors can sell using the Amazon site now provides a significant slice of revenue for the company.

Could this be applied to computing power & resources? There is no reason why not – sellers of product could advertise their products and organisations could ‘click and buy’.

As these products are commoditised then the next logical step is the construction of a Cloud Exchange with many buyers and sellers in the market with a constantly changing price depending on demand and supply. A futures market of computing power with derivatives trading on trends and market intelligence would surely follow.

Cloud computing is growing significantly and Forrester expect it to expand from $40B today to $240B by 2020 (differing slightly from the Gartner view) with the significant part of that Software as a Service. We think that Business processes as a Service are the area where a significant advantage can be created from the Cloud and are currently working with our partners to build a strategy to progress this.

It will be interesting to see if and how far organisations will use the cloud for their core revenue generating applications – currently the appetite is very limited due to concerns over security, control & reliability as highlighted in Amazon’s cloud recent outage.

Concerns over hidden costs (such as transition/migration costs, architectural changes & integration aspects) are starting to rise as maturity of the model progresses.

The book is open on how long it is before Cloud becomes so part of the normal world that it ceases to be mentioned – on the ‘Plateau of Productivity’ in Gartner’s words.