Highlights of 2014 and some Predictions for 2015 in Financial Technology

Posted on : 22-12-2014 | By : richard.gale | In : Innovation

Tags: , , , , , , , , , , ,

0

A number of emerging technology trends have impacted financial services in 2014. Some of these will continue to grow and enjoy wider adoption through 2015 whilst additional new concepts and products will also appear.

Financial Services embrace the Start-up community

What has been apparent, in London at least, is the increasing connection between tech and FS. We have been pursuing this for a number of years by introducing great start-up products and people to our clients and the growing influence of TechMeetups, Level39 etc within the financial sector follows this trend. We have also seen some interesting innovation with seemingly legacy technology  – Our old friend Lubo from L3C offers mainframe ‘on demand’ and cut-price, secure Oracle databases an IBM S3 in the cloud! Innovation and digital departments are the norm in most firms now staffed with clever, creative people encouraging often slow moving, cumbersome organisations to think and (sometimes) act differently to embrace different ways of thinking. Will FS fall out of love with Tech in 2015 – we don’t think so. There will be a few bumps along the way but the potential, upside and energy of start-ups will start to move deeper into large organisations.

Cloud Adoption

FS firms are finally facing up to the cloud. Over the last five years we have bored too many people within financial services talking about the advantages of the cloud. Our question ‘why have you just built a £200m datacentre when you are a bank not an IT company?’ was met with many answers but two themes were ‘Security’ and ‘We are an IT company’…. Finally, driven by user empowerment (see our previous article on ‘user frustration vs. empowerment) banks and over financial organisations are ’embracing’ the cloud mainly with SaaS products and IaaS using private and public clouds. The march to the cloud will accelerate over the coming years. Looking back from 2020 we see massively different IT organisations within banks. The vast majority of infrastructure will be elsewhere, development will take place by the business users and the ‘IT department’ will be a combination of rocket scientist data gurus and procurement experts managing and tuning contracts with vendors and partners.

Mobile Payments

Mobile payments have been one of the discussed subjects of the past year. Not only do mobile payments enable customers to pay without getting their wallets out but using a phone or wearable will be the norm in the future. With new entrants coming online every day, offering mobile payment solutions that are faster and cheaper than competitors is on every bank’s agenda. Labelled ‘disruptors’ due to the disruptive impact they are having on businesses within the financial service industry (in particular banks), many of these new entrants are either large non-financial brands with a big customer-base or start-up companies with fresh new solutions to existing issues.

One of the biggest non-financial companies to enter the payments sector in 2014 was Apple. Some experts believe that Apple Pay has the power to disrupt the entire sector. Although Apple Pay has 500 banks signed up and there is competition from card issuers to get their card as the default card option under Apple devices, some banks are still worried that Apple Pay and other similar service will make their branches less important. If Apple chose to go into retail banking seriously by offering current accounts then the banks would have plenty more to worry them.

Collaboration

The fusion of development, operations and business teams to provide agile, focussed solutions has been one of the growth areas in 2014. The ‘DevOps’ approach has transformed many otherwise slow, ponderous IT departments into talking to their business & operational consumers of their systems and providing better, faster and closer-fit applications and processes. This trend is only going to grow and 2015 maybe the year it really takes off. The repercussions for 2016 are that too many projects will become ‘DevOpped’ and start failing through focussing on short term solutions rather than long term strategy.

Security

Obviously the Sony Pictures hack is on everyone’s mind at the moment but protection against cyber attack from countries with virtually unlimited will, if not resources, is a threat that most firms cannot protect against. Most organisations have had a breach of some type this year (and the others probably don’t know it’s happened). Security has risen up to the boardroom and threat mitigation is now published on most firms annual reports. We see three themes emerging to combat this.

– More of the same, more budget and resource is focussed on organisational protection (both technology and people/process)
– Companies start to mitigate with the purchase of Cyber Insurance
– Governments start to move from defence/inform to attacking the main criminal or political motivated culprits

We hope you’ve enjoyed our posts over the last few years and we’re looking forward to more in 2015.

Twitter.com/broadgateview

 

 

Broadgate Big Data Dictionary

Posted on : 28-10-2014 | By : richard.gale | In : Data

Tags: , , , , , , , , , , ,

0

A couple of years back we were getting to grips with big data and thought it would be worthwhile putting a couple of articles together to help explain what the fuss was all about. Big Data is still here and the adoption of it is growing so we thought it would be worthwhile updating and re-publishing. Let us know what you think?

We have been interested in Big Data concepts and technology for a while. There is a great deal of interest and discussion with our clients and associates on the subject of obtaining additional knowledge & value from data.

As with most emerging ideas there are different interpretations and meanings for some of the terms and technologies (including the thinking that ‘big data’ isn’t new at all but just a new name for existing methods and techniques).

With this in mind we thought it would be useful to put together a few terms and definitions that people have asked us about recently to help frame Big Data.

We would really like to get feedback, useful articles & different views on these to help build a more definitive library of Big Data resources.

Analytics 

Big Data Analytics is the processing and searching through large volumes of unstructured and structured data to find hidden patterns and value. The results can be used to further scientific or commercial research, identify customer spending habits or find exceptions in financial, telemetric or risk data to indicate hidden issues or fraudulent activity.

Big Data Analytics is often carried out with software tools designed to sift and analyse large amounts of diverse information being produced at enormous velocity. Statistical tools used for predictive analysis and data mining are utilised to search and build algorithms.

Big Data

The term Big Data describes amounts of data that are too big for conventional data management systems to handle. The volume, velocity and variety of data overwhelm databases and storage. The result is that either data is discarded or unable to be analysed and mined for value.

Gartner has coined the term ‘Extreme Information Processing’ to describe Big Data – we think that’s a pretty good term to describe the limits of capability of existing infrastructure.

There has always been “big data” in the sense that data volumes have always exceeded the ability for systems to process it. The tool sets to store & analyse and make sense of the data generally lag behind the quantity and diversity of information sources.

The actual amounts and types of Big Data this relates to is constantly being redefined as database and hardware manufacturers are constantly moving those limits forward.

Several technologies have emerged to manage the Big Data challenge. Hadoop has become a favourite tool to store and manage the data, traditional database manufacturers have extended their products to deal with the volumes, variety and velocity and new database firms such as ParAccel, Sand & Vectorwise have emerged offering ultra-fast columnar data management systems. Some firms, such as Hadapt, have a hybrid solution utilising tools from both the relational and unstructured world with an intelligent query optimiser and loader which places data in the optimum storage engine.

Business Intelligence

The term Business Intelligence(BI) has been around for a long time and the growth of data and then Big Data has focused more attention in this space. The essence of BI is to obtain value from data to help build business benefits. Big Data itself could be seen as BI – it is a set of applications, techniques and technologies that are applied to an entities data to help produce insight and value from it’s data.

There are a multitude of products that help build Business Intelligence solutions – ranging from the humble Excel to sophisticated (aka expensive) solutions requiring complex and extensive infrastructure to support. In the last few years a number of user friendly tools such as Qlikview and Tableau have emerged allowing tech-savvy business people to exploit and re-cut their data without the need for input from the IT department.

Data Science

This is, perhaps, the most exciting area of Big Data. This is where the Big Value is extracted from the data. One of our data scientist friends described it as follows: ” Big Data is plumbing and that Data Science is the value driver…”

Data Science is a mixture of scientific research techniques, advance programming and statistical skills (or hacking), philosophical thinking (perhaps previously known as ‘thinking outside the box’) and business insight. Basically it’s being able to think about new/different questions to ask, be technically able to intepret them into a machine based format, process the result, interpret them and then ask new questions based from the results of the previous set…

A diagram by blogger Drew Conway  describes some of the skills needed – maybe explains the lack of skills in this space!

 

In addition Pete Warden (creator of the Data Science Toolkit) and others have raised caution on the term Data Science “Anything that needs science in the name is not a real science” but confirms the need to have a definition of what Data Scientists do.

Database

Databases can generally be divided into structured and unstructured.

Structured are the traditional relational database management systems such as Oracle, DB2 and SQL-Server which are fantastic at organising large volumes of transactional and other data with the ability to load and query the data at speed with an integrity in the transactional process to ensure data quality.

Unstructured are technologies that can deal with any form of data that is thrown at them and then distribute out to a highly scalable platform. Hadoop is a good example of this product and a number of firms now produce, package and support the open-source product.

Feedback Loops

Feedback loops are systems where the output from the system are fed back into it to adjust or improve the system processing. Feedback loops exist widely in nature and in engineering systems – think of an oven – heat is applied to warm to a specific temperature and is measured by a thermostat – once the correct temperature is reached the thermostat informs the heating element and it shuts down until feedback from the thermostat says it is getting too cold and it turns on again… and so on.

Feedback loops are an essential part of extracting value from Big Data. Building in feedback and then incorporating Machine Learning methods start to allow systems to become semi-autonomous, this allows the Data Scientists to focus on new and more complex questions whilst testing and tweaking the feedback from their previous systems.

Hadoop

Hadoop is one of the key technologies to support the storage and processing of Big Data. Hadoop emerged from Google and its distributed Google File System and Mapreduce processing tools. It is an open source product under the Apache banner but, like Linux, is distributed by a number of commercial vendors that add support, consultancy and advice on top of the products.

Hadoop is a framework for running applications on large clusters of commodity hardware. The Hadoop framework transparently provides applications both reliability and data motion. Hadoop implements a computational paradigm named map/reduce, where the application is divided into many small fragments of work, each of which may be executed or re-executed on any node in the cluster. In addition, it provides a distributed file system that stores data on the compute nodes, providing very high aggregate bandwidth across the cluster. Both map/reduce and the distributed file system are designed so that node failures are automatically handled by the framework.

So Hadoop could almost be seen as a (big) bucket where you can throw any form and quantity of data into it and it will organise and know where that data resides and can retrieve and process it. It also accepts that there may be holes in the bucket and can patch them up by using additional resources to patch itself up – all in all very clever bucket!!

Hadoop runs on a scheduling basis so when a question is asked it breaks up the query and shoots them out to different parts of the distributed network in parallel and then waits and collates the answers.

Hive

Hive provides a high level, simple, SQL type language to enable processing of and access to data stored in Hadoop files. Hive can provide analytical and business intelligence capability on top of Hadoop. The Hive queries are translated into a set of MapReduce jobs to run against the data. The technology is used by many large technology firms in their products including Facebook and Last.FM. The latency/batch related limitations of MapReduce are present in Hive too but the language allows non-Java programmers to access and manipulate large data sets in Hadoop.

Machine Learning

Machine learning is one of the most exciting concepts in the world of data. The idea is not new at all but the focus on utilising feedback loops of information and algorithms that take actions and change depending on the data without manual intervention could improve numerous business functions. The aim is to find new or previously unknown patterns & linkages between data items to obtain additional value and insight. An example of machine learning in action is Netflix which is constantly trying to improve its movie recommendation system based on a user’s previous viewing, their characteristics and also the features of their other customers with a similar set of attributes.

MapReduce

Mapreduce is a framework for processing large amounts of data across a large number of nodes or machines.

http://code.google.com/edu/parallel/img/mrfigure.png
Map Reduce diagram (courtesy of Google)

Mapreduce works by splitting out (or mapping) requests into multiple separate tasks to be performed on many nodes of the system and then collates and summarises the results back (or reduces) to the outputs.

Mapreduce based on the java language and is the basis of a number of the higher level tools (Hive, Pig) used to access and manipulate large data sets.

Google (amongst others) developed and use this technology to process large amounts of data (such as documents and web pages trawled by its web crawling robots). It allows the complexity of parallel processing, data location and distribution and also system failures to be hidden or abstracted from the requester running the query.

MPP

MPP stands for massively parallel processing and it is the concept which gives the ability to process the volumes (and velocity and variety) of data flowing through systems. Chip processing capabilities are always increasing but to cope with the faster increasing amounts of data processing needs to be split across multiple engines. Technology that can split out requests into equal(ish) chunks of work, manage the processing and then join the results has been difficult to develop.  MPP can be centralised with a cluster of chips or machines in a single or closely coupled cluster or distributed where the power of many distributed machines are used (think ‘idle’ desktop PCs overnight usage as an example). Hadoop utilises many distributed systems for data storage and processing and also has fault tolerance built in which enables processing to continue with the loss of some of those machines.

NoSQL

NoSQL really means ‘not only SQL’, it is the term used for database management systems that do not conform to the traditional RDBMS model (transactional oriented data management systems based on the ACID principle). These systems were developed by technology companies in response to challenges raised by the high volumes of data. Amazon, Google and Yahoo built NoSQL systems to cope with the tidal wave of data generated by their users.

Pig

Apache Pig is a platform for analysing huge data sets. It has a high-level language called Pig Latin which is combined with a data management infrastructure which allows high levels of parallel processing. Again, like Hive, the Pig Latin is compiled into MapReduce requests. Pig is also flexible so additional functions and processing can be added by users for their own specific needs.

Real Time

The challenges in processing the “V”‘s in big data (volume, velocity and variety) have meant that some requirements have been compromised. It the case of Hadoop and Mapreduce this has been the interactive or instant availability of the results. Mapreduce is batch orientated in the sense that requests are sent for processing where they are then scheduled to be run and then the output summarised. This works fine for the original purposes but now the ability to become more real-time or interactive are growing. With a ‘traditional’ database or application users expect the results to be available instantly or pretty close to instant. Google and others are developing more interactive interfaces to Hadoop. Google has Drill and Twitter has release Storm. We see this as one of the most interesting areas of development in the Big Data space at the moment.

 

Over the next few months we have some guest contributors penning their thoughts on the future for big data, analytics and data science.  Also don’t miss Tim Seears’s (TheBigDataPartnership) article on maximising value from your data “Feedback Loops” published here in June 2012.

For the technically minded Damian Spendel also published some worked examples using ‘R’ language on Data Analysis and Value at Risk calculations.

These are our thoughts on the products and technologies – we would welcome any challenges or corrections and will work them into the articles.

 

Big Data – Can it win Big Games?

Posted on : 30-04-2014 | By : richard.gale | In : Data

Tags: , , , , ,

0

Malcolm Lewis is in the news with his new book on high frequency trading. He also wrote a book, Moneyball, on sports data back in 2003 on how the Oakland A’s baseball club made extensive use of data research to out gun much higher spending rivals and get to the play-offs two years in a row.

Now with Liverpool Football Club riding high at the top of the league, much credit should be given to its manager, Brendon Rodgers, but there is a vast backroom team sifting through terabytes of data which is assisting him. English Premier League football is big business and the spoils of winning are worth tens of millions. Anything which can give an edge in team selection or insight into the opposition is worth it’s weight in silverware. A new breed of technology companies track every pass, movement and save to gain insight and work out the best way to win.

There are millions to be won or lost in the Premier League. A massive television audiences make it the world’s richest league, expected to make £3.5 billion in revenues this season. But although Premier League clubs make a lot, they spend most of it too. Buying the top talent is pricey – players salaries drain more than 70% of an average club’s takings, and for some that figure is 90% plus. Not buying the best talent and so get relegated is very expensive and costs tens of millions to a club. Getting back up can be a struggle and teams are left struggling with huge wage bills and not enough income to cover the costs. Relegation can result in bankruptcy. Fielding the right team is essential and clubs employee spotters that travel the globe looking for the right talent at a good price. These researchers use their experience and knowledge to pick the next generation talent, often using instinct over data. But now companies such as Opta and Prozone collect reams of helpful data, selling them to the clubs and media for a fee. Pitch-side analysts log every tackle, pass and goal, typically collecting information on 2,000 events per match. Above the stadium, arrays of cameras track players’ movements, logging their distance, speed and acceleration.

The capturing and analysis of data can offer new insight into players value. Gareth Bale, one of the finest players in the world is seen as a strong, fast goal scorer with amazing power and accuracy from a distance. Analysing the data shows that he also plays an important defensive contribution to his club, something that may not be obvious from the play. That helped his previous club Spurs as much as his goals did.

If clubs use the same criteria to crunch the data on players in the lower leagues they may be able to purchase ‘Bale’ quality skills at a lower price. Chelsea have carried out a lot of research in this space with data on all players in fifteen leagues around the globe.

American sports show that this approach can work. As mentioned at the top of the article, at the turn of the century Oakland Athletics, a poor badly resourced team, were playing badly. Then they started to analyse the huge data sets available in baseball to spot under priced players, getting them at budget prices. It worked: in 2002 the Athletics enjoyed a record-breaking 20-game winning streak. With reams of new data available, Premier League clubs are taking notice. Last year Liverpool recruited a data scientist with a PhD in biological physics.

Still, computers are not going to make traditional scouting redundant just yet. Human analysis can take into account the context of play including quality of support a player has which can change his behaviour – such as booting up a long ball if he thinks his midfielders are of low quality and will lose the ball…

 

From a single view of a customer to a global view of an individual – bespoke banking for the mass market

Posted on : 02-09-2013 | By : richard.gale | In : Innovation

Tags: , , , , , , , , , , ,

0

Customer interaction with banks can be complex. Historically this has resulted in lost opportunities for both institutions and their clients with neither obtaining full value from the relationship. Forward looking banks are addressing this through changes in thinking and technology.

Banks have many touch points with their existing & potential clients;

  • Accounts –  such as current, saving, loan, share trading, business/personal, mortgages
  • Products – such as life insurance, pensions or advisory services
  • Channels – face-to-face, telephone, ATM, web application, mobile, social media and a multitudes of formats in advertising and marketing
  • History – banks have bought and absorbed many different, divergent firms and may not have fully integrated across people, process & systems

The complexity potential of this interaction combined with the sometimes disjointed nature of these organisations mean that connections are not made and so opportunities can be lost, customers can feel undervalued whilst increasing the potential for fraud.

Change – Cultural & organisational integration

Most banks are huge organisations with thousands of staff based around the globe. To scale the organisation, roles have become more specialised and most people have deep skills in relatively narrow fields of the banks overall capability.

This has worked well and has enabled the global growth of the organisation but opportunities are being missed to further grow customers and clients through the consolidation of information and consistency of customer experience.

That additional value can be enabled by a cultural shift towards a ‘one bank’ philosophy, most banks have these programmes in place and seem to work at the infrastructure level but a different way of thinking that gives an incentive to think about other areas that could help their customer.

To enable this to work there would need to be a supporting framework in place;

  1. Knowledge of the other areas/business units/geography – a simple view of a complex environment is critical
  2. Open & effective communication channels – the mechanism is less important than the knowledge that it is available and there are people listening and willing to help
  3. Communication needs to be valued and seen to be valued by all levels with the business

Improve – Customer Relations

Timely, accurate & complete customer intelligence  is critical. Who, what where are your customers? What do they do, what do they like & dislike and what are their dreams? Gaining this insight into your customer’s mind and tailoring communications & solutions to match this will make them want to do more business with you.

A major factor in achieving this will be to collate & analyse all possible information and so having a single point (such as customer relationship team)  accountable for ensuring its accuracy & completeness will help this process.

Having a more complete set of information in regard to your customer will help understand their needs and, with a consistent approach to communication, also help avoid alienating them through providing inaccurate or inappropriate information or advice.

As important to consistency & completeness is the longevity  of the relationship. Customers in the past have generally stayed with the same bank for a considerable time, this ‘stickiness’ is now being eroded through;

  • Improved knowledge – of other options available
  • Legislation – forcing switching of accounts to me made easier
  • Changing attitudes – people are commoditising purchasing and usage based on value and quality ahead buying from a single company
  • Technology – information from many sources & companies are available on a phone or tablet

The relationship between a customer and a bank is similar to any long term partnership, it’s based on a set of core features; trust, openness, well-being. equality amongst others.

Thinking about these principles when engaging with a customer will only help the relationship endure.

Integrate – Infrastructure, systems & applications

Large scale, standardised technology has been the norm for banks interacting with their customers. This works and has been the only real way to handle the millions of transactions from thousands of customers in the past.

That same core technology still underpins the banking world but with the advances in capability & speed and parallel reduction in cost there is an opportunity to build a view of the individual and then start providing bespoke services on a manufacturing scale.

The move to more customer centric technology should enable the standard bank account holder to experience a ‘Saville Row’ world for a Marks & Spencer price.

An impact of this may be that the Private banking and Wealth management divisions of banks will have to raise their level of service to differentiate from the ‘norm’.

The use of data analytics to search through the volumes of data and analyse and extract insight and value from it are essential tools to achieve the bespoke solution.

Big Data databases and tool-kits can help provide the framework but knowledgeable teams of people with both the understanding of the customers and technology will be required to provide answers and the next set of questions to achieve an even greater level of customer satisfaction, retention and growth.

Sinking in a data storm? Ideas for investment companies

Posted on : 30-06-2013 | By : richard.gale | In : Data

Tags: , , , , , , , , , , , , , ,

1

All established organisations have oceans of data and only very basic ways to navigate a path through it

This data builds up over time through interaction with clients, suppliers and other organisations. It is usually stored in different ways on disconnected systems and documents

Trying to Identify what it means on a single system is a big enough challenge, trying to do this across a variety of applications is a much bigger problem with different meaning and interpretations of the fields and terms in the system

How can a company get a ‘360’ view of their client when they have different identifiers in various applications and there is no way of connecting them together. How can you measure the true value of your client when you can only see a small amount of the information you hold about them.

Many attempts have been made to join and integrate these data sets (through architected common data structures, data warehouses, messaging systems, business intelligence applications etc) but it has proved a very expensive and difficult problem to solve. These kind of projects take a long time to implement and the business has often moved on by the time they are ready. In addition early benefits are hard to find so these sorts of projects can often fall victim to termination if a round of cost cutting is required.

So what can be done? Three of the key problems are identification of value from data, duration & costs of data projects and ability to deal with a changing business landscape.

There is no silver bullet but we have been working with a number of Big Data firms and have found a key value from them is the ability to quickly load large volumes of data (both traditional database and unstructured documents, text, multi-media). This technology is relatively cheap and the hardware required is both generic and cheap and again can be easily sourced from cloud vendors.

Using a Hadoop based data store on Amazon cloud or a set of spare servers enables large amounts of data to be uploaded and made available for analysis.

So that can help with the first part, having disparate data in one place. So how to start extracting additional value from that data?

We have found a good way is to start asking questions of the data – “what is the total value of business client X does with my company?” or “what is our overall risk if this counterparty fails?” or “what is my cost of doing business with supplier A vs. supplier B?” if you start building question sets against the data and test & retest you can refine the questions, data and results and answers with higher levels of confidence start appearing. What often happens is that the answers create new questions and so answers etc.

There is nothing new about using data sets to enquire and test but the emerging Big Data technologies allow larger, more complex sets of data to be analysed and cheaper cloud ‘utility’ computing power makes the experimentation economically viable.

What is also good about this is that as the business grows and moves on – to new areas, systems or processes then loading the new data sets should be straightforward and fast. The questions can be re-run and results reappraised quickly and cheaply.

As we have discussed previously we think the most exciting areas within Big Data are the Data science and analytics – find which questions to ask and refining the results.

Visualisation of these results is another area where we see some exciting developments and we will be writing an article on this soon.

 

 

Broadgate Big Data Dictionary Part Two

Posted on : 31-08-2012 | By : richard.gale | In : Cloud, Data

Tags: , , , , ,

4

Last month we started our Big Data Dictionary to help define terms and concepts in the world of data management. Many of our readers have said it has been useful so we are continuing it this month and will focus more on some of the products and technologies that are emerging in this space.

Over the next few months we have some guest contributors penning their thoughts on the future for big data, analytics and data science.  Also don’t miss Tim Seears’s (TheBigDataPartnership) article on maximising value from your data “Feedback Loops” published here in June.

So continuing the theme from last month:

Hive

Hive provides a high level, simple, SQL type language to enable processing of and access to data stored in Hadoop files. Hive can provide analytical and business intelligence capability on top of Hadoop. The Hive queries are translated into a set of MapReduce jobs to run against the data. The technology is used by many large technology firms in their products including Facebook and Last.FM. The latency/batch related limitations of MapReduce are present in Hive too but the language allows non-Java programmers to access and manipulate large data sets in Hadoop.

Machine Learning

Machine learning is one of the most exciting concepts in the world of data. The idea is not new at all but the focus on utilising feedback loops of information and algorithms that take actions and change depending on the data without manual intervention could improve numerous business functions. The aim is to find new or previously unknown patterns & linkages between data items to obtain additional value and insight. An example of machine learning in action is Netflix which is constantly trying to improve its movie recommendation system based on a user’s previous viewing, their characteristics and also the features of their other customers with a similar set of attributes.

MapReduce

Mapreduce is a framework for processing large amounts of data across a large number of nodes or machines.

http://code.google.com/edu/parallel/img/mrfigure.png

Map Reduce diagram (courtesy of Google)

Mapreduce works by splitting out (or mapping) requests into multiple separate tasks to be performed on many nodes of the system and then collates and summarises the results back (or reduces) to the outputs.

Mapreduce based on the java language and is the basis of a number of the higher level tools (Hive, Pig) used to access and manipulate large data sets.

Google (amongst others) developed and use this technology to process large amounts of data (such as documents and web pages trawled by its web crawling robots). It allows the complexity of parallel processing, data location and distribution and also system failures to be hidden or abstracted from the requester running the query.

MPP

MPP stands for massively parallel processing and it is the concept which gives the ability to process the volumes (and velocity and variety) of data flowing through systems. Chip processing capabilities are always increasing but to cope with the faster increasing amounts of data processing needs to be split across multiple engines. Technology that can split out requests into equal(ish) chunks of work, manage the processing and then join the results has been difficult to develop.  MPP can be centralised with a cluster of chips or machines in a single or closely coupled cluster or distributed where the power of many distributed machines are used (think ‘idle’ desktop PCs overnight usage as an example). Hadoop utilises many distributed systems for data storage and processing and also has fault tolerance built in which enables processing to continue with the loss of some of those machines.

NoSQL

NoSQL really means ‘not only SQL’, it is the term used for database management systems that do not conform to the traditional RDBMS model (transactional oriented data management systems based on the ACID principle). These systems were developed by technology companies in response to challenges raised by the high volumes of data. Amazon, Google and Yahoo built NoSQL systems to cope with the tidal wave of data generated by their users.

Pig

Apache Pig is a platform for analysing huge data sets. It has a high-level language called Pig Latin which is combined with a data management infrastructure which allows high levels of parallel processing. Again, like Hive, the Pig Latin is compiled into MapReduce requests. Pig is also flexible so additional functions and processing can be added by users for their own specific needs.

Real Time

The challenges in processing the “V”‘s in big data (volume, velocity and variety) have meant that some requirements have been compromised. It the case of Hadoop and Mapreduce this has been the interactive or instant availability of the results. Mapreduce is batch orientated in the sense that requests are sent for processing where they are then scheduled to be run and then the output summarised. This works fine for the original purposes but now the ability to become more real-time or interactive are growing. With a ‘traditional’ database or application users expect the results to be available instantly or pretty close to instant. Google and others are developing more interactive interfaces to Hadoop. Google has Drill and Twitter has release Storm. We see this as one of the most interesting areas of development in the Big Data space at the moment.

These are our thoughts on the products and technologies – we would welcome any challenges or corrections and will work them into the articles.

 

 

Broadgate Big Data Dictionary Part One

Posted on : 26-07-2012 | By : richard.gale | In : Data

Tags: , , , , , , , , , , , , , , , ,

2

We have been interested in Big Data concepts and technology for a while. There is a great deal of interest and discussion with our clients and associates on the subject of obtaining additional knowledge & value from data.

As with most emerging ideas there are different interpretations and meanings for some of the terms and technologies (including the thinking that ‘big data’ isn’t new at all but just a new name for existing methods and techniques).

With this in mind we thought it would be useful to put together a few terms and definitions that people have asked us about recently to help frame Big Data.

We would really like to get feedback, useful articles & different views on these to help build a more definitive library of Big Data resources.  We’ve started with a few basic terms and next month with cover some of the firms developing solutions – this is just a starting point…

Analytics 

Big Data Analytics is the processing and searching through large volumes of unstructured and structured data to find hidden patterns and value. The results can be used to further scientific or commercial research, identify customer spending habits or find exceptions in financial, telemetric or risk data to indicate hidden issues or fraudulent activity.

Big Data Analytics is often carried out with software tools designed to sift and analyse large amounts of diverse information being produced at enormous velocity. Statistical tools used for predictive analysis and data mining are utilised to search and build algorithms.

Big Data

The term Big Data describes amounts of data that are too big for conventional data management systems to handle. The volume, velocity and variety of data overwhelm databases and storage. The result is that either data is discarded or unable to be analysed and mined for value.

Gartner has coined the term ‘Extreme Information Processing’ to describe Big Data – we think that’s a pretty good term to describe the limits of capability of existing infrastructure.

There has always been Big Data in the sense that data volumes have always exceeded the ability for systems to process it. The tool sets to store & analyse and make sense of the data generally lag behind the quantity and diversity of information sources.

The actual amounts and types of Big Data this relates to is constantly being redefined as database and hardware manufacturers are constantly moving those limits forward.

Several technologies have emerged to manage the Big Data challenge. Hadoop has become a favourite tool to store and manage the data, traditional database manufacturers have extended their products to deal with the volumes, variety and velocity and new database firms such as ParAccel, Sand & Vectorwise have emerged offering ultra-fast columnar data management systems. Some firms, such as Hadapt, have a hybrid solution utilising tools from both the relational and unstructured world with an intelligent query optimiser and loader which places data in the optimum storage engine.

Business Intelligence

The term Business Intelligence(BI) has been around for a long time and the growth of data and then Big Data has focused more attention in this space. The essence of BI is to obtain value from data to help build business benefits. Big Data itself could be seen as BI – it is a set of applications, techniques and technologies that are applied to an entities data to help produce insight and value from it’s data.

There are a multitude of products that help build Business Intelligence solutions – ranging from the humble Excel to sophisticated (aka expensive) solutions requiring complex and extensive infrastructure to support. In the last few years a number of user friendly tools such as Qlikview and Tableau have emerged allowing tech-savvy business people to exploit and re-cut their data without the need for input from the IT department.

Data Science

This is, perhaps, the most exciting area of Big Data. This is where the Big Value is extracted from the data. One Data Scientist partner of ours described as follows: ” Big Data is plumbing and that Data Science is the value driver…”

Data Science is a mixture of scientific research techniques, advance programming and statistical skills (or hacking), philosophical thinking (perhaps previously known as ‘thinking outside the box’) and business insight. Basically it’s being able to think about new/different questions to ask, be technically able to intepret them into a machine based format, process the result, interpret them and then ask new questions based from the results of the previous set…

A diagram by blogger Drew Conway  describes some of the skills needed – maybe explains the lack of skills in this space!

In addition Pete Warden (creator of the Data Science Toolkit) and others have raised caution on the term Data Science “Anything that needs science in the name is not a real science” but confirms the need to have a definition of what Data Scientists do.

Database

Databases can generally be divided into structured and unstructured.

Structured are the traditional relational database management systems such as Oracle, DB2 and SQL-Server which are fantastic at organising large volumes of transactional and other data with the ability to load and query the data at speed with an integrity in the transactional process to ensure data quality.

Unstructured are technologies that can deal with any form of data that is thrown at them and then distribute out to a highly scalable platform. Hadoop is a good example of this product and a number of firms now produce, package and support the open-source product.

Feedback Loops

Feedback loops are systems where the output from the system are fed back into it to adjust or improve the system processing. Feedback loops exist widely in nature and in engineering systems – think of an oven – heat is applied to warm to a specific temperature and is measured by a thermostat – once the correct temperature is reached the thermostat informs the heating element and it shuts down until feedback from the thermostat says it is getting too cold and it turns on again… and so on.

Feedback loops are an essential part of extracting value from Big Data. Building in feedback and then incorporating Machine Learning methods start to allow systems to become semi-autonomous, this allows the Data Scientists to focus on new and more complex questions whilst testing and tweaking the feedback from their previous systems.

Hadoop

Hadoop is one of the key technologies to support the storage and processing of Big Data. Hadoop emerged from Google and its distributed Google File System and Mapreduce processing tools. It is an open source product under the Apache banner but, like Linux, is distributed by a number of commercial vendors that add support, consultancy and advice on top of the products.

Hadoop is a framework for running applications on large clusters of commodity hardware. The Hadoop framework transparently provides applications both reliability and data motion. Hadoop implements a computational paradigm named map/reduce, where the application is divided into many small fragments of work, each of which may be executed or re-executed on any node in the cluster. In addition, it provides a distributed file system that stores data on the compute nodes, providing very high aggregate bandwidth across the cluster. Both map/reduce and the distributed file system are designed so that node failures are automatically handled by the framework.

So Hadoop could almost be seen as a (big) bucket where you can throw any form and quantity of data into it and it will organise and know where that data resides and can retrieve and process it. It also accepts that there may be holes in the bucket and can patch them up by using additional resources to patch itself up – all in all very clever bucket!!

Hadoop runs on a scheduling basis so when a question is asked it breaks up the query and shoots them out to different parts of the distributed network in parallel and then waits and collates the answers.

 

We will continue this theme next month and then start discussing some of the technology organisations involve in more detail, such as covering Hive, Machine Learning, MapReduce, NoSQL and Pig.

 

Feedback Loops: How to maximise value from your Big Data

Posted on : 27-06-2012 | By : richard.gale | In : General News

Tags: , , , , , , , , , , , , , , , , ,

3

Closing the feedback loop

With businesses becoming increasingly sensitive to customer opinion of their brand, monitoring consumer feedback is becoming ever more important.  Additionally, the recognition of social media as an important and valid source of customer opinion has brought about a need for new systems and a new approach.

Traditional approaches of reactive response to any press coverage by a PR department, or conducting infrequent customer surveys whether online or by phone are all part of extremely slow-cycle feedback loops, no longer adequate to capture the ever-changing shifts in public sentiment.

They represent a huge hindrance to any business looking to improve brand relations; delay in feedback can cost real money.  Inevitably, the manual sections of traditional approaches create huge delays in information reaching its users.  These days, we need constant feedback and we need low-latency – the information needs to be almost real-time.  Wait a few moments too long, and suddenly the intelligence you captured could be stale and useless.

 

A social media listening post

Witness the rise of the “social media listening post”: a new breed of system designed to plug directly in to social networks, constantly watching for brand feedback automatically around the clock.  Some forward-thinking companies have already built such systems.  How does yours keep track right now?  If your competitors have it and you don’t, does that give them a competitive advantage over you?

I’d argue for the need for most big brands to have such a system these days.  Gone are the days when businesses could wait months for surveys or focus groups to trickle back with a sampled response from a small select group.  In that time, your brand could have been suffering ongoing damage, and by the time you find out, valuable customers have been lost.  Intelligence is readily available these days on an near-instantaneous basis, can you afford not to use it?

Some emerging “Big Data” platforms offer the perfect tool for monitoring public sentiment toward a company or brand, even in the face of the rapid explosion in data volumes from social media, which could easily overwhelm traditional BI analytics tools.  By implementing a social media “listening post” on cutting-edge Big Data technology, organisations now have the opportunity to unlock a new dimension in customer feedback and insight into public sentiment toward their brands.

Primarily, we must design the platform for low-latency continuous operation to allow complete closure of the feedback loop – that is to say, events (news, ad campaigns etc) can be monitored for near-real time positive/negative/neutral response by the public – thus bringing rapid response, corrections in strategy, into the realm of possibility.  Maybe you could just pull that new ad campaign early if something disastrous and unexpected happened to public reaction to the material?  It’s also about understanding trends and topics of interest to a brand audience, and who are the influencers.  Social media platforms like Twitter offer a rich granular model for exploring this complex web of social influence.

The three main challenges inherent in implementing a social media listening post are:

  • Data volume
  • Complexity of data integration – e.g. unstructured, semi-structured, evolving schema etc
  • Complexity of analysis – e.g. determining sentiment: is it really a positive or negative statement with respect to the brand?

To gain a complete picture of public opinion towards your brand or organisation through social media, many millions of web sites and data services must be consumed, continuously around the clock.  They need to be analysed in complex ways, far beyond traditional data warehouse query functionality.  Even just a sentiment analysis capability on its’ own poses a considerable challenge, and as a science is still an emerging discipline, but even more advanced techniques in Machine Learning may prove necessary to correctly interpret all signals from the data.  Data format will vary greatly among social media sources, ranging from regular ‘structured’ data through semi-and unstructured forms, to complex poly-structured data with many dimensions.  This structural complexity poses extreme difficulty for traditional data warehouses and up-front ETL (Extract-Transform-Load) approaches, and demands a far more flexible data consumption platform.

So how do we architect a system like this?  Generally speaking, at its core you will need some kind of distributed data capture and analysis platform.  Big Data platforms were designed to address problems where you have Volume, Variety, or Velocity of data – and most often, all three.  In this particular use-case, we need to look towards the cutting-edge of the technology, and look for a platform which supports near-real time, streaming data capture and analysis, with the capability to implement Machine Learning algorithms for the analytics/sentiment analysis component.

For the back-end, a high-throughput data capture/store/query capability is required, suitable for continuous streaming operation, probably with redundancy/high-availability, and a non-rigid schema layer capable of evolving over time as the data sources evolve.  So-called “No-SQL” database systems (which in fact stands for “Not Only SQL” rather than NO SQL) such as Cassandra, HBase or MongoDB offer excellent properties for high-volume streaming operation, and would be well suited to the challenge, or there are also commercial derviatives of some of these platforms on the market, such as the excellent Acunu Data Platform which commercialises Cassandra.

Additionally a facility for complex analytics, most likely via parallel, shared-nothing computation (due to the extreme data volumes) will be required to derive any useful insight from the data you capture.  For this component, paradigms like MapReduce are a natural choice, offering the benefits of linear scalability and unlimited flexibility in implementing custom algorithms, and libraries of Machine Learning such as the great Apache Mahout project have grown up around providing a toolbox of analytics on top of the MapReduce programming model.  Hadoop is an obvious choice when it comes to exploiting the MapReduce model, but since the objective here is to achieve near-real time streaming capability, it may not always be the best choice.  Cassandra and HBase (which in fact runs on Hadoop) can be a good choice since they offer the low-latency characteristics, coupled with MapReduce analytic capabilities.

Finally, some form of front-end visualization/analysis layer will be necessary to graph and present results in a usable visual form.  There are some new open-source BI Analytics tools around which might do the job, or a variety of commercial offerings in this area.  The exact package to be selected for this component is strongly dependent on the desired insight and form of visualization and so is probably beyond the scope of this article, but of course requirements are clearly that it needs to interface with whatever back-end storage layer you choose.

Given the cutting-edge nature of many of the systems required, a solid operational team is really essential to maintain and tune the system for continuous operation.  Many of these products have complex tuning requirements demanding specialist skill with dedicated headcount.  Some of the commercial open-source offerings have support packages that can help mitigate this requirement, but either way, the need for operational resource must never be ignored if the project is to be a success.

The technologies highlighted here are evolving rapidly, with variants or entirely new products appearing frequently, as such it would not be unreasonable to expect significant advancement in this field within the 6-12 month timeframe.  This will likely translate into advancement on two fronts: increased and functional capability of the leading distributed data platforms in areas such as query interface and indexing capability, and reduced operational complexity and maintenance requirements.

Tim Seears – CTO Big Data Partnership.

www.bigdatapartnership.com

Big Data Partnership and Broadgate Consultants are working together to help organisations unlock the value in their big data.  This partnership allows us to provide a full suite of services from thought-leadership, strategic advice and consultancy through delivery and implementation of robust, supportable big data solutions.  We would be delighted to discuss further the use case outlined above should you wish to explore adapting these concepts to your own business.

How Much is Your Data Worth?

Posted on : 29-05-2012 | By : richard.gale | In : Data

Tags: , , , , , ,

0

 

Data is like Oil,  sort of…

  • We are completely dependent upon it to go about our daily lives
  • It is difficult and expensive to locate and extract and vast tracts of it are currently inaccessible.
  • As technology improves we are able to obtain more of it but the demand constantly outpaces supply.
  • The raw material is not worth much and it is the processing which provides the value, fuels & plastics in the case of oil and business intelligence from data.
  • It lubricates the running of an organisation in the same way as oil does for a car.
  • The key difference between oil and data is that the supply of data is increasing at an ever faster rate whilst the amount of oil is fixed.

 

So how can data be valued and what exploration mechanisms are available to exploit this asset?

The recent valuation of Facebook at ~$100B shows the value of data to the market. Facebook itself has tangible (accounts friendly) assets of around $8-10B but the potential value of its data and growth gives rise to the high price investors are willing to pay.

The Facebook and other social media IPO valuations has highlighted that calculating a company’s data worth or value has not built into most more established organisations price. The economic value of a firm’s information assets has recently been termed ‘data equity’ and a new economics discipline, Infonomics, is emerging to provide a structure and foundation of measuring value in data.

The value and so price of organisations could radically alter as the value of its data becomes more transparent. Data equity will at some point be added to the balance sheet of established firms potentially significantly affecting the share price – think about Dun & Bradstreet, the business intelligence service – they have vast amounts of information on businesses and individuals which is sold to help organisations make decisions in terms of credit worthiness. Does the price of D&B reflect the value of that data? Probably not.

Organisations are starting appreciate the value locked up in their data and are utilising technologies to process and analyse the Big Data both within and external to them. These Big Data tools are like the geological maps and exploration platforms for the information world.

Some of these tools were covered in our previous blog, but it is worth remembering the fundamentals which give rise to the Big Data challenge:

  • The volume of data is rising at an ever increasing rate
  • The velocity of that data rushing into and past organisations is accelerating
  • The variety of data has overwhelmed conventional indexing systems

Innovative technology and methods are improving the odds to finding and getting value from that data.

How can an organisation gain value from its data? What are forward thinking firms doing to invest and protect its data?

1. Agree a Common Language

Data is and does mean many things to different firms, departments and people. If there is no common understanding of what a ‘client’ or ‘sale’ or an ‘asset’ is then at the very least confusion will reign and most likely that poor business decisions will be made from the poor data.

This task is not to be underestimated. As organisations grow they build new functions with different thinking, they acquire or are bought themselves and the ‘standard’ definitions of what data means can change and blur. Getting a handle on organisation wide data definitions is a critical and complex set of tasks that need leadership and buy-in. Building a data fabric into an organisation is a thankless but necessary activity in order to achieve longer term value from the firm’s data.

2.Quality, Quality, Quality

The old adage of rubbish in, rubbish out still rings true. All organisations have multiple ‘golden sources’ of data often with legacy transformation and translation rules shunting the data between systems – if a new delivery mechanism is built it is often implemented by reverse engineering the existing feeds to make it the same rather than looking at the underlying data quality and logic. The potential for issues with one of the many consuming systems makes it too risky to do anything else. An alternative is to build a new feed for each new consumer system which de-risks the issue in one sense but builds a bewildering array of pipes crossing an organisation. With any organisation of size it is worth accepting that there will be multiple golden copies of data but the challenge is to make sure they are consistent and have quality checks built in. Reconciling sets of data across systems is great but doesn’t actually check if the data is correct, just that it matches another system….

3. Timeliness

Like most things, data has a time value. As one Chief Data Officer of a large bank recently commented ‘data has a half-life’ – the value decays over time and so ensuring the right data is in the correct place and the right time is essential and out of date/valueless data needs to be identified as such. For example; A correct prediction of tomorrow’s weather is useful, today’s weather is interesting and a report of yesterday’s weather has little value.

4. Organisational Culture

Large organisations are always ‘dealing’ with data problems and providing new solutions to improve data quality. Many large, expensive programmes have been started to solve ‘data’. Thinking about data needs to be more pervasive than that it needs to be part of the culture and fabric of the organisation. Thinking about data (accuracy, ownership, consistency, and time value) needs to be incorporated into organisations as part of the culture, articulating the value of data can help immensely with this.

5.Classification

Understanding what is important rather than having a blanket way of dealing with data is important. Some data doesn’t matter if it is wrong or not up to date because either not consumed (obvious question is – then why have it?) or irrelevant for process.  Other data is critical for a business to survive so a risk based approach to data quality needs to be used and data graded and classified on its value.

6. Data ownership

Someone needs to be accountable for and owner of data and data governance within an organisation. It does not mean that they have to manage each piece but they need to set the strategy and vision for data. More large organisations are now creating a Chief Data Officer role to ensure there is this ownership, strategy and discipline with regard to their data.

Data is the core of a business and there is a growing acknowledgement of its potential value.

As the ability to extract information and intelligence from data improves there will be some disruptive changes in the market value of firms that have  the sort of data which can improve the organisations market share, profitability and potentially traded.

Companies that have huge amounts of information regarding their customers: banks, shops, telecoms firms will be well positioned to take advantage of this information if they can manage to organise and exploit it.

 

Education – How can technology help?

Posted on : 28-03-2012 | By : richard.gale | In : General News

Tags: , , , , , , , , , ,

1

The development of the Raspberry Pi, (a £30 computer designed to give the next generation of children programming skills) started a few of us at Broadgate thinking about technology and education – Are there ways that schools and other organisations could utilise some of the current technology trends?

 

Background

ICT in the classroom has changed radically over the last 30 years. In the 1980s there existed  ‘the school computer’ where a select group of students could spend lunch-times and evenings writing programmes in incomprehensible languages resulting in simple calculators or battleship type games. Now computers are embedded in homes, offices and schools – the UK GCSE ICT course now includes a full project management lifecycle study from initial requirements gathering to system implementation. Outside the classroom computers are used for all the usual business processes including pupil records, finance, scheduling and communications.

In the UK the Professor Steve Furber of Royal Society  criticised the skills of ICT teachers (for example only 35% have a specific qualification in the subject contrasting with 74% of maths teachers) and teaching and proposed the standalone subject be scrapped. He said that IT was so important it should be part of the core curriculum integrated into schools to improve digital literacy alongside reading, writing and arithmetic.

 

Our Broad Thoughts

Integrating technology into the core of the curriculum is key and we think the opportunities for technology to improve, accelerate and enhance the educational experience for both pupils and teachers are huge.

A few of our ideas are below and we’d welcome your thoughts on these and other areas.

 

1. Social Media – collaborative approach

This is an area were the pupils excel and, as a rule, are ahead of the teachers. These technical natives have grown up with technology and the use of social networks is a natural extension of them. They are used for updating friends, promoting themselves, discussing & arguing and sharing information. Are there ways schools can utilise this technology and more importantly energy & enthusiasm?

The key element of Twitter, Facebook, Pinterest etc etc. is socialising and sharing ideas. Discussions started in the classroom can be extended to home/remote working. These often happen informally amongst pupils but could have added value if teachers could interact and assist. Schools could create ecosystems for collaborative working. Initially it may be difficult to attract pupils to the school created areas so a more successful approach may be for the pupils to create and teachers to join. Obviously there are risks to this but the idea that there be a shared area for thoughts and ideas without negativity in a safe space.

 

2. BYOD/Mobility – help or hindrance?

Many pupils now carry smartphones some are starting to carry iPads too. These can be viewed negatively from a school perspective as they can, at worst, be a distraction in class and potentially a cheating and bullying device.

So, accepting they are not going away, how can the positive aspects of smartphones be utilised?

Simple techniques such as using calendar facilities to upload the class timetables, reminders for homework, coursework etc. Alerts for taking in gym kit could be pushed out to pupils (and parents) devices. Obviously this does not completely remove ‘The dog ate my blackberry’ issue for teachers but it should help!

Coursework, homework and useful reference material & links can be also pushed up to phones to consolidate knowledge and aide pupils.

Even more useful would be to think how people use their phones and tablets, as well as communicating they are great research tools and could be used within the classroom situation helping finding different viewpoints on historical events for instance (and so helping improve the critical thinking of children as there are so many different and potentially inaccurate ‘facts’ out there –  “Always check your sources!” as my history teacher used to say).

Tablets and iPads in particular are very exciting tools for learning. They move away from the conformity of keyboards and mice and can make learning truly interactive. They are starting to be adopted in schools but we think there is a great potential to radically change the classroom and learning experience.

Obviously not all pupils can afford smartphones so to avoid technology related poverty trap, less well-off pupils should be provided with the same phones/tablets. Cash rich technology organisations should be approached to assist and a need mechanism could be introduced such as that for school dinners. Also parents’ wishes need to be taken into account as the age that a child is allowed to use a phone can vary widely.

 

3. Data Intelligence – Capturing Trends

As with any organisation there are large amounts of data contained in multiple stores. Also as with any other organisation that data is often not connected with other relevant sources so the information value of that data is lost.

One of our colleagues moved from financial services to education and was surprised by the lack of management information available to the teaching team. The data is there but it was not being translated into meaningful information.

There must be potential to link an individual teachers/class/subject results to identify trends. E.g. if the interim test results for the year 8 history class is going down, is it because the course work has been modified, there is a new teacher or the pupils socio-economic make up has changed? A good business intelligence application can trawl the data to identify the causes and so the appropriate remedial actions taken.

Similarly if maths A level results suddenly improve, what are the reasons for this and how can then they be applied elsewhere (internally or externally see Communications below)

If an individual pupils attainment levels started dropping off then additional attention could be provided to that student to help them get back on track and also identify and help hopefully resolve the underlying cause of the issue.

Other areas which may be more radical may involve gathering the information and identifying the better performing areas within or across schools including measurements such as a ‘cost per GCSE’ or ‘Entry/Exit attainment improvement’ of pupils.

 

4. Communications – sharing

Schools can sometimes be inward looking. Often teachers stay in one school for a considerable time. This is great for continuity and progression but may result in lost opportunities for innovation and changes that are happening in the extended educational community. Some schools encourage visits to other schools, conferences and courses can help here and there is big opportunity to take this further.

Businesses utilise management consultants to help improve organisations for efficiency or growth with the view to build revenue and profits.

Could information sharing, more inter-school communications, best practice and teaching artefact sharing help schools and teaching? Information is now available locally, nationally and internationally so can be shared and used amongst educational establishments.

 

5. Cloud Computing – Who needs infrastructure?

Most schools have a room/office with the computers/servers. As IT requirements grew in terms of finance, pupils’ records, assessments, operational and staff information the amount and complexity of equipment expanded often requiring dedicated resources to support and change. As we have been saying to our clients, with the advent of Cloud and Software as a Service the need for this is reducing to the point where the default should be for someone else to host, manage and support a schools technology infrastructure.

Obviously, as with any sensitive information, the question of student data privacy and security needs to be addressed. This should already be the case and the existing policies should be proved by any potential vendor and tested regularly by the educational authority.

 

6. Security – Paramount

The most important part of the use of technology is pupil safety and confidentiality. This is obvious and needs to be kept in the forefront of any discussion in regard to the introduction of a system whether it is IT or other mechanism.

 

Final thoughts

The opportunities for technology to help improve schools is both immense and exciting, this is not an area we have worked in but are really interested in stimulating a debate and seeing if we can assist in any way. Every time we help people outside our core business areas of finance IT we find not only do we enjoy it but we too learn a great deal from different working structures and cultures.

“If we teach today as we taught yesterday, we rob our children of tomorrow” John Dewey – innovation & technology can help us help the next generation.