Search: A Key Building Block for Business Applications

As I wrote previously, the ECM market is going through significant transformation and continued consolidation.  The convergence of 4 different markets are contributing to this evolution.  One of the interesting aspects of this transformation is the emergence of search as a key enabler for business applications.

Basic Premise

Future winners in the evolving ECM market will need to embrace new composition frameworks that can rapidly deliver solutions at the intersection of content, social, search and analytics, and business process management.  Those composition platforms will also need to support hybrid cloud delivery models to deliver optimal business values based on the customer needs.

Search and the Composite Application Platform

The concept of search based applications is defined by Sue Feldman and her team at IDC as "applications that combine search and or text analytics with collaborative technologies, workflow, domain knowledge, business intelligence or relevant web services".

The definition above introduces key distinctions from the traditional definition of search:

1. Search based applications combine a broad range of capabilities:  I find this aspect of the definition particularly interesting.  From my perspective, this makes search technology one of the building blocks of an application composition platform.  It offers both information management capabilities such as information categorization, text mining, indexing, fuzzy matching and information navigation capabilities such as faceted / relational navigation and data visualization in the context of business applications.
 
2. Information is integrated with business processes: this is a critical aspect of search based applications and this is where I believe there is strong overlap with decision management or case management.  In a business context, search supports a decision process.  Those decisions might be more project centric which will require more collaboration and ad-hoc discovery.  They might also be more process or case centric in which a network of processes will manage the orchestration of the decision process.

This requires that search capabilities be weaved into the fabric of business applications.

What about some examples?

First let's look at some examples for best of breed search vendors and how they leverage the concept of search based applications to address their customer problems.

New account opening and single view of the customer

Attivio provides the case study of a large global bank challenged to enhance customer service across 40 countries, 50,000 employees and 100 separate sources of information.  The bank decided to leverage a search based application to provide a unified view of customers resulting in the enhanced productivity of the customer facing agent, improved customer experience, and a higher percentage of first call resolution rates, while decreasing overall support and maintenance cost.

Warranty and quality monitoring

High warranty costs have received significant exposure lately. Think about Johnson and Johnson having to recall 22 products in 19 months adding up to over $900M in revenue loss.  Those are nightmares manufacturing executives cannot ignore.  Getting the information to identify warranty issues before those organizations are at risk requires visibility throughout the entire supply chain.  By integrating data from multiple sources, search based applications provide the right tools to support the discovery and early identification of product quality issues.  Endeca offers Endeca Latitude for Warranty to address this problem.

A New Kind of Business Application Composition Platform

The examples above illustrate the power of search based application.  Those applications become even more powerful when they are combined with an action engine that enables the tracking and processing of warranty issues, the collaboration on criminal investigations or complex financial products.  With xCP, our view is that great search and data visualization capabilities need to be combined with the orchestration of content and information that will drive optimal business decisions: for instance a product recall based on the early identification of warranty issues.

Samir Batla, product manager for search and analytics at EMC IIG has posted a series of blogs posts on how xCP exposes a wide variety of technologies that enables optimal business decisions and how from early warnings, customers can take action to make pro-active decision for their businesses.  As shown below, the anatomy of an optimal business decision requires much more than search.  The combination of the broad set of capabilities of xCP is what enables these optimal business decisions.

The Need for Cloud Integration

The cloud is everywhere those days and the software industry is going through some drastic transformations.  According to IDC, “Software as a Service 2010-2014” report, software will never be the same.  The on-premise software market will go down 19% during this period from a market share of 66%.  In the same period, shares of the public cloud are expected to go up 12% to 22% of the software market revenue.  The Platform as a Service market is expected to grow between 39% and 55% a year for the next 5 years (IDC forecast vs. Forrester forecast).  This is quite some shift...

Forrester Research defines Platform as a Service (PaaS) as:

A complete application platform for multitenant cloud environments that includes development tools, runtime, and administration and management tools and services. PaaS combines an application platform with managed cloud infrastructure services.

A key value of PaaS is the ability to focus on the business application and not on platform technologies.  As PaaS becomes more widely adopted and business applications increasingly built on top of such platforms, the need for cloud integration will become pervasive.

Modern applications require the ability to expose enterprise information in the context of the task at hand.  Adaptive contextual user experiences will become increasingly critical for business users to rapidly make optimal business decisions.  In a PaaS or SaaS world, this will require robust integration with the needed information sources such as:

• Cloud based applications: CRM systems like Salesforce.com, HR systems like Success Factor or Workday, public cloud email systems and productivity applications
• Enterprise applications:  ERP systems like SAP or Oracle Business Applications
• Content management systems internally hosted or in the public cloud
• Legacy systems and applications

However, cloud integration as some serious challenges it needs to overcome.  A July 2010 survey by Saugatuck Technology confirms that data security and privacy as well as data integration concerns are top concerns in deploying cloud based business solutions.

Let’s have a look at some of those challenges:

• Data integrity and information security.  Art Coviello from RSA articulates the problem nicely:  "While cloud computing offers tremendous benefits in cost and agility, it breaks down some of the traditional means of ensuring visibility and control of infrastructure and information. Forcing enterprises to develop trusted relationships individually with each cloud service provider they wish to use is cumbersome and will not scale. New thinking in security and compliance is required to provide a future in which organizations can consume services from a wide variety of cloud service providers on-demand and for all their application needs." 
• Application security integration.  Anyone who has tried to integrate applications together knows that one major integration issue is around access control and particularly role based security.  Protocols  like OAuth have emerged to provide a simple way to publish and interact with protected data.  Other standards like XACML provide flexible solutions for enforcing granular authorization.  Unfortunately, those technologies are not broadly adopted by enterprise application making authorization integration a challenge.
• Data integration.  Many enterprise systems still require proprietary integration mechanisms.  But even cloud based applications require integration with their proprietary APIs.  Cloud based integration should provide a flexible mechanism for data mapping and integration.  As Mark Brennan, Director of Business Applications at Pandora puts it: "traditional approaches to integration are no longer adequate when you have literally dozens of SaaS applications, all needing to talk to each other. If every change, customization or refinement becomes a project, we can’t keep up. To changes all that, we need a solution that elevates us out of code level and putting the power directly into our hands".   Some of those data integration challenges may involve:
• Bi-directional data synchronization
• Transactional integrity and service orchestration
• Solutions to deal with information caching and system latency

Clearly, addressing those challenges will be key to enterprises moving and trusting their enterprise systems to the public cloud.

A New Platform to Deliver Superior Customer Experiences

A few weeks ago, I talked about the new platform and the new technology era of business applications. The new platform must enable business applications to drive optimal business decisions and improve customer experiences. It must provide a breadth of application services that enable businesses to rapidly build intelligent business applications that can adjust rapidly to changes in their business environment.

Several weeks ago, at EMC World 2011, I gave a presentation on How to leverage Documentum xCP and the 5 Cs (correspondences, capture, case, content and compliance) to deliver superior customer experiences while improving business efficiencies.

In my presentation I started with a personal experience. A few months ago, I got an automated phone call from my bank telling me that my debit card had been deactivated and to press 1 if I wanted to reactivate it. Puzzled, I decided to call my bank instead. First, regular customer services was closed; so I decided to get routed to the service where you report card losses. After a 15 minute process, I finally got in touch with a customer representative who told me that this was a scam they were aware of and that they had issued a press release to warn their customers about the scam. Yes, you read right: a press release... As educated and responsible consumers and customers, we are expected to monitor the press releases of the companies we do business with, no less.

Well, in the day and age of personalized services and hyper social interactions, needless to say we expect more. This is where the new platform comes into play.  Imagine what my bank could have done had they been using that new platform:

1. They could have been monitoring social channels and their own customer data to look for patterns that their brand was being used for mischievous purposes.  That's where Big Data comes in.   Leveraging data computing appliances and predictive models, corporations can be on the look for significant patterns or events that are likely to impact their businesses.
2. Once, my bank has found evidence that a scam is underway, they could have leveraged the intelligent correspondence management and process capabilities of xCP to send me multi-channel personalized communications and to make sure that I was aware of the scam.  In those communications, they would have provided me with instructions on how to deal with the situation.
3. They could have been proactive in identifying customers that were victims of the scam and opened cases to help those customers deal with the situation, cancel their debit cards and issue new ones
4. They would be delivering better customer experiences, ensuring the loyalty of their customers while protected their business against potential charges of negligent behavior.

This is the power of the new platform.  Gain insight from social and business data, and take actions on that insight to deliver superior customer experiences.  As the success of companies like Zappos shows, companies that can successfully deliver those superior experiences create long term sustainable competitive advantage.

The Evolution of the ECM Market

When Documentum was founded, the founders' vision was to manage the world's information to enable innovation and scientific breakthrough (like finding a cure for cancer). Since then, what started as a Document Management opportunity has gone a long way. Today, Enterprise Content Management is a mature market providing breadth of technologies that enable organizations to reshape the way project or transactional decisions are made. The ECM market is also a market that has gone through significant transformation lately and continued consolidation.

I see the convergence of 4 markets contributing to the evolution of ECM targeting different interaction types (from adhoc to structured) and application types (from horizontal to vertical).

The Commoditization of Content Services

The unprecedented success of Microsoft SharePoint has commoditized content services and significantly changed the content management market. SharePoint is now a force to be reckoned with. The sprawl of SharePoint sites has also created tremendous opportunities for information governance and given rise to a new generation of anti-SharePoint offerings such as Box.net leveraging the cloud to simplify information sharing.

The Emergence of Social Business Platforms

With the launch of their new Alfresco Enterprise release, Alfresco embraced the concept of Social Content Management. Jive and IBM refer to the same opportunity as Social Business. Whatever the term, in its latest magic quadrant, Gartner recognizes the impact of social interactions as social media create masses of unmanaged content.

Jive and IBM are the clear leaders in Social Business Platforms and the recent partnership between Jive and Alfresco recognizes the need for Social Business Platforms to provide robust content management capabilities. Social Business Platform vendors like Jive have been successful against Microsoft by focusing exclusively on social features and how they can help certain types of employees (salespeople, developers, etc.). Microsoft has also recognized the impact of social on content management and collaboration with SharePoint 2010.

Vendors for Social Business Platforms include: Jive, IBM, Microsoft, Cisco Quad, Acquia / Drupal

When ECM Meets Search Based Applications

The concept of search based applications is defined by Sue Feldman and her team at IDC as "applications that combine search and or text analytics with collaborative technologies, workflow, domain knowledge, business intelligence or relevant web services". This definition is being embraced by search vendors such as Endeca, Sinequa and Exalead (now part of Dassault Systemes). Gregory Greffenstette, Exalead chief science officer, offers a glance into concrete examples of search based applications from logistic track and trace type applications, to CRM, 360 view of the customer and decision intelligence.

At EMC, we have recognized the importance of search based applications and are incorporating search and text mining functionality into our products from CenterStage to Documentum xCP leveraging our Documentum xPlore, Content Intelligence Services and Federated Search Services, now core features of the EMC Documentum Platform.

Other ECM vendors from OpenText to IBM have recognized this trend as well with significant investments and acquisitions in this area. Key to delivering the promises of search based applications will be tight integration with composition tools for a new breed of business savvy developers that will enable the rapid creation of search enabled applications.

The following search vendors have embraced this trend: Autonomy, Microsoft / Fast, Attivio, Endeca, Sinequa, Exalead (Dassault Systemes)

Towards Composite Content Application Frameworks and Dynamic Case Management

As Gartner points out in its latest magic quadrant, there is increasingly more pressure on ECM vendors for ready to use solutions which solve real business problems rather than generic content platforms composed of dozens of loosely coupled modules. The need for more business solutions coupled with the convergence of the ECM and BPM markets has seen the rise of new opportunities for what Gartner calls Composite Content Application frameworks and Forrester calls Dynamic Case Management. While composite content application frameworks address the needs of content solutions, dynamic case management focuses on the convergence of ECM, BPM, business analytics and event processing. The reality is that there is a spectrum of solutions from content to case that require a new generation of composition tools to rapidly deliver business value with the agility to evolve quickly as business needs change. Organizations need more than a repository, they need an application platform and the solutions that come with it.

The following vendors provide composite content application platforms: EMC Documentum xCP, IBM, OpenText, Adobe, SpringCM, Hyland, Nuxeo

Towards the New Information Fabric

As the delivery of composite content application platforms transition to the cloud, this creates an opportunity for a new information fabric that will deliver tremendous business value and flexibility while reducing barriers to adoption. The industry is clearly at an inflection points in the adoption of cloud services. While Public cloud services offer tremendous promises, hybrid cloud deployments are most likely to provide the right balance between information control and flexibility.

Future winners in the evolving ECM market will need to embrace new composition frameworks that can rapidly deliver solutions at the intersection of content, social, search and analytics, and BPM (from content to case). Those composition platforms will also need to support hybrid cloud delivery models to deliver optimal business values based on the customer needs.

Do you share this perspective on the evolution of ECM market? Share your thoughts on this blog or on twitter.

The New Technology Era of Business Applications

After a long hiatus, I have decided that it was time to start blogging again. Last time I posted a blog post was more than 2 years ago. Time flies...

We live in an exciting technology era. Technology transformation is accelerating and what is unique about this transformation is that all layers of the technology stack are experiencing significant changes at the same time. Jeetu Patel wrote a good blog post introducing the Post PC-era. For business applications this has significant implications.

1. The new platform: Business applications require a new platform. That new platform must be scalable, elastic, and rapidly respond to the needs of the business. It also needs a rich set of services to enable businesses to rapidly deliver applications that meet their most critical needs. At EMC Information Intelligence Group, we are investing on both those fronts:
   1. At EMC World, we announced EMC OnDemand to take EMC Documentum into the New Platform era.
   2. Documentum xCP was mentioned in every one of our executives keynotes (by Joe Tucci, Pat Gelsinger, Paul Maritz). xCP provides the breadth of services that enables to rapidly deliver business applications that meet the critical needs of the business. At EMC World, I had a presentation on leveraging the 5Cs (correspondence, capture, case, content and compliance). Those are cornerstone services for the new platform.
2. The new developer: The new developer needs new tools to create applications at a new level of abstraction and to enable her to focus on solving business problems rapidly and not as much on coding or the technology. The new developer needs powerful composition tools. Check out how xCP can empower the new developer to rapidly create sophisticated business applications.
3. The new user: Choice computing is critical to the new user. Check out Jeetu Patel's Blog post on the new user. The new user wants pervasive access to information, user interfaces that are catered to her needs. She needs to make decisions rapidly and needs the right insight to make the right decision with confidence. To that effect, EMC IIG had some significant partnership announcements at EMC World. But beyond the right user interface, the new user needs instant access to the right data, with the insight that enables her to take the right action. She needs to be able to turn information into business advantage. I presented at EMC World on how xCP can enable the new user to do just that

Those are exciting times at EMC IIG, the innovation machines is humming and we are taking part of the transformation of the business application platform on those 3 fronts. This represents an incredible opportunity for the future. I am exciting to be part of this transformation and to lead the development of next generation technology that will enable it.

Business and Cultural Challenges of a Next Generation Collaborative Solution

Cross posting from my blog post on the EMC Knowledge Worker blog.

Last Monday, at Momentum Prague 2008, I held a Product Advisory Forum (PAF) with 27 EMC customers. The overall objective of the Knowledge Worker PAF was to assess the characteristics of a next generation collaborative solution. As an introduction, we reviewed a set of disruptive trends around:

• The changing nature of the Enterprise
• New work habits, social and cultural changes
• Disruptive technologies that support new patterns of collaboration
• The impact of the changing economic climate

See the presentation for more details on each trends.

3 teams of participants were then asked to assess business, cultural challenges and solutions that would help their organizations adapt to such trends.

Overall Needs and Business Challenges
The following needs and business challenges were identified:

• Organizations need to improve collaboration within their organizations and with external parties. This raises issues around the proper level of openness and how to balance such a need against the needs for better security.
  • Open systems are necessary to enable expertise location and finding information
  • Integration between enterprise services is required to break information silos and encourage collaboration across various types of content in the enterprise
  • Improving connections between people and content provides additional context to the information and fosters better interaction between knowledge workers
• Organizations need to improve the use of information structures and reuse of already existing information
  • Formalizing knowledge within organizations can improve its reuse. In order to do so, organizations needs better mechanisms for classifying and organizing information. Both taxonomies and folksonomies have a role to play.
  • Reusing content in new ways can open new business opportunities and improve the speed of delivery of new projects.
  • Organizationally, there is no substitute for producing lessons learned and synthesizing information. Organizations need to adopt models that encourage information reuse and developed template based information models.
    • This provides a framework for producing new quality information more quickly and on-ramping on new subjects and projects more quickly.
• How do organizations make sure that the information they manage is valid and of quality and how do they effectively manage their information lifecycle?
  • Organizations can rely on informal validation where the collective intelligence of the community can be leveraged to produce and elevate the most valuable information
  • In parallel, some categories of content will never be properly assessed by the community. Formal processes are required for such content (e.g. Standard Operating Procedures)
  • In addition, organizations need to put systems and policies in place that preserve information readability over the years and retain only content that is relevant to the organization. How does an organization properly identify the content that needs to be retained, in particular when it pertains to the organization knowledge?
  • Organizations should look at technology that helps with concepts extraction and leverages such capabilities for improved information classification
• Organizations must deploy technologies and solutions that empower their business users:
  • Business users should be able to configure applications in a way that meets their needs
  • Access to information should be ubiquitous

Cultural and Organization Challenges
Software does not drive organizational changes, people do. Too often, budgets get sucked into technical implementations to the detriment of investments in driving adoption and culture changes. Investments in driving adoption are critical and software vendors can help by delivering better out of the box solutions that allow organizations to focus on adoption and less on implementation. Some of the cultural challenges organizations have to confront include:

• Fear. Fear of consequences and fear of rejection. Organizations need to assess how they value knowledge. Often, people are reluctant to contribute for fear that it will undermine their own value. Organizations needs to adopt processes that better value and reward sharing information.
• Privacy. People are also concerned that we will share information and loose control of how the information will be used. Vendors need to provide features that help end users better understand how the information they contribute is being reused.

Organizations need to leverage the viral aspect of Web 2.0 technologies and move towards a Discover / Adopt / Adapt model. Only when the information is adopted and adapted does it start to deliver significant value to the enterprise.

Being able to justify the ROI of Web 2.0 and social networking technology is a challenge. Organizations must be able to measure the adoption of the content being shared. Measuring direct benefits should focus on productivity benchmarks with or without such technologies.

EMC Knowledge Worker Strategy Fit
Overall, EMC Knowledge Worker strategy is a great fit for many of the business challenges identified during the workshop. EMC's investments in CenterStage will provide improved patterns of collaboration that empower their business users and extend the reach of their virtual organization.

In addition, EMC Documentum provides a robust framework for managing the information lifecycle and the reuse of information. At the core of EMC strategy is a strong emphasis on information intelligence to permeate all of its collaborative capabilities. Based on advanced concept extraction and combined with an understanding of people social networks and interaction with content, such information intelligence will foster information reuse and provide better models for information classification and retention.

In order to help customers with the adoption of such technologies, EMC will work with its partners and its own consulting organization to provide best practices on how to most rapidly deliver business value.

CenterStage: A Technical Overview

Cross posting from my blog post on the EMC Knowledge Worker blog.

With CenterStage, EMC delivers an interactive web experience, together with the associated computational resources and web services, for accessing and managing communities and team workspaces within the framework of an enterprise information infrastructure. Designed as a rich internet application, CenterStage leverages AJAX (asynchronous JavaScript and XML) and web services to provide a dynamic user experience. Let's take a look at CenterStage high level architecture:


For version 1.0 release of CenterStage, two CenterStage clients will be made available in Q1 2009: CenterStage Essentials and CenterStage Pro. CenterStage Essentials provides basic content services; CenterStage Pro builds upon CenterStage Essentials to provide rich Web 2.0 and Social Networking capabilities. CenterStage Pro leverages CenterStage Essentials sub-systems and builds upon them.

Server Infrastructure
CenterStage leverages the robust customization infrastructure of the EMC Documentum Platform components as well as core services to provide both data model and services.
The following server components are provided with CenterStage:

• Content server - This core server component provides data model management, security services and content management services
• Full text index - Provides CenterStage with full-text search functionality
• Thumbnail generator - This component generates thumbnails for common file formats. More advanced rendition capabilities (advanced formats, multi-paging) require customers to upgrade to the CTS framework and its Media Transformation Services (MTS) or Advanced Documents Transformation Services (ADTS)
• Federated search - Allows users to aggregate searches across multiple sources. CenterStage Pro will provide search adapters for Google, ODBC/JDBC, Open Directory, and Open Search to name a few
• Classification and Entity Extraction - The classification and entity extraction server components extract metadata from the content being uploaded in CenterStage based on semantic analysis of the content itself. The extracted metadata provides additional dimensions used by users to filter content

In addition to the above server components, the following modules need to be deployed to support CenterStage functionality:

• Collaboration services: Provide the core collaboration infrastructure and is also used as the underpinning for the collaboration functionality available in Webtop
• Rich media and transformation services: Support the ability to preview content in context within CenterStage
• Extended search services: Provide the core set of services for search and clustering capabilities
• CenterStage Essentials and Pro services: Provide the additional core services and data model for both CenterStage Essentials and Pro

Services Infrastructure
CenterStage application services are built on top of Documentum Foundation Services (DFS) providing CenterStage with a strong service orientation . CenterStage application services provide coarse-grained APIs built to address the needs of the user interface. Elements of the services infrastructure include:

• The Rich Content Management Platform (RCMP) services, which handles the presentation layer requirement for:
  • Component delivery - the component registry and configuration management for the various UI components (or Widgets) exposed in the CenterStage UI
  • Container manager: the infrastructure for managing the user interface layouts
  • Bundle support: the packaging and deployment model leveraging OSGi at its core
• The CenterStage Essentials and Pro application services which provide a broad set of APIs to address the needs of the user interface and support its team productivity, information discovery, business process and web 2.0 functionality

User Interface Infrastructure
Built as a true rich internet application, CenterStage leverages browser - based technology to bind to the proper services and render the user interface:

• A thin UI infrastructure allows CenterStage to render the layout definition provided by the services. Service binding is achieved via either DWR (Direct Web Remoting), REST or SOAP depending on the UI technology leveraged in a particular UI widget. CenterStage leverages the ExtJS toolkit for most its user interface, but widgets can be built using a different UI technology such as Flex.
• A plugin infrastructure is also provided to enable seamless integration with the desktop when working with files.

Many of the architectural subsystems described above are in place in CenterStage Essentials Beta. Other sub-systems will be introduced with the formal release of CenterStage. Building upon the strength of the EMC Documentum Platform and standard rich internet technologies, CenterStage will provide a strong foundation for social, intelligent content enabled applications.