Mark Smith's Analyst Perspectives

The technology analyst industry has ended its barrage of exuberant predictions for the new year. After digesting all of these commentaries (some were a challenge, I admit), we need to stop our predicting and instead start a discussion about what all of us ought to be doing. From my perspective, that should be an annual statement of direction, what we here call the research agenda. I have decided it is time to remind all of us what this important activity is and why it is really important to see from industry analysts.

It should be a standard practice in our business for each firm to issue a research agenda that is supposed to guide its analysts or practices in the coming year. Ours have been available to clients and now will become public record as they are annually. Unfortunately, most analyst firms have excused themselves from performing this important step; instead, they just bring forward predictions and what I have heard from many of my colleagues as educated guesses from their opinion and inquiries from dozens of clients that are looking to advance their technology. The absence of a research agenda is not a good idea, for the simplest of reasons: Clients pay for services, and they have a right to expect that the analysts whose time they pay for will set out what their knowledge and experience leads them to conclude are the areas on which they should focus so the clients can evaluate how it matches their own objectives.

At Ventana Research we are guided in setting our research agenda by what our clients need to advance their competency and maturity. In practice, for each market segment this requires comprehensive research across hundreds of organizations. We do this using our benchmarking methodology, which provides a framework for conducting, analyzing and presenting the research. Using this approach we are able to evaluate the level of maturity of an organization in a particular realm and also to drill into the maturity of its specific people, process, information and technology competencies. Another required element of how we establish competency and maturity is that we conduct research in both the business and IT sides of an organization. Researching only one of these two audiences can significantly skew the research findings and thus guidance provided to clients and the industry.

We’re hardly the only firm that claims to conduct research. But we are unique in how our research is conducted within a consistent framework and is guided by clearly stated overall objectives. I’ve pointed out years ago (See: “Why Bad Research Could Jeopardize Your Business”) that conducting surveys without a methodology and not as part of an overall research agenda can easily rob the results of actionable value. Our benchmark research, in contrast, provides us with a strong foundation on which to build our education efforts and to use in advising business and IT buyers.

Our research and the insights it yields makes it possible for us to determine best and worst practices, providing a perspective on what to do and what to avoid. Most analyst firms fall far short in identifying these because they do not conduct primary research, depending instead only on client inquiries and in most cases exclusively from IT. Unfortunately, this can bias their view of the market. Recently one of the largest IT analyst firms, one that does not do primary research across business and IT, admitted publicly that it was late to the table in covering mobile technologies. It did not see that the business side is driving the adoption of mobile technologies in the enterprise because it only advises the IT side, which has not yet been pushing to adopt these in more than a corporate standard for electronic mail.

Nor is just doing buyer research sufficient. One needs to understand the technology itself in order to evaluate its fit for the people and process needs of an organization. Doing so requires not just supplier/vendor briefings and attending product demonstrations and reading product reviews but actually engaging into trying the combination of software and technology. It still amazes me that the majority of analysis who do not like to actually see, try and evaluate the technology they offer advice. It is important that analyst firms offering advice and guidance have undertaken a detailed assessment of different vendors and their products to assess their merits in order to be able to match them to buyer needs and evaluate how well they address the overall market.

We do this using an approach that is framed as a balanced RFP (request for proposal). It includes evaluation categories such as usage, capabilities and management as well as cost, benefit and an examination of other customers’ deployments. Some approaches spend more time looking at the vision of the vendor and the future of their products or looking at their financials and number of customers. While these are important, they should not comprise much more than 20 percent of your evaluation review of vendors and products. If you are weighting these vendor criteria much more than this amount, you could be doing your business and technology environment a disservice.

Why I am starting a conversation on the science of research and what you should expect from technology analyst firms? The answer is to let you, the buyer, know that there is a wide spectrum of research methods across analyst firms, and to ensure that you understand this to be able to determine the value of the services and advice you are receiving. Does the analyst with which you work have sufficient experience in the segment or technology that matters to you? Does that analyst derive knowledge from in-depth buyer research, supplier research, or both? Does his or her perspective come from being a counselor to organizations or only from reading research and education that is already published? Does the advice you’re given speak directly to your current competencies and those of your organization, or does it just regurgitate advice drawn from a spread of technology adoption laggards to leaders?

All of these are questions you should ask yourself to ensure you get the advice you need – and, more importantly, the quality of knowledge transfer that can enable you to decide your needs and to act on them quickly. It is time for you to be more knowledgeable on what you are hearing and reading to ensure you can trust the information and advice you are utilizing for your business and IT organizations. I may be biased in my framing of this dialogue, of course, but after 20 years of seeing the good, the bad and the ugly of analyst firms and worked for one other than Ventana Research, I thought it would be good to start a discussion on what you as a buyer should expect in analyst firms and their research to provide you with the service you need and deserve.

Let me know your thoughts or come and collaborate with me on Facebook, LinkedIn and Twitter.

Regards,

Mark Smith – CEO & EVP Research