Subject: [NEXTEL-1] Market research providers confront credibility concerns (Attention Good House Keeping)

From: [email-masked]
Date: Tue, 14 Oct 2003 13:53:42 -0400

Market research providers confront credibility concerns

By THOMAS HOFFMAN
OCTOBER 13, 2003

Recently published reports that have prompted questions about the credibility of
some market research firms have spurred two of the biggest names in the field to
make substantial changes to their ethics and disclosure policies.

And that's a good thing, users say. They want to see market research firms state
their ethics policies more clearly on their Web sites and through other
distribution channels, according to an exclusive Computerworld survey of 133 IT
professionals that was conducted last week.
Last month, some IT professionals reacted angrily to a Microsoft-funded report
released by Cambridge, Mass.-based Forrester Research Inc. that concluded that
developing and deploying Web-based portal applications is substantially less
expensive using Microsoft technology than it is using a Linux/J2EE combination
[QuickLink 41320].

Two weeks later, the release of an anti-Microsoft security report led to the
firing of one of its co-authors, who harshly criticized the vendor's dominance of
the software industry [QuickLink 41691].

Russ Lewis, CIO at GFInet Inc.

Interviews with 15 CIOs last week indicated that many IT executives continue to
rely on market research firms as one component of making product purchasing
decisions or setting strategic plans. But some CIOs say they have soured on the
use of market research because of credibility concerns, high costs and doubts
about the reports' value.

For instance, Russ Lewis said he made use of research from Forrester and Gartner
Inc. quite often when he was CIO at New York investment bank Jefferies & Co. from
1994 to 1999. But he "found their research to be somewhat limiting and narrow in
focus," said Lewis, who is now CIO at GFInet Inc., a subsidiary of New York-based
brokerage services provider GFI Group Inc.

"I'm not a big fan of any of those market research firms," Lewis said. "I don't
believe them to be independent, and I don't believe their research to be valuable
except at a very high level for very large firms."

For its part, Forrester is making substantial changes to its ethics and disclosure
policies following the release of the Microsoft-sponsored report and controversy
surrounding another recent report that was funded by PeopleSoft Inc.

PeopleSoft had sponsored a survey of more than 600 business and IT users that
examined users' satisfaction with products from several enterprise application
vendors, including PeopleSoft, SAP AG, Oracle Corp. and Siebel Systems Inc.
PeopleSoft put out a press release on Sept. 15 in which it boasted that it
"outscored SAP, Oracle, and Siebel in the overall ability to deliver a superior
enterprise application ownership experience" [QuickLink a3710]. The release
attributed the findings to "an independent research study completed by Forrester
Research" but failed to mention that PeopleSoft had paid for the study.

 George Colony, CEO at Forrester Research Inc.
Steve Swasey, director of corporate public relations at PeopleSoft, said there was
no point in disclosing the source of the funding. "What's the need? The research
was objective and unbiased," he said. "We would have done the same if the research
was sponsored by someone else."

Allowing PeopleSoft to publish the results in the press release was a mistake, a
Forrester spokeswoman said last week.

In an interview last week, Forrester CEO George Colony said that the company has
taken steps to "tighten" its internal processes and its integrity policy. In fact,
Forrester "will no longer accept projects that involve paid-for, publicized
product comparisons," according to a statement from Colony that's now posted on
the company's Web site.

"We erred in the Microsoft and PeopleSoft cases, and we're correcting our
processes," Colony said.
Forrester isn't alone. Boston-based Aberdeen Group Inc. plans over the next 45
days to "sharpen and enhance" the ethics policies posted on its Web site in order
to more clearly state its research methodologies, said James L. Bedard, who became
Aberdeen's president and CEO in August. The decision is partly a response to a
June 2002 Wall Street Journal story criticizing firms that offer "praise for pay."
In that story, the previous Aberdeen management defended the practice of
publishing favorable research reports for vendors.

"It's critical that [customers] have the highest degree of belief in our
integrity," said Bedard.
Executives at other leading market research firms, including Gartner, Meta Group
Inc. and IDC, say they haven't made any recent changes to their ethics policies or
how they go about disclosing them.
"If anything, we're often asked [by vendors] to change [our disclosure policies]
in the other direction" and make them less restrictive, said David Yockelson,
executive vice president and director of technology research services at Stamford,
Conn.-based Meta Group.

Meta Group "rarely" takes on research sponsored by a single vendor, said
Yockelson. And though the vendor does have the right to review Meta Group's
findings in these types of reports, Yockelson said, "that doesn't give the vendor
the right to not publish it or change it. They can come back and say they
disagree. But we won't withhold it because it's funded by the vendor."

If Meta Group is conducting internally funded research in which a particular
vendor receives a "substantial mention," the researcher will allow the vendor to
fact-check the material and recommend changes for the sake of accuracy.

"We don't offer a vendor the guarantee that if they don't like what we write they
can change it. As long as we are factually correct, we will publish whatever it is
that we've created," said Yockelson.
Jim Shepherd, senior vice president at AMR Research Inc. in Boston, said
vendor-sponsored research isn't an issue for his firm, at least from an ethics
standpoint. "If we do research for a [vendor] client, it is not published and can
only be used by a client internally," he said. "And we don't do much of that."
Like Meta Group, AMR will also let vendors fact-check its so-called deep-dive
product review reports. "But we won't cede editorial control in any sense," said
Shepherd.

Framingham, Mass.-based IDC, a sister company of Computerworld, "always discloses"
in writing on the research document who the source of funding is for a particular
piece of research, said CEO Kirk Campbell.

In the rare instances IDC does conduct vendor-sponsored research, said Campbell,
the vendor has the right to review the findings. "But IDC has the final
determination on all of its research content and how and where it is published,"
he said, adding that IDC hasn't made any changes to its research objectivity
policies since they were established in the early 1990s.

Joseph Baylock, group vice president of vendor relations at Stamford, Conn.-based
Gartner, said the company doesn't conduct any vendor-sponsored research, although
its Gartner Consulting unit does conduct proprietary studies for vendors and
groups of vendors. But those reports aren't published for public consumption, he
said.

Fewer Secrets
Company Discloses source of research funding Posts ethics policies online
Forrester Research Inc. Yes Yes
Meta Group Inc. Upon client approval Posts a citation policy; ethics policy to be
posted next month
AMR Research Inc. Not applicable No, but posts a client privacy policy
Gartner Inc. Not applicable Yes
Aberdeen Group Inc. Plans to within next 45 days Plans to within next 45 days
IDC Yes Yes

For what reasons do you rely upon market research?

 How credible do you feel the information you receive from analysts is?

Do you think that research firms should have published, clearly stated ethics
policies governing their vendor/client relationships?

------------------------ Yahoo! Groups Sponsor ---------------------~-->
Rent DVDs from home.
Over 14,500 titles. Free Shipping
& No Late Fees. Try Netflix for FREE!
http://us.click.yahoo.com/ArdFIC/hP.FAA/3jkFAA/61FxlB/TM
---------------------------------------------------------------------~->

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/