December 10, 2008

    Modern spreadsheet applications such as Microsoft Excel and Calc are marvelous tools. In addition to their natural utility as business reporting tools they can also be a great extension to performance and capacity management tools.

    In his presentation, "Pivot Tables/Charts - Magic Beans without Living in a Fairy Tale," John S. Van Wagenen of Caterpillar Corporation gave a useful demonstration of how the PivotTables feature in Excel ( Calc calls it DataPilot) can be used to dice and slice time series performance data, such as the data collected by TeamQuest Manager, and present it just they way management wants it.

    We provide a similar and modest version of this capability in the "Chart Paging" feature of TeamQuest IT Service Analyzer, where lots of data can be "paged" by any identifier in the data such as server, virtual machine, LPAR, zone, workload, etc.

    I find it interesting, but not surprising, that several session speakers at CMG this year mention spreadsheet applications as their favorite management communication tool for performance and capacity reports. Being firmly entrenched in the business world, what better place to plug in technical data than spreadsheet applications in the quest to connect IT with the Business.

    BTW, TeamQuest Model and TeamQuest IT Service Analyzer and Reporter let you take the data you're looking at over to Excel with the click of a button.

    Paul Strong of eBay Research Labs gave a very interesting talk on "The Shape of Infrastructure to Come" where he presented the infrastructure powering the eBay website that we all know.

    He also gave his views on cloud computing, which seems to be the buzzword du jour, a companion to "virtualization." Behind the website is actually a trading cloud developed by eBay for eBay. The three major services used by the website are the auction service, the payment service (now PayPal), and the search service. The eBay programmers develop these major services using APIs of the eBay trading cloud. The cloud is an abstraction of the physical IT resources that collectively form the cloud.

    Cumulus clouds

    But it was not always so. Around the year 2000 (after some high profile outages of the website) eBay realized that to cope with their exponential growth they had to make a radical change to the IT infrastructure, particularly the core component: the auction items database.

    They decided to break down their large, vertically scaled vendor hardware, "virtualize" the database, and spread it across horizontally scaled commodity hardware. The three big services drawing power from the cloud were updated to also scale horizontally and now use metadata to make calls for the "real" data.

    The concept of a "virtual database" is very powerful. When we introduced TeamQuest IT Service Analyzer and Reporter, we decided to do the same thing as eBay, although for different reasons. I call our database a federated performance and capacity database.

    This new database is an aggregation of metadata representing the actual data stored in hundreds, maybe thousands of TeamQuest Manager databases in your environment. As with the eBay web services, Analyzer and Reporter use metadata to request data from the “real” Manager databases.

    Strong pointed out that although the scaling problem at eBay was now addressed, new challenges surfaced. With horizontally scaled, modular systems with lots of sharing of IT resources and millions of relationships and interdependencies, finding the point of failure or congestion is a much more complex and time-consuming task than before. Assuring good performance has also become more challenging.

    This confirms our position at TeamQuest that no matter how much you virtualize your data center and introduce layers of abstraction, there will always be collections of physical IT resources out on the floor that will require proper instrumentation and tools for performance and capacity management.

    Finally, and perhaps most interesting, Strong made the prediction that cloud computing will lower the barriers to entrepreneurship. With easy access any time to just the right amount of computing resources you need without the need for your own data center, we will see more creative and innovative ideas comes to life.


    PS. Fun trivia: in the early stages of development of TeamQuest IT Service Analyzer and Reporter, their code names were Cirrus and Stratus :)

    Category: cmg-2008