New York, NY — Though Earth’s inhabitants are just coming to terms with the implications of global warming and how to adapt to changes in the climate, the 60 research scientists at Columbia University’s International Research Institute (IRI) for Climate and Society have been studying these weighty topics for years. The IRI staff develops and implements strategies to manage climate-related risks and opportunities in developing countries. Since December of 2005, the institute’s Data Library has been powered by an Xsan-based solution. The Data Library is a powerful and freely accessible online tool that allows users to view, manipulate, and download more than 400 climate-related data sets. At least 60,000 users from more than 120 countries use the library each year, and they can depend on Apple’s solution to help them quickly locate and download the data they need.
The IRI Data Library is implemented using a number of services, including web proxy and database servers, and a custom application (“Ingrid”) that acts as a web application server for information retrieval. The data can be accessed using web browsers or other HTTP client applications. Additionally, the data is user-selectable, and can be delivered in a wide variety of formats, such as standard data files, tables, plots, and animated GIF files. If desired, the data may be processed (that is averaged or sampled) before its delivery.
Several aspects of the Library cause the application to be very I/O (input/output) intensive. For example, a user may request a very large dataset, or a user may request many datasets at once. The Data Library must be capable of handling all such requests in a timely fashion. As much as a terabyte of data may be needed to fulfill a request. Clearly, this mandates that the back-end systems support extremely fast I/O, to ensure that results can be delivered within satisfactory timeframes. Plus, system reliability is of paramount importance, in order to deliver 24-hour-a-day, 7-day-a-week data access by Columbia’s researchers — as well as the international research community that visits the Library daily.
Xsan Offers Cost-Effective, Powerful Solution
IRI Computing Support Manager Leo Ostwald is responsible for the hardware and operating system aspects of the IRI Data Library. He was part of the committee that evaluated SAN (storage area network) solutions prior to the Apple purchase. He said the decision to go with Xsan came down to two key factors: performance and price.
“When you’re looking at other solutions, you really don’t know if what comes in the door is what was advertised,” Ostwald says. “With Xsan, the big draw for us was that Apple could actually demonstrate the performance they promised — several hundred megabytes per second, in terms of I/O, and the ability to host large file systems. The I/O performance was definitely a driving factor in going with Xsan.
“Also,” Ostwald continues, “the solution was affordable; the Xsan software is very competitively priced against other SAN solutions. Another advantage of Xsan was its interoperability with StorNext FX, which allows us to use some of our current data library systems as Xsan clients. We saw that this would help us control our costs, and minimize the time and effort needed to make the new storage system work with our existing software.”
