Since the beginning of the NSFNet effort (1986), the number and complexity of distributed applications has exploded, and each must create its own method for providing diagnostic tools and performance metrics. These distributed services have become increasingly dependent not only on the system and network infrastructures that they are built upon, but also each other. Effectiveness of the diagnostician is seriously hindered by the difficulty of access to diagnostic data. However, even if access can be gained, it exposes the daunting challenge of correlating a myriad of different data formats and an incredible amount of data both in static files and real time streams.
To say that diagnosis of distributed system is complex and difficult is a vast understatement, and the task is getting tougher every day. These challenges expose a void of tools and methods that has been growing for years. The search for a new methodology should not be a modest effort, but a call to action not only for higher education, but also the commercial and government sectors in a highly coordinated way.
These considerations led to the following goals for CyDAT:
Carnegie Mellon has developed a vision of becoming the world’s most sensed campus and the most diagnosable. In studying processes of early medical practitioners who had extremely similar objectives of diagnosing a largely unknown distributed system, we have gained insight on their initial methods and have initiated three coordinated efforts which are the pillars of CyDAT;
1. Technology (EDDY) - A platform for tool development that will expose diagnostic information directly to the diagnostician in a form they can use, when they want it. The tool platform will be based on a common diagnostic event record and an event orchestration technology. Both will be developed through an industry collaborative that will:
2. Observatory - Use the tool platform in real world environments to provide a campus wide repository and data orchestration center to import, transform and route diagnostic information for historical and real-time analysis. In essence a laboratory where scientists and engineers have data orchestration resources acting as a test bed for experimenting with new techniques and methods to address the challenges of distributed diagnostics of large scale systems.
3. Research - Constantly verify and modify the tool platform and the methods constructed around it both from a domain agnostic and highly focused domain specific perspective.
This work was initially sponsored through the generous support of the National Science Foundation. Thus far, there has been significant interest from both within the commercial community and from government, commercial and other external organizations, especially in developing and evolving existing data and orchestration standards. Carnegie Mellon University is spearheading the development effort with participation from multiple internal schools and organizations as well as several other universities.
Further inquires into the details of the effort or comments are welcome. Please address questions or comments to the project principle investigators:
Chas DiFatta, Research Scientist; Carnegie Mellon University
Mark Poepping, Director of Architecture, Discovery, and Integration; Carnegie Mellon University