10.1184/R1/6619466.v1 David G. Andersen David G. Andersen Nick Feamster Nick Feamster Challenges and Opportunities in Internet Data Mining (CMU-PDL-06-102) Carnegie Mellon University 2006 network monitoring databases data management data mining 2006-01-01 00:00:00 Journal contribution https://kilthub.cmu.edu/articles/journal_contribution/Challenges_and_Opportunities_in_Internet_Data_Mining_CMU-PDL-06-102_/6619466 Internet measurement data provides the foundation for the operation and planning of the networks that comprise the Internet, and is a necessary component in research for analysis, simulation, and emulation. Despite its critical role, however, the management of this data—from collection and transmission to storage and its use within applications—remains primarily ad hoc, using techniques created and re-created by each corporation or researcher that uses the data. This paper examines several of the challenges faced when attempting to collect and archive large volumes of network measurement data, and outlines an architecture for an Internet data repository—the datapository—designed to create a framework for collaboratively addressing these challenges.