Background:
One of my friend has discussed with me about how to gather real-time data from original devices ,such as CDR in telecom industry or performance data in electronic industry, he tend to using a real-time database as a buffer between devices and the database that really store data to use and analysis, because there is a good deal of data would be produced in a short time.
The following is the content of the mail that I send back to him.
There are two solutions I suggested, but at the first, you should know how often and how many data will be gather, and what level of data quality we should consider to.
Most of vendors of switcher using file system to store Signal Call Data Record (SCDR) ,It's about 10M per minutes, so the first solution is: using file to store the data that gathered from the original device, each device has it's own file-name conversion. The oracle collects data from those files at interval of 15 minutes. The benefit of this solution is have no need to using expensive Real-time Database. But on the contrary, we should control the transactions by ourselves, it is a terrible work.
The second solution is using a fast and light database (we call it "mid-db") to gather data from devices, same as the previous solution, the oracle collect data from the mid-db at interval of some certain times. But we should consider the money we paid on the mid-db.
If this project is one of your private projects. I recommend you choose the first solution to saving cost.
If you choose the second solution, I recommend choosing OODB (object oriented database ) as your mid-db, because one of our friend, *****, is agency of this production, may be you can get a fairly discount from him. On the other hand, the OODB have some success case in electronic industry on collecting performance data from devices. Through your interesting, you may found some materials from Google,.