In-Memory database

What is In-Memory database? How is it used in telecommunications and real time applications.

8 thoughts on “In-Memory database

  1. mykh74

    In-Memory database is a database which stores data in main memory of a computer. The main advantage of in-memory database is fast access to data and speed of processing as an application does not need to request data from I/O-based persistent media (disk, flash, etc.).

    This feature allows using In-Memory Database Systems (IMDSs) in real-time embedded applications. Many of modern Real-Time Operational Systems such as network routing, telecom devices and other industrial devices utilize IMDSs. In-Memory databases are also useful in non-embedded applications where exceptional performance is important. IMDSs provide necessary speed of data manipulation in trading and financial applications. In addition, IDMSs can be operated in multi-user Web applications such as e-commerce and social media networks to cache back-end functionality.

  2. Vinay Venkatachala

    General database management systems use disk storage. In-memory databases use main memory to store data. As the memory costs became cheaper, variety of new in-memory databases are getting developed. One of the key reasons for the interest in in-memory database is to get faster responses to queries which otherwise would be limited by the speed of the disk storage systems. In applications which require faster response times, these databases are very helpful. Some examples:
    • SAP HNA
    • Oracle TimesTen
    • MS SQL server
    • solidDB etc
    As these databases are relatively simpler compared to on-disk databases, they require significantly lower memory and CPU requirements.
    In domains like telecommunications where there is a need for-
    • Very fast data access, storage, modification
    • Systems which don’t have disk but need ability to manage large amounts of data
    In-memory databases play a pivotal role. Network routing, switching are some of the examples where in-memory databases running on real time operating systems provide fast turnaround times. As switches and routers are resource-constrained, in-memory databases will be very useful because of their low CPU usage compared to on-disk databases.

  3. rahul

    In-memory database is a database which is sitting primarily in the physical RAM. They provide all the ACID properties with the Durability achieved by many methods, transactions logging in to disks being one of them. Since the actual read and write is in the RAM, in-memory database is extremely fast (response time of us) and gives very high through-put. In-memory database is not optimized for disk access – which is fast for serial access of data compared to random access. In-memory database also differs from the full-cache disk based databases with the high response being the main differentiating factor.
    Examples of in-memory database solutions available include Oracle’s TimesTen, extermeDb,EhCache etc.
    In-memory database is important to telecommunication because of the features like low response time, high availability and scalability. With the technology advances from 2G to 3G to 4G the response time expected is very less and the throughput required is much high. This cannot be achieved using a disk based DB since the seek time of disk will prevent the telecom operators from achieving these targets. When a subscriber is making a call an operator has to check the balance, his subscription plans, caller tunes etc .and have to respond fast. Another application where less delay is of upmost importance is IMS based services like VoIP. Customers will move to a different operator who has a lesser delay and subsequently better call quality.
    The best example of real time application using in-memory database is the finance sector. Here the requirements are high availability, fault tolerance with minimum latency and scalability. The applications include solutions that track market data, market risk, online share trading. Another application is to find fraud in electronic payment systems. All these require very less response time with ability to go through a large number of entries, which can only be provided by a in-memory database
    Another important advantage provided by in-memory data base is the lesser time to market for telecommunication and real time applications. Without an in-memory database the developers have to replicate some of the data to memory tables to achieve the high response time. This requires careful design of the data required and optimizations which will take huge effort.

  4. manishekar

    Traditional databases use disk for storing database information. But, In-Memory database uses main memory for storing the database information. Since the information is stored in main memory, the data access will be faster compared to conventional databases. The faster data access is achieved due to the lesser number of CPU instructions involved. In this database, the data has to be stored in the main memory effectively. The available memory has to be used efficiently since the main memory in a system will usually be less compared to the disk memory.

    In this database, the primary data is stored in the volatile main memory. The information stored will be lost after the device is powered down. These In-Memory databases are ACID complaint. The atomicity of the database is maintained i.e., a transaction succeeds only if all the parts of the transaction succeed. Consistency feature ensures that only valid data will be written to the database. Isolation feature ensures that parallel executions do not affect each other. Durability ensures that the information stored in the database will not be deleted without any user confirmation.

    In-memory databases are used mainly in defense and telecommunication where the response time for the system is of high priority. They are used in applications that require real-time performance and low latency.

    solidDB is a product developed by IBM that uses in-memory database. The read and write of data happens on the main memory. These databases have a small footprint and are hence optimized for use in portable devices. The information in this database could be a contact list or cost information etc. eXtremeDB is another in-memory database that is used in telecommunication field.

    Empress Embedded Database is another database that falls under the category of in-memory database. It is mainly targeted for real-time embedded systems. These real-time system ranges from flight control to sensors. These systems require consuming less power and providing highly reliable data. Raima Database manager is another in-memory database that is used in real-time systems. The database, which is composed of multiple files, is stored in the main memory of the system and hence it provides very high performance.

    One disadvantage with In-Memory database is that the data in the main memory could be lost easily due to any system error.

  5. ramya

    IN-Memory database
    In-memory database is a database that uses a system’s main memory to store the data rather than the disk-based storage typically utilized by traditional databases. In-memory databases, or IMDBs, are frequently employed under two circumstances:
    1) in high-volume environments and when lots of data has to be stored and
    2) where response time is critical,
    This is because processing a database request is faster when system’s main memory is used as opposed to disk storage, particularly hard drive storage as the internal optimization algorithms are simpler and execute fewer CPU instructions. All the data required is stored in computers main memory so this eliminates seek time when querying the data.
    In-memory databases are also referred to as main memory database systems, or MMDBs, and have become more popular in recent years for handing High-Performance Computing (HPC) and Big Data applications. The distinction between traditional databases and in-memory databases has been blurred recently with the advent of hybrid databases, which support both in-memory and disk-based storage in order to maximize performance as well as reliability.
    In memory database used in telecommunications and real time applications.
    Disk-based database systems offer a very reliable storage option but are not able to attain the high performance required by a real-time application. Telecommunication applications are well-known for the need of high-performance real-time responses for their transactions. Main-memory database systems have been designed to achieve this goal. These systems offer most of the features of a disk-based database while having an architecture designed to manipulate main-memory resident data.
    Telecom applications demand high performance especially for their real-time applications. Location management is one of many telecom applications that fall into this category. The location problem in the telecom industry can be handled by making use of main memory database systems. The main-memory systems are designed to achieve the reliability of a traditional database without sacrificing its performance. The main-memory databases are used for location management in wireless infrastructure
    Main-memory databases deliver real-time performance by changing the assumptions around where the data resides. By managing data in memory, and optimizing data structures and access algorithms accordingly, database operations execute with maximum efficiency, achieving dramatic gains in responsiveness and throughput, even compared to a fully cached RDBMS.
    Some principles involved in main-memory database architecture are:
    Direct access to data: the database is mapped into the virtual address space of the process, allowing users to efficiently access the information through pointers to the memory location.
    No inter-process communication for basic system services: concurrency control and logging are provided via shared memory instead of calling external services. This eliminates expensive remote procedure calls significantly improving performance.
    Reduced volume of persistent logging: most of the logs are written to the memory. Just transactions that are ready to commit write a redo log to disk, reducing disk access considerably.

    Telecom applications such as location/roaming management could greatly benefit from a main memory database architecture. Both HLR/VLR store a limited amount of information for each mobile user which makes their architecture very simple to manage. With an appropriate set of database files and a robust storage manager, HLR/VLR databases could become more flexible and manageable. Such implementation would most probably not impact their performance and
    make them easily integrated to the remaining systems.

    NOTE:
    HLR -> Home Location Register
    VLR-> Visitor Location Register
    The roaming management strategies are supported by two main databases, the
    Home Location Register (HLR) and the Visitor Location Register (VLR). Whenever a subscriber is visiting a network out of its home location range, the corresponding VLR of the visited location queries the HLR for the subscriber information. This process is called registration.

  6. leekris

    An in-memory database (IMDB) is a database management system whose primary focus is on main memory (RAM) for storing the data. It stores data completely in main memory rather than storing on disk which a traditional dbms does. It loads data into RAM while booting. The core idea is traditional disk based storage systems have huge random access penalties which do not apply to DRAM. Data can be index/organized in a random access optimized way which is not feasible using traditional RDBMS data caching schemes. It is also called main memory database system (MMDB) or real time database (RTDB). As memory is faster than hard disk, they are used in applications where response time is critical in terms of data access, storage and manipulation.

    They employ internal optimization algorithms & data structures which require very few I/O (CPU requirements) and provide event, data and transaction management in the application tier. They have been developed to achieve very low response time and high throughput for performance-critical systems. All its data is stored and manipulated exactly in the form used by the application, removing overheads associated with caching and translation. Another advantage of in-memory databases comes into picture when transactional data and analysis data is stored in the same database. It is very easy to perform ad-hoc analysis as all of the data required to analyze a business case is written to a single database. However main drawback of MMDB is that the data is stored on volatile memory devices which tend to lose information when power is lost or reset. So out of the ACID properties, it lacks the support for durability. But providers are finding alternatives by introducing snapshot files, transaction logs, non-volatile memory/replication.

    An important usage of in-memory database systems is in real-time embedded systems. IMDSs present in real-time operating systems (RTOSs) provide the required responsiveness in various applications including control systems, multi media servers for real-time streaming. In Tele communication industry, IMDBs find their usage in routers and network management, telecom switching, real-time event aggregation, billing, fraud detection/prevention etc. They are also employed in managing music databases in MP3 players and in handling programming data in set-top boxes. In-memory databases’ have very small memory and CPU footprint which make suitable for most of the embedded systems as they are highly resource-constrained.

    Applications which are not embedded require very high performance and are major growth area for IMDBs. For example, algorithmic trading and other applications for financial markets use IMDSs to provide instant manipulation of data, in order to identify and leverage market opportunities. Some multi-user Web applications – such as e-commerce and social networking sites – use in-memory databases to cache portions of their back-end on-disk database systems. These enterprise-scale applications sometimes require very large in-memory data stores, and this need is met by 64-bit IMDS editions. An in memory database is also useful in game programming and business analytics, the most popular application for IMDB. Other popular applications for in-memory databases include supply chain management, sensor data management and geo-spatial applications.

    Another application use case is a distributed IP intrusion detection system which captures IP packages and validates them against known patterns. Also these systems are used to shut down and prevent unwanted network traffic. Data speed is one of the most challenging aspects of such kind of systems. Additionally they need to trigger real-time alarms to network administrators and notify other IP units about black listed traffic origins. With the RDM embedded database true in-memory and circular data buffering, IP intrusion units efficiently add dynamic, transactional safe data logging with real-time pattern matching. Also real-time alarms are pushed upstream to network administrators with ease, and distributed black list updates can be performed at a transactional level.

  7. Niki

    An in-memory database is a database management system that primarily relies on main memory for computer data storage. It is contrasted with database management systems which employ a disk storage mechanism. Main memory databases are faster than disk-optimized databases since the internal optimization algorithms are simpler and execute fewer CPU instructions. Accessing data in memory eliminates seek time when querying the data, which gives faster and more predictable performance than disk. In applications where response time is critical, such as telecommunications network equipment and mobile advertising networks, main memory databases are often used.

    Disk-based database systems offer a very reliable storage option but are not able to attain the high performance required by a real-time application. Every access to these databases incurs network latency, disk latency and buffer manager overhead. Telecommunication applications are well-known for the need of high-performance real-time responses for their transactions. In-memory database systems have been designed to achieve this goal.

    Principles involved in main-memory database architecture are:
    • Direct access to data: the database is mapped into the virtual address space of the process, allowing users to efficiently access the information through pointers to the memory location.
    • No inter-process communication for basic system services: concurrency control and logging are provided via shared memory instead of calling external services. This eliminates expensive remote procedure calls significantly improving performance.
    • Reduced volume of persistent logging: most of the logs are written to the memory. Just transactions that are ready to commit write a redo log to disk, reducing disk access considerably.
    The main memory resident of data have important implications in
    • Data Representation
    • Data Access Algorithms – Query Processing
    • Recovery
    • Concurrency control

    In case of MMDB, physical entity that corresponds to database is either shared memory segment or memory mapped file. These Inter Process Communication (IPC) mechanisms allow sharing of memory across processes. For multi process access, we should either go with shared memory IPC or memory mapped IPC mechanism. Usually the control information is stored separately from the data records to avoid data corruption of control information.

  8. sthiaga1

    In-memory database stores the data in the main memory of the computer rather than the traditional way of storing the data’s in the disk. They are used in an environment which has high-volume of data’s and the response time is critical in transacting the database as they reduce the time in accessing the data (from external disks) since they reside at the main memory itself . Due to this feature, they provide high-performance computing and can run big data – datasets. They are referred to as the IMDB (In –Memory database or main memory database), which are often used in telecommunications and certain other real time applications such as the mobile advertising agents, financial and trading where these kinds of databases are used in abundance. They communicate with the data stored in the main memory through the inter-process communication (IPC). In these kinds of databases, by eliminating physical disk access as well as caching logic and other sources of processing overhead, IMDSs can deliver real-time responsiveness demanded in many application categories that cannot tolerate the traditional database systems. Many IT companies have started shifting to the IMDB which has greater back-end functionality and unique shared memory scalability.

Leave a Reply