Data Storage System for High-End CCTV Infrastructures (5,000 HD cameras and above)
Data Storage in Video Surveillance
Data storage in Video Surveillance implies multi-thread sequential operations. Each IP-camera generates a sequential data stream that is written into a video archive storage. This results in a considerable multi-thread write workload.
Playing video from the archive entails sequential read operations triggered by single or multiple streams. In terms of workloads, the storage performs its best when a project does not involve permanent video playback by various operators from multiple workstations. Real-time monitoring does not require permanent data reading, and archive videos are viewed one at a time — occasionally or on a regular basis. In this scenario, the predominant workload — 90 percent or above — will have to do with writing, and the data storage will reveal no significant reading load.
In an opposite case, when multiple videos are played back in parallel from versatile cameras (e.g., casino, stadium, mall, etc.), the system processes a multitude of concurrent read streams and boosts the overall storage workload. As a result, the same number of cameras will require a higher performing storage system, otherwise the administrator may need to cut down the quantity of cameras per array (controller).
While operating under high workloads, it’s crucial to balance out storage performance, density and costs. System performance depends on the number of drives and individual performance of each drive. As a rule, high-speed software-defined technology utilizing HDDs fully complies with sequential workload requirements. In combination with high-density JBOD enclosures, storage professionals create effective configurations that cater to a large number of parallel threads or high-definition streams.
Greater scalability requires implementation of cluster solutions, such as the system based on the RAIDIX management software and HyperFS. Aside from high performance and low latencies, the list of key features expected from a fully functional Scale-Out solution includes:
- Single namespace for multiple storage clusters
- Concurrent access via versatile protocols
- File and block access to the same data.
Hereinafter, one will find standard requirements for data storage infrastructures servicing 5,000–10,000 parallel HD streams and the corresponding solutions offered by the RAIDIX technology.
Infrastructure and Capacity Requirements
The modern video surveillance market grows exponentially and demands brand-new solutions that keep up with the pace. Apparently, CCTV system functionality should go beyond camera capacity and software capabilities and factor in the IT infrastructure required for servicing cutting-edge professional equipment.
Should it come to hundreds and thousands of video cameras, a single server or boxed solution will fail to deliver the coveted reliability — even more so when a system involves video analytics (detection, tracking and recognition), integration with POS solutions, access control, etc. These scenarios call for specialized surveillance software — VMS — that delivers high scalability, smooth support for a multitude of IP cameras as well essential video functions.
Deploying VMS requires a dedicated IT infrastructure — a server farm processing multiple video streams. Processing a large quantity of high-definition threads demands maximum computation power available.
Let’s evaluate the requirements spanning system resources and the storage subsystem in large-scale Video Surveillance projects. This evaluation builds on the assumption that video servers receive, process and write video streams into an archive, and perform other essential VMS functions — without employing resource-intensive video analytics.
Picture display from IP-cameras for real-time monitoring and video playback should be performed from dedicated remote workstations, which will allow video servers a 50 percent workload cut. Remote workstations in Video Surveillance stand for powerful computers, such as graphic stations. These machines run special client software to ensure VMS connectivity and provide an ability to display a multitude of pictures on large screens.
As a basis for further calculations, we’ll take a stream generated by a single IP-camera via ONVIF (an open standard that ensures interoperability of IP-cameras with VMS), with 1MPX HD definition, basic H.265 codec, frequency rate of 25 fps and high frame activity. Average traffic volume in this configuration constitutes 2.84 Mbit/s.
The computing power required for servicing 5,000–10,000 video streams can be provided with 4-core Intel Xeon E3-2637 processors and 256GB RAM.
Challenges and the Solution
The RAIDIX software and COTS components
In the foundation of the suggested CCTV storage lies the RAIDIX Software-Defined Storage technology that enables the user to build reliable, high-performance, and fault-tolerant storage infrastructures. The use of commodity x64 hardware and high-density chassis allows a RAIDIX-based system to attain minimum TCO, $/GB and $/IOPS.
RAIDIX-powered commodity-off-the-shelf equipment ensures flexible, high-speed, fault-tolerant data storage for Video Surveillance needs. RAIDIX offers ad hoc solutions tailored to a specific number of cameras and individual workloads, eliminating the need to employ costly industrial systems, whose capacity and functionality may greatly exceed genuine IT needs of the customer.
A 5,000+ HD camera configuration may include х64 servers of choice and the following components:
- 2 Intel Xeon processors and a corresponding RAM volume.
- Single or multiple SAS HBAs for connecting internal and/or external disk enclosures.
- Single or multiple interfaces for cache synchronization in a dual controller configuration: SAS, InfiniBand, or Ethernet.
- Interfaces for SAN and/or NAS connectivity: Ethernet, InfiniBand, Fibre Channel.
- SAS/SATA HDDs of large (3,5”) or small (2,5”) form factor: any model from any manufacturer without limitations.
- Server platform that fits the abovementioned hardware requirements.
The RAIDIX technology resolves the following video surveillance issues:
Exponential growth of data streams for write/read in case of black operations or terrorist threats
Leveraging multi-gigabyte volumes of cache memory, RAIDIX allows for simultaneous processing of hundreds of heavy data streams with millisecond latencies. For cache data consistency, RAIDIX employs high-performance interfaces that may be combined into a trunk avoiding bottlenecks on cache synchronization. Cache protection against power failure is ensured with extra SPS modules.
Scalability on expanding CCTV coverage
RAIDIX enables flexible scalability of the video surveillance infrastructure. The system scales linearly up to 1800ТB (raw volume), and up to 16ZB when using the high-performance scale-out technology (HyperFS). At that, infrastructure extension does not entail any IO downtime and performs with full transparency to applications communicating with the storage system. The system can scale with the increments of a 12-disk batch, 60-disk JBOD enclosure, or a separate node.
Processing and storage of video streams generated by HD/Full HD/2K/4K/8К cameras from versatile vendors
The RAIDIX performance is optimized for storing large volumes of video data. Peak performance may reach 40GB/s for a standard rack in 42U. RAIDIX also delivers maximum efficiency when using commodity-off-the-shelf hardware by paralleling input/output operations.
High availability of the RAIDIX storage builds on using HA configurations of RAID 6, RAID 7.3 and RAID N+M on read/write. The technology sustains high performance even if hardware fails or the system undergoes array reconstruction.
Ensuring data integrity
Even partial data loss on saving video information may lead to losing an entire video stream. The use of proprietary RAID calculation algorithms, such as RAID 6 or RAID 7.3, allows the system to accomplish record-high speeds and data integrity. The array remains fully functional and reveals no performance degradation even if three disks fail in a RAID group.
Fault-tolerance on data writing
The data storage system ensures full fault-tolerance in case of multiple disk or enclosure failures, hidden errors in disk sectors or human error. The RAIDIX solutions have no single point of failure (SPOF) and ensure high data availability at 99,999 and higher.
Increased requirements for media servers running analytics software
The Advanced Reconstruction feature allows the system to sustain required data rate even in degraded mode and during RAID reconstruction. The reconstruction process does not impact other disks or overall system performance. RAIDIX can permanently track 1/2048th part of the disk and restore damaged data only — on the fly. Also, reconstruction priorities may be tweaked in real time.
The use of standard x64 hardware and high-density chassis allows for minimal ТСО, $/GB and $/IOps. Compliance with COTS hardware enables the user to pick the optimal configuration for building high-performance fault-tolerant data storage systems in Video Surveillance.
|Supported RAID levels||RAID 0/5/6/7.3/10/N+M|
|Max. number of drives in a RAID||64|
|Max. number of drives in the system||600|
|Scalability unit||12 drives|
|Hot spare||Dedicated reserve disks and shared access disks|
|Max. LUN size||Unlimited|
|Max. number of LUNs||487|
|iSCSI||MPIO, ACLs, CHAP-authorization, LUN masking, СRC Digest|
|Supported number of sessions||1024|
|Max. number of hosts in case of direct connection||32|
|Supported operating systems||Mас OS Х 10.6.8 and higher, Microsoft® Windows® Server 2008/ 2008 R2/ 2012, Microsoft ® Windows® XP/Vista/7/8; Red Hat Linux, SuSE, ALT Linux, Cent OS Linux, Ubuntu Linux; Solaris 10|
|Supported virtualization platforms||VMware ESX 3.5/4.0/4.1/5.0/5.1/5.5/6.0; KVM (Kernel-based Virtual Machine); RHEV (Red Hat Enterprise Virtualization), Microsoft Hyper-V Server, XenServer|
|Supported high-performance interface||Fibre Channel 8Gb, 16Gb; InfiniBand (FDR, QDR, DDR, EDR); iSCSI; 12G SAS|
|Supported NAS interfaces||SMB, NFS, FTP, AFP|
|Integration with MS Active Directory||Yes|
|WORM (Write Once – Read Many)||Yes|
|Number of nodes||2 in the Active/Active mode|
|Data caching||Two-tier: RAM and Flash, WriteBack and Read Ahead for multiple streams|
|QoS support||On the host/application level|
Organization of a storage network
Connecting video servers to the data storage system is performed via NAS (NFS, SMB) or SAN (iSCSI, FC, etc.) protocols and requires a dedicated data storage network. For this purpose, every video server should be equipped with corresponding physical adapters, preferably dedicated and duplicated. The core of this storage network is comprised of two allocated switches that connect a multitude of video servers to the data storage system. Physical decoupling of a data storage network from other data transfer networks and organization of allocated equipment with duplication (switches and adapters) will guarantee ease-of-use, transparency, security, required throughput and fault-tolerance.
In a simple scenario, a storage network can build on two high-performance 10GbE (Ethernet, 10Gbit/s) switches and two allocated 1–10GbE ports for each video server. The NFS and iSCSI protocols may be used for connectivity. Some cases demanding greater performance may involve convergent Ethernet or InfiniBand adapters with RDMA (SRP, iSER, RoCE) support, as well as corresponding switches. At that, 10Gbit/s ports will suffice on the server level whereas the switch level will require no less than 40Gbit/s.
Sample storage architecture (for 10,000 HD cameras)
In this scenario, we opt in for a large-scale RAIDIX-based storage system that enables JBOD connectivity, high performance and support for multiple HD threads. Full data integrity is achieved with the use of RAIDIX patented algorithms, including a double-parity RAID 6 level. RAID 6 ensure data continuity and sustainable performance even in case two drives fail in a RAID-group.
As data storage management software, RAIDIX operates with commodity х86–64 components (casing, drives, interface controllers, memory, processors, etc.) and allows the end customer to customize RAID arrays for specific tasks and decrease overall implementation and maintenance costs.
Below, one can find a CCTV storage configuration for 10,000 HD cameras (1MPX) when using RAID 6:
(1) 10 video servers employed by the customer
(2) 2 metadata controllers powered by RAIDIX (AIC HA401-LB platform)
(3) AIC HA401-LB platform
(4) HGST JBOD 4U60
This configuration encompasses 10,000 cameras (1MPX), h.265 codec, and the frequency of 25 FPS.
As we mentioned before, this task of building high-end storage clusters may go beyond the functionality of traditional file systems (FS). Classic FS impose the following limitations:
- Metadata and data are stored on the same partitions.
- Files are ‘scattered’ across the partition, causing access latencies.
- No protection against defragmentation.
- Low scalability by capacity, performance, file number, directory depth, etc.
- Lack of ‘native’ cross-platform support.
These issues can be resolved with the aid of cluster file systems. The HyperFS system from Scale Logic, for instance, ensures high scalability with full process transparency for the customer and shared access to data from various OS’s (in particular, through a dedicated NAS-gateway). Integrated with HyperFS, the RAIDIX software allows for a single namespace for SAN and NAS.
Technical benefits of the comprehensive solution include:
- Up to 4B files in a single directory
- Up to 4096 partitions that can be consolidated within a single FS
- No single point of failure (SPOF)
- Dynamic FS scalability by volume and performance
- Support for the latest versions of popular OS — Mac/Windows/Linux.
The HyperFS SAN file system ensures required redundancy, high data availability, mirroring of paths and data. HyperFS for SAN helps to transform multiple FS or iSCSI drive arrays into a unified storage cluster. This cluster enables parallel editing and playback from several client machines, as well as high performance and shared data access within a single namespace.
The system encompasses an optional metadata controller (MDC) with redundancy structure as well as a full-redundancy SAN-structure with metadata mirroring. HyperFS SAN also supports multi-path configurations in the Fibre Channel and iSCSI environments. The system reveals no SPOF and ensures high storage reliability.
The capabilities of Scale-Out NAS systems employed in major M&E infrastructures involve consolidation of up to 64 nodes in a single cluster, concurrent access via versatile protocols (SMB v2/v3, NFS v3/v4, FTP/FTPS, HTTP/HTTPS/WebDAV), workload balancing across the nodes (Round-Robin, Connection Count, Load node), and Active Directory support.
Enhanced RAIDIX and HyperFS features include:
- System optimization for large and small files
- Support for user and folder quotes
- SNMP monitoring by SNMP for SONG and MDC
- LDAP/Active Directory — an opportunity to use a local user database or integrate with Active Directory
- ACL support — an opportunity to utilize ACL on all supported operating systems.
The system based on RAIDIX and HyperFS provides customers with a high-performance solution featuring a single namespace, concurrent access via various protocols, low latencies, high scalability, file and block access to the same data.
Technical characteristics of the scale-out system
|System capacity||64 ZB (theoretical limit)|
|Max. number of files/objects/folders||Up to 4,000,000,000 when using 4TB metadata volume|
|File size||64 ZB (theoretical limit)|
|File name length||Windows: 255 ASCII characters; Linux/Mac: 255 ASCII characters|
|Directory depth||Windows: 244 characters; Linux: 4096 Bytes|
|Max. number of LUNs||4093|
|Number of metadata controllers (MDC)||Up to 2, can be configured in HA mode|
|Number of concurrent file systems||16|
|Full redundancy configuration||Supported: No single point of failure|
|Dynamic file system expansion||Yes, LUNs can be added with no downtime|
|Supported SAN client OSs||Windows 7 32/x86_64/Win 8, Win 10
Windows 2008/2008_R2/2012/2012_R2/Server 32/x86_64/2016_x86_64
SUSE 11 SP1-3
OS X 10.7-10.12
Building on RAIDIX Data Storage and modern commodity hardware, system integrators offer efficient storage solutions to CCTV infrastructures ranging from entry-level to large-scale.
RAIDIX stands for the following business benefits:
- Data availability.
- Full support for required data volumes along with low TCO.
- Record performance with multiple parallel video threads.
- Flexible scalability on increasing data volumes. The systems scales up by adding new enclosures and controllers with no prejudice to IO processes or applications interacting with the system.
- Decreased hardware overheads due to universal compatibility.
The RAIDIX technology in combination with the HyperFS file system complies with the high performance and fault-tolerance requirements. The use of the RAIDIX technology allows the user to minimize hardware overheads when building a storage cluster — by providing effective scale-out of existing infrastructures with no downtime or performance slump.