Professional Documents
Culture Documents
Data Transfers Between Processes in An SMP System: by Jayesh Patel
Data Transfers Between Processes in An SMP System: by Jayesh Patel
Data Transfers Between Processes in An SMP System: by Jayesh Patel
By Jayesh Patel
Content
Introduction Data transfer techniques
Copying through Shared Buffers Copying through Message Queues Copying with the Ptrace System Call Copying Using a Kernel Module (Kaput) Copying Using a NIC
Introduction
SMP is multiprocessor system with centralized shared memory called as main memory, operating under single operating system with two or more homogeneous processor.
Shared memory
Process 2
Receive queue
Shared Memory
Process 1
Process 1
kernel Memory
NIC is Network Interface Controller. This mechanism is same as kaput difference is instead of kernel module NIC do the copy operation. In all previous methods when source processor perform the copy operation it replace the cache data with the data to be copied on shared memory region. Which increases the cache miss after the copy operation. NIC doesn't read the data from the cache of processor instead it read the data from the memory allocated to that processor in the shared memory region using RDMA(Remote Direct Memory Access). Advantages: Very less cache miss. No source processor involvement. Disadvantage: Since NIC uses I/O bus instead of system bus the latency is high.
Performance comparisons
Testing environment: Dual SMP 2 GHz Xeon node 4 GB of maim memory 512 kb 8 way associate L2 cache OS : Linux 2.6.10 Myrinet 2000 PC164C NIC 1. Latency:
Shared buffer
Message queue
Kaput put Kaput get NIC-copy put NIC-copy get Ptrace
3.3
2.1 2.1 11.6 14.3 20.2
Performance comparisons
2. Throughput 3. Cache misses
Coclusion
In this paper, we have described five mechanisms for transferring data between processes in an SMP machine and evaluated them based on bandwidth, latency and their impact on the application's cache. Not all mechanisms may be available in all environments. For instance, users typically would not be able to load a kernel module for the Kaput mechanism. The NIC-copy mechanism we analyzed in this paper was performed with a NIC that is a few years old. Faster NICs and communication subsystems, such as Myricom's MX, can provide up to 495 MBps bandwidth and latency down to 2.6 s.