e-VLBI: Science over High-Performance Networks

APEC-TEL APGrid Workshop September 6, 2005

Masaki Hirabaru and Yasuhiro Koyama {masaki,koyama}@nict.go.jp

Radio Telescopes
Shanghai 25m

NICT Kashima Space Center 34m Perks 64m (right) Australia Telescope National Facility

Onsala Space Observatory 20m (left) MIT Haystack 18m

VLBI Applications
Geophysics and Plate Tectonics

400 200 0 -200 -400 1984

Kashima-Kauai Baseline Length -63.5 0.5 mm/year

5k 4 m Fairbanks 0 0 4k 7m 0 0 5k 7m 0 0 Kauai
Kashima

1986

1988

1990

1992

1994

Fairbanks-Kauai Baseline Length

400 200 0 -200 -400 1984 1986 1988 1990

-46.1 0.3 mm/year

1992

1994

400 200 0 -200

Kashima-Fairbanks Baseline Length 1.3 0.5 mm/year

-400 1984

1986

1988

1990

1992

1994

VLBI Applications (2)

Radio Astronomy : High Resolution Imaging, Astro-dynamics Reference Frame : Celestial / Terrestrial Reference Frame Earth Orientation Parameters, Dynamics of Earths Inner Core

HalcaMuses-B

Radio Telescope Satellite Halca and its images

NGC4261

Earth Orientation Parameters 4

VLBI (Very Long Baseline Interferometry)

radio signal from a star

e-VLBI geographically distributed observation, interconnecting radio antennas over the world

A
A/D

B
clock
~Gbps

ASTRONOMY GEODESY

Gigabit / real-time VLBI multi-gigabit rate sampling Large Bandwidth-Delay Product Network issue

A/D ~Gbps

clock

Internet

correlator

A B

VLBI System Transitions

2nd Generation 1st Generation
64Mbps 256Mbps

3rd Generation
12Gbps

K4 Terminal K5 Data Acquisition Terminal 2002~ PC-based System Hard-disk Storage Software Correlator e-VLBI over Internet 6

K3 Correlator (Center) K3 Recorder (Right)

1983~ Open-Reel Tape Hardware Correlator

K4 Correlator 1990~ Cassette Tape Hardware Correlator e-VLBI over ATM

ADS1000

Recent e-VLBI System Developments K5 by NICT

PC-VSI Board
(Supports VSI-H specifications)

(1024Msample/sec 1ch 1bit or 2bits)

VSI

Correlator other DAS Internet

VSI-E RTP/RTCP

VSI

IP-VLBI Board ADS2000

(~16Msample/chsec, ~4ch, ~8bits)

(64Msample/chsec, 16ch, 1bit or 2bits)

PC : Data Acquisition Correlation

Motivations
MIT Haystack NICT Kashima e-VLBI Experiment on August 27, 2003 to measure UT1-UTC in 24 hours 41.54 GB NICT MIT 107 Mbps (~50 mins) 41.54 GB MIT NICT 44.6 Mbps (~120 mins) RTT ~220 ms, UDP throughput 300-400 Mbps However TCP ~6-8 Mbps (per session, tuned) BBFTP with 5 x 10 TCP sessions to gain performance

HUT NICT Kashima Gigabit VLBI Experiment - RTT ~325 ms, UDP throughput ~70 Mbps However TCP ~2 Mbps (as is), ~10 Mbps (tuned) - Netants (5 TCP sessions with ftp stream restart extension)

There was bandwidth available but we could not utilize.

8

VLBI - Characteristics
Observing Bandwidth Data rate -1 (Precision of Time Delay) 1/2 (SNR)
Faster Data Rate = Higher Sensitivity

Wave Length / Baseline Length Angular Resolution -1 Baseline Length (EOP Precision)
Longer Distance = Better Resolution
9

Long Distant Rover Control

(at least) 7 minutes one way delay

Earth

Mars

When operator saw collision, it was too late.

10

Long-Distance End-to-End Congestion Control

B
Sender (JP) A C Overflow

Merge (Bottleneck) A+B > C

Feedback
200ms round trip delay

Receiver (US)

BWDP: Amount of data sent but not yet acknowledged 64Kbps x 200ms = 1600B ~ 1 Packet 1Gbps x 200ms = 25MB ~ 16700 Packets
11

Example
How much speed can we get?
Sender

1G

HighSpeed Backbone

1G

L2/L3 SW

100M

Receiver

Delay at light speed: 100ms

Average TCP Throughput less than 20Mbps

Q=50 12

Analyzing Advanced TCP Dynamic Behavior in a Real Network (Example: From Tokyo to Indianapolis at 1G bps with HighSpeed TCP)

Throughput RTT

Window Sizes

Packet Losses

The data was obtained during e-VLBI demonstration at Internet2 Member Meeting in October 2003.

13

Performance Measurement Platform for High-Performance Scientific Data Transfer

Seoul XP

Korea
KOREN
Kwangju

10G Daejon Taegu Busan 1G

250km 100km

Kashima

JGNII
1G (10G) Koganei U of Tokyo

bwctl server perf server e-vlbi server

SWITCH

2.5G

Tokyo XP / JGN II I-NOC

1G(10G)

2.5G SONET

GEANT JGN II APII/JGN II

Kitakyushu
1,000km

10G
7,000km

10G

10G
9,000km

Chicago MIT Haystack

Fukuoka

1G

1G (10G)

TransPAC Abilene
Pittsburgh

Genkai XP

Fukuoka Japan

2.4G Washington DC

10G

4,000km

U. Hawaii

Los Angeles

Indianapolis

*Performance Measurement Point Directory http://e2epi.internet2.edu/pipes/pmp/pmp-dir.html

Locate the problem International collaboration to support for science applications

14

Solutions by Advanced TCPs

How can wee foresee collision (queue overflow)?

Loss-Based AQM (Advanced Queue Management) Reno, Scalable, High-Speed, BIC, Delay-Based Vegas, FAST Explicit Router Notification ECN, XCP, Quick Start, SIRENS, MaxNet
15

TCP Performance with Different Queue Sizes

16

Measuring Bottleneck Queue Sizes

Sender Capacit yC Receiver

packet train

lost packet measured packet

Queue Size = C x (Delaymax Delaymin) Switch / Router Queue Size Measurement Result Device Switch A Switch B Queuing Delay (s) 6161 22168 Capacity (Mbps) 100* 100*
Estimated Queue Size (1500B)

50p/75KB 180p/270KB
* set to 100M for measurement

Switch C
Switch D Switch E Router F

20847
738 3662 148463

100*
1000 1000 1000

169p/254KB
60p/90KB 298p/447KB 12081p/18MB
cross traffic injected for measurement

Router G

188627

1000

15350p/23MB

17

Typical Bottleneck Cases

b-1)
Queue ~100 Queue ~1000

a)

VLANs
Switch Router

Switch

Router
1Gbps (10G)

100Mbps (1G)

b-2)

9.5G WAN-PHY 802.1q Tag

Switch/ Router

10G LAN-PHY Ethernet Untag 18

e-VLBI Demonstration in JGN II Osaka (Jan. 2005)

Dr. Koyama
Osaka 1G CPU x8 1G x4 1G x4 #1,#2 #3,#4 Raid Disks #5,#6 1G /10G
NICT Kashima

10G 1G Tokyo Abilene (10G) WAS

CHI

1G/2.5G
MIT Haystack

JGN II Intl (10G)

4 Apple G5 Server machines (8 CPUs in Total)

#7,#8 Raid Disks *TCP parameters were tuned for the path.

Software Cross Correlation ~240Mbps per station e-VLBI data transfer achieved ~700Mbps from Haystack to Osaka ~900Mbps from Kashima to Osaka

19

VLBI Antenna Locations in North-East Asia

Shintotsukawa 3.8m Nanshan (Urumqi) 25m 70km from Urumqi Miyun (Beijing) 50m 50km from Beijing

Tomakomai 11m, FTTH (100M) 70km from Sapporo Nobeyama 45m OC48/ATM Galaxy Mizusawa 10m 20m 118km from Sendai Tsukuba 32m, OC48/ATMx2 SuperSINET Kashima 34m, 1Gx2 JGN II, OC48/ATM Galaxy Koganei 34m, 1Gx2 JGN II, OC48/ATM Galaxy Gifu 11m 3m, OC48/ATMx2 SuperSINET Iriki 20m Kagoshima 6m Yamaguchi 32m Aira 10m 1G, 75M SINET

Seoul 20m Yonsei U

2Mbps

Daejon 14m Taeduk Jeju 20m Tamna U

Usuda 64m, OC48/ATM Galaxy

Ulsan 20m U Ulsan

Observatory is on CSTNET at 100M Sheshan (Shanghai) 25m 30km from Shanghai

2Mbps

Yunnan (Kunming) 3m (40m) 10km from Kunming Ishigaki 20m

Ogasawara 20m Chichijima 10m

Legend
connected not yet connected antenna under construction

20

e-VLBI Data Transfer

Common e-VLBI file transfer
Carry the disk to the nearest station to put on-line

Real-time e-VLBI flat-rate live data transfer

Internet

Correlation

Synchronize

Future e-VLBI infrastructure multicast and automated

Correlate among many combinations concurrently to get more precise data (like a virtual huge antenna)
21

Summary
High-performance scientific data transfer faces on network issues we need to work out. Big science applications like e-VLBI and HighEnergy Physics need cooperation with network and Grid researchers. Deployment of performance measurement Infrastructure over research networks is ongoing on world-wide basis.

22