CANARIE

“The Customer Empowered Networking

Revolution”

http://www.canarie.ca
http://www.canet3.net

Background Papers on Gigabit to

The Home and Optical Internet
Architecture Design Available Bill.St.Arnaud@canarie.ca
Optical Internet News list:
Send e-mail to Bill@Canarie.ca
 Outline
 The Message
 CANARIE
 CA*net 3
 CA*net 4
 Gigabit Internet to the School and Home
 The Message
 In mid 1990s the prevailing wisdom was that commercial sector would drive
design of Internet infrastructure
 R&E networks would focus on applications or specialized services
 As a result in North America R&E networks were commercialized or discontinued
 e.g NSFnet & CA*net
 However new network technologies and most importantly dark fiber is allowing
R&E networks to once again redefine telecommunications not only for themselves
but also for businesses and most importantly the last mile to the home
 R&E networks may become the cornerstone of municipal fiber to the home
networks
 Over time the current hierarchical “connection oriented” telecom environment will
look more like the Internet which is made up of autonomous peering networks.
 These new concepts in customer empowered networking are starting in the same
place as the Internet started – the university and research community.
 CANARIE Inc
 Mission: To facilitate the development of Canada’s communications
infrastructure and stimulate next generation products, applications
and services
 Canadian equivalent to Internet 2 and NGI
 private-sector led, not-for-profit consortium
 consortium formed 1993
 federal funding of $300m (1993-99)
 total project costs estimated over $600 M
 currently over 140 members; 21 Board members
 CA*net 3 National Optical Internet
Consortium Partners:
Bell Nexxia CA*net 3 Primary Route
Nortel
Cisco CA*net 3 Diverse Route
JDS Uniphase
Newbridge GigaPOP
ORAN
Deploying a 4
channel CWDM Deploying a 4
Gigabit Ethernet channel Gigabit
Condo Fiber network – 400 km Condo Dark Fiber Ethernet transparent
Networks Multiple Customer
Network linking all optical DWDM–
connecting Owned Dark Fiber
universities and 1500 km
universities and Networks
hospital Netera SRnet MRnet connecting
schools
universities and
schools ACORN
BCnet Calgary St. John’s
Regina
Winnipeg
ONet RISQ Charlottetown
Fredericton
Vancouver 16 channel DWDM Montreal
-8 wavelengths @OC-192 Halifax
reserved for CANARIE Ottawa
Seattle
-8 wavelengths for carrier
and other customers STAR TAP
Los Angeles Toronto
Chicago
New York
 Customer Empowered Networks
 School boards and municipalities throughout North America are deploying
their own dark fiber networks in partnership with next generation carrier
 Individual institutions – the customers – own and control their own strands
of fiber
 Fiber are configured in point to point private networks; or
 Connect to local ISP or carrier hotel
 Low cost LAN architectures and optics are used to light the fiber
 Control and management of the optics and wavelengths is under the domain of the
LAN customer at the edge, as opposed to the traditional carrier in the center
 These new concepts in customer empowered networking are starting in the same
place as the Internet started – the university and research community.
 Customers will start with dark fiber but will eventually extend further
outwards with customer control and ownership of wavelengths
 Extending the Internet model of autonomous peering networks to the telecom
world
 Examples
Customer Empowered Networks
 Universities in Quebec are building their own 3500km “condominium” fiber
network in partnership with 6 next gen carriers- $US 2million
 Will deploy and manage their own optics and long haul transmission gear
 Universities in Alberta are deploying their own 400 km 4xGbe dark fiber
network - $US 200K
 Deploy and manage their own optics and long haul transmission gear
 City of Montreal is second most fibered city in the world because of
municipal owned open access conduit
 In Ottawa is deploying a 85km- 144 strand “condominium” network
connecting 26 institutions – cost $1m US
 Peel County – Missassuaga & Brampton has built a 200km public sector
fiber network - $US 5m
 Many other cities including Ashland OR, Halifax, Toronto are looking at
similar initiatives
 Market Drivers
 First - low cost
 Up to 1000% reduction over current telecom prices. 6-12 month payback
 Second - LAN invades the WAN – no complex SONET or ATM required in
network
 Network Restoral & Protection can be done by customer using a variety of
techniques such as wireless backup, or relocating servers to a multi-homed site,
etc
 Third - Enables new applications and services not possible with traditional
telecom service providers
 Relocation of servers and extending LAN to central site
 Out sourcing LAN and web servers to a 3rd party because no performance impact
 IP telephony in the wide area (Spokane)
 HDTV video
 Fourth – Allows access to new competitive low cost telecom and IT
companies at carrier neutral meet me points
 Much easier to out source servers, e-commerce etc to a 3 rd party at a carrier
neutral collocation facility
 Quebec University Condo Network
Construit

Projet démarré
À venir
Bande passante louée

Val d’Or/Rouyn

MAN de Montréal

MAN d’Ottawa/Hull
MAN de Québec

MAN de Sherbrooke

Observatoire Mont-Mégantic
 Lionel-Groulx Lanaudière Sorel-Tracy

Montreal Public Sector Marie-Victorin

Condominium Networks

Rosemont
Montmorency
Ahuntsic Maisonneuve
Édouard-Montpetit
Bois-de-Boulogne
Vers Québec

St-Laurent/Vanier

Champlain
Vieux-Montréal
Gérald-Godin

Construit
Dawson
Projet démarré
John-Abbott À venir

André-Laurendeau Bande passante louée

Schoolboard Condominium Builds
 Typical Capital Costs
 Fixed One Time Capital Costs Include
 Management, engineering and construction costs
 Negotiating support structure agreements
 Fiber optic cables
 Fusing of fibers
 OTDR sweeps, Premise termination, etc.
 Average total cost between $7 and $15 per meter as follows:
 Engineering and Design:
 $1 - $3 per meter for engineering, design, supervision, splicing
 Plus Installation:
 $7 to $10 per meter for install in existing conduit; or
 $3 to $6 per meter for install on existing poles
 Plus Premise termination:
 Average $5k each
 Plus cost of fiber:
 15¢ per strand per meter for 36 strands or less
 12¢ per strand per meter for 96 strands or less
 10¢ per strand per meter 192 strands or less
 5¢ per strand per meter over 192 strands
 Examples of Dark Fiber costs
 University network Urban Fiber Builds
 Varennes: 50 km - $406K (maintenance $26K/year)
 Montreal East: 14 km - $120K (maintenance $9K/year)
 Laval: 33km - $213K (maintenance $15K/year)
 University network Rural Fiber Builds
 Sorel: 54km - $266K (maintenance $19K/year)
 Megantic: 40km -$273K (maintenance $14K/year)
 Schoolboards
 Victoriaville school board -Average price for fiber(s) $2 - $7 per meter
 Spokane School District - $US 800/mo for first 5 years then $US 400/mo
 Over 50 schools
 Stockholm - $1200/mo – over 100 schools
 Las Vegas School district – 240 schools – Telcordia (Bellcore) prime contractor
 Many, many others in the works
 Companies like Telcordia (Bellcore), IBM, etc are now leading development of
dark fiber networks for schools
 Condo Fiber Build Examples
 Des affluents: Total cost $1,500,00 ($750,00 for schools)
 70 schools
 12 municipal buildings
 204 km fiber
 $1,500,000 total cost
 average cost per building - $18,000 per building
 Mille-Isles: Total cost $2,100,000 ($1,500,000 for schools)
 80 schools
 18 municipal buildings
 223km
 $21,428 per building
 Laval: Total cost $1,800,000 ($1,000,000 for schools)
 111 schools
 45 municipal buildings
 165 km
 $11,500 per building
 Peel county: Total cost $5m – 100 buildings
 Cost per building $50,000
 Ottawa Fiber Condominium
 Consortium consists of 16 members from various sectors including
businesses, hospitals, schools, universities, research institutes
 26 sites
 Point-to-point topology
 144 fibre pairs
 Route diversity requirement for one member
 85 km run
 $11k - $50K per site
 Total project cost $CDN 1.25 million
 Cost per strand less than $.50 per strand per meter
 80% aerial
 Due to overwhelming response to first build – planning for second
build under way
 Typical Payback for school
(Real example – des affluents – north of Montreal)
 DSL to 100 schools - $400 per month per school
 Over 3 years total expenditure of $1,440,000 for DSL service
 Total cost of dark fiber network for 100 schools $1,350,000
 Additional condominium participants were brought in to lower cost to
school board to $750,000
 School board can now centralize routers and network servers at each
school
 Estimated savings in travel and software upgrades $800,000
 Payback typically 8 –16 months
 Independent Study by Group Secor available upon request
Reduction in the number of
servers
Before
After
fiber
fiber
Antennas 78
0
Novell Servers 82 1
SQL Servers 13 3
Lotus Notes Servers 2
1
Tape Backup Servers 12 4
Ethernet switches/hubs 10 98
Routers 108
3
Cache/proxy (Linux) 12
 CA*net 4

Optional Layer 3 aggregation service

Dedicated
Wavelength or
St. John’s
SONET channel
Regina Winnipeg

Charlottetown
Calgary Europe
Vancouver Montreal
Large channel
WDM system Fredericton Halifax
Seattle
OBGP switches
Ottawa

Chicago
New York
Toronto
 Overall Objective
 To deploy a novel new optical network that allows GigaPOPs at the
edge of the network (and ultimately their participating institutions) to
setup and manage their own wavelengths across the network and thus
allow direct peering between GigaPOPs on dedicated wavelengths and
optical cross connects that they control and manage
 To allow the establishment of wavelengths by the GigaPOPs and their
participating institutions in support of QoS and grid applications
 To allow connected regional and community networks to setup transit
wavelength peering relationships with similar like minded networks to
reduce the cost of Internet transit
 To offer an “optional” layer 3 aggregation service for those networks
that require or want such a facility
 O-BGP (Optical BGP)
 Control of optical routing and switches across an optical cloud is by the customer – not the
carrier
 Use BGP peering at network configuration stage for process to establish light path cross
connects
 Customers control of portions of OXC which becomes part of their AS
 Optical cross connects look like BGP speaking peers
 All customer requires from carrier is dark fiber, dim wavelengths, dark spaces and dumb
switches
 Building “carrier free” networks
 Traditional BGP gives no indication of route congestion or QoS, but with DWDM wave
lengths edge router will have a simple QoS path of guaranteed bandwidth
 May allow smaller ISPs and R&E networks to route around large ISPs that dominate the
Internet by massive direct peerings with like minded networks
 Wavelengths will become new instrument for settlement and exchange eventually leading
to futures market in wavelengths
 The biggest challenge of all…
To foster and accelerate
broadband Internet to the home
 The basic assumptions
 The good, the bad and the ugly..
 Monopolies are bad
 Duopolies are ugly
 Facilities based competition is good
 The private sector, in an open competitive market, is far more effective at
responding to consumer’s needs and introducing new services at lower
prices than any kind of government regulation
 But government has a responsibility to foster competition and ensure a
level playing field
 Where a natural monopoly exists government has a responsibility to
regulate that monopoly, but only as a last resort
 First it should make every attempt to develop mechanisms for
introducing private sector competition rather than depending on
legislative fiat
 Regulation should be seen as a last resort
 Facilities based competition in
the residential neighborhood?
 Facilities based competition is alive and well in downtown core
 The biggest challenge for governments is manage and coordinate the
digging up of streets
 Outside of downtown in big cities
 Usually only a monopoly telecom provider
 At best a duopoly
 How do we introduce facilities based competition into this market (or at
least come as close as possible to true facilities based competition)?
 As well how can we assure scalable high speed Internet services to the
home that eventually will support Gigabit speeds or higher?
 Critical role for governments and
universities
 Municipal dark fiber networks increases facilities based competition, levels
the playing field and provides greater choice to the consumer
 Governments can play a critical role in paying for dark fiber to all public
sector buildings
 Private sector can extend the fiber to businesses and homes ( via wireless,
fiber, DSL, etc)
 Universities can play critical role in organizing municipal condominium
fiber builds in their community and serve as the “anchor tenant”
 Governments and universities can also encourage building carrier neutral
collocation facilities
 In downtown cores will likely be done by private sector
 In suburbs will probably have to be public facility like school board
office, university, etc
 Networked Nation
CA*net 4 Usually one GigaPOP per province

Provincial research and education network Usually one access facility in every major town and city

Commercial Commercial
Internet Internet

SuperNodes
School board office Colo City Hall Colo Colo University

Splice Box
Nodes
School Library Hospital Splice Box School Colo
School

Homes
Option A: Home owners and
businesses have fused
connections all the way to
service provider at supernode Option B: Home owners
are aggregated at node by
service provider of their
choice
 Benefits to Industry
 For cablecos and telcos it help them accelerate the deployment of high speed
internet services into the community
 Currently deployment of DSL and cable modem deployment is hampered by
high cost of deploying fiber into the neighbourhoods
 Cable companies need fiber to every 250 homes for cable modem service, but
currently only have fiber on average to every 5000 homes
 Telephone companies need to get fiber to every 250 homes to support VDSL or
FSAN technologies
 Wireless companies need to get fiber to every 250 homes for new high
bandwidth wireless services and mobile Internet
 It will provide opportunities for small innovative service providers to offer service
to public institutions as well as homes
 For e-commerce and web hosting companies it will generate new business in out
sourcing and web hosting
 For Canadian optical manufacturing companies it will provide new opportunities for
sales of optical technology and components
 CANARIE's 6th Advanced Networks Workshop
"The Networked Nation"
November 28 and 29, 2000
Palais des Congrès
Montreal, Quebec - Canada

 "The Networked Nation", will focus on application architectures ("grids") made

up of customer owned dark fiber and next generation Internet networks like
CA*net 3 that will ultimately lead to the development of the networked nation
where eventually every school, home and business will have high bandwidth
connection to the Internet.
 Three tracks:
 Customer owned dark fiber for schools, hospitals, businesses and homes.
 Next generation optical Internet architectures that will be a natural and seamless
extension of the customer owned dark fiber networks being built for schools,
homes and businesses.
 "application grids", which are a seamless integration of dark fiber and optical
networks to support specific collaborative research and education applications.