Acceleration

WARNING: AT LEAST ONE DOCUMENT COULD NOT BE INCLUDED!
You were not billed for these documents.

Please see below.
Document Number Document Description Pages Document Error

Document 1 Attachment 57 DOCUMENT COULD NOT BE RETRIEVED!
attachment However, it may still be viewable individually.
Case 5:19-cv-04133-SVK Document 1 Filed 07/18/19 Page 1 of 25
1 David P. Enzminger (SBN: 137065)

denzminger@winston.com
2 Michael A. Tomasulo (SBN: 179389)
mtomasulo@winston.com
3 WINSTON & STRAWN LLP
333 South Grand Avenue, 38th Floor
4 Los Angeles, CA 90071-1543
Telephone: (213) 615-1700
5 Facsimile: (213) 615-1750
6 Louis L. Campbell (SBN:221282)

llcampbell@winston.com
7 Matthew R. McCullough (SBN: 301330)
mrmccullough@winston.com
8 WINSTON & STRAWN LLP
275 Middlefield Road, Suite 205
9 Menlo Park, California 94025-4004
Telephone: (650) 858-6500
10 Facsimile: (650) 858-6550
11 Attorneys for Plaintiff

EPIC GAMES, INC.
12
13 UNITED STATES DISTRICT COURT
14 NORTHERN DISTRICT OF CALIFORNIA
15 EPIC GAMES, INC., a Maryland Corporation, Case No.: 3:19-cv-04133

16 Plaintiff, COMPLAINT FOR DECLARATORY
JUDGMENT OF NON-INFRINGEMENT OF
17 v. U.S. PATENT NOS. 6,701,344, 6,714,966,
6,829,634, 6,732,147, 6,910,069 6,920,497, AND
18 ACCELERATION BAY LLC, a Delaware 7,412,537
Limited Liability Corporation,
19 JURY TRIAL DEMANDED
Defendant.
20
21
22
23
24
25
26
27
28
COMPLAINT FOR DECLARATORY JUDGMENT OF PATENT NON-INFRINGEMENT - CASE NO. 3:19-CV-04133

1 Plaintiff Epic Games, Inc. (“Epic” or “Plaintiff”) by and through its attorneys, alleges
2 against Defendant Acceleration Bay, LLC (“Acceleration” or “Defendant”) as follows:
3 PARTIES AND BACKGROUND
4 1. Plaintiff Epic is one of the leading video game companies in the world. It develops
5 and publishes, among other things, the “Fortnite” video game. Epic publicly released Fortnite’s
6 cooperative Save the World game mode in or around July 2017. Epic publicly released Fortnite’s
7 free-to-play Battle Royale game mode in or around September 2017. Fortnite is an extremely
8 popular game with over two hundred and fifty million players.
9 2. On information and belief, sometime around February 2015, Defendant Acceleration
10 acquired certain rights to certain patents previously assigned to The Boeing Company. Those patents
11 include U.S. Patent Nos. 6,701,344 (“the ’344 Patent”), 6,714,966 (“the ’966 Patent”), 6,829,634
12 (“the ’634 Patent”), 6,732,147 (“the ’147 Patent”), 6,910,069 (“the ’069 Patent”), 6,920,497 (“the
13 ’497 Patent”) and 7,412,537 (“the ’537 Patent) (collectively, the “Asserted Patents”). The Asserted
14 Patents are attached as Exhibits 1-7.
15 3. On December 28, 2018, Acceleration sent a letter to Epic’s address. The letter states
16 without explanation that Epic’s videogame Fortnite “requires a license.” The December 28 letter
17 makes an explicit and direct charge of infringement, stating that six of the Asserted Patents are
18 “Patents Infringed By Epic Games.”
19 4. The December 28 letter appears to have been a “cut-and-paste” from a letter sent to a
20 totally unrelated company. For instance, it states that “[UNRELATED COMPANY] has an
21 affirmative responsibility to ensure that it has secured all necessary patent rights to sell the products
22 and services identified in this letter” and concludes by stating “[w]e look forward to your working
23 with [UNRELATED COMPANY] on a business solution, and thank you in advance for your prompt
24 attention to this matter.” Therefore, on information and belief, Acceleration sent the same generic,
25 substance-free form letter alleging infringement to at least one, and perhaps many, other companies.
26 5. On March 4, 2019, Acceleration sent a second letter to Epic’s address. This letter
27 states without explanation that “[o]ur analysis of our patent portfolio and Epic Games’ multi-player
28 games indicates use of Acceleration Bay patented technologies, at minimum, the following games
1
1 require a license: Fortnite: Save the World, Fortnite Battle Royale, and potentially other games.”
2 (hereinafter, the “Accused Products”).
3 6. As with the prior letter, the March 4 letter makes an explicit and direct charge of
4 infringement, stating that six of the Asserted Patents are “Patents Infringed By Epic Games.”
5 7. Neither of these letters mention the ’537 Patent, but Acceleration subsequently
6 advised that the ’537 Patent was at issue as well.
7 8. On our around July 10, 2019, representatives of Epic and Acceleration had a
8 teleconference.
9 9. None of Acceleration’s communications provided an explanation for its repeated
10 charges of infringement. No claim charts have been provided. Neither letter explains what claims
11 Epic allegedly infringes, or which patents or claims are supposedly infringed by each of the games
12 named in the letters.
13 10. In addition, on information and belief, both letters contain statements about
14 Acceleration itself that are incorrect or, at best, misleading. For instance, the letters state that
15 Acceleration was “[f]ounded in 2012” and that it is a “technology incubator” that “partners with
16 inventors, corporations and entrepreneurs to accelerate growth in creating innovative companies.”
17 11. On information and belief, and based on publicly available records and publicly
18 available Court-filings, Acceleration is not a “technology incubator” and was not “[f]ounded in
19 2012.”
20 12. Acceleration was formed by its litigation counsel in August 2014 for the purpose of
21 acquiring and enforcing the Asserted Patents.
22 13. Acceleration is not a “technology incubator” and does not “partner” with “inventors,
23 corporations and entrepreneurs” and does not “accelerate growth in creating innovative companies.”
24 14. Publicly available information does not indicate that Acceleration provides any
25 product or service or has any non-litigation-based income or revenue.
26 15. On information and belief, Acceleration is exclusively in the business of monetizing
27 patents acquired from third parties.
28 16. Epic’s games do not infringe the Asserted Patents.
2
1 17. The claims of six of the seven Asserted Patents (all but the ’497) all relate to a
2 computer network and/or broadcast channel with an m-regular, incomplete topology.
3 18. These six Asserted Patents require a specific “peer to peer” computer network where
4 computers in the network communicate directly with each other as opposed to communicating
5 through a central server.
6 19. Certain of the claims are directed to a computer network.
7 20. Computer networks existed and were in public use prior to the alleged inventions of
8 the Asserted Patents.
9 21. Certain of the claims are directed to a broadcast channel.
10 22. The Asserted Patents state that “[e]ach computer that is connected to the broadcast
11 channel receives all messages that are broadcast while it is connected.” See, e.g., Ex. 1, ’344 Patent
12 at 4:12–14.
13 23. Broadcast channels existed and were in public use prior to the alleged inventions of
14 the Asserted Patents.
15 24. The “Background” section of the Asserted Patents identifies “client/server
16 middleware systems” as prior art to the Asserted Patents. See, e.g., Ex. 1, Background ’344 Patent
17 at 1:58–2:14.
18 25. The Asserted Patents state that “The client/server middleware systems provide a
19 server that coordinates the communications between the various clients who are sharing the
20 information.” See, e.g., Ex. 1, Background ’344 Patent at 1:58–60.
21 26. Client-server middleware systems where a server coordinates the communications
22 between the various clients who are sharing the information existed and were in public use prior to
23 the alleged inventions of the Asserted Patents.
24 27. The main purported point of novelty for these six patents is that the claimed network
25 forms an m-regular, incomplete graph instead of the prior art client-server or full mesh networks.
26 28. The claims of these patents all require that (1) each participant/computer in the
27 network must have connections to at least three other neighboring participants.
28
3
1 29. The claims of these patents all require that (2) the network must be “m-regular” where
2 each participant is connected to the exact same number, m, of neighbor participants.
3 30. The claims of these patents all require that (3) the network must be incomplete – m
4 must be at least two less than the total number of participants.
5 31. In other words, each participant must be connected to at least three neighbor
6 participants, and no participant can be connected to all of the other participants in the network.
7 32. Before the technologies covered by the Asserted Patents were allegedly invented,
8 other types of networks such as client-server and “full mesh” networks were well known and used
9 for videogames.
10 33. Multiplayer videogames using computer networks existed and were in public use prior
11 to the alleged inventions of the Asserted Patents.
12 34. Multiplayer videogames using broadcast channels existed and were in public use prior
13 to the alleged inventions of the Asserted Patents.
14 35. The Asserted Patents state that client-server systems are “not particularly well suited
15 to sharing of information among many participants.” See, e.g., Ex. 1, Background section of ’344
16 Patent at 1:65–67.
17 36. The Asserted Patents state that client-server networks are inferior and different from
18 the claimed invention because, with a client-server network, “a failure at a single computer (i.e., the
19 server) would prevent communications between any of the clients.” See, e.g., Ex. 1, Background
20 section of ’344 Patent at 1:58–2:14.
21 37. In contrast, the Asserted Patents contend that “[t]he use of a 4-regular graph means
22 that a computer would become disconnected from the broadcast channel only if all four of the
23 connections to its neighbors fail.” See, e.g., Ex. 1, Background section of ’344 Patent at 4:39-42.
24 38. A comparison of the three network types is shown below:
25
26
27
28
4
8 39. At all times, the Accused Products have used the non-infringing, client-server
9 architecture that was well known in the prior art.
10 40. In this architecture, Fortnite players are connected to a server and send and receive
11 information only to/from a server.
12 41. Fortnite players are not connected to other Fortnite players and do not directly
13 exchange information with other Fortnite players.
14 42. The final Asserted Patent is the ’497 Patent. Neither Epic nor the Accused Products
15 infringe this patent.
16 43. The claims of the ’497 Patent require, among other things, the use of a non-random
17 “port ordering algorithm” where “the communications ports selected by the port ordering algorithm”
18 are re-ordered.
19 44. Thus, for instance, Claim 9 of the ’497 Patent claims a “component in a computer
20 system for locating a call-in port of a portal computer” which includes “means for selecting the call-
21 in port of the identified portal computer using a port ordering algorithm” and “means for re-ordering
22 the communications ports selected by the port ordering algorithm.”
23 45. Fortnite does not “select” a “call-in port” of an identified portal computer using a non-
24 random port ordering algorithm and does not include means for re-ordering the communications
25 ports selected by any non-random port ordering algorithm.
26 46. For these and many other reasons, Epic does not infringe any claim of the Asserted
27 Patents.
28
5
1 NATURE OF THE ACTION
2 47. This is an action brought pursuant to the Declaratory Judgment Act, 28 U.S.C. § 2201,
3 for a declaratory judgment that Epic does not infringe any of the Asserted Patents, including through
4 its Fortnite products.
5 48. Epic seeks a declaratory judgment that neither it nor the Accused Products infringe
6 any of the Asserted Patents under the Patent Laws of the United States, 35 U.S.C. §§ 101, et seq.
7 PARTIES, JURISDICTION AND VENUE
8 49. The parties to this lawsuit are Plaintiff Epic and Defendant Acceleration.
9 50. Plaintiff Epic is a Maryland corporation with its principal place of business at 620
10 Crossroads Blvd, Cary, North Carolina, 27518.
11 51. Defendant Acceleration, upon information and belief, is a Delaware limited liability
12 company, which has at all times represented that it has a principal place of business within the
13 Northern District of California. In multiple complaints against other companies, Acceleration has
14 pleaded that it has a principal place of business in this District. Presently, Acceleration’s website
15 states that its principal and only place of business is located at 951 Mariners Island Blvd, San Mateo,
16 California, 94404. Both of the letters that Acceleration sent to Epic bear this address as well.
17 52. This is an action under the Federal Declaratory Judgments Act, 28 U.S.C. §§ 2201
18 and 2202, against Defendant for a declaration that, pursuant to the patent laws of the United States,
19 35 U.S.C. §§ 1 et seq., the disputed claims of U.S. Patent Nos. 6,701,344, 6,714,966, 6,829,634,
20 6,732,147, 6,910,069, 7,412,537, and 6,920,497 are not infringed by Plaintiff or its games.
21 Jurisdiction as to these claims is conferred on this Court by 28 U.S.C. §§ 1331 and 1338(a).
22 53. This Court has personal jurisdiction over Defendant Acceleration at least because of
23 its continuous and systematic contacts with the State of California and with this District. As noted
24 above, Acceleration’s principal place of business is located within this District. Acceleration has
25 also registered to do business in California with the California Secretary of State since March 13,
26 2015, and keeps a designated agent for service of process in Foster City, California. Acceleration
27 has established minimum contacts with the forum and the exercise of jurisdiction would not offend
28 traditional notions of fair play and substantial justice.
6
1 54. Venue is proper in this jurisdiction under 28 U.S.C. §§ 1391 and 1400(b), at least
2 because Defendant maintains its principal place of business in this district, a substantial part of the
3 events or omissions giving rise to the claims occurred in this district, and Defendant purposefully
4 directed activities to this district.
5 55. An actual controversy exists between Defendant and Plaintiff as to whether Plaintiff
6 infringes the Asserted Patents. In both its December 28, 2018 and March 4, 2019 letters,
7 Acceleration identifies specific patent numbers, made an explicit and direct statement that Epic is
8 infringing the Asserted Patents, and asserted that Fortnite “require[s] a license.” The March 4 letter
9 further adds that Acceleration’s “analysis” of “Epic Games’ multi-player games indicates use of
10 Acceleration Bay patented technologies.”
11 56. Epic denies that any of its activities or the Accused Products infringe any claim of the
12 Asserted Patents purportedly owned by Defendant.
13 57. Epic is aware that Acceleration has asserted the same six patents against other major
14 videogame publishers, even though Epic can see no applicability of the claimed technology to the
15 videogame industry. In addition, Acceleration’s December 28, 2018 letter to Epic also included
16 claims regarding an unrelated third party in the videogame industry. This, in combination with
17 Acceleration’s litigious history with other companies in the industry, suggests that Acceleration’s
18 business model is limited to enforcing these patents.
19 58. On information and belief, Acceleration engages in no business activity other than
20 seeking to enforce the Asserted Patents. As a result of Defendant’s actions and repeated licensing
21 demands, Plaintiff has a reasonable apprehension that Defendant will commence litigation against
22 Plaintiff on the Asserted Patents.
23 59. There is an immediate, real, and substantial justiciable controversy between Plaintiff
24 and Defendant as to its purported right to threaten or maintain suit for infringement of the Asserted
25 Patents; as to the scope and enforceability thereof; and as to whether Plaintiff or any Accused
26 Product infringes any enforceable claims of the Asserted Patents. This controversy is of such
27 immediacy and reality as to warrant declaratory relief so that the parties may ascertain their rights
28 and duties with respect to the Asserted Patents. Therefore, without waiver of any rights, including
7
1 the right to challenge prudential standing, Plaintiff brings this declaratory judgment action seeking
2 a declaration that it and the Accused Products do not infringe any of the Asserted Patents.
3 INTRADISTRICT ASSIGNMENT
4 60. This is an intellectual property action subject to district-wide assignment pursuant to
5 Local Rules 3-2(c) and 3-5(b).
6 CAUSES OF ACTION
7 FIRST CAUSE OF ACTION
8 (Declaratory Judgment of Non-Infringement of the ’344 Patent)
9 61. Plaintiff incorporates herein by reference and re-alleges all preceding paragraphs as
10 though fully set forth herein.
11 62. Defendant Acceleration has alleged and continues to assert that Plaintiff and its
12 Accused Products and services infringe the ’344 Patent.
13 63. Claims 1–11 and 16–19 of the ’344 Patent were cancelled, and claim 21 added in inter
14 partes review proceedings before the USPTO. See Ex. 1.
15 64. Dependent claim 12 and the surviving independent claims of the ’344 Patent are
16 shown below (with claim element enumeration added for convenience).
17
Dependent Claim 12 (inheriting all limitations from Claim 1)
18 12-a. A computer network for providing a game environment for a plurality of participants,
12-b. each participant having connections to at least three neighbor participants,
19 12-c. wherein an originating participant sends data to the other participants by sending the
data through each of its connections to its neighbor participants and
20 12-d. wherein each participant sends data that it receives from a neighbor participant to its
other neighbor participants,
21 12-e. further wherein the network is m-regular, where m is the exact number of neighbor
participants of each participant and
22 12-f. further wherein the number of participants is at least two greater than m thus resulting
in a non-complete graph.
23
12-g. The computer network of claim 1 wherein the interconnections of participants form a
24 broadcast channel for a game of interest.
Claim 13
25 13-a. A distributed game system comprising:
13-b. a plurality of broadcast channels, each broadcast channel for playing a game,
26 13-c. each of the broadcast channels for providing game information related to said game to
a plurality of participants,
27 13-d. each participant having connections to at least three neighbor participants,
13-e. wherein an originating participant sends data to the other participants by sending the
28 data through each of its connections to its neighbor participants and
8
1 13-f. wherein each participant sends data that it receives from a neighbor participant to its
neighbor participants,
2 13-g. further wherein the network is m-regular, where m is the exact number of neighbor
3 13-h. further wherein the number of participants is at least two greater than m thus resulting
in a non-complete graph;
4 13-i. means for identifying a broadcast channel for a game of interest;
13-j. and means for connecting to the identified broadcast channel.
5 Substitute Claim 21
21-a. A computer network for providing a game environment for a plurality of gaming
6 participants, each gaming participant having connections to at least three neighbor gaming
participants,
7 21-b. wherein an originating gaming participant sends gaming data to the other gaming
participants by sending the gaming data through each of its connections to its neighbor
8 gaming participants and
21-c. wherein each gaming participant sends gaming data that it receives from a neighbor
9 gaming participant to its other neighbor gaming participants,
21-d. further wherein the network is m-regular, where m is the exact number of neighbor
10 gaming participants of each gaming participant,
21-e. further wherein the number of gaming participants is at least two greater than m thus
11 resulting in a non-complete graph,
21-f. further wherein the connections between the gaming participants are peer-to-peer
12 connections,
13 21-g. further wherein the network is formed through a broadcast channel that overlays an
underlying network
14 21-h. further wherein the game environment is provided by at least one game application
program executing on each computer of the computer network that interacts with the
15 broadcast channel, and
21-i. further wherein gaming participants can join and leave the network using the broadcast
16 channel.
65. Plaintiff does not infringe any claim of the ’344 Patent, at least because, by way of
17
non-limiting example, Plaintiff does not make, use, or sell the “computer network” or “distributed
18
game system” as required by the claims of the ’344 Patent.
19
66. Plaintiff does not infringe any of the claims of the ’344 Patent, at least because, by
20
way of non-limiting example, the Accused Products do not meet the following claim limitations of
21
the claims shown above: claim 12, elements 12-a through 12-f; claim 13, elements 13-a through 13-
22
j; claim 18, elements 18-a through 18-e; and claim 21, elements 21-a through 21-i.
23
67. For instance, each “participant” in Fornite is not connected to at least three “neighbor”
24
participants as required by elements 12-b, 13-d and 21-a.
25
68. Fortnite does not form m-regular and incomplete/non-complete networks as required
26
by elements 12-e, 12-f, 13-g, 13-h, 18-e, 21-d, and 21-e.
27
28
9
1 69. Fortnite does not send data to the other participants and nodes as required by elements
2 12-c, 12-d, 13-e, 13-f, 18-c, 18-d, 21-b, and 21-c.
3 70. Plaintiff’s activities, Accused Products, and services have not and do not directly
4 infringe, do not infringe by inducement, and do not contributorily infringe any enforceable claims
5 of the ’344 patent.
6 71. Plaintiff’s activities, Accused Products, and services have not and do not infringe,
7 willfully or otherwise, any enforceable claims of the ’344 Patent.
9 literally or under the doctrine of equivalents, any enforceable claims of the ’344 Patent.
10 73. In 2006, Boeing licensed the ’344 Patent to Sony.
11 74. Sony makes multiplayer videogame products that, according to Acceleration, meet
12 the limitations of the claims of the ’344 patent.
13 75. Sony did not mark the ’344 Patent number on any such Sony products.
14 76. Acceleration has not provided constructive notice of infringement of the ’344 Patent
15 to Plaintiff.
16 77. As to the ’344 Patent, Plaintiff has not complied with the requirements of 35 U.S.C.
17 § 287(a).
18 78. An actual controversy thus exists between Plaintiff and Defendant as to whether the
19 Accused Products infringe the ’344 Patent.
20 79. Plaintiff seeks a declaratory judgment from this Court under Rule 57 of the Federal
21 Rules of Civil Procedure and 28 U.S.C. § 2201 declaring that Plaintiff and each of the Accused
22 Products is not infringing and has not infringed the ’344 Patent and granting Plaintiff all other
23 declaratory relief to which it may be entitled.
24 SECOND CAUSE OF ACTION
28
10
3 82. Claims 1–11 and 16–17 of the ’966 Patent were cancelled, and claim 19 added in inter
5 83. Dependent claim 12 and the surviving independent claims of the ’966 Patent are
6 shown below (with claim element enumeration added for convenience).
7 Dependent Claim 12 (inheriting all limitations from Claim 1)

12-a. A computer network for providing an information delivery service for a plurality of
8 participants,
12-b. each participant having connections to at least three neighbor participants,
9 12-c. wherein an originating participant sends data to the other participants by sending the
10 12-d. wherein each participant sends data that it receives from a neighbor participant to its
other neighbor participants,
11 12-e. further wherein the network is m-regular, where m is the exact number of neighbor
13
12-g. The computer network of claim 1 wherein the interconnections of participants form a
14 broadcast channel for a topic of interest.
Claim 13
15 13-a. An information delivery service comprising:
13-b. a plurality of broadcast channels, each broadcast channel for distributing information
16 relating to a topic,
13-c. each of the broadcast channels for providing said information related to a topic to a
17 plurality of participants,
13-d. each participant having connections to at least three neighbor participants,
18 13-e. wherein an originating participant sends data to the other participants by sending the
19 13-f. wherein each participant sends data that it receives from a neighbor participant to its
neighbor participants,
20 13-g. further wherein the network is m-regular, where m is the exact number of neighbor
21 13-h. further wherein the number of participants is at least two greater than m thus resulting
in a non-complete graph;
22 13-i. means for identifying a broadcast channel for a topic of interest; and
13-j. means for connecting to the identified broadcast channel.
23 Substitute Claim 19
19-a. A computer network for providing an information delivery service for a plurality of
24 participants, each participant having connections to at least three neighbor participants,
19-b. wherein an originating participant sends data to the other participants by sending the
25 data through each of its connections to its neighbor participants and
19-c. wherein each participant sends data that it receives from a neighbor participant to its
26 other neighbor participants,
19-d. further wherein the network is m-regular, where m is the exact number of neighbor
27 participants of each participant, and further wherein the number of participants is at least
two greater than m thus resulting in a non-complete graph
28 19-e. further wherein the connections are peer-to-peer connections,
11
1 19-f. further wherein the network is formed through a broadcast channel that overlays an
underlying network,
2 19-g. further wherein the information delivery service is provided by at least one
information delivery service application program executing on each computer of the
3 computer network that interacts with the broadcast channel,
19-h. and further wherein participants can join and leave the network using the broadcast
4 channel.
5 84. Plaintiff does not infringe any claim of the ’966 Patent, at least because, by way of
6 non-limiting example, Plaintiff does not make, use, or sell the “computer network” or “information
7 delivery service” as required by the claims of the ’966 Patent.
8 85. Plaintiff does not infringe any of the asserted claims, at least because, by way of non-
9 limiting example, the Accused Products do not meet the following limitations of the claims shown
10 above: claim 12, elements 12-a to 12-e, claim 13, elements 13-a to 13-g and 13-i to 13-j, and claim
11 19, elements 19-a to 19-h.
12 86. For instance, each “participant” in Fornite is not connected to at least three “neighbor”
13 participants as required by elements 12-b, 13-d and 19-a.
14 87. Fortnite does not form m-regular and incomplete/non-complete networks as required
15 by elements 12-e, 12-f, 13-g, 13-h, and 19-d.
17 12-c, 12-d, 13-e, 13-f, 19-b, and 19-c.
20 of the ’966 Patent.
28 to Plaintiff.
12
2 § 287(a).
4 literally or under the doctrine of equivalents, any enforceable claims of the ’966 Patent. An actual
5 controversy thus exists between Plaintiff and Defendant as to whether the Accused Products infringe
6 the ’966 Patent.
11 THIRD CAUSE OF ACTION
17 100. Claims 1–9 of the ’634 Patent were cancelled, and substitute claim 25 added in inter
19 101. Claims 19 and 22 have been ruled invalid for covering ineligible subject matter by the
20 District Court for the District of Delaware. Specifically, the claimed computer-readable medium in
21 claims 19 and 22 was construed to include carrier waves. See e.g., Acceleration Bay LLC v.
22 Activision Blizzard, Inc., 324 F.Supp.3d 470, 487 (D. Del. 2018). Claims 19 and 22 were also held
23 invalid as being indefinite. See id. at 479. Accordingly, claim 19 and all claims depending from
24 independent claim 19 have effectively been held invalid, subject to any appeal Acceleration may be
25 entitled to make.
26 102. The surviving independent claims of the ’634 Patent are original claim 10 and
27 substitute claim 25, both of which is shown below (with claim element enumeration added for
28 convenience).
13
1
Claim 10
2 10-a. A non-routing table based broadcast channel for participants, comprising:
10-b. a communications network that provides peer-to-peer communications between the
3 participants connected to the broadcast channel; and
10-c. for each participant connected to the broadcast channel, an indication of four neighbor
4 participants of that participant; and
10-d. a broadcast component that receives data from a neighbor participant using the
5 communications network and that sends the received data to its other neighbor participants
to effect the broadcasting of the data to each participant of the to broadcast channel,
6 10-e. wherein the network is m-regular and m-connected, where m is the number of
neighbor participants of each participant, and
8
Substitute Claim 25
9 25-a. A non-routing table based computer network having a plurality of participants, each
participant being an application program, and each participant having connections to at least
10 three neighbor participants,
25-b. wherein an originating participant sends data to the other participants by sending the
11 data through each of its connections to its neighbor participants, wherein each participant
sends data that it receives from a neighbor participant to its other neighbor participants,
12 25-c. wherein data is numbered sequentially so that data received out of order can be queued
and rearranged,
13 25-d. further wherein the network is m-regular and m-connected, where m is the number of
neighbor participants of each participant, and further wherein the number of participants is
14 at least two greater than m thus resulting in a non-complete graph,
25-e. further wherein the connections are peer-to-peer connections, further wherein the
15 network is an overlay network that overlays an underlying network,
25-f. further wherein each participant can interact with a broadcast channel with a channel
16 type and a change instance, and
25-g. further wherein the network is dynamic and participants can join and leave the
17 network using the broadcast channel.
18 103. Plaintiff does not infringe any claim of the ’634 Patent, at least because, by way of
19 non-limiting example, Plaintiff does not make, use, or sell the “computer network” or
20 “communications network” as required by the claims of the ’634 Patent.
21 104. Plaintiff does not infringe any of the claims of the ’634 Patent, at least because, by
22 way of non-limiting example, the Accused Products do not meet the following claim limitations of
23 the claim shown above: claim 10, elements 10-a to 10-e; claim 25, elements 25-a to 25-g.
24 105. For instance, each “participant” in Fornite does not have “four neighbor participants”
25 as required by element 10-c and each participant does not have “connections to at least three
26 neighbor participants” as required by element 25-a.
27 106. Fortnite does not form m-regular and incomplete/non-complete networks as required
28 by elements 10-e, 10-f, and 25-d.
14
2 10-d and 25-b.
15 to Plaintiff.
17 § 287(a).
24 FOURTH CAUSE OF ACTION
28
15
3 120. The independent claims of the ’069 Patent read as follows (with claim element
4 enumeration added for convenience):
5 Claim 1
1-a. A computer-based, non-routing table based, non-switch based method for adding a
6 participant to a network of participants, each participant being connected to three or more
other participants, the method comprising:
7 1-b. identifying a pair of participants of the network that are connected
1-c. wherein a seeking participant contacts a fully connected portal computer,
8 1-d. which in turn sends an edge connection request to a number of randomly selected
neighboring participants to which the seeking participant is to connect;
9 1-e. disconnecting the participants of the identified pair from each other; and
1-f. connecting each participant of the identified pair of participants to the seeking
10 participant.
11 Claim 14
14-a. A computer-based, non-switch based method for adding nodes to a graph that is m-
12 regular and m-connected to maintain the graph as m-regular, where m is four or greater, the
method comprising:
13 14-b. identifying p pairs of nodes of the graph that are connected, where p is one half of m,
wherein a seeking node contacts a fully connected portal node, which in turn sends an edge
14 connection request to a number of randomly selected neighboring nodes to which the
seeking node is to connect;
15 14-c. disconnecting the nodes of each identified pair from each other; and
14-d. connecting each node of the identified pairs of nodes to the seeking node.
16
17
18
the claims shown above: claim 1, elements 1-a to 1-f; claim 14, elements 14-a to 14-d.
19
122. Each participant in Fortnite is not “connected to three or more other participants” as
20
required by element 1-a and each “node” is not connected to four or greater other “nodes” as
21
required by element 14-a.
22
123. Fortnite does not identify a pair of participants which are connected as required by
23
elements 1-b and 14-b.
24
124. Plaintiff’s activities, Accused Products, and services have not and do not directly
25
infringe, do not infringe by inducement, and do not contributorily infringe any enforceable claims
26
of the ’069 Patent.
27
125. Plaintiff’s activities, Accused Products, and services have not and do not infringe,
28
willfully or otherwise, any enforceable claims of the ’069 Patent.
16
8 to Plaintiff.
10 § 287(a).
17 FIFTH CAUSE OF ACTION
23 136. Claims 11, 15–16 have been held invalid as covering ineligible subject matter by the
24 District Court for the District of Delaware.
25 137. The claimed computer-readable medium of claims 11, 15, and 16 was construed to
26 include carrier waves. See e.g., Acceleration Bay LLC v. Activision Blizzard, Inc., 324 F.Supp.3d
27 470, 487 (D. Del. 2018). Accordingly, claim 11 and all claims depending from independent claim
28 11 have been effectively held invalid, subject to any appeal Acceleration may be entitled to make.
17
1 138. The surviving independent claim of the ’147 Patent is shown below (with claim
2 element enumeration added for convenience).

Claim 1
3 1-a. A method of disconnecting a first computer from a second computer, the first computer
and the second computer being connected to a broadcast channel, said broadcast channel
4 forming an m-regular graph where m is at least 3, the method comprising:
1-b. when the first computer decides to disconnect from the second computer, the first
5 computer sends a disconnect message to the second computer, said disconnect message
including a list of neighbors of the first computer;
6 1-c. and when the second computer receives the disconnect message from the first computer,
the second computer broadcasts a connection port search message on the broadcast channel
7 to find a third computer to which it can connect in order to maintain an m-regular graph,
1-d. said third computer being one of the neighbors on said list of neighbors.
8
9
10
the claim shown above: claim 1, elements 1-a to 1-d.
11
140. Fortnite does not form m-regular networks when m is at least 3 as required by element
12
1-a, and each computer is not connected to at least three other computers, as required by this claim
13
element.
14
141. In Fortnite, when a first computer decides to disconnect from a second computer, the
15
first computer does not “send a disconnect message to the second computer, said disconnect
16
message including a list of neighbors of the first computer” as required by element 1-b.
17
142. In Fortnite, even if a server receives a disconnection message, the server would not
18
“broadcast[] a connection port search message on the broadcast channel to find a third computer to
19
which it can connect in order to maintain an m-regular graph” as required by element 1-c.
20
143. Plaintiff’s activities, Accused Products, and services have not and do not directly
21
infringe, do not infringe by inducement, and do not contributorily infringe any enforceable claims
22
of the ’147 Patent.
23
24
25
26
literally or under the doctrine of equivalents, any enforceable claims of the ’147 Patent.
27
146. In 2006, Boeing licensed the ’147 Patent to Sony.
28
18
5 to Plaintiff.
7 § 287(a).
12 Products is not infringing and have not infringed the ’147 Patent and granting Plaintiff all other
14 SIXTH CAUSE OF ACTION
20 155. Claims 1, 3–5, and 7–8 of the ’497 Patent were cancelled in inter partes review
21 proceedings before the USPTO. See Ex. 6.
22 156. The surviving claims of the ’497 Patent depend from either cancelled claim 1 and
23 claim 9, both of which are shown below (with claim element enumeration added for convenience).
24 Claim 1 (cancelled)
1-a. A method in a computer for locating a computer through which to connect to a network,
25 the method comprising:
1-b. providing an identification of a portal computer or a plurality of portal computers, the
26 portal computer or the plurality of portal computers having a communications port or
communications ports with a call-in port being enabled for communications when the portal
27 computer or the plurality of portal computers is in a state to coordinate the connection of a
seeking computer to the network,
28 1-c. wherein the call-in port is a type of communications port;
19
1 1-d. selecting the communications port or communications ports of the portal computer or
the plurality of portal computers and attempting to communicate with the selected
2 communications port or communications ports until communications with the call-in port is
successful,
3 1-e. wherein a port ordering algorithm is used to identify the call-in port, and
1-f. wherein the communications ports selected by the port ordering algorithm may be
4 reordered; and
1-g. using the call-in port to request that the portal computer or the plurality of portal
5 computers coordinate the connecting of the seeking computer to the network.
Claim 9
6 9-a. A component in a computer system for locating a call-in port of a portal computer,
comprising:
7 9-b. means for identifying the portal computer, the portal computer having a dynamically
selected call-in port for communicating with other computers;
8 9-c. means for identifying the call-in port of the identified portal computer by repeatedly
trying to establish a connection with the identified portal computer through contacting a
9 communications port or communications ports until a connection is successfully established;
9-d. means for selecting the call-in port of the identified portal computer using a port
10 ordering algorithm; and
9-e. means for re-ordering the communications ports selected by the port ordering algorithm.
11
157. Plaintiff does not infringe claim 9 of the ’497 Patent, at least because, by way of non-
12
limiting example, Plaintiff does not make, use or sell the “component in a computer system for
13
locating a call-in port of a portal computer” as required by element 9-a.
14
15
16
the claims shown above: claim 1, elements 1-b to 1-g; and claim 9, elements 9-b to 9-e.
17
159. Fortnite does not use a port ordering algorithm as required by elements 1-e and 9-d.
18
160. Fortnite does not reorder the ports selected by the port ordering algorithm as required
19
by elements 1-f and 9-e.
20
161. Plaintiff’s activities, products, and services have not and do not directly infringe, do
21
not infringe by inducement, and do not contributorily infringe any enforceable claims of the ’497
22
Patent.
23
24
25
26
literally or under the doctrine of equivalents, any enforceable claims of the ’497 Patent.
27
164. In 2006, Boeing licensed the ’497 Patent to Sony.
28
20
5 to Plaintiff.
7 § 287(a).
14 SEVENTH CAUSE OF ACTION
20 173. The sole independent claim 1 of the ’537 Patent is shown below (with claim element
21 enumeration added for convenience).
22 Claim 1
1-a. A method in a computer system for determining a diameter of a broadcast channel, the
23 broadcast channel having computers, each computer connected to at least three neighbor
computers, the method comprising:
24 1-b. receiving a message from a neighbor computer;
1-c. identifying a distance traveled from the received message;
25 1-d. setting an estimated diameter based on the identified distance traveled amount, wherein
setting the estimated diameter sets the estimated diameter to the distance traveled whenever
26 the identified distance traveled is greater than the current estimated diameter;
1-e. incrementing the distance traveled in the message; and
27 1-f. sending the message with the incremented distance traveled to a neighbor computer.
28
21
1 174. Plaintiff does not infringe any of the claims of the ’537 Patent, at least because, by
2 way of non-limiting example, the Accused Products do not meet the following claim limitations of
3 the claim shown above: claim 1, elements 1-a, 1-c to 1-f.
4 175. First, each participant in Fortnite is not “connected to at least three neighbor
5 computers,” as required by element 1-a. Second, Fortnite does not “identify[] a distance traveled
6 from the received message,” as required by element 1-c. Third, Fortnite does not “set[] an estimated
7 diameter based on the identified distance traveled amount,” as required by element 1-d. Fourth,
8 Fortnite does not “increment[] the distance traveled in the message,” as required by element 1-e.
9 And, Fortnite does not “send[] the message with the incremented distance traveled to a neighbor
10 computer,” as required by element 1-f.
18 179. As discussed above, after making multiple, direct charges of infringement as to the
19 six Asserted Patents discussed above, Acceleration identified the ’537 Patent as an additional patent
20 in the same patent portfolio Acceleration contends Epic needs a license to.
27 REQUEST FOR RELIEF
28 WHEREFORE, Plaintiff Epic respectfully prays for relief against Defendant as follows:
22
1 1. For a declaratory judgment that:
2 a. Plaintiff has not and does not infringe, willfully or otherwise, any enforceable
3 claim of the Asserted Patents; and
4 b. Defendant, and those acting in concert with it or acting with knowledge of
5 the judgment herein, are without right or authority to threaten or maintain suit against Plaintiff, or
6 users of Plaintiff’s products or services, for alleged infringement of the Asserted Patents.
7 2. Enjoining Defendant, its officers, agents, servants, employees, and all persons acting
8 in concert or participation with Defendant from initiating infringement litigation against, and from
9 threatening, Plaintiff or purchasers or users of Plaintiff’s products or services with infringement
10 litigation or charging any of them verbally or in writing with infringement of the Asserted Patents,
11 or representing to any of them that infringement has occurred, because of any activities of Plaintiff;
12 3. Determining that this is an exceptional case under 35 U.S.C. § 285;
13 4. For Plaintiff’s costs and reasonable attorneys’ fees incurred herein; and
14 5. For such other and further relief as the Court may deem appropriate.
15
16 Dated: July18, 2019 WINSTON & STRAWN LLP

17
By /s/ Michael A. Tomasulo
18 David P. Enzminger
Michael A. Tomasulo
19 Louis L. Campbell
Matthew R. McCullough
20
Attorneys for Plaintiff
21 EPIC GAMES, INC.
22
23
24
25
26
27
28
23
1 DEMAND FOR JURY TRIAL
2 Plaintiff Epic requests a trial by jury of all issues so triable.
4 Dated: July 18, 2019 WINSTON & STRAWN LLP
5
By /s/ Michael A. Tomasulo
6 David P. Enzminger
Michael A. Tomasulo
7 Louis L. Campbell
8
Attorneys for Plaintiff
9 EPIC GAMES, INC.
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
24
Case 5:19-cv-04133-SVK Document 1-2 Filed 07/18/19 Page 1 of 59
EXHIBIT 2
USOO67.14966B1
(12) United States Patent (10) Patent No.: US 6,714,966 B1

Holt et al. (45) Date of Patent: *Mar. 30, 2004
(54) INFORMATION DELIVERY SERVICE 5,734,865 A 3/1998 Yu

5,737,526 A 4/1998 Periasamy et al.
(75) Inventors: Fred B. Holt, Seattle, WA (US); Virgil 5,754,830 A 5/1998 Butts et al.
E. Bourassa, Bellevue, WA (US) 5,761,425 A 6/1998 Miller
5,764,756 A 6/1998 Onweller
(73) Assignee: The Boeing Company, Seattle, WA 5,790,548 A 8/1998 Sistanizadeh et al.
(US) 5,790,553 A 8/1998 Deaton, Jr. et al.
5,799,016 A 8/1998 Onweller
(*) Notice: Subject to any disclaimer, the term of this 5,802.285 A 9/1998 Hirviniemi
patent is extended or adjusted under 35 5,850,592 A 12/1998 Ramanathan .................. 455/7
U.S.C. 154(b) by 467 days. 5,864,711 A 1/1999 Mairs et al.
5,867,660 A 2/1999 Schmidt et al.
5,867,667 A 2/1999 Butman et al.
This patent is Subject to a terminal dis- 5,870,605 A 2/1999 Bracho et al.
claimer.
(List continued on next page.)
(21) Appl. No.: 09/629,043 OTHER PUBLICATIONS
(22) Filed: Jul. 31, 2000 -- -- -
PR Newswire, “Microsoft Boosts Accessibility to Internet
7 Gaming Zone with Latest Release, Apr. 27, 1998, pp. 1 ff.*
(51) Int. Cl." ................................................ G06F 15/16 PR NewSwire, “Microsoft Announces Launch Date for
(52) U.S. Cl. ....................... 7091204. 709/205,709,203 UltraCorps. Its Second Premium Title for the Internet Gam
709/243; 463/92 ing Zone.” Ma 27, 1998, pp. 1 ff.:*
Business Wire, “Boeing and Panthesis Complete SWAN
(58) Field of Search ................................. 709/204, 205, Transaction,” Jul. 22, 2002, pp. 1 ff.*
709/227, 243, 203; 463/40, 42 (List continued on next page.)
(56) References Cited
Primary Examiner Dung C. Dinh
U.S. PATENT DOCUMENTS ASSistant Examiner Brad Edelman
4,912,656. A 3/1990 Cain et al. (74) Attorney, Agent, or Firm-Perkins Coie LLP
5,056,085. A 10/1991 Vu
5,058,105 A * 10/1991. Mansour et al. ............ 370/228 (57) ABSTRACT
2. A : 5/1994
5. : Rhia
5,309.437 A E.
et al... 3.s AService
computer network for providing an information delivery
for a plurality of participants over the network is
5.426,637 A 6/1995 Derby et al disclosed. Each participant has connections to at least three
5.459.75 A 10/1995 Bodner et al. - - - - - - - - - - - - - - 370/390 neighbor participants. An originating participant Sends data
5,471,623 A * 11/1995 Napolitano, Jr. ............ 709/243 to the other participants by sending the data through each of
5.535,199 A 7/1996 Amri et al. its connections to its neighbor participants. Further, each
5,568,487. A 10/1996 Sitbon et al. participant Sends data that it receives from a neighbor
5,636,371 A 6/1997 Yu participant to its other neighbor participants. The network is
5,644,714 A 7/1997 Kikinis ....................... 709/219 m-regular where m is the exact number of neighbor partici
5,673,265 A 9/1997 Gupta et al. pants of each participant and the network is an incomplete
5,696,903 A 12/1997 Mahany graph
5,732,074 A 3/1998 Spaur et al.
5,732,086 A * 3/1998 Liang et al. ................ 370/410
5,732.219 A 3/1998 Blumer et al. 17 Claims, 39 Drawing Sheets
Distribute message
broadcast message from neighbor
Select next neighbor
4O2
All neighbor
Selected
N
2403
Send internal
message
US 6,714,966 B1
Page 2
U.S. PATENT DOCUMENTS U.S. patent application Ser. No. 09/629,577, Bourassa et al.,
5,874,960 A 2/1999 Mairs et al. filed Jul. 31, 2000.
5,899.980 A 5/1999 Wilf et al. U.S. patent application Ser. No. 09/629,575, Bourassa et al.,
5,907,610 A 5/1999 Onweller filed Jul. 31, 2000.
5,925,097 A * 7/1999 Gopinath et al. ........... 709/200 U.S. patent application Ser. No. 09/629,572, Bourassa et al.,
5,928,335 A 7/1999 Morita filed Jul. 31, 2000.
5,935,215 A 8/1999 Bell et al. U.S. patent application Ser. No. 09/629,023, Bourassa et al.,
5.948,054 A 9/1999 Nielsen filed Jul. 31, 2000
5,949.975 A 9/1999 Batty et al. -yu.
5,956.484 A 9/1999 Rosenberg et al. U.S. patent application Ser. No. 09/629,576, Bourassa et al.,
5,970.232 A * 10/1999 Passint et al. .............. 709,238 filed Jul. 31, 2000.
5.974.043 A 10/1999 Solomon U.S. patent application Ser. No. 09/629,024, Bourassa et al.,
5.987,506 A 11/1999 Carter et al. filed Jul. 31, 2000.
6,003,088 A 12/1999 Houston et al. U.S. patent application Ser. No. 09/629,042, Bourassa et al.,
6,013,107 A 1/2000 Blackshear et al. filed Jul. 31, 2000.
6,023,734. A 2/2000 Ratcliff et al. r. 66
6,029,171 A 2/2000 Smiga et al. Murphy, Patricia, A., “The Next Generation Networking
6,032,188 A 2/2000 Mairs et al. Paradigm: Producer/Consumer Model.” Dedicated Systems
6,038,602 A 3/2000 Ishikawa Magazine-2000 (pp. 26-28).
6,047.289 A 4/2000 Thorne et al. The Gamer's Guide, “First-Person Shooters,” Oct. 20, 1998
6,094,676 A 7/2000 Gray et al. (4 pages).
6,115,580 A * 9/2000 Chuprun et al. ............... 455/1 The O'Reilly Network, “Gnutella: Alive, Well, and Chang
6,167,432 A * 12/2000 Jiang ................... ... 709/204 ing Fast. Jan. 25, 2001 (5 es) http:// open2p.com/
6,173,314 B1 1/2001 Kurashima et al. ......... 709/204 g FaSL, Jan. Z, pag p://www.openzp.
6,199,116 B1 3/2001 May et al. lpt/... Accessed Jan. 29, 2002).
6,216,177 B1 4/2001 Mairs et al. Oram, Andy, “Gnutella and Freenet Represents True Tech
6,223,212 B1 4/2001 Batty et al. nological Innovation,” May 12, 2000 (7 pages) The O'Reily
6.243,691 B1 6/2001 Fisher et al. Network http://www.oreillynet.com/lpt... Accessed Jan. 29,
6,268.855 B1 7/2001 Mairs et al. 2002).
6,271,839 B1 8/2001 Mairs et al. Internetworking Technologies Handbook, Chapter 43 (pp.
6,272.548 B1 * 8/2001 Cotter et al................. 709/239
6,285,363 B1 9/2001 Mairs et al. 43-1-43-16).
6,304,928 B1 10/2001 Mairs et al. Oram, Andy, "Peer-to-Peer Makes the Internet Interesting
6,321,270 B1 * 11/2001 Crawley ..................... 709/238 Again,” Sep. 22, 2000 (7 pages) The O'Reilly Network
6,463,078 B1 * 10/2002 Engstrom et al. . ... 370/466 http://linux.oreillynet.com/lpt. Accessed Jan. 29, 2002).
6,524,189 B1 2/2003 Rautila ........................ 463/40 Monte, Richard, “The Random Walk for Dummies, MIT
2002/0027896 A1 * 3/2002 Hughes et al............... 370/342 Undergraduate Journal of Mathematics (pp. 143-148).
OTHER PUBLICATIONS Srinivasan, R., “XDR: External Data Representation Stan
Azar et al., “Routing Strategies for Fast Networks,” May
dard.” Sun Microsystems, Aug. 1995 (20 pages) Internet
RFC/STD/FYI/BCP Archives http://www.faqs.org/rfcs/
1992, INFOCOM 92, Eleventh Annual Joint Conference of rfc1832.html Accessed Jan. 29, 2002).
the IEEE Computer and Communications Societies, vol. 1, A Databeam Corporate White Paper, “A Primer on the T120
pp. 170–179.* Series Standards,” Copyright 1995 (pp. 1-16).
Cho et al., “A Flood Routing Method for Data Networks,”
Sep.1997, Proceedings of 1997 International Conference on Kessler, Gary, C., “An Overview of TCP/IP Protocols and
Information, Communications, and Signal Processing, Vol. the Internet,” Apr. 23, 1999 (23 pages) Hill Assoicates, Inc.,
3, pp. 1418-1422.* http://www.hill.com/library/publications/t... Accessed Jan.
Komine et al., “A Distributed Restoration Algorithm for 29, 2002).
Multiple-Link and Node Failures of Transport Networks,” Bondy, J.A., and Murty, U.S.R., “Graph Theory with Appli
Dec. 199 Global Telecommunications Conference, 1990, cations,” Chapters 1-3 (pp. 1-47), 1976 American Elsevier
and Exhibition, IEEE, vol. 1, pp. 459–463.* Publishing Co., Inc., New York, New York.
Peercy et al., “Distributed Algorithms for Shortest-Path, Cormen, Thomas H. et al., Introduction to Algorithms,
Deadlock-Free Routing and Broadcasting in Arbitrarily Chapter 5.3 (pp. 84-91), Chapter 12 (pp. 218-243), Chapter
Faulty Hypercubes,” Jun. 1999, 20' International Sympo 13 (p. 245), 1990, The MIT Press, Cambridge, Massachu
sium of Fault-Tolerant Computing, 1990, pp. 218-225.* setts, McGraw-Hill Book Company, New York.
Alagar, S. and Venkatesan, S., “Reliable Broadcast in The Common Object Request Broker: Architecture and
Mobile Wireless Networks,” Department of Computer Sci Specification, Revision 2.6, Dec. 2001, Chapter 12 (pp.
ence, University of Texas at Dallas, Military Communica 21-1-12-10), Chapter 12 (pp. 13-1-13-56), Chapter 16
tions Conference, 1995, MILCOM 95 Conference Record, (pp. 16-1-16-26), Chapter 18 (pp. 18-1-18-52), Chapter
IEEE San Diego, California, Nov. 5–8, 1995 (pp. 236-240). 20 (pp. 20-1-20-22).
International Search Report for The Boeing Company, Inter The University of Warwick, Computer Science Open Days,
national patent application No. PCT/US01/24240, Jun. 5, “Demonstration on the Problems of Distributed Systems.”
2002 (7 pages). http://www.dcs. Warwick.ac.u... Accessed Jan. 29, 2002).
U.S. patent application Ser. No. 09/629,570, Bourassa et al.,
filed Jul. 31, 2000. * cited by examiner
U.S. Patent Mar. 30, 2004 Sheet 2 of 39 US 6,714,966 B1

U.S. Patent Mar. 30, 2004 Sheet 7 of 39



U.S. Patent Mar. 30, 2004 Sheet 10 Of 39

O<
N
le
S9
O
O an

Channel Instance,
Connect Aux Info)
801
O
Open call in port
802 Fig. 8
803
Seek portal - computer
(channel type channel
instance)
804
<C- Y
805 806
Contacts
- ---- Y Achieve connection
O
807
Install external dispatcher
Connect request
Seek portal Channel Type

computer Channel Instance
902
Select next depth
903
All depths selected Y Return (failure)
Fig. 9
All portal computers
selected
N 906
Dial portal computer
907
Success
Y 908
cone poes 909

Hang up selected portal
computer
910
Check for external Selected portal
call computer connected
Y
Return (success)
Contact process
1001
Fig. 10
1 OO2
Receive external message
1 OO3
N-O Return )
1005 1004 1006
Add as connected portal Y Answering process N Add as fellow seeking
computer s connected - Computer
Connect request
Fig. II 1101 1102
Was a fully N
connected portal found
Dial call in port of portal

computer
1104
N
Y 1105
Send external message
1106
1112
Ready to connect Add neighbor
N 1113
Check for externa

call
12O1 Fig. 12
Answer
1202
ype F = seeking
connection call
Success
Add other as fellow seeker

Achieve connection
1301
Fig. 13
Connection - state = fully
connected s
Notify fellow seekers

1303
Invoke connect call back
External dispatcher Fig. 4

1401
Pick up and receive
external message
Message
Seeking connection call Handle seeking

connection call
Connection request call
Edge proposal call Handle edge proposal

call
Port connect call

Handle port
connection call
Connected statement
Condition repair Handle condition

StatChent repair statement
Handle seeking
connection call
Fig. 15
1502 1503
Set message to indicate Set message to not
connected connected
Add other as fellow

seeking process
1505
andle connection
s request call
Connecteddin
Set newcomer's
holes to expect
1601
1603
1604
C Return
Return )
SO
Set diameter estimate in

response
Fig. 16
Set ready in response
606
Sent external message
connect request resp.
16O7
Set newcomer's
holes to fill
1609
Add neighbor
161O
Newcomer's
holes to fill --
diameter >
Forward connection
edge search
C Holes to fill - = Z.
1616 1617
ewcomer's Y
Qoles to fill > 9 >- Fill hole (requestor)
NK
Add neighbor
1701
air
Identifies calling party
Sets neighbor to
messages pending
7 O2
F.l9. I 7
1703 1704
singes
<Seeking connectio
N
W Connection state -
partially connected
1705
Add as neighbor
17O6
Install interal dispatcher
for new neighbor
1707 1708
1 O
1712
Y Purge pending edges
Forward connection requestor

edge search distance remaining
Fig. 18
Distance
remaining > 0
neighbors
>1
1804
neighbor = Select random neighbor
requestor
All neighbors
selected
1806
Send internal message
1808
Note connection edge
search call
Handle edge in message

proposal call out message
Fig. 19
at end of edges a
peighbor
N 1903
create edge (pending)
proposed
neighbors
1911 pending
Send external message N
1907
1912
N 1908
Holes odd N
Y
1913 Y 1909
Fill hol
1910
addego
Handle port
connection call
Fig. 20
2003
Caller is not Send external message
neighbor (point-connect-resp
not ok)

(point-connect-resp, ok)
2006
Add neighbor
2008
Connect request
U.S. Patent Mar. 30, 2004 Sheet 27 Of 39 US 6,714,966 B1
Fill hole
O 2101
F3. 2I Initialize internal
message
102
party the request
ing part
2104 2103
Handle connection Distribute internal
ports search edit message
Return
Internal
dispatcher
22O1
Fig. 22
22O2
Assess diameter
process = -
originating
Y Partially connected
pending connection buffer 2005
Type
= r broadcast Handle broadcast
Statement message
Type 22O6 2OO7

= - shutdown Y Handle shutdown
Statement - Statenelt
N
Y Pending
connection buffer
an N
21 O
Is
message queue
empty 2212
Y Receive response ( )
Handle broadcast origin

message from neighbor
Fl9. 23 23O1 message
Process out of order
message
2302
Distribute broadcast
message
Clear out of order info

Distribute message
Fig. 24 broadcast message from neighbor
24O1
All neighbor
Selected
24O3
Send internal
meSSage
Handle connection from neighbor

for search message
26O1
Distribute internal
message Fig. 26
6O2
N
603 2604
Is requestor N Court neighbor
a neighbor
Return
enerate
condition check
message w/neighbors
26O7
to requestor
Court neighbor Prospect
Fig. 27
Is prospect
a neighbor
27O2
Dial prospect
703
Send and receive

external message
Add neighbor
27O6
Hang up prospect

edge search call message
Fig. 28 8O1
N Not
my message ll
holes >= Z.
2813
Message Y
N(fromholes
this=pi&&.
= 8O2 Forward
2803
Remaining Y connection second
Y distance > 0 edge (requestor
2814 remaining dist -l)
Fill hole (self)
A> C Requestor
equ Forward
ti
end 1nterna 2815 is neighbor or edge E.it is
message (from reserved O)
neighbor, ack N 2806 VA/
K)
2807
Send and receive
external message
808
is edge acceptable
Reserve edge of from

neighbor
Handle edge search origin

resp. from neighbor
F.l. 29 2901 message
search response
902
Edge selected N
Success
Y 2908
Fill hole (self)
eS
Broadcast Sag e
Fig. 30
3002
Generate internal
message
3003
Set message sequence
number
3004
Distribute internal
message
Acquire message message
31 O1
Fig. 31 Pop message queue
Message Return false

retrieved
Return true
Handle condition check
Fig. 32
Samc set of
neighbors
32O5
elect a neighbor
Set up message with list of sending process
of neighbors not my neighbor
3206
Send internal message to selected neighbor
32O7
Add neighbor
Return
Handle condition
repair statement
Fig. 33 3301
Holes = =
Select a neighbor not

involved in condition
3303
Remove selected
neighbor
3304
Add neighbor
Return
Handle condition
double check
Same set of
neighbors
N
N 3404
34O7
to-from neighbor
3405
US 6,714,966 B1
1 2
INFORMATION DELIVERY SERVICE to be shared at the Server, each other client would need to
poll the Server to determine that new information is being
CROSS-REFERENCE TO RELATED shared. Such polling places a very high overhead on the
APPLICATIONS communications network. Alternatively, each client may
This application is related to U.S. patent application Ser.
5 register a callback with the Server, which the Server then
invokes when new information is available to be shared.
No. 09/629,576, entitled “BROAD CASTING Such a callback technique presents a performance bottleneck
NETWORK,” filed on Jul. 31, 2000; U.S. patent application because a Single Server needs to call back to each client
Ser. No. 09/629,570, entitled “JOINING A BROADCAST whenever new information is to be shared. In addition, the
CHANNEL,” filed on Jul. 31, 2000; U.S. patent application reliability of the entire Sharing of information depends upon
Ser. No. 09/629,577, “LEAVING A BROADCAST the reliability of the Single Server. Thus, a failure at a single
CHANNEL,” filed on Jul. 31, 2000; U.S. patent application computer (i.e., the server) would prevent communications
Ser. No. 09/629,575, entitled “BROADCASTING ON A between any of the clients.
BROADCAST CHANNEL, filed on Jul. 31, 2000; U.S. The multicasting network protocols allow the Sending of
patent application Ser. No. 09/629,572, entitled “CON 15 broadcast messages to multiple recipients of a network. The
TACTING A BROADCAST CHANNEL, filed on Jul. 31, current implementations of Such multicasting network pro
2000; U.S. patent application Ser. No. 09/629,023, entitled tocols tend to place an unacceptable overhead on the under
“DISTRIBUTED AUCTION SYSTEM,” filed on Jul. 31, lying network. For example, UDP multicasting would
2000; U.S. patent application Ser. No. 09/629,024, entitled Swamp the Internet when trying to locate all possible par
“DISTRIBUTED CONFERENCING SYSTEM,” filed on ticipants. IP multicasting has other problems that include
Jul. 31, 2000; and U.S. patent application Ser. No. 09/629, needing special-purpose infrastructure (e.g., routers) to Sup
042, entitled “DISTRIBUTED GAME ENVIRONMENT,” port the Sharing of information efficiently.
filed on Jul. 31, 2000, the disclosures of which are incor
porated herein by reference. The peer-to-peer middleware communications Systems
rely on a multicasting network protocol or a graph of
25 point-to-point network protocols. Such peer-to-peer middle
TECHNICAL FIELD
ware is provided by the T120 Internet standard, which is
The described technology relates generally to a computer used in Such products as Data Connection's D.C.-share and
network and more particularly, to a broadcast channel for a Microsoft's NetMeeting. These peer-to-peer middleware
Subset of a computers of an underlying network. Systems rely upon a user to assemble a point-to-point graph
BACKGROUND
of the connections used for Sharing the information. Thus, it
is neither Suitable nor desirable to use peer-to-peer middle
There are a wide variety of computer network communi ware Systems when more than a Small number of partici
cations techniqueS Such as point-to-point network protocols, pants is desired. In addition, the underlying architecture of
client/server middleware, multicasting network protocols, 35
the T120 Internet standard is a tree structure, which relies on
and peer-to-peer middleware. Each of these communications the root node of the tree for reliability of the entire network.
techniques have their advantages and disadvantages, but That is, each message must pass through the root node in
none is particularly well Suited to the Simultaneous sharing order to be received by all participants.
of information among computers that are widely distributed. It would be desirable to have a reliable communications
For example, collaborative processing applications, Such as 40
network that is Suitable for the Simultaneous Sharing of
a network meeting programs, have a need to distribute information among a large number of the processes that are
information in a timely manner to all participants who may widely distributed.
be geographically distributed. BRIEF DESCRIPTION OF THE DRAWINGS
The point-to-point network protocols, such as UNIX
pipes, TCP/IP, and UDP, allow processes on different com 45 FIG. 1 illustrates a graph that is 4-regular and 4-connected
puters to communicate via point-to-point connections. The which represents a broadcast channel.
interconnection of all participants using point-to-point FIG. 2 illustrates a graph representing 20 computers
connections, while theoretically possible, does not Scale well connected to a broadcast channel.
as a number of participants grows. For example, each FIGS. 3A and 3B illustrate the process of connecting a
participating process would need to manage its direct con 50 new computer Z to the broadcast channel.
nections to all other participating processes. Programmers, FIG. 4A illustrates the broadcast channel of FIG. 1 with
however, find it very difficult to manage Single connections, an added computer.
and management of multiple connections is much more FIG. 4B illustrates the broadcast channel of FIG. 4A with
complex. In addition, participating processes may be limited an added computer.
to the number of direct connections that they can Support. 55 FIG. 4C also illustrates the broadcast channel of FIG. 4A
This limits the number of possible participants in the Sharing with an added computer.
of information.
The client/server middleware systems provide a server FIG. 5A illustrates the disconnecting of a computer from
that coordinates the communications between the various the broadcast channel in a planned manner.
clients who are sharing the information. The Server functions 60 FIG. 5B illustrates the disconnecting of a computer from
as a central authority for controlling access to shared the broadcast channel in an unplanned manner.
resources. Examples of client/server middleware Systems FIG. 5C illustrates the neighbors with empty ports con
include remote procedure calls (“RPC"), database servers, dition.
and the common object request broker architecture FIG. 5D illustrates two computers that are not neighbors
(“CORBA). Client/server middleware systems are not par 65 who now have empty ports.
ticularly well Suited to sharing of information among many FIG. 5E illustrates the neighbors with empty ports con
participants. In particular, when a client Stores information dition in the Small regime.
US 6,714,966 B1
3 4
FIG. 5F illustrates the situation of FIG. 5E when in the DETAILED DESCRIPTION
large regime.
FIG. 6 is a block diagram illustrating components of a A broadcast technique in which a broadcast channel
overlays a point-to-point communications network is pro
computer that is connected to a broadcast channel. Vided. The broadcasting of a message over the broadcast
FIG. 7 is a block diagram illustrating the Sub-components channel is effectively a multicast to those computers of the
of the broadcaster component in one embodiment. network that are currently connected to the broadcast chan
FIG. 8 is a flow diagram illustrating the processing of the nel. In one embodiment, the broadcast technique provides a
connect routine in one embodiment. logical broadcast channel to which host computers through
FIG. 9 is a flow diagram illustrating the processing of the their executing processes can be connected. Each computer
Seek portal computer routine in one embodiment. that is connected to the broadcast channel can broadcast
FIG. 10 is a flow diagram illustrating the processing of the messages onto and receive messages off of the broadcast
contact process routine in one embodiment. channel. Each computer that is connected to the broadcast
FIG. 11 is a flow diagram illustrating the processing of the channel receives all messages that are broadcast while it is
15 connected. The logical broadcast channel is implemented
connect request routine in one embodiment. using an underlying network System (e.g., the Internet) that
FIG. 12 is a flow diagram of the processing of the check allows each computer connected to the underlying network
for external call routine in one embodiment. System to Send messages to each other connected computer
FIG. 13 is a flow diagram of the processing of the achieve using each computer's address. Thus, the broadcast tech
connection routine in one embodiment. nique effectively provides a broadcast channel using an
FIG. 14 is a flow diagram illustrating the processing of the underlying network System that sends messages on a point
external dispatcher routine in one embodiment. to-point basis.
FIG. 15 is a flow diagram illustrating the processing of the The broadcast technique overlays the underlying network
handle Seeking connection call routine in one embodiment. System with a graph of point-to-point connections (i.e.,
FIG. 16 is a flow diagram illustrating processing of the 25 edges) between host computers (i.e., nodes) through which
handle connection request call routine in one embodiment. the broadcast channel is implemented. In one embodiment,
FIG. 17 is a flow diagram illustrating the processing of the each computer is connected to four other computers, referred
to as neighbors. (Actually, a process executing on a com
add neighbor routine in one embodiment. puter is connected to four other processes executing on this
FIG. 18 is a flow diagram illustrating the processing of the or four other computers.) To broadcast a message, the
forward connection edge Search routine in one embodiment. originating computer Sends the message to each of its
FIG. 19 is a flow diagram illustrating the processing of the neighbors using its point-to-point connections. Each com
handle edge proposal call routine. puter that receives the message then sends the message to its
FIG.20 is a flow diagram illustrating the processing of the three other neighbors using the point-to-point connections.
handle port connection call routine in one embodiment. 35 In this way, the message is propagated to each computer
FIG.21 is a flow diagram illustrating the processing of the using the underlying network to effect the broadcasting of
fill hole routine in one embodiment. the message to each computer over a logical broadcast
FIG.22 is a flow diagram illustrating the processing of the channel. A graph in which each node is connected to four
internal dispatcher routine in one embodiment. other nodes is referred to as a 4-regular graph. The use of a
40 4-regular graph means that a computer would become
FIG.23 is a flow diagram illustrating the processing of the disconnected from the broadcast channel only if all four of
handle broadcast message routine in one embodiment. the connections to its neighborS fail. The graph used by the
FIG.24 is a flow diagram illustrating the processing of the broadcast technique also has the property that it would take
distribute broadcast message routine in one embodiment. a failure of four computers to divide the graph into disjoint
FIG. 26 is a flow diagram illustrating the processing of the 45 Sub-graphs, that is two Separate broadcast channels. This
handle connection port Search Statement routine in one property is referred to as being 4-connected. Thus, the graph
embodiment. is both 4-regular and 4-connected.
FIG.27 is a flow diagram illustrating the processing of the FIG. 1 illustrates a graph that is 4-regular and 4-connected
court neighbor routine in one embodiment. which represents the broadcast channel. Each of the nine
FIG. 28 is a flow diagram illustrating the processing of the 50 nodes A-I represents a computer that is connected to the
handle connection edge Search call routine in one embodi broadcast channel, and each of the edges represents an
ment. "edge” connection between two computers of the broadcast
FIG.29 is a flow diagram illustrating the processing of the channel. The time it takes to broadcast a message to each
handle connection edge Search response routine in one computer on the broadcast channel depends on the Speed of
embodiment. 55 the connections between the computers and the number of
FIG.30 is a flow diagram illustrating the processing of the connections between the originating computer and each
broadcast routine in one embodiment. other computer on the broadcast channel. The minimum
FIG.31 is a flow diagram illustrating the processing of the number of connections that a message would need to
acquire message routine in one embodiment. traverse between each pair of computerS is the “distance'
FIG. 32 is a flow diagram illustrating processing of the
60 between the computers (i.e., the shortest path between the
two nodes of the graph). For example, the distance between
handle condition check message in one embodiment. computers A and F is one because computer A is directly
FIG. 33 is a flow diagram illustrating processing of the connected to computer F. The distance between computerSA
handle condition repair Statement routine in one embodi and B is two because there is no direct connection between
ment. 65 computers A and B, but computer F is directly connected to
FIG.34 is a flow diagram illustrating the processing of the computer B. Thus, a message originating at computer A
handle condition double check routine. would be sent directly to computer F, and then sent from
US 6,714,966 B1
S 6
computer F to computer B. The maximum of the distances pairs is broken, and a connection between computer Z and
between the computers is the “diameter' of broadcast chan each of computers B, C, D, and E is established as indicated
nel. The diameter of the broadcast channel represented by by FIG.3B. The process of breaking the connection between
FIG. 1 is two. That is, a message Sent by any computer two neighbors and reconnecting each of the former neigh
would traverse no more than two connections to reach every bors to another computer is referred to as "edge pinning” as
other computer. FIG. 2 illustrates a graph representing 20 the edge between two nodes may be considered to be
computers connected to a broadcast channel. The diameter Stretched and pinned to a new node.
of this broadcast channel is 4. In particular, the Shortest path Each computer connected to the broadcast channel allo
between computers 1 and 3 contains four connections (1-12, cates five communications ports for communicating with
12–15, 15–18, and 18–3). other computers. Four of the ports are referred to as “inter
The broadcast technique includes (1) the connecting of nal ports because they are the ports through which the
computers to the broadcast channel (i.e., composing the messages of the broadcast channels are sent. The connec
graph), (2) the broadcasting of messages over the broadcast tions between internal ports of neighbors are referred to as
channel (i.e., broadcasting through the graph), and (3) the “internal” connections. Thus, the internal connections of the
disconnecting of computers from the broadcast channel (i.e., 15 broadcast channel form the 4-regular and 4-connected
decomposing the graph) composing the graph. graph. The fifth port is referred to as an “external' port
Composing the Graph because it is used for Sending non-broadcast messages
To connect to the broadcast channel, the computer Seeking between two computerS. Neighbors can Send non-broadcast
the connection first locates a computer that is currently fully messages either through their internal ports of their connec
connected to the broadcast channel and then establishes a tion or through their external ports. A Seeking computer uses
connection with four of the computers that are already external ports when locating a portal computer.
connected to the broadcast channel. (This assumes that there In one embodiment, the broadcast technique establishes
are at least four computerS already connected to the broad the computer connections using the TCP/IP communications
cast channel. When there are fewer than five computers protocol, which is a point-to-point protocol, as the underly
connected, the broadcast channel cannot be a 4-regular 25 ing network. The TCP/IP protocol provides for reliable and
graph. In Such a case, the broadcast channel is considered to ordered delivery of messages between computers. The TCP/
be in a “small regime.” The broadcast technique for the IP protocol provides each computer with a “port space” that
small regime is described below in detail. When five or more is shared among all the processes that may execute on that
computers are connected, the broadcast channel is consid computer. The ports are identified by numbers from 0 to
ered to be in the "large regime.” This description assumes 65,535. The first 2056 ports are reserved for specific appli
that the broadcast channel is in the large regime, unless cations (e.g., port 80 for HTTP messages). The remainder of
Specified otherwise.) Thus, the process of connecting to the the ports are user ports that are available to any proceSS. In
broadcast channel includes locating the broadcast channel, one embodiment, a set of port numbers can be reserved for
identifying the neighbors for the connecting computer, and use by the computer connected to the broadcast channel. In
then connecting to each identified neighbor. Each computer 35 an alternative embodiment, the port numbers used are
is aware of one or more "portal computers' through which dynamically identified by each computer. Each computer
that computer may locate the broadcast channel. A Seeking dynamically identifies an available port to be used as its
computer locates the broadcast channel by contacting the call-in port. This call-in port is used to establish connections
portal computers until it finds one that is currently fully with the external port and the internal ports. Each computer
connected to the broadcast channel. The found portal com 40 that is connected to the broadcast channel can receive
puter then directs the identifying of four computers (i.e., to non-broadcast messages through its external port. A Seeking
be the Seeking computer's neighbors) to which the Seeking computer tries “dialing the port numbers of the portal
computer is to connect. Each of these four computers then computers until a portal computer "answers,” a call on its
cooperates with the Seeking computer to effect the connect call-in port. A portal computer answers when it is connected
ing of the Seeking computer to the broadcast channel. A 45 to or attempting to connect to the broadcast channel and its
computer that has started the process of locating a portal call-in port is dialed. (In this description, a telephone meta
computer, but does not yet have a neighbor, is in the phor is used to describe the connections.) When a computer
“Seeking connection State.” A computer that is connected to receives a call on its call-in port, it transferS the call to
at least one neighbor, but not yet four neighbors, is in the another port. Thus, the Seeking computer actually commu
"partially connected State.” A computer that is currently, or 50 nicates through that transfer-to port, which is the external
has been, previously connected to four neighbors is in the port. The call is transferred So that other computers can place
“fully connected state.” calls to that computer via the call-in port. The Seeking
Since the broadcast channel is a 4-regular graph, each of computer then communicates via that external port to
the identified computers is already connected to four com request the portal computer to assist in connecting the
puters. Thus, Some connections between computers need to 55 Seeking computer to the broadcast channel. The Seeking
be broken So that the Seeking computer can connect to four computer could identify the call-in port number of a portal
computers. In one embodiment, the broadcast technique computer by Successively dialing each port in port number
identifies two pairs of computers that are currently con order. AS discussed below in detail, the broadcast technique
nected to each other. Each of these pairs of computers breaks uses a hashing algorithm to Select the port number order,
the connection between them, and then each of the four 60 which may result in improved performance.
computers (two from each pair) connects to the seeking A Seeking computer could connect to the broadcast chan
computer. FIGS. 3A and 3B illustrate the process of a new nel by connecting to computers either directly connected to
computer Z connecting to the broadcast channel. FIG. 3A the found portal computer or directly connected to one of its
illustrates the broadcast channel before computer Z is con neighbors. A possible problem with Such a Scheme for
nected. The pairs of computers B and E and computers C and 65 identifying the neighbors for the Seeking computer is that the
Dare the two pairs that are identified as the neighbors for the diameter of the broadcast channel may increase when each
new computer Z. The connections between each of these Seeking computer uses the same found portal computer and
US 6,714,966 B1
7 8
establishes a connection to the broadcast channel directly When the broadcast channel is in a steady state (i.e., no
through that found portal computer. Conceptually, the graph computers connecting or disconnecting from the broadcast
becomes elongated in the direction of where the new nodes channel), out-of-order messages are not a problem because
are added. FIGS. 4A-4C illustrate that possible problem. each computer will eventually receive both messages and
FIG. 4A illustrates the broadcast channel of FIG. 1 with an can queue messages until all earlier ordered messages are
added computer. Computer J was connected to the broadcast received. If, however, the broadcast channel is not in a
channel by edge pinning edges C-D and E-H to computer Steady State, then problems can occur. In particular, a com
J. The diameter of this broadcast channel is still two. FIG.
4B illustrates the broadcast channel of FIG. 4A with an puter may connect to the broadcast channel after the Second
added computer. Computer K was connected to the broad message has already been received and forwarded on by its
cast channel by edge pinning edges E-J and B-C to com new neighbors. When a new neighbor eventually receives
puter K. The diameter of this broadcast channel is three, the first message, it sends the message to the newly con
because the Shortest path from computer G to computer Kis nected computer. Thus, the newly connected computer will
through edges G-A, A-E, and E-K. FIG. 4C also illustrates receive the first message, but will not receive the Second
the broadcast channel of FIG. 4A with an added computer. message. If the newly connected computer needs to process
Computer K was connected to the broadcast channel by edge 15 the messages in order, it would wait indefinitely for the
pinning edges D-G and E-J to computer K. The diameter of Second message.
this broadcast channel is, however, still two. Thus, the One Solution to this problem is to have each computer
Selection of neighbors impacts the diameter of the broadcast queue all the messages that it receives until it can Send them
channel. To help minimize the diameter, the broadcast in their proper order to its neighbors. This Solution, however,
technique uses a random Selection technique to identify the may tend to Slow down the propagation of messages through
four neighbors of a computer in the Seeking connection State. the computers of the broadcast channel. Another Solution
The random Selection technique tends to distribute the that may have leSS impact on the propagation Speed is to
connections to new Seeking computers throughout the com queue messages only at computers who are neighbors of the
puters of the broadcast channel which may result in Smaller newly connected computers. Each already connected neigh
overall diameters. 25 bor would forward messages as it receives them to its other
Broadcasting Through the Graph neighbors who are not newly connected, but not to the newly
AS described above, each computer that is connected to connected neighbor. The already connected neighbor would
the broadcast channel can broadcast messages onto the only forward messages from each originating computer to
broadcast channel and does receive all messages that are the newly connected computer when it can ensure that no
broadcast on the broadcast channel. The computer that gaps in the messages from that originating computer will
originates a message to be broadcast sends that message to occur. In one embodiment, the already connected neighbor
each of its four neighbors using the internal connections. may track the highest Sequence number of the messages
When a computer receives a broadcast message from a already received and forwarded on from each originating
neighbor, it sends the message to its three other neighbors. computer. The already connected computer will Send only
Each computer on the broadcast channel, except the origi 35 higher numbered messages from the originating computers
nating computer, will thus receive a copy of each broadcast to the newly connected computer. Once all lower numbered
message from each of its four neighbors. Each computer, messages have been received from all originating
however, only sends the first copy of the message that it computers, then the already connected computer can treat
receives to its neighbors and disregards Subsequently the newly connected computer as its other neighbors and
received copies. Thus, the total number of copies of a 40 Simply forward each message as it is received. In another
message that is Sent between the computerS is 3N+1, where embodiment, each computer may queue messages and only
N is the number of computers connected to the broadcast forwards to the newly connected computer those messages
channel. Each computer Sends three copies of the message, as the gaps are filled in. For example, a computer might
except for the originating computer, which Sends four copies receive messages 4 and 5 and then receive message 3. In
of the message. 45 Such a case, the already connected computer would forward
The redundancy of the message Sending helps to ensure queue messages 4 and 5. When message 3 is finally received,
the overall reliability of the broadcast channel. Since each the already connected computer will Send messages 3, 4, and
computer has four connections to the broadcast channel, if 5 to the newly connected computer. If messages 4 and 5 were
one computer fails during the broadcast of a message, its Sent to the newly connected computer before message 3,
neighbors have three other connections through which they 50 then the newly connected computer would process messages
will receive copies of the broadcast message. Also, if the 4 and 5 and disregard message 3. Because the already
internal connection between two computers is slow, each connected computer queues messages 4 and 5, the newly
computer has three other connections through which it may connected computer will be able to proceSS message 3. It is
receive a copy of each message Sooner. possible that a newly connected computer will receive a Set
Each computer that originates a message numbers its own 55 of messages from an originating computer through one
messages Sequentially. Because of the dynamic nature of the neighbor and then receive another Set of message from the
broadcast channel and because there are many possible Same originating computer through another neighbor. If the
connection paths between computers, the messages may be Second Set of messages contains a message that is ordered
received out of order. For example, the distance between an earlier than the messages of the first Set received, then the
originating computer and a certain receiving computer may 60 newly connected computer may ignore that earlier ordered
be four. After Sending the first message, the originating message if the computer already processed those later
computer and receiving computer may become neighbors ordered messages.
and thus the distance between them changes to one. The first Decomposing the Graph
message may have to travel a distance of four to reach the A connected computer disconnects from the broadcast
receiving computer. The Second message only has to travel 65 channel either in a planned or unplanned manner. When a
a distance of one. Thus, it is possible for the Second message computer disconnects in a planned manner, it sends a
to reach the receiving computer before the first message. disconnect message to each of its four neighbors. The
US 6,714,966 B1
9 10
disconnect message includes a list that identifies the four The condition check message includes a list of the neighbors
neighbors of the disconnecting computer. When a neighbor of the Sending computer. When the receiving computer
receives the disconnect message, it tries to connect to one of receives the list, it compares the list to its own list of
the computers on the list. In one embodiment, the first neighbors. If the lists are different, then this condition has
computer in the list will try to connect to the Second occurred in the large regime and repair is needed. To repair
computer in the list, and the third computer in the list will this condition, the receiving computer will Send a condition
try to connect to the fourth computer in the list. If a computer repair request to one of the neighbors of the Sending com
cannot connect (e.g., the first and Second computers are puter which is not already a neighbor of the receiving
already connected), then the computers may try connecting computer. When the computer receives the condition repair
in various other combinations. If connections cannot be request, it disconnects from one of its neighbors (other than
established, each computer broadcasts a message that it the neighbor that is involved with the condition) and con
needs to establish a connection with another computer. nects to the computer that Sent the condition repair request.
When a computer with an available internal port receives the Thus, one of the original neighbors involved in the condition
message, it can then establish a connection with the com will have had a port filled. However, two computers are still
puter that broadcast the message. FIGS. 5A-D illustrate the 15 in need of a connection, the other original neighbor and the
disconnecting of a computer from the broadcast channel. computer that is now disconnected from the computer that
FIG. 5A illustrates the disconnecting of a computer from the received the condition repair request. Those two computers
broadcast channel in a planned manner. When computer H Send out port connection requests. If those two computers
decides to disconnect, it sends its list of neighbors to each of are not neighbors, then they will connect to each other when
its neighbors (computers A, E, F and I) and then disconnects they receive the requests. If, however, the two computers are
from each of its neighbors. When computers A and I receive neighbors, then they repeat the condition repair proceSS until
the message they establish a connection between them as two non-neighbors are in need of connections.
indicated by the dashed line, and Similarly for computers E It is possible that the two original neighbors with the
and F. condition may have the same set of neighbors. When the
When a computer disconnects in an unplanned manner, 25 neighbor that receives the condition check message deter
Such as resulting from a power failure, the neighbors con mines that the Sets of neighbors are the Same, it sends a
nected to the disconnected computer recognize the discon condition double check message to one of its neighbors
nection when each attempts to Send its next message to the other than the neighbor who also has the condition. When
now disconnected computer. Each former neighbor of the the computer receives the condition double check message,
disconnected computer recognizes that it is short one con it determines whether it has the same Set of neighbors as the
nection (i.e., it has a hole or empty port). When a connected Sending computer. If So, the broadcast channel is in the Small
computer detects that one of its neighbors is now regime and the condition is not a problem. If the Set of
disconnected, it broadcasts a port connection request on the neighbors are different, then the computer that received the
broadcast channel, which indicates that it has one internal condition double check message Sends a condition check
port that needs a connection. The port connection request 35 message to the original neighbors with the condition. The
identifies the call-in port of the requesting computer. When computer that receives that condition check message directs
a connected computer that is also short a connection receives one of it neighbors to connect to one of the original
the connection request, it communicates with the requesting neighbors with the condition by Sending a condition repair
computer through its external port to establish a connection message. Thus, one of the original neighbors with the
between the two computers. FIG. 5B illustrates the discon 40 condition will have its port filled.
necting of a computer from the broadcast channel in an FIG. 5C illustrates the neighbors with empty ports con
unplanned manner. In this illustration, computer H has dition. In this illustration, computer H disconnected in an
disconnected in an unplanned manner. When each of its unplanned manner, but computers F and I responded to the
neighbors, computers A, E, F, and I, recognizes the port connection request of the other and are now connected
disconnection, each neighbor broadcasts a port connection 45 together. The other former neighbors of computer H, com
request indicating that it needs to fill an empty port. AS puters A and E, are already neighbors, which gives rise to the
shown by the dashed lines, computers F and I and computers neighbors with empty ports condition. In this example,
A and E respond to each other's requests and establish a computer E received the port connection request from
connection. computer A, recognized the possible condition, and Sent
It is possible that a planned or unplanned disconnection 50 (since they are neighbors via the internal connection) a
may result in two neighbors each having an empty internal condition check message with a list of its neighbors to
port. In Such a case, Since they are neighbors, they are computer A. When computer A received the list, it recog
already connected and cannot fill their empty ports by nized that computer E has a different set of neighbor (i.e., the
connecting to each other. Such a condition is referred to as broadcast channel is in the large regime). Computer A
the “neighbors with empty ports' condition. Each neighbor 55 Selected computer D, which is a neighbor of computer E and
broadcasts a port connection request when it detects that it Sent it a condition repair request. When computer D received
has an empty port as described above. When a neighbor the condition repair request, it disconnected from one of its
receives the port connection request from the other neighbor, neighbors (other than computer E), which is computer G in
it will recognize the condition that its neighbor also has an this example. Computer D then connected to computer A.
empty port. Such a condition may also occur when the 60 FIG. 5D illustrates two computers that are not neighbors
broadcast channel is in the Small regime. The condition can who now have empty ports. Computers E and G now have
only be corrected when in the large regime. When in the empty ports and are not currently neighbors. Therefore,
Small regime, each computer will have less than four neigh computers E and G can connect to each other.
bors. To detect this condition in the large regime, which FIGS. 5E and 5F further illustrate the neighbors with
would be a problem if not repaired, the first neighbor to 65 empty ports condition. FIG.5E illustrates the neighbors with
receive the port connection request recognizes the condition empty ports condition in the Small regime. In this example,
and sends a condition check message to the other neighbor. if computer E disconnected in an unplanned manner, then
US 6,714,966 B1
11 12
each computer broadcasts a port connection request when it to be connected to multiple broadcast channels that are
detects the disconnect. When computer A receives the port uniquely identified by channel type and channel instance.
connection request form computer B, it detects the neigh The algorithm may be “seeded with channel type and
bors with empty ports condition and Sends a condition check channel instance in order to generate a unique ordering of
message to computer B. Computer B recognizes that it has port numbers for each broadcast channel. Thus, a Seeking
the same set of neighbors (computer C and D) as computer computer will dial the ports of a portal computer in the same
A and then Sends a condition double check message to order as the portal computer used when allocating its call-in
computer C. Computer C recognizes that the broadcast port.
channel is in the Small regime because is also has the same If many computers are at the same time Seeking connec
Set of neighbors as computers A and B, computer C may then tion to a broadcast channel through a single portal computer,
broadcast a message indicating that the broadcast channel is then the ports of the portal computer may be busy when
in the Small regime. called by Seeking computers. The Seeking computers would
FIG. 5F illustrates the situation of FIG. 5E when in the typically need to keep on redialing a busy port. The process
large regime. AS discussed above, computer C receives the of locating a call-in port may be significantly slowed by Such
condition double check message from computer B. In this 15 redialing. In one embodiment, each Seeking computer may
case, computer C recognizes that the broadcast channel is in each reorder the first few port numbers generated by the
the large regime because it has a set of neighbors that is hashing algorithm. For example, each Seeking computer
different from computer B. The edges extending up from could randomly reorder the first eight port numbers gener
computer C and D indicate connections to other computers. ated by the hashing algorithm. The random ordering could
Computer C then Sends a condition check message to also be weighted where the first port number generated by
computer B. When computer B receives the condition check the hashing algorithm would have a 50% chance of being
message, it sends a condition repair message to one of the first in the reordering, the Second port number would have
neighbors of computer C. The computer that receives the a 25% chance of being first in the reordering, and So on.
condition repair message disconnects from one of its Because the Seeking computers would use different
neighbors, other than computer C, and tries to connect to 25 orderings, the likelihood of finding a busy port is reduced.
computer B and the neighbor from which it disconnected For example, if the first eight port numbers are randomly
tries to connect to computer A. Selected, then it is possible that eight Seeking computers
Port Selection could be simultaneously dialing ports in different Sequences
As described above, the TCP/IP protocol designates ports which would reduce the chances of dialing a busy port.
above number 2056 as user ports. The broadcast technique Locating a Portal Computer
uses five user port numbers on each computer: one external Each computer that can connect to the broadcast channel
port and four internal ports. Generally, user ports cannot be has a list of one or more portal computers through which it
Statically allocated to an application program because other can connect to the broadcast channel. In one embodiment,
applications programs executing on the same computer may each computer has the same Set of portal computers. A
use conflicting port numbers. As a result, in one 35 Seeking computer locates a portal computer that is connected
embodiment, the computers connected to the broadcast to the broadcast channel by Successively dialing the ports of
channel dynamically allocate their port numbers. Each com each portal computer in the order Specified by an algorithm.
puter could simply try to locate the lowest number unused A Seeking computer could Select the first portal computer
port on that computer and use that port as the call-in port. A and then dial all its ports until a call-in port of a computer
Seeking computer, however, does not know in advance the 40 that is fully connected to the broadcast channel is found. If
call-in port number of the portal computers when the port no call-in port is found, then the Seeking computer would
numbers are dynamically allocated. Thus, a Seeking com Select the next portal computer and repeat the proceSS until
puter needs to dial ports of a portal computer Starting with a portal computer with Such a call-in port is found. A
the lowest port number when locating the call-in port of a problem with Such a Seeking technique is that all user ports
portal computer. If the portal computer is connected to (or 45 of each portal computer are dialed until a portal computer
attempting to connect to) the broadcast channel, then the fully connected to the broadcast channel is found. In an
Seeking computer would eventually find the call-in port. If alternate embodiment, the Seeking computer Selects a port
the portal computer is not connected, then the Seeking number according to the algorithm and then dials each portal
computer would eventually dial every user port. In addition, computer at that port number. If no acceptable call-in port to
if each application program on a computer tried to allocate 50 the broadcast channel is found, then the Seeking computer
low-ordered port numbers, then a portal computer may end Selects the next port number and repeats the process. Since
up with a high-numbered port for its call-in port because the call-in ports are likely allocated at lower-ordered port
many of the low-ordered port numbers would be used by numbers, the Seeking computer first dials the port numbers
other application programs. Since the dialing of a port is a that are most likely to be call-in ports of the broadcast
relatively slow process, it would take the Seeking computer 55 channel. The Seeking computers may have a maximum
a long time to locate the call-in port of a portal computer. To search depth, that is the number of ports that it will dial when
minimize this time, the broadcast technique uses a port Seeking a portal computer that is fully connected. If the
ordering algorithm to identify the port number order that a Seeking computer exhausts its Search depth, then either the
portal computer should use when finding an available port broadcast channel has not yet been established or, if the
for its call-in port. In one embodiment, the broadcast tech 60 Seeking computer is also a portal computer, it can then
nique uses a hashing algorithm to identify the port order. The establish the broadcast channel with itself as the first fully
algorithm preferably distributes the ordering of the port connected computer.
numbers randomly through out the user port number Space When a Seeking computer locates a portal computer that
and only Selects each port number once. In addition, every is itself not fully connected, the two computers do not
time the algorithm is executed on any computer for a given 65 connect when they first locate each other because the
channel type and channel instance, it generates the same port broadcast channel may already be established and accessible
ordering. AS described below, it is possible for a computer through a higher-ordered port number on another portal
US 6,714,966 B1
13 14
computer. If the two seeking computers were to connect to Because of the local nature of the information maintained
each other, then two disjoint broadcast channels would be by each computer connected to the broadcast channel, the
formed. Each Seeking computer can share its experience in computers need not generally be aware of the diameter of the
trying to locate a portal computer with the other Seeking broadcast channel. In one embodiment, each message Sent
computer. In particular, if one Seeking computer has through the broadcast channel has a distance traveled field.
Searched all the portal computers to a depth of eight, then the Each computer that forwards a message increments the
one Seeking computer can share that it has Searched to a distance traveled field. Each computer also maintains an
depth of eight with another Seeking computer. If that other estimated diameter of the broadcast channel. When a com
Seeking computer has Searched to a depth of, for example, puter receives a message that has traveled a distance that
only four, it can Skip Searching through depths five through indicates that the estimated diameter is too Small, it updates
eight and that other Seeking computer can advance its its estimated diameter and broadcasts an estimated diameter
Searching to a depth of nine. message. When a computer receives an estimated diameter
In one embodiment, each computer may have a different message that indicates a diameter that is larger than its own
Set of portal computers and a different maximum Search
depth. In Such a situation, it may be possible that two disjoint estimated diameter, it updates its own estimated diameter.
broadcast channels are formed because a Seeking computer 15 This estimated diameter is used to establish the distance that
cannot locate a fully connected port computer at a higher an edge connection request message should travel.
depth. Similarly, if the Set of portal computers are disjoint, External Data Representation
then two separate broadcast channels would be formed. The computers connected to the broadcast channel may
Identifying Neighbors for a Seeking Computer internally Store their data in different formats. For example,
AS described above, the neighbors of a newly connecting one computer may use 32-bit integers, and another computer
computer are preferably Selected randomly from the Set of may use 64-bit integers. AS another example, one computer
currently connected computers. One advantage of the broad may use ASCII to represent text and another computer may
cast channel, however, is that no computer has global use Unicode. To allow communications between heteroge
knowledge of the broadcast channel. Rather, each computer neous computers, the messages Sent over the broadcast
has local knowledge of itself and its neighbors. This limited 25 channel may use the XDR (“External Data Representation”)
local knowledge has the advantage that all the connected format.
computers are peers (as far as the broadcasting is concerned) The underlying peer-to-peer communications protocol
and the failure of any one computer (actually any three may send multiple messages in a Single message Stream. The
computers when in the 4-regular and 4-connect form) will traditional technique for retrieving messages from a stream
not cause the broadcast channel to fail. This local knowledge has been to repeatedly invoke an operating System routine to
makes it difficult for a portal computer to randomly Select retrieve the next message in the Stream. The retrieval of each
four neighbors for a Seeking computer. message may require two calls to the operating System: one
To Select the four computers, a portal computer Sends an to retrieve the size of the next message and the other to
edge connection request message through one of its internal retrieve the number of bytes indicated by the retrieved size.
connections that is randomly Selected. The receiving com 35 Such calls to the operating System can, however, be very
puter again Sends the edge connection request message Slow in comparison to the invocations of local routines. To
through one of its internal connections that is randomly overcome the inefficiencies of Such repeated calls, the broad
Selected. This Sending of the message corresponds to a cast technique in one embodiment, uses XDR to identify the
random walk through the graph that represents the broadcast message boundaries in a stream of messages. The broadcast
channel. Eventually, a receiving computer will decide that 40 technique may request the operating System to provide the
the message has traveled far enough to represent a randomly next, for example, 1,024 bytes from the stream. The broad
Selected computer. That receiving computer will offer the cast technique can then repeatedly invoke the XDR routines
internal connection upon which it received the edge con to retrieve the messages and use the Success or failure of
nection request message to the Seeking computer for edge each invocation to determine whether another block of 1,024
pinning. Of course, if either of the computers at the end of 45 bytes needs to be retrieved from the operating System. The
the offered internal connection are already neighbors of the invocation of XDR routines do not involve system calls and
Seeking computer, then the Seeking computer cannot connect are thus more efficient than repeated System calls.
through that internal connection. The computer that decided M-Regular
that the message has traveled far enough will detect this In the embodiment described above, each fully connected
condition of already being a neighbor and Send the message 50 computer has four internal connections. The broadcast tech
to a randomly Selected neighbor. nique can be used with other numbers of internal connec
In one embodiment, the distance that the edge connection tions. For example, each computer could have 6, 8, or any
request message travels is established by the portal computer even number of internal connections. AS the number of
to be approximately twice the estimated diameter of the internal connections increase, the diameter of the broadcast
broadcast channel. The message includes an indication of 55 channel tends to decrease, and thus propagation time for a
the distance that it is to travel. Each receiving computer message tends to decrease. The time that it takes to connect
decrements that distance to travel before Sending the mes a Seeking computer to the broadcast channel may, however,
Sage on. The computer that receives a message with a increase as the number of internal connections increases.
distance to travel that is Zero is considered to be the When the number of internal connectors is even, then the
randomly Selected computer. If that randomly Selected com 60 broadcast channel can be maintained as m-regular and
puter cannot connect to the Seeking computer (e.g., because m-connected (in the steady State). If the number of internal
it is already connected to it), then that randomly selected connections is odd, then when the broadcast channel has an
computer forwards the edge connection request to one of its odd number of computers connected, one of the computers
neighbors with a new distance to travel. In one embodiment, will have less than that odd number of internal connections.
the forwarding computer toggles the new distance to travel 65 In Such a situation, the broadcast network is neither
between Zero and one to help prevent two computers from m-regular nor m-connected. When the next computer con
Sending the message back and forth between each other. nects to the broadcast channel, it can again become
US 6,714,966 B1
15 16
m-regular and m-connected. Thus, with an odd number of each internal connection, an acquire message component
internal connections, the broadcast channel toggles between 704 and a broadcast component 712. The application pro
being and not being m-regular and m-connected. gram may provide a connect callback component 710 and a
Components receive response component 711 that are invoked by the
FIG. 6 is a block diagram illustrating components of a broadcaster component. The application program invokes
computer that is connected to a broadcast channel. The the connect component to establish a connection to a des
above description generally assumed that there was only one ignated broadcast channel. The connect component identi
broadcast channel and that each computer had only one fies the external port and installs the external dispatcher for
connection to that broadcast channel. More generally, a handling messages that are received on the external port.
network of computers may have multiple broadcast The connect component invokes the Seek portal computer
channels, each computer may be connected to more than one component 705 to identify a portal computer that is con
nected to the broadcast channel and invokes the connect
broadcast channel, and each computer can have multiple request component 706 to ask the portal computer (if fully
connections to the same broadcast channel. The broadcast connected) to select neighbor processes for the newly con
channel is well Suited for computer processes (e.g., appli necting process. The external dispatcher receives external
cation programs) that execute collaboratively, Such as net 15 messages, identifies the type of message, and invokes the
work meeting programs. Each computer process can connect appropriate handling routine 707. The internal dispatcher
to one or more broadcast channels. The broadcast channels receives the internal messages, identifies the type of
can be identified by channel type (e.g., application program message, and invokes the appropriate handling routine 708.
name) and channel instance that represents separate broad The received broadcast messages are Stored in the broadcast
cast channels for that channel type. When a proceSS attempts message queue 709. The acquire message component is
to connect to a broadcast channel, it seeks a process cur invoked to retrieve messages from the broadcast queue. The
rently connected to that broadcast channel that is executing broadcast component is invoked by the application program
on a portal computer. The Seeking process identifies the to broadcast messages in the broadcast channel.
broadcast channel by channel type and channel instance. An Information Delivery Service
Computer 600 includes multiple application programs 25 In one embodiment, an information delivery Service
601 executing as Separate processes. Each application pro application is implemented using the broadcast channel. The
gram interfaces with a broadcaster component 602 for each information delivery Service allows participants to monitor
broadcast channel to which it is connected. The broadcaster messages as they are broadcast on the broadcast channel.
component may be implement as an object that is instanti Each participant may function as a producer of information,
ated within the proceSS Space of the application program. as a consumer of information, or both. The producers
Alternatively, the broadcaster component may execute as a broadcast messages on the broadcast channel, and consum
Separate process or thread from the application program. In erS receive the broadcast messages. For example, a Sports
broadcast channel may be used to disseminate the results of
one embodiment, the broadcaster component provides func sporting events. Certain organizations, such as the National
tions (e.g., methods of class) that can be invoked by the Football League, may be authorized to broadcast results of
application programs. The primary functions provided may 35 Sporting events on the broadcast channel. The operators of
include a connect function that an application program the broadcast channel may sell Subscriptions to the broadcast
invokes passing an indication of the broadcast channel to channel to Sports enthusiasts. The information delivery
which the application program wants to connect. The appli Service may be used to distribute a broad range of content
cation program may provide a callback routine that the including news articles, Stock prices, weather alerts, medical
broadcaster component invokes to notify the application 40 alerts, traffic reports, and So on.
program that the connection has been completed, that is the The information delivery service may provide a directory
proceSS enters the fully connected State. The broadcaster web site where consumers can locate and Subscribe to
component may also provide an acquire message function broadcast channels of interest. The directory may provide a
that the application program can invoke to retrieve the next hierarchical organization of topics of the various broadcast
message that is broadcast on the broadcast channel. 45 channels. When a user decides to Subscribe to a broadcast
Alternatively, the application program may provide a call channel, the broadcaster component and information deliv
back routine (which may be a virtual function provided by ery Service application program may be downloaded to the
the application program) that the broadcaster component user's computer if not already available on the user's
invokes to notify the application program that a broadcast computer. Also, the channel type and channel instance
asSociated with that broadcast channel and the identification
message has been received. Each broadcaster component 50
of the portal computers for that broadcast channel may be
allocates a call-in port using the hashing algorithm. When downloaded to the subscriber's computer. The information
calls are answered at the call-in port, they are transferred to delivery service may also provide a subscriber identifier that
other ports that Serve as the external and internal ports. may be used by a portal computer to authorize access to or
The computers connecting to the broadcast channel may track who has connected to the broadcast channel.
include a central processing unit, memory, input devices 55 The information delivery service web site may also allow
(e.g., keyboard and pointing device), output devices (e.g., an entity to create new broadcast channels. For example, the
display devices), and storage devices (e.g., disk drives). The NFL may want a broadcast channel dedicated to the dis
memory and Storage devices are computer-readable medium Semination of information under its control. In which case,
that may contain computer instructions that implement the the entity would interact with the web site to create the
broadcaster component. In addition, the data structures and 60 broadcast channel. The creation of the broadcast channel
message Structures may be Stored or transmitted via a signal would entail the generation of a channel type and channel
transmitted on a computer-readable media, Such as a com instance, the specification of Security level (e.g., encrypted
munications link. messages), the specification of Subscriber qualifications, and
FIG. 7 is a block diagram illustrating the Sub-components SO O.
of the broadcaster component in one embodiment. The 65 A user may Subscribe to a broadcast channel for an
broadcaster component includes a connect component 701, individual topic, which corresponds to a leaf node in the
an external dispatcher 702, an internal dispatcher 703 for hierarchy, or a user may Subscribe to a category of topics,
US 6,714,966 B1
17 18
which corresponds to a non-leaf node in the hierarchy. For Flow Diagrams
example, a user may Subscribe to a category of Sports Scores FIGS. 8-34 are flow diagrams illustrating the processing
or subscribe to the topic of NFL scores. In one embodiment, of the broadcaster component in one embodiment. FIG. 8 is
each topic would have its own broadcast channel. AS a a flow diagram illustrating the processing of the connect
result, the Subscribing to a category of topics would mean routine in one embodiment. This routine is passed a channel
Subscribing to multiple broadcast channels. Alternatively, a type (e.g., application name) and channel instance (e.g.,
category of topics may have a single broadcast channel. If a Session identifier), that identifies the broadcast channel to
user Subscribes to just one topic in the category, the infor
mation delivery Service application program executing at the which this proceSS wants to connect. The routine is also
Subscriber's computer would simply disregard messages not passed auxiliary information that includes the list of portal
related to the topic. computers and a connection callback routine. When the
Many different fee structures can be used by the infor connection is established, the connection callback routine is
mation delivery Service. A Subscriber may be charged a fixed invoked to notify the application program. When this pro
fee per month for Subscribing to a topic. Alternatively, a ceSS invokes this routine, it is in the Seeking connection
Subscriber may be charged based on time actually con State. When a portal computer is located that is connected
nected. For example, when a Subscriber's computer is 15 and this routine connects to at least one neighbor, this
connected, it might broadcast an identification message process enters the partially connected State, and when the
every hour or So. A billing computer could monitor the process eventually connects to four neighbors, it enters the
broadcast and record the connect time based on the identi fully connected state. When in the small regime, a fully
fication messages. If the billing computer does not receive connected process may have less than four neighbors. In
an identification message for a certain time period, it block 801, the routine opens the call-in port through which
assumes that the Subscriber's computer has disconnected. the proceSS is to communicate with other processes when
Also, the operator of the broadcast channel may derive establishing external and internal connections. The port is
revenue from advertisements broadcast over the broadcast Selected as the first available port using the hashing algo
channel. The fee for advertising on a broadcast channel may rithm described above. In block 802, the routine sets the
vary based on the number of subscribers connected to the 25 connect time to the current time. The connect time is used to
broadcast channel at the time the advertisement is broadcast. identify the instance of the process that is connected through
The following tables list messages Sent by the broadcaster this external port. One proceSS may connect to a broadcast
components. channel of a certain channel type and channel instance using
Message Type Description

EXTERNAL MESSAGES
seeking connection call Indicates that a seeking process would like to know whether the
receiving process is fully connected to the broadcast channel
connection request call Indicates that the sending process would like the receiving
process to initiate a connection of the sending process to the
broadcast channel
edge proposal call Indicates that the sending process is proposing an edge through
which the receiving process can connect to the broadcast
channel (i.e., edge pinning)
port connection call Indicates that the sending process is proposing a port through
channel
connected stimt Indicates that the sending process is connected to the broadcast
channel
condition repair stimt Indicates that the receiving process should disconnect from one
of its neighbors and connect to one of the processes involved in
the neighbors with empty port condition
INTERNAL MESSAGES
broadcast stimt Indicates a message that is being broadcast through the

broadcast channel for the application programs
connection port search stimt Indicates that the designated process is looking for a port
through which it can connect to the broadcast channel
connection edge search call Indicates that the requesting process is looking for an edge
connection edge search resp Indicates whether the edge between this process and the
sending neighbor has been accepted by the requesting
party
diameter estimate stimt Indicates an estimated diameter of the broadcast channel
diameter reset stimt Indicates to reset the estimated diameter to indicated
diameter
disconnect stimt Indicates that the sending neighbor is disconnecting from
the broadcast channel
condition check stimt Indicates that neighbors with empty port condition have
been detected
condition double check stimt Indicates that the neighbors with empty ports have the
same set of neighbors
shutdown stimt Indicates that the broadcast channel is being shutdown
US 6,714,966 B1
19 20
one call-in port and then disconnects, and another proceSS routine hangs up on the Selected portal computer. In decision
may then connect to that same broadcast channel using the block 910, if the answering process is fully connected to the
Same call-in port. Before the other proceSS becomes fully broadcast channel, then the routine returns a Success
connected, another process may try to communicate with it indicator, else the routine continues at block 911. In block
thinking it is the fully connected old process. In Such a case, 911, the routine invokes the check for external call routine
the connect time can be used to identify this situation. In to determine whether an external call has been made to this
block 803, the routine invokes the seek portal computer process as a portal computer and processes that call. The
routine passing the channel type and channel instance. The routine then loops to block 904 to select the next portal
Seek portal computer routine attempts to locate a portal computer.
computer through which this proceSS can connect to the FIG. 10 is a flow diagram illustrating the processing of the
broadcast channel for the passed type and instance. In contact process routine in one embodiment. This routine
decision block 804, if the seek portal computer routine is determines whether the process of the Selected portal com
Successful in locating a fully connected proceSS on that puter that answered the call-in to the Selected port is fully
portal computer, then the routine continues at block 805, else connected to the broadcast channel. In block 1001, the
the routine returns an unsuccessful indication. In decision 15 routine sends an external message (i.e., seeking
block 805, if no portal computer other than the portal connection call) to the answering process indicating that a
computer on which the proceSS is executing was located, Seeking process wants to know whether the answering
then this is the first process to fully connect to broadcast process is fully connected to the broadcast channel. In block
channel and the routine continues at block 806, else the 1002, the routine receives the external response message
routine continues at block 808. In block 806, the routine from the answering process. In decision block 1003, if the
invokes the achieve connection routine to change the State of external response message is Successfully received (i.e.,
this process to fully connected. In block 807, the routine Seeking connection resp), then the routine continues at
installs the external dispatcher for processing messages block 1004, else the routine returns. Wherever the broadcast
received through this process external port for the passed component requests to receive an external message, it Sets a
channel type and channel instance. When a message is 25 time out period. If the external message is not received
received through that external port, the external dispatcher is within that time out period, the broadcaster component
invoked. The routine then returns. In block 808, the routine checks its own call-in port to see if another process is calling
installs an external dispatcher. In block 809, the routine it. In particular, the dialed proceSS may be calling the dialing
invokes the connect request routine to initiate the process of process, which may result in a deadlock situation. The
identifying neighbors for the Seeking computer. The routine broadcaster component may repeat the receive request Sev
then returns. eral times. If the expected message is not received, then the
FIG. 9 is a flow diagram illustrating the processing of the broadcaster component handles the error as appropriate. In
Seek portal computer routine in one embodiment. This decision block 1004, if the answering process indicates in its
routine is passed the channel type and channel instance of response message that it is fully connected to the broadcast
the broadcast channel to which this proceSS wishes to 35 channel, then the routine continues at block 1005, else the
connect. This routine, for each Search depth (e.g., port routine continues at block 1006. In block 1005, the routine
number), checks the portal computers at that Search depth. If adds the Selected portal computer to a list of connected
a portal computer is located at that Search depth with a portal computers and then returns. In block 1006, the routine
process that is fully connected to the broadcast channel, then adds the answering process to a list of fellow Seeking
the routine returns an indication of Success. In blockS 40 processes and then returns.
902-911, the routine loops selecting each search depth until FIG. 11 is a flow diagram illustrating the processing of the
a process is located. In block 902, the routine selects the next connect request routine in one embodiment. This routine
Search depth using a port number ordering algorithm. In requests a process of a portal computer that was identified as
decision block 903, if all the search depths have already being fully connected to the broadcast channel to initiate the
been Selected during this execution of the loop, that is for the 45 connection of this process to the broadcast channel. In
currently Selected depth, then the routine returns a failure decision block 1101, if at least one process of a portal
indication, else the routine continues at block 904. In blocks computer was located that is fully connected to the broadcast
904-911, the routine loops selecting each portal computer channel, then the routine continues at block 1103, else the
and determining whether a process of that portal computer routine continues at block 1102. A process of the portal
is connected to (or attempting to connect to) the broadcast 50 computer may no longer be in the list if it recently discon
channel with the passed channel type and channel instance. nected from the broadcast channel. In one embodiment, a
In block 904, the routine selects the next portal computer. In Seeking computer may always Search its entire Search depth
decision block 905, if all the portal computers have already and find multiple portal computers through which it can
been selected, then the routine loops to block 902 to select connect to the broadcast channel. In block 1102, the routine
the next Search depth, else the routine continues at block 55 restarts the process of connecting to the broadcast channel
906. In block 906, the routine dials the selected portal and returns. In block 1103, the routine dials the process of
computer through the port represented by the Search depth. one of the found portal computers through the call-in port.
In decision block 907, if the dialing was successful, then the In decision block 1104, if the dialing is successful, then the
routine continues at block 908, else the routine loops to routine continues at block 1105, else the routine continues at
block 904 to select the next portal computer. The dialing will 60 block 1113. The dialing may be unsuccessful if, for example,
be successful if the dialed port is the call-in port of the the dialed process recently disconnected from the broadcast
broadcast channel of the passed channel type and channel channel. In block 1105, the routine sends an external mes
instance of a proceSS executing on that portal computer. In Sage to the dialed process requesting a connection to the
block 908, the routine invokes a contact process routine, broadcast channel (i.e., connection request call). In block
which contacts the answering process of the portal computer 65 1106, the routine receives the response message (i.e.,
through the dialed port and determines whether that proceSS connection request resp). In decision block 1107, if the
is fully connected to the broadcast channel. In block 909, the response message is Successfully received, then the routine
US 6,714,966 B1
21 22
continues at block 1108, else the routine continues at block 1403, else the routine hangs up on the external port in block
1113. In block 1108, the routine sets the expected number of 1415 and returns. In decision block 1403, if the message
holes (i.e., empty internal connections) for this process type is for a process Seeking a connection (i.e., Seeking
based on the received response. When in the large regime, connection call), then the routine invokes the handle Seek
the expected number of holes is zero. When in the small ing connection call routine in block 1404, else the routine
regime, the expected number of holes varies from one to continues at block 1405. In decision block 1405, if the
three. In block 1109, the routine sets the estimated diameter message type is for a connection request call (i.e.,
of the broadcast channel based on the received response. In connection request call), then the routine invokes the
decision block 1111, if the dialed process is ready to connect handle connection request call routine in block 1406, else
to this process as indicated by the response message, then the routine continues at block 1407. In decision block 1407,
the routine continues at block 1112, else the routine contin if the message type is edge proposal call (i.e., edge
ues at block 1113. In block 1112, the routine invokes the add proposal call), then the routine invokes the handle edge
neighbor routine to add the answering process as a neighbor proposal call routine in block 1408, else the routine contin
to this process. This adding of the answering process typi ues at block 1409. In decision block 1409, if the message
cally occurs when the broadcast channel is in the Small 15 type is port connect call (i.e., port connect call), then the
regime. When in the large regime, the random walk Search routine invokes the handle port connection call routine in
for a neighbor is performed. In block 1113, the routine hangs block 1410, else the routine continues at block 1411. In
up the external connection with the answering proceSS decision block 1411, if the message type is a connected
computer and then returns. Statement (i.e., connected Stint), the routine invokes the
FIG. 12 is a flow diagram of the processing of the check handle connected Statement in block 1112, else the routine
for external call routine in one embodiment. This routine is continues at block 1212. In decision block 1412, if the
invoked to identify whether a fellow Seeking process is message type is a condition repair Statement (i.e.,
attempting to establish a connection to the broadcast channel condition repair Stint), then the routine invokes the handle
through this process. In block 1201, the routine attempts to condition repair routine in block 1413, else the routine loops
answer a call on the call-in port. In decision block 1202, if 25 to block 1414 to process the next message. After each
the answer is Successful, then the routine continues at block handling routine is invoked, the routine loops to block 1414.
1203, else the routine returns. In block 1203, the routine In block 1414, the routine hangs up on the external port and
receives the external message from the external port. In continues at block 1401 to receive the next message.
decision block 1204, if the type of the message indicates that FIG. 15 is a flow diagram illustrating the processing of the
a Seeking process is calling (i.e., Seeking connection call), handle Seeking connection call routine in one embodiment.
then the routine continues at block 1205, else the routine This routine is invoked when a Seeking process is calling to
returns. In block 1205, the routine sends an external message identify a portal computer through which it can connect to
(i.e., Seeking connection resp) to the other Seeking pro the broadcast channel. In decision block 1501, if this process
ceSS indicating that this proceSS is also is Seeking a connec is currently fully connected to the broadcast channel iden
tion. In decision block 1206, if the sending of the external 35 tified in the message, then the routine continues at block
message is Successful, then the routine continues at block 1502, else the routine continues at block 1503. In block
1207, else the routine returns. In block 1207, the routine 1502, the routine Sets a message to indicate that this process
adds the other Seeking process to a list of fellow Seeking is fully connected to the broadcast channel and continues at
processes and then returns. This list may be used if this block 1505. In block 1503, the routine sets a message to
proceSS can find no process that is fully connected to the 40 indicate that this proceSS is not fully connected. In block
broadcast channel. In which case, this process may check to 1504, the routine adds the identification of the seeking
See if any fellow Seeking process were Successful in con process to a list of fellow Seeking processes. If this proceSS
necting to the broadcast channel. For example, a fellow is not fully connected, then it is attempting to connect to the
Seeking process may become the first process fully con broadcast channel. In block 1505, the routine sends the
nected to the broadcast channel. 45 external message response (i.e., Seeking connection resp)
FIG. 13 is a flow diagram of the processing of the achieve to the Seeking process and then returns.
connection routine in one embodiment. This routine Sets the FIG. 16 is a flow diagram illustrating processing of the
State of this process to fully connected to the broadcast handle connection request call routine in one embodiment.
channel and invokes a callback routine to notify the appli This routine is invoked when the calling process wants this
cation program that the process is now fully connected to the 50 process to initiate the connection of the process to the
requested broadcast channel. In block 1301, the routine sets broadcast channel. This routine either allows the calling
the connection State of this process to fully connected. In process to establish an internal connection with this proceSS
block 1302, the routine notifies fellow seeking processes (e.g., if in the Small regime) or starts the process of identi
that it is fully connected by Sending a connected external fying a process to which the calling process can connect. In
message to them (i.e., connected stint). In block 1303, the 55 decision block 1601, if this process is currently fully con
routine invokes the connect callback routine to notify the nected to the broadcast channel, then the routine continues
application program and then returns. at block 1603, else the routine hangs up on the external port
FIG. 14 is a flow diagram illustrating the processing of the in block 1602 and returns. In block 1603, the routine sets the
external dispatcher routine in one embodiment. This routine number of holes that the calling process should expect in the
is invoked when the external port receives a message. This 60 response message. In block 1604, the routine Sets the
routine retrieves the message, identifies the external mes estimated diameter in the response message. In block 1605,
Sage type, and invokes the appropriate routine to handle that the routine indicates whether this process is ready to connect
message. This routine loops processing each message until to the calling process. This process is ready to connect when
all the received messages have been handled. In block 1401, the number of its holes is greater than Zero and the calling
the routine answers (e.g., picks up) the external port and 65 process is not a neighbor of this process. In block 1606, the
retrieves an external message. In decision block 1402, if a routine Sends to the calling process an external message that
message was retrieved, then the routine continues at block is responsive to the connection request call (i.e.,
US 6,714,966 B1
23 24
connection request resp). In block 1607, the routine notes routine continues at block 1711. In block 1710, the routine
the number of holes that the calling process needs to fill as invokes the achieve connected routine to indicate that this
indicated in the request message. In decision block 1608, if process is fully connected. In decision block 1711, if the
this process is ready to connect to the calling process, then number of holes for this process is Zero, then the routine
the routine continues at block 1609, else the routine contin continues at block 1712, else the routine returns. In block
ues at block 1611. In block 1609, the routine invokes the add 1712, the routine deletes any pending edges and then
neighbor routine to add the calling proceSS as a neighbor. In returns. A pending edge is an edge that has been proposed to
block 1610, the routine decrements the number of holes that this process for edge pinning, which in this case is no longer
the calling proceSS needs to fill and continues at block 1611. needed.
In block 1611, the routine hangs up on the external port. In FIG. 18 is a flow diagram illustrating the processing of the
decision block 1612, if this process has no holes or the forward connection edge Search routine in one embodiment.
estimated diameter is greater than one (i.e., in the large This routine is responsible for passing along a request to
regime), then the routine continues at block 1613, else the connect a requesting process to a randomly Selected neigh
routine continues at block 1616. bor of this process through the internal port of the Selected
In blocks 1613–1615, the routine loops forwarding a 15 neighbor, that is part of the random walk. In decision block
request for an edge through which to connect to the calling 1801, if the forwarding distance remaining is greater than
process to the broadcast channel. One request is forwarded Zero, then the routine continues at block 1804, else the
for each pair of holes of the calling process that needs to be routine continues at block 1802. In decision block 1802, if
filled. In decision block 1613, if the number of holes of the the number of neighbors of this proceSS is greater than one,
calling process to be filled is greater than or equal to two, then the routine continues at block 1804, else this broadcast
then the routine continues at block 1614, else the routine channel is in the Small regime and the routine continues at
continues at block 1616. In block 1614, the routine invokes block 1803. In decision block 1803, if the requesting process
the forward connection edge Search routine. The invoked is a neighbor of this process, then the routine returns, else the
routine is passed to an indication of the calling proceSS and routine continues at block 1804. In blocks 1804–1807, the
the random walk distance. In one embodiment, the distance 25 routine loops attempting to Send a connection edge Search
is twice in the estimated diameter of the broadcast channel. call internal message (i.e., connection edge Search call)
In block 1614, the routine decrements the holes left to fill by to a randomly selected neighbor. In block 1804, the routine
two and loops to block 1613. In decision block 1616, if there randomly Selects a neighbor of this process. In decision
is still a hole to fill, then the routine continues at block 1617, block 1805, if all the neighbors of this process have already
else the routine returns. In block 1617, the routine invokes been Selected, then the routine cannot forward the message
the fill hole routine passing the identification of the calling and the routine returns, else the routine continues at block
process. The fill hole routine broadcasts a connection port 1806. In block 1806, the routine sends a connection edge
Search Statement (i.e., connection port Search Stint) for a Search call internal meSSage to the Selected neighbor. In
hole of a connected process through which the calling decision block 1807, if the sending of the message is
proceSS can connect to the broadcast channel. The routine 35 Successful, then the routine continues at block 1808, else the
then returns. routine loops to block 1804 to select the next neighbor.
FIG. 17 is a flow diagram illustrating the processing of the When the Sending of an internal message is unsuccessful,
add neighbor routine in one embodiment. This routine adds then the neighbor may have disconnected from the broadcast
the process calling on the external port as a neighbor to this channel in an unplanned manner. Whenever Such a situation
process. In block 1701, the routine identifies the calling 40 is detected by the broadcaster component, it attempts to find
process on the external port. In block 1702, the routine sets another neighbor by invoking the fill holes routine to fill a
a flag to indicate that the neighbor has not yet received the Single hole or the forward connecting edge Search routine to
broadcast messages from this process. This flag is used to fill two holes. In block 1808, the routine notes that the
ensure that there are no gaps in the messages initially Sent to recently Sent connection edge Search call has not yet been
the new neighbor. The external port becomes the internal 45 acknowledged and indicates that the edge to this neighbor is
port for this connection. In decision block 1703, if this reserved if the remaining forwarding distance is less than or
proceSS is in the Seeking connection State, then this proceSS equal to one. It is reserved because the Selected neighbor
is connecting to its first neighbor and the routine continues may offer this edge to the requesting process for edge
at block 1704, else the routine continues at block 1705. In pinning. The routine then returns.
block 1704, the routine sets the connection state of this 50 FIG. 19 is a flow diagram illustrating the processing of the
process to partially connected. In block 1705, the routine handle edge proposal call routine. This routine is invoked
adds the calling process to the list of neighbors of this when a message is received from a proposing process that
process. In block 1706, the routine installs an internal proposes to connect an edge between the proposing proceSS
dispatcher for the new neighbor. The internal dispatcher is and one of its neighbors to this proceSS for edge pinning. In
invoked when a message is received from that new neighbor 55 decision block 1901, if the number of holes of this process
through the internal port of that new neighbor. In decision minus the number of pending edges is greater than or equal
block 1707, if this process buffered up messages while not to one, then this process still has holes to be filled and the
fully connected, then the routine continues at block 1708, routine continues at block 1902, else the routine continues at
else the routine continues at block 1709. In one embodiment, block 1911. In decision block 1902, if the proposing process
a process that is partially connected may buffer the messages 60 or its neighbor is a neighbor of this process, then the routine
that it receives through an internal connection So that it can continues at block 1911, else the routine continues at block
Send these messages as it connects to new neighbors. In 1903. In block 1903, the routine indicates that the edge is
block 1708, the routine sends the buffered messages to the pending between this process and the proposing process. In
new neighbor through the internal port. In decision block decision block 1904, if a proposed neighbor is already
1709, if the number of holes of this process equals the 65 pending as a proposed neighbor, then the routine continues
expected number of holes, then this process is fully con at block 1911, else the routine continues at block 1907. In
nected and the routine continues at block 1710, else the block 1907, the routine sends an edge proposal response as
US 6,714,966 B1
25 26
an external message to the proposing process (i.e., edge block 2203 A, if the process is partially connected, then the
proposal resp) indicating that the proposed edge is routine continues at block 2203 B, else the routine continues
accepted. In decision block 1908, if the sending of the at block 2204. In block 2203 B, the routine adds the message
message was Successful, then the routine continues at block to the pending connection buffer and continues at block
1909, else the routine returns. In block 1909, the routine 2204. In decision blocks 2204-2207, the routine decodes the
adds the edge as a pending edge. In block 1910, the routine message type and invokes the appropriate routine to handle
invokes the add neighbor routine to add the proposing the message. For example, in decision block 2204, if the
proceSS on the external port as a neighbor. The routine then type of the message is broadcast Statement (i.e., broadcast
returns. In block 1911, the routine Sends an external message Stmt), then the routine invokes the handle broadcast message
(i.e., edge proposal resp) indicating that this proposed routine in block 2205. After invoking the appropriate han
edge is not accepted. In decision block 1912, if the number dling routine, the routine continues at block 2208. In deci
of holes is odd, then the routine continues at block 1913, else sion block 2208, if the partially connected buffer is full, then
the routine returns. In block 1913, the routine invokes the fill the routine continues at block 2209, else the routine contin
hole routine and then returns. ues at block 2210. The broadcaster component collects all its
FIG.20 is a flow diagram illustrating the processing of the 15 internal messages in a buffer while partially connected So
handle port connection call routine in one embodiment. This that it can forward the messages as it connects to new
routine is invoked when an external message is received neighbors. If, however, that buffer becomes full, then the
then indicates that the Sending proceSS wants to connect to process assumes that it is now fully connected and that the
one hole of this process. In decision block 2001, if the expected number of connections was too high, because the
number of holes of this proceSS is greater than Zero, then the broadcast channel is now in the small regime. In block 2209,
routine continues at block 2002, else the routine continues at the routine invokes the achieve connection routine and then
block 2003. In decision block 2002, if the sending process continues in block 2210. In decision block 2210, if the
is not a neighbor, then the routine continues at block 2004, application program message queue is empty, then the
else the routine continues to block 2003. In block 2003, the routine returns, else the routine continues at block 2212. In
routine Sends a port connection response external message 25 block 2212, the routine invokes the receive response routine
(i.e., port connection resp) to the Sending process that passing the acquired message and then returns. The received
indicates that it is not okay to connect to this process. The response routine is a callback routine of the application
routine then returns. In block 2004, the routine sends a port program.
connection response external message to the Sending proceSS FIG. 23 is a flow diagram illustrating the processing of the
that indicates that is okay to connect this process. In decision handle broadcast message routine in one embodiment. This
block 2005, if the sending of the message was successful, routine is passed an indication of the originating process, an
then the routine continues at block 2006, else the routine indication of the neighbor who sent the broadcast message,
continues at block 2007. In block 2006, the routine invokes and the broadcast message itself. In block 2301, the routine
the add neighbor routine to add the Sending process as a performs the out of order processing for this message. The
neighbor of this process and then returns. In block 2007, the 35 broadcaster component queues messages from each origi
routine hangs up the external connection. In block 2008, the nating process until it can Send them in Sequence number
routine invokes the connect request routine to request that a order to the application program. In block 2302, the routine
proceSS connect to one of the holes of this process. The invokes the distribute broadcast message routine to forward
routine then returns. the message to the neighbors of this process. In decision
FIG.21 is a flow diagram illustrating the processing of the 40 block 2303, if a newly connected neighbor is waiting to
fill hole routine in one embodiment. This routine is passed receive messages, then the routine continues at block 2304,
an indication of the requesting process. If this proceSS is else the routine returns. In block 2304, the routine sends the
requesting to fill a hole, then this routine Sends an internal messages in the correct order if possible for each originating
message to other processes. If another proceSS is requesting process and then returns.
to fill a hole, then this routine invokes the routine to handle 45 FIG.24 is a flow diagram illustrating the processing of the
a connection port Search request. In block 2101, the routine distribute broadcast message routine in one embodiment.
initializes a connection port Search Statement internal mes This routine Sends the broadcast message to each of the
Sage (i.e., connection port Search Stnt). In decision neighbors of this process, except for the neighbor who sent
block 2102, if this process is the requesting process, then the the message to this process. In block 2401, the routine
routine continues at block 2103, else the routine continues at 50 Selects the next neighbor other than the neighbor who sent
block 2104. In block 2103, the routine distributes the the message. In decision block 2402, if all such neighbors
message to the neighbors of this process through the internal have already been Selected, then the routine returns. In block
ports and then returns. In block 2104, the routine invokes the 2403, the routine sends the message to the selected neighbor
handle connection port Search routine and then returns. and then loops to block 2401 to select the next neighbor.
FIG.22 is a flow diagram illustrating the processing of the 55 FIG. 26 is a flow diagram illustrating the processing of the
internal dispatcher routine in one embodiment. This routine handle connection port Search Statement routine in one
is passed an indication of the neighbor who sent the internal embodiment. This routine is passed an indication of the
message. In block 2201, the routine receives the internal neighbor that Sent the message and the message itself. In
message. This routine identifies the message type and block 2601, the routine invokes the distribute internal mes
invokes the appropriate routine to handle the message. In 60 Sage which sends the message to each of its neighbors other
block 2202, the routine assesses whether to change the than the sending neighbor. In decision block 2602, if the
estimated diameter of the broadcast channel based on the number of holes of this proceSS is greater than Zero, then the
information in the received message. In decision block 2203, routine continues at block 2603, else the routine returns. In
if this process is the originating process of the message or decision block 2603, if the requesting process is a neighbor,
the message has already been received (i.e., a duplicate), 65 then the routine continues at block 2605, else the routine
then the routine ignores the message and continues at block continues at block 2604. In block 2604, the routine invokes
2208, else the routine continues at block 2203 A. In decision the court neighbor routine and then returns. The court
US 6,714,966 B1
27 28
neighbor routine connects this process to the requesting removes the Sending neighbor as a neighbor. In block 2812,
process if possible. In block 2605, if this process has one the routine hangs up the external port and continues at block
hole, then the neighbors with empty ports condition exists 2815. In decision block 2813, if this process is the requesting
and the routine continues at block 2606, else the routine process and the number of holes of this process equals one,
returns. In block 2606, the routine generates a condition then the routine continues at block 2814, else the routine
check message (i.e., condition check) that includes a list of continues at block 2815. In block 2814, the routine invokes
this process neighbors. In block 2607, the routine sends the the fill hole routine. In block 2815, the routine sends an
message to the requesting neighbor. connection edge Search response message (i.e., connection
FIG.27 is a flow diagram illustrating the processing of the edge Search response) to the Sending neighbor indicating
court neighbor routine in one embodiment. This routine is acknowledgement and then returns. The graphs are Sensitive
passed an indication of the prospective neighbor for this to parity. That is, all possible paths starting from a node and
process. If this proceSS can connect to the prospective ending at that node will have an even length unless the graph
neighbor, then it sends a port connection call external has a cycle whose length is odd. The broadcaster component
message to the prospective neighbor and adds the prospec uses a toggle indicator to vary the random walk distance
tive neighbor as a neighbor. In decision block 2701, if the 15 between even and odd distances.
prospective neighbor is already a neighbor, then the routine FIG.29 is a flow diagram illustrating the processing of the
returns, else the routine continues at block 2702. In block handle connection edge Search response routine in one
2702, the routine dials the prospective neighbor. In decision embodiment. This routine is passed as indication of the
block 2703, if the number of holes of this process is greater requesting process, the Sending neighbor, and the message.
than Zero, then the routine continues at block 2704, else the In block 2901, the routine notes that the connection edge
routine continues at block 2706. In block 2704, the routine Search response (i.e., connection edge Search resp) has
sends a port connection call external message (i.e., port been received and if the forwarding distance is less than or
connection call) to the prospective neighbor and receives equal to one unreserves the edge between this proceSS and
its response (i.e., port connection resp). ASSuming the the sending neighbor. In decision block 2902, if the request
response is successfully received, in block 2705, the routine 25 ing process indicates that the edge is acceptable as indicated
adds the prospective neighbor as a neighbor of this proceSS in the message, then the routine continues at block 2903, else
by invoking the add neighbor routine. In block 2706, the the routine returns. In block 2903, the routine reserves the
routine hangs up with the prospect and then returns. edge between this proceSS and the Sending neighbor. In
FIG. 28 is a flow diagram illustrating the processing of the block 2904, the routine removes the sending neighbor as a
handle connection edge Search call routine in one embodi neighbor. In block 2905, the routine invokes the court
ment. This routine is passed a indication of the neighbor who neighbor routine to connect to the requesting process. In
Sent the message and the message itself This routine either decision block 2906, if the invoked routine was
forwards the meSSage to a neighbor or proposes the edge unsuccessful, then the routine continues at block 2907, else
between this process and the Sending neighbor to the the routine returns. In decision block 2907, if the number of
requesting process for edge pinning. In decision block 2801, 35 holes of this proceSS is greater than Zero, then the routine
if this proceSS is not the requesting proceSS or the number of continues at block 2908, else the routine returns. In block
holes of the requesting process is still greater than or equal 2908, the routine invokes the fill hole routine and then
to two, then the routine continues at block 2802, else the returns.
routine continues at block 2813. In decision block 2802, if FIG.30 is a flow diagram illustrating the processing of the
the forwarding distance is greater than Zero, then the random 40 broadcast routine in one embodiment. This routine is
walk is not complete and the routine continues at block invoked by the application program to broadcast a message
2803, else the routine continues at block 2804. In block on the broadcast channel. This routine is passed the message
2803, the routine invokes the forward connection edge to be broadcast. In decision block 3001, if this process has
Search routine passing the identification of the requesting at least one neighbor, then the routine continues at block
proceSS and the decremented forwarding distance. The rou 45 3002, else the routine returns since it is the only process
tine then continues at block 2815. In decision block 2804, if connected to be broadcast channel. In block 3002, the
the requesting process is a neighbor or the edge between this routine generates an internal message of the broadcast
proceSS and the Sending neighbor is reserved because it has statement type (i.e., broadcast stmt). In block 3003, the
already been offered to a process, then the routine continues routine Sets the Sequence number of the message. In block
at block 2805, else the routine continues at block 2806. In 50 3004, the routine invokes the distribute internal message
block 2805, the routine invokes the forward connection edge routine to broadcast the message on the broadcast channel.
Search routine passing an indication of the requesting party The routine returns.
and a toggle indicator that alternatively indicates to continue FIG.31 is a flow diagram illustrating the processing of the
the random walk for one or two more computers. The routine acquire message routine in one embodiment. The acquire
then continues at block 2815. In block 2806, the routine dials 55 message routine may be invoked by the application program
the requesting process via the call-in port. In block 2807, the or by a callback routine provided by the application pro
routine sends an edge proposal call external message (i.e., gram. This routine returns a message. In block 3101, the
edge proposal call) and receives the response (i.e., edge routine pops the message from the message queue of the
proposal resp). ASSuming that the response is Successfully broadcast channel. In decision block 3102, if a message was
received, the routine continues at block 2808. In decision 60 retrieved, then the routine returns an indication of Success,
block 2808, if the response indicates that the edge is else the routine returns indication of failure.
acceptable to the requesting process, then the routine con FIGS. 32-34 are flow diagrams illustrating the processing
tinues at block 2809, else the routine continues at block of messages associated with the neighbors with empty ports
2812. In block 2809, the routine reserves the edge between condition. FIG. 32 is a flow diagram illustrating processing
this process and the sending neighbor. In block 2810, the 65 of the handle condition check message in one embodiment.
routine adds the requesting process as a neighbor by invok This message is sent by a neighbor process that has one hole
ing the add neighbor routine. In block 2811, the routine and has received a request to connect to a hole of this
US 6,714,966 B1
29 30
process. In decision block 3201, if the number of holes of What is claimed is:
this process is equal to one, then the routine continues at 1. A computer network for providing an information
block 3202, else the neighbors with empty ports condition delivery Service for a plurality of participants, each partici
does not exist any more and the routine returns. In decision pant having connections to at least three neighbor
block 3202, if the sending neighbor and this process have the participants, wherein an originating participant Sends data to
same set of neighbors, the routine continues at block 3203, the other participants by Sending the data through each of its
else the routine continues at block 3205. In block 3203, the connections to its neighbor participants and wherein each
routine initializes a condition double check message (i.e., participant Sends data that it receives from a neighbor
condition double check) with the list of neighbors of this participant to its other neighbor participants, further wherein
process. In block 3204, the routine Sends the message the network is m-regular, where m is the exact number of
internally to a neighbor other than Sending neighbor. The neighbor participants of each participant and further wherein
routine then returns. In block 3205, the routine selects a the number of participants is at least two greater than m thus
neighbor of the Sending process that is not also a neighbor resulting in a non-complete graph.
of this process. In block 3206, the routine sends a condition 2. The computer network of claim 1 wherein each par
repair message (i.e., condition repair stmt) externally to 15
ticipant is connected to 4 other participants.
the selected process. In block 3207, the routine invokes the 3. The computer network of claim 1 wherein each par
add neighbor routine to add the Selected neighbor as a ticipant is connected to an even number of other participants.
neighbor of this process and then returns. 4. The computer network of claim 1 wherein the network
FIG. 33 is a flow diagram illustrating processing of the is m-connected, where m is the number of neighbor partici
handle condition repair Statement routine in one embodi pants of each participant.
ment. This routine removes an existing neighbor and con 5. The computer network of claim 1 wherein the network
nects to the process that Sent the message. In decision block is m-regular and m-connected, where m is the number of
3301, if this process has no holes, then the routine continues neighbor participants of each participant.
at block 3302, else the routine continues at block 3304. In 6. The computer network of claim 1 wherein all the
block 3302, the routine selects a neighbor that is not participants are peers.
involved in the neighbors with empty ports condition. In 25 7. The computer network of claim 1 wherein the connec
block 3303, the routine removes the selected neighbor as a tions are peer-to-peer connections.
neighbor of this process. Thus, this process that is executing 8. The computer network of claim 1 wherein the connec
the routine now has at least one hole. In block 3304, the tions are TCP/IP connections.
routine invokes the add neighbor routine to add the proceSS 9. The computer network of claim 1 wherein each par
that Sent the message as a neighbor of this process. The ticipant is a proceSS eXecuting on a computer.
routine then returns. 10. The computer network of claim 1 wherein a computer
FIG.34 is a flow diagram illustrating the processing of the hosts more than one participant.
handle condition double check routine. This routine deter 11. The computer network of claim 1 wherein each
mines whether the neighbors with empty ports condition participant Sends to each of its neighbors only one copy of
really is a problem or whether the broadcast channel is in the 35 the data.
small regime. In decision block 3401, if this process has one 12. The computer network of claim 1 wherein the inter
hole, then the routine continues at block 3402, else the connections of participants form a broadcast channel for a
routine continues at block 3403. If this process does not have topic of interest.
one hole, then the Set of neighbors of this process is not the 13. An information delivery Service comprising:
Same as the Set of neighbors of the Sending process. In 40 a plurality of broadcast channels, each broadcast channel
decision block 3402, if this proceSS and the Sending proceSS for distributing information relating to a topic, each of
have the same Set of neighbors, then the broadcast channel the broadcast channels for providing Said information
is not in the Small regime and the routine continues at block related to a topic to a plurality of participants, each
3403, else the routine continues at block 3406. In decision participant having connections to at least three neigh
block 3403, if this process has no holes, then the routine 45 bor participants, wherein an originating participant
returns, else the routine continues at block 3404. In block Sends data to the other participants by Sending the data
3404, the routine sets the estimated diameter for this process through each of its connections to its neighbor partici
to one. In block 3405, the routine broadcasts a diameter reset pants and wherein each participant Sends data that it
internal message (i.e., diameter reset) indicating that the receives from a neighbor participant to its neighbor
estimated diameter is one and then returns. In block 3406, 50 participants, further wherein the network is m-regular,
the routine creates a list of neighbors of this process. In where m is the exact number of neighbor participants of
block 3407, the routine sends the condition check message each participant and further wherein the number of
(i.e., condition check Stmt) with the list of neighbors to participants is at least two greater than m thus resulting
the neighbor who sent the condition double check message in a non-complete graph;
and then returns. 55 means for identifying a broadcast channel for a topic of
From the above description, it will be appreciated that interest; and
although specific embodiments of the technology have been means for connecting to the identified broadcast channel.
described, various modifications may be made without devi 14. The information delivery service of claim 13 wherein
ating from the Spirit and Scope of the invention. For means for identifying a topic of interest includes accessing
example, the communications on the broadcast channel may 60 a web server that maps topics to corresponding broadcast
be encrypted. Also, the channel instance or Session identifier channel.
may be a very large number (e.g., 128 bits) to help prevent 15. The information deliver service of claim 13 wherein
an unauthorized user to maliciously tap into a broadcast a broadcast channel is formed by Subscriber computers that
channel. The portal computer may also enforce Security and are each interconnected to at least three other Subscriber
not allow an unauthorized user to connect to the broadcast 65 computers.
channel. Accordingly, the invention is not limited except by 16. A computer network for providing an information
the claims. delivery Service for a plurality of participants, each partici
US 6,714,966 B1
31 32
pant having connections to exactly four neighbor network is in a stable 4-regular State and wherein there are
participants, wherein an originating participant Sends data to at least Six participants to result in a non-complete graph.
the other participants by Sending the data through each of its 17. The computer network of claim 16 wherein a com
connections to its neighbor participants and wherein each puter hosts more than one participant.
participant Sends data that it receives from a neighbor 5
participant to its neighbor participants, further wherein the k . . . .
UNITED STATES PATENT AND TRADEMARK OFFICE
CERTIFICATE OF CORRECTION
PATENT NO. : 6,714,966 B1 Page 1 of 1

DATED : March 30, 2004
INVENTOR(S) : Fred B. Holt et al.
It is certified that error appears in the above-identified patent and that said Letters Patent is
hereby corrected as shown below:
Column 9
Line 8, "(e.g.” Should be -- (e.g., --,
Column 21
Line 55, "Stint Should be -- Stimt --
Column 22
Lines 19 and 23, “Stint Should be -- Stmt --
Column 23
Line 33, “Stint Should be -- Stimt --
Column 25,
Line 67, delete space between “2203' and “A”
Column 26
Line 1, delete space between “2203' and “A';
Lines 2 and 3, delete space between “2203' and “B”;
Column 27
Line 32, insert period between “itself and “This”;
Column 29
Line 62, "(e.g.” Should be -- (e.g., --,
Signed and Sealed this

Twenty-ninth Day of June, 2004
WDJ
JON W. DUDAS
Acting Director of the United States Patent and Trademark Office
EXHIBIT 3
USOO6829634B1

Holt et al. (45) Date of Patent: Dec. 7, 2004
(54) BROADCASTING NETWORK 5,864,711 A 1/1999 Mairs et al.

5,867,660 A 2/1999 Schmidt et al.
(75) Inventors: Fred B. Holt, Seattle, WA (US); Virgil 5,867,667 A 2/1999 Butman et al.
E. Bourassa, Bellevue, WA (US) 5,870,605 A 2/1999 Bracho et al.
5,874,960 A 2/1999 Mairs et al.
rr. A 5,899.980 A 5/1999 Wilf et al.
(73) ASSignee: T Boeing Company, Seattle, WA 5,907,610 A 5/1999 Onweller
(US) 5,928,335 A 7/1999 Morita
(*) Notice: patent
Subjectistoextended
any disclaimer, the term of this
or adjusted under 35
3. A
5,949.975 A
to it al.
9/1999 Batty et al.
U.S.C. 154(b) by 737 days. 5,953,318 A * 9/1999 Nattkemper et al. ........ 370/236
5,956.484 A 9/1999 Rosenberg et al.
(21) Appl. No.: 09/629,576 5.974.043 A 10/1999 Solomon
9 5.987,506 A 11/1999 Carter et al.
(22) Filed: Jul. 31, 2000
(51) Int. Cl." ................................................ G06F 15/16
(52) U.S. Cl. ....................... 709/204; 709/205; 709/203; OTHER PUBLICATIONS
709/243; 709/201; 709/238; 709/319; 709/225; Alagar, S. and Venkatesan, S., “Reliable Broadcast in
370/236 Mobile Wireless Networks,” Department of Computer Sci
(58) Field of Search ................................. 709/106, 201, CICC, University of Texas at Dallas, Military Communica
709/238, 319 tions Conference, 1995, MILCOM 95 Conference Record,
(56) References Cited
IEEE San Diego, California, Nov. 5–8, 1995 (pp. 236-240).
U.S. PATENT DOCUMENTS
4.912,656 A 3/1990 Cain et al. Primary Examiner-Hosain Alam
5,056,085 A 10/1991 Vu Assistant Examiner Young N. Won
5,309.437 A 5/1994 Perlman et al. (74) Attorney, Agent, or Firm-Perkins Coie LLP
5,426,637 A 6/1995 Derby et al. (57) ABSTRACT
5.535,199 A 7/1996 Amri et al.
5,568.487 A 10/1996 Sitbon et al. A technique for broadcasting data acroSS a network is
5,636,371 A 6/1997 Yu provided. An originating participant Sends data to another
5,673,265 A 9/1997 Gupta et al. participant, which in turn Sends the data that it receives from
5,696,903 A 12/1997 Mahany a neighbor participant to its other neighbor participants.
5,732,074 A 3/1998 Spaur et al.
5,732.219 A 3/1998 Blumer et al. Communication in the broadcast network is controlled by a
5,734,865 A 3/1998 Yu contact module that locates the neighbor participants to
5,737,526 A 4/1998 Periasamy et al. which the Seeking participant can be connected and by a join
5,754,830 A 5/1998 Butts et al. module that establishes the connection between the neighbor
5,761,425 A 6/1998 Miller participants and the Seeking participant. Data is numbered
5,764,756 A 6/1998 Onweller Sequentially So that data that is received out of order can be
5,790,548 A 8/1998 Sistanizadeh et al. queued and rearranged.
5,790,553 A 8/1998 Deaton, Jr. et al.
5,799,016 A 8/1998 Onweller
5,802.285 A 9/1998 Hirviniemi 24 Claims, 39 Drawing Sheets
US 6,829,634 B1
Page 2
U.S. PATENT DOCUMENTS The O'Reilly Network, “Gnutella: Alive, Well, and Chang
6,003,088 A 12/1999 Houston et al.
ing Fast,” Jan. 25, 2001 (5 pages) http://www.open2p.com/
6,013,107 A 1/2000 Blackshear et al. 1.pt/ . . . Accessed Jan. 29, 2002).
6,023,734. A 2/2000 Ratcliff et al. Oram, Andy, “Gnutella and Freenet Represents True Tech
6,029,171 A 2/2000 Smiga et al. nological Innovation,” May 12, 2000 (7 pages) The O'Reilly
6,032,188 A 2/2000 Mairs et al. Network http://www.oreillynet.com/1pt . . . Accessed Jan.
6,038,602 A 3/2000 Ishikawa 29, 2002).
6,047.289 A 4/2000 Thorne et al. Internetworking Technologies Handbook, Chapter 43 (pp.
6,094,676 A 7/2000 Gray et al.
6,199,116 B1 3/2001 May et al. 43-1-43-16).
6,216,177 B1 4/2001 Mairs et al. Oram, Andy, “Peer-to-Peer Makes the Internet Interesting
6,223,212 B1 4/2001 Batty et al. Again,” Sep. 22, 2000 (7 pages) The O'Reilly Network
6,243,691 B1 6/2001 Fisher et al. http://linux.oreillynet.com/1pt. . . Accessed Jan. 29, 2002).
6,268.855 B1 7/2001 Mairs et al.
6,271,839 B1 8/2001 Mairs et al. Monte, Richard, “The Random Walk for Dummies,'MIT
6,285,363 B1 9/2001 Mairs et al. Undergraduate Journal of Mathematics (pp. 143-148).
6,304,928 B1 10/2001 Mairs et al. Srinivasan, R., “XDR: External Data Representation Stan
6,611,872 B1 8/2003 McCanne ................... 709/238 dard.” Sun Microsystems, Aug. 1995 (20 pages) Internet
OTHER PUBLICATIONS RFC/STD/FYI/BCP Archives http://www.faqs.org/rfcs/
rfc1832.html Accessed Jan. 29, 2002).
International Search Report for The Boeing Company, Inter A Databeam Corporate White Paper, “A Primer on the T120
national Patent Application No. PCT/US01/24240, Jun. 5, Series Standards,” Copyright 1995 (pp. 1-16).
2002 (7 pages). Kessler, Gary, C., “An Overview of TCP/IP Protocols and
.S. patent application Ser. No. 09/629,570, Bourassa et al., the Internet,” Apr. 23, 1999 (23 pages) Hill Associates, Inc.
led Jul. 31, 2000. http://www.hill.com/library/publications/t . . . Accessed
.S. patent application Ser. No. 09/629,577, Bourassa et al., Jan. 29, 2002).
led Jul. 31, 2000. Bondy, J.A., and Murty, U.S.R., “Graph Theory with Appli
.S. patent application Ser. No. 09/629,575, Bourassa et al., cations,” Chapters 1-3 (pp. 1-47), 1976 American Elsevier
led Jul. 31, 2000. Publishing Co., Inc., New York, New York.
S. patent application Ser. No. 09/629,572, Bourassa et al.,
led Jul. 31, 2000. Cormen, Thomas H. et al., Introduction to Algorithms,
.S. patent application Ser. No. 09/629,023, Bourassa et al., Chapter 5.3 (pp. 84-91), Chapter 12 (pp. 218-243), Chapter
led Jul. 31, 2000. 13 (p. 245), 1990, The MIT Press, Cambridge, Massachu
.S. patent application Ser. No. 09/629,043, Bourassa et al., setts, McGraw-Hill Book Company, New York.
led Jul. 31, 2000. The Common Object Request Broker: Architecture and
.S. patent application Ser. No. 09/629,024, Bourassa et al., Specification, Revision 2.6, Dec. 2001, Chapter 12 (pp.
led Jul. 31, 2000. 12-1-12-10), Chapter 13 (pp. 13-1-13-56), Chapter 16
.S. patent application Ser. No. 09/629,042, Bourassa et al., (pp. 16-1-16-26), Chapter 18 (pp. 18-1-18-52), Chapter
filed Jul. 31, 2000. 20 (pp. 20-1-20-22).
Murphy, Patricia, A., “The Next Generation Networking The University of Warwick, Computer Science Open Days,
Paradigm: Producer/Consumer Model.” Dedicated Systems “Demonstration on the Problems of Distributed Systems.”
Magazine-2000 (pp. 26-28). http://www.dcs. Warwick.ac.u. . . . Accessed Jan. 29, 2002).
The Gamer's Guide, “First-Person Shooters,” Oct. 20, 1998
(4 pages). * cited by examiner
U.S. Patent Dec. 7, 2004 Sheet 1 of 39

U.S. Patent Dec. 7, 2004 Sheet 2 of 39 US 6,829,634 B1

U.S. Patent Dec. 7, 2004 Sheet 3 Of 39



U.S. Patent US 6,829,634 B1



s
º:Z?|-09

U.S. Patent Dec. 7, 2004 Sheet 14 Of 39 US 6,829,634 B1
(Channel Type,
Channel Instance,
801
Connect Aux Info)
Open call in port
802 Fig. 8
Set connect-time
803
instance)
804
<> y
806
N
e BO7
808
809
Connect request
Channel Type
Channel Instance
902
Select next depth
All depths selected

selected
Check for external

(Return (success)
1001
Send external message Fig. 10
1002
Y
1005 OO4 1006
computer connected computer
Connect request
1102
Y
Set expect holes from

response
Set diameter from response

111 1112
N 1113
Check for externa

call
Fig. 12
connection call
send external message

N
O Return D
Achieve connection
1301
Fig. 13
Connection - state F fully
connected
1302
1303
Invoke connect callback
Port connect call
Handle connected
Connected statement Stat t

satee - repair statement
Handle seeking
connection call
Fig. 15
1502
connected connected
Add other as fellow

seeking process
1505
andle connection
request call
1601 SO
<GSSet newcomer's
S
Return
holes to Cxpect
response
S. Fig. 16
SOC
S
Set newcomer's
holes to fill
608 SS
Ready
Newcomer's
holes to fill
Forward connection
edge search
Holes to fill- = Z.
holes to fill > 9 - Fil) hole (requestor)
Return
Add neighbor
Fig. 17
Sets neighbor to
messages pending
93 Y Connection state =
Seeking connectio partially connected
N
pe L play
AddAdd asas neighbor
neighbor O
for new neighbor
707 O
Connecting buffer)Y
N
1709
Holes = = Y
expected hole
N
1711
Purge pending edges
1 if

neighbors
>
neighbors
requestor
808
search cal)

Fig. 19
91
1907
1912 1908
Y
1909
ill hol
Fill hole Add edge aspending
1910
Graum D as
Handle port
connection call

not ok)

2006
Add neighbor
2008
connecteus
Fill hole

Internal
220
Fig. 22 22
Assess diameter
O3
N
22O3A
Partially connected
Insert -In S N
pending connection buffer
Type
F = broadcast
2007
y
connection buffer
message queue
S-1 EI is
Fig. 23
Fig. 24

for search message
260
Distribute internal
message Fig. 26
6O2
Is requestor
a neighbor
enerate
condition check
message wineighbors
26O7
to requestor
Fig. 27
Send and receive

external message
Add neighbor
2813
N Message SO
Akfrom this pt. &&. O .
oles = F connection second
Y 2814
Fill hole (self)
A> 2815 connection edge
end intel2
message (from search custo,
O
neighbor, ack
2806
2807
Send and receive
external message
U.S. Patent Dec. 7, 2004 Sheet 34 0f 39 US 6,829,634 B1

resp. from neighbor
Fig. 29 message
Fig. 30
message
31 O1
Fig. 3i Pop message queue
andle condition check

Fig. 32
elect a neighbor
Send internal message Send external message

to selected neighbor
Add neighbor
Handle condition
repair statement
Fig. 33

Handle condition
double check

to-from neighbor
US 6,829,634 B1
1 2
BROADCASTING NETWORK ticularly well Suited to sharing of information among many
participants. In particular, when a client Stores information
CROSS-REFERENCE TO RELATED to be shared at the Server, each other client would need to
APPLICATIONS poll the Server to determine that new information is being
5 shared. Such polling places a very high overhead on the
This application is related to U.S. patent application Ser. communications network. Alternatively, each client may
No. 09/629,570, entitled “JOINING A BROADCAST register a callback with the Server, which the Server then
CHANNEL,” filed on Jul. 31, 2000 U.S. patent application invokes when new information is available to be shared.
Ser. No. 09/629,577, “LEAVING A BROADCAST Such a callback technique presents a performance bottleneck
CHANNEL,” filed on Jul. 31, 2000 currently patented. U.S. because a Single Server needs to call back to each client
patent application Ser. No. 09/629,575, entitled “BROAD whenever new information is to be shared. In addition, the
CASTING ON A BROADCAST CHANNEL, filed on Jul. reliability of the entire Sharing of information depends upon
31, 2000; U.S. patent application Ser. No. 09/629,572, the reliability of the Single Server. Thus, a failure at a single
entitled “CONTACTING A BROADCAST CHANNEL,” computer (i.e., the server) would prevent communications
filed on Jul. 31, 2000; U.S. patent application Ser. No. 15 between any of the clients.
09/629,023, entitled “DISTRIBUTED AUCTION The multicasting network protocols allow the Sending of
SYSTEM," filed on Jul. 31, 2000 now under appeal. U.S. broadcast messages to multiple recipients of a network. The
patent application Ser. No. 09/629,043, entitled “AN current implementations of Such multicasting network pro
INFORMATION DELIVERY SERVICE,” filed on Jul. 31, tocols tend to place an unacceptable overhead on the under
2000 currently patented; U.S. patent application Ser. No. lying network. For example, UDP multicasting would
09/629,024, entitled “DISTRIBUTED CONFERENCING Swamp the Internet when trying to locate all possible par
SYSTEM," filed on Jul. 31, 2000; and U.S. patent applica ticipants. IP multicasting has other problems that include
tion Ser. No. 09/629,042, entitled “DISTRIBUTED GAME needing special-purpose infrastructure (e.g., routers) to Sup
ENVIRONMENT,” filed on Jul. 31, 2000 currently port the Sharing of information efficiently.
patented, the disclosures of which are incorporated herein by 25 The peer-to-peer middleware communications Systems
reference. rely on a multicasting network protocol or a graph of
TECHNICAL FIELD
point-to-point network protocols. Such peer-to-peer middle
of the connections used for sharing the information. Thus, it
BACKGROUND is neither Suitable nor desirable to use peer-to-peer middle
ware Systems when more than a Small number of partici
There are a wide variety of computer network communi 35 pants is desired. In addition, the underlying architecture of
cations techniqueS Such as point-to-point network protocols, the T120 Internet standard is a tree structure, which relies on
client/server middleware, multicasting network protocols, the root node of the tree for reliability of the entire network.
and peer-to-peer middleware. Each of these communications That is, each message must pass through the root node in
techniques have their advantages and disadvantages, but order to be received by all participants.
none is particularly well Suited to the Simultaneous sharing 40 It would be desirable to have a reliable communications
of information among computers that are widely distributed. network that is Suitable for the Simultaneous Sharing of
For example, collaborative processing applications, Such as information among a large number of the processes that are
a network meeting programs, have a need to distribute widely distributed.
information in a timely manner to all participants who may
be geographically distributed. 45
SUMMARY OF THE INVENTION
The point-to-point network protocols, such as UNIX Embodiments of the invention deal with a non-routing
pipes, TCP/IP, and UDP, allow processes on different com table based method for broadcasting messages in a network.
puters to communicate via point-to-point connections. The More Specifically, a network in which each participant has at
interconnection of all participants using point-to-point least three neighbor participants broadcasts data through
connections, while theoretically possible, does not Scale well 50 each of its connections to neighbor participants, which in
as a number of participants grows. For example, each turn Send the data that it receives to its other neighbor
participating process would need to manage its direct con participants. The data is numbered Sequentially So that data
nections to all other participating processes. Programmers, that is received out of order can be queued and rearranged.
however, find it very difficult to manage Single connections, Communication within the broadcast channel is con
and management of multiple connections is much more 55 trolled by a contact module and by a join module. The
complex. In addition, participating processes may be limited contact module locates a portal computer and requests the
to the number of direct connections that they can Support. located portal computer to provide an indication of neighbor
This limits the number of possible participants in the Sharing participants to which the participant can be connected. The
of information. join module receives the indication of the neighbor partici
The client/server middleware systems provide a server 60 pants and establishes a connection between the Seeking
that coordinates the communications between the various participant and each of the indicated neighbor participants.
clients who are sharing the information. The Server functions Each participant in the network is connected to neighbor
as a central authority for controlling access to shared participants, and the participants and connections between
resources. Examples of client/server middleware Systems them forman m-regular graph, where m is greater than 2. In
include remote procedure calls (“RPC"), database servers, 65 addition, when a participant receives data from a neighbor
and the common object request broker architecture participant, it sends the data to its other neighbor partici
(“CORBA). Client/server middleware systems are not par pants.
US 6,829,634 B1
3 4
BRIEF DESCRIPTION OF THE DRAWINGS FIG.24 is a flow diagram illustrating the processing of the
FIG. 1 illustrates a graph that is 4-regular and 4-connected distribute
FIG. 26
broadcast message routine in one embodiment.
is a flow diagram illustrating the processing of the
which represents a broadcast channel.
handle
FIG. 2 illustrates a graph representing 20 computers 5 embodiment. connection port Search Statement routine in one
connected to a broadcast channel.
FIG. 27 is a flow diagram illustrating the processing of the
FIGS. 3A and 3B illustrate the process of connecting a court neighbor routine in one embodiment.
new computer Z to the broadcast channel. FIG. 28 is a flow diagram illustrating the processing of the
FIG. 4A illustrates the broadcast channel of FIG. 1 with
an added computer. 1O handle connection edge Search call routine in one embodi
ment.
FIG. 4B illustrates the broadcast channel of FIG. 4A with
an added computer. FIG.29 is a flow diagram illustrating the processing of the
handle connection edge Search response routine in one
FIG. 4C also illustrates the broadcast channel of FIG. 4A embodiment.
with an added computer. 15
FIG.30 is a flow diagram illustrating the processing of the
FIG. 5A illustrates the disconnecting of a computer from broadcast routine in one embodiment.
the broadcast channel in a planned manner. FIG.31 is a flow diagram illustrating the processing of the
FIG. 5B illustrates the disconnecting of a computer from acquire message routine in one embodiment.
the broadcast channel in an unplanned manner. FIG. 32 is a flow diagram illustrating processing of the
FIG. 5C illustrates the neighbors with empty ports con- 20 handle condition check message in one embodiment.
dition. FIG. 33 is a flow diagram illustrating processing of the
FIG.5D illustrates two computers that are not neighbors handle condition repair Statement routine in one embodi
who now have empty ports. ment.
FIG. 5E illustrates the neighbors with empty ports con 25 FIG. 34 is a flow diagram illustrating the processing of the
dition in the Small regime. handle condition double check routine.
FIG. 5F illustrates the situation of FIG. 5E when in the
large regime. DETAILED DESCRIPTION
computer that is connected to a broadcast channel. 3O overlays a point-to-point communications network is pro
FIG. 7 is a block diagram illustrating the Sub-components Vided. The broadcasting of a message over the broadcast
of the broadcaster component in one embodiment. channel is effectively a multicast to those computers of the
FIG. 8 is a flow diagram illustrating the processing of the network
nel. In
that are currently connected to the broadcast chan
one embodiment, the broadcast technique provides a
connect routine in one embodiment. 35
logical broadcast channel to which host computers through
contact process routine in one embodiment. channel. Each computer that is connected to the broadcast
FIG. 11 is a flow diagram illustrating the processing of the 40 channel receives all messages that are broadcast while it is
connect request routine in one embodiment. connected. The logical broadcast channel is implemented
FIG. 12 is a flow diagram of the processing of the check using an underlying network System (e.g., the Internet) that
for external call routine in one embodiment. allows each computer connected to the underlying network
System
FIG. 13 is a flow diagram of the processing of the achieve 45 using eachto Send messages to each other connected computer
connection routine in one embodiment. computer's address. Thus, the broadcast tech
nique effectively provides a broadcast channel using an
FIG. 15 is a flow diagram illustrating the processing of the The broadcast technique overlays the underlying network
handle Seeking connection call routine in one embodiment. 50 System with a graph of point-to-point connections (i.e.,
FIG. 16 is a flow diagram illustrating processing of the edges) between host computers (i.e., nodes) through which
FIG. 17 is a flow diagram illustrating the processing of the each computer is connected to four other computers, referred
add neighbor routine in one embodiment. to as neighbors. (Actually, a process executing on a com
FIG. 18 is a flow diagram illustrating the processing of the SS puter is connected to four other processes executing on this
forward connection edge Search routine in one embodiment. or four other computers.) To broadcast a message, the
FIG. 19 is a flow diagram illustrating the processing of the originating
neighbors
computer Sends the message to each of its
using its point-to-point connections. Each com
handle edge proposal call routine. puter that receives the message then sends the message to its
FIG.20 is a flow diagram illustrating the processing of the 60 three other neighbors using the point-to-point connections.
handle port connection call routine in one embodiment. In this way, the message is propagated to each computer
FIG.22 is a flow diagram illustrating the processing of the channel. A graph in which each node is connected to four
internal dispatcher routine in one embodiment. 65 other nodes is referred to as a 4-regular graph. The use of a
FIG.23 is a flow diagram illustrating the processing of the 4-regular graph means that a computer would become
handle broadcast message routine in one embodiment. disconnected from the broadcast channel only if all four of
US 6,829,634 B1
S 6
the connections to its neighborS fail. The graph used by the puter then directs the identifying of four computers (i.e., to
broadcast technique also has the property that it would take be the Seeking computer's neighbors) to which the Seeking
a failure of four computers to divide the graph into disjoint computer is to connect. Each of these four computers then
Sub-graphs, that is two Separate broadcast channels. This cooperates with the Seeking computer to effect the connect
property is referred to as being 4-connected. Thus, the graph ing of the Seeking computer to the broadcast channel. A
is both 4-regular and 4-connected. computer that has started the process of locating a portal
FIG. 1 illustrates a graph that is 4-regular and 4-connected computer, but does not yet have a neighbor, is in the
which represents the broadcast channel. Each of the nine “Seeking connection State.” A computer that is connected to
nodes A-I represents a computer that is connected to the at least one neighbor, but not yet four neighbors, is in the
broadcast channel, and each of the edges represents an "partially connected State.” A computer that is currently, or
"edge” connection between two computers of the broadcast has been, previously connected to four neighbors is in the
channel. The time it takes to broadcast a message to each “fully connected state.”
computer on the broadcast channel depends on the Speed of Since the broadcast channel is a 4-regular graph, each of
the connections between the computers and the number of the identified computerS is already connected to four com
connections between the originating computer and each 15 puters. Thus, Some connections between computers need to
other computer on the broadcast channel. The minimum be broken So that the Seeking computer can connect to four
number of connections that a message would need to computers. In one embodiment, the broadcast technique
traverse between each pair of computerS is the “distance' identifies two pairs of computers that are currently con
between the computers (i.e., the shortest path between the nected to each other. Each of these pairs of computers breaks
two nodes of the graph). For example, the distance between the connection between them, and then each of the four
computers A and F is one because computer A is directly computers (two from each pair) connects to the seeking
connected to computer F. The distance between computers A computer. FIGS. 3A and 3B illustrate the process of a new
and B is two because there is no direct connection between computer Z connecting to the broadcast channel. FIG. 3A
computers A and B, but computer F is directly connected to illustrates the broadcast channel before computer Z is con
computer B. Thus, a message originating at computer A 25 nected. The pairs of computers B and E and computers C and
would be sent directly to computer F, and then sent from Dare the two pairs that are identified as the neighbors for the
computer F to computer B. The maximum of the distances new computer Z. The connections between each of these
between the computers is the “diameter' of broadcast chan pairs is broken, and a connection between computer Z and
nel. The diameter of the broadcast channel represented by each of computers B, C, D, and E is established as indicated
FIG. 1 is two. That is, a message Sent by any computer by FIG.3B. The process of breaking the connection between
would traverse no more than two connections to reach every two neighbors and reconnecting each of the former neigh
other computer. FIG. 2 illustrates a graph representing 20 bors to another computer is referred to as "edge pinning” as
computers connected to a broadcast channel. The diameter the edge between two nodes may be considered to be
of this broadcast channel is 4. In particular, the Shortest path Stretched and pinned to a new node.
between computers 1 and 3 contains four connections (1-12, 35 Each computer connected to the broadcast channel allo
12-15, 15-18, and 18-3). cates five communications ports for communicating with
The broadcast technique includes (1) the connecting of other computers. Four of the ports are referred to as “inter
computers to the broadcast channel (i.e., composing the nal ports because they are the ports through which the
graph), (2) the broadcasting of messages over the broadcast messages of the broadcast channels are sent. The connec
channel (i.e., broadcasting through the graph), and (3) the 40 tions between internal ports of neighbors are referred to as
disconnecting of computers from the broadcast channel (i.e., “internal” connections. Thus, the internal connections of the
decomposing the graph) composing the graph. broadcast channel form the 4-regular and 4-connected
graph. The fifth port is referred to as an “external' port
To connect to the broadcast channel, the computer Seeking 45 between two computerS. Neighbors can Send non-broadcast
are at least four computerS already connected to the broad 50 the computer connections using the TCP/IP communications
connected, the broadcast channel cannot be a 4-regular ing network. The TCP/IP protocol provides for reliable and
small regime is described below in detail. When five or more 55 is shared among all the processes that may execute on that
broadcast channel includes locating the broadcast channel, 60 one embodiment, a Set of port numbers can be reserved for
then connecting to each identified neighbor. Each computer an alternative embodiment, the port numbers used are
computer locates the broadcast channel by contacting the 65 call-in port. This call-in port is used to establish connections
connected to the broadcast channel. The found portal com that is connected to the broadcast channel can receive
US 6,829,634 B1
7 8
non-broadcast messages through its external port. A Seeking receives to its neighbors and disregards Subsequently
computer tries “dialing the port numbers of the portal received copies. Thus, the total number of copies of a
computers until a portal computer "answers,” a call on its message that is Sent between the computerS is 3N+1, where
call-in port. A portal computer answers when it is connected N is the number of computers connected to the broadcast
to or attempting to connect to the broadcast channel and its channel. Each computer Sends three copies of the message,
call-in port is dialed. (In this description, a telephone meta except for the originating computer, which Sends four copies
phor is used to describe the connections.) When a computer of the message.
receives a call on its call-in port, it transferS the call to The redundancy of the message Sending helps to ensure
another port. Thus, the Seeking computer actually commu the overall reliability of the broadcast channel. Since each
nicates through that transfer-to port, which is the external computer has four connections to the broadcast channel, if
port. The call is transferred So that other computers can place one computer fails during the broadcast of a message, its
calls to that computer via the call-in port. The Seeking neighbors have three other connections through which they
computer then communicates via that external port to will receive copies of the broadcast message. Also, if the
request the portal computer to assist in connecting the internal connection between two computers is slow, each
Seeking computer to the broadcast channel. The Seeking 15 computer has three other connections through which it may
computer could identify the call-in port number of a portal receive a copy of each message Sooner.
computer by Successively dialing each port in port number Each computer that originates a message numbers its own
order. AS discussed below in detail, the broadcast technique messages Sequentially. Because of the dynamic nature of the
uses a hashing algorithm to Select the port number order, broadcast channel and because there are many possible
which may result in improved performance. connection paths between computers, the messages may be
A Seeking computer could connect to the broadcast chan received out of order. For example, the distance between an
nel by connecting to computers either directly connected to originating computer and a certain receiving computer may
the found portal computer or directly connected to one of its be four. After Sending the first message, the originating
neighbors. A possible problem with Such a Scheme for computer and receiving computer may become neighbors
identifying the neighbors for the Seeking computer is that the 25 and thus the distance between them changes to one. The first
diameter of the broadcast channel may increase when each message may have to travel a distance of four to reach the
Seeking computer uses the same found portal computer and receiving computer. The Second message only has to travel
establishes a connection to the broadcast channel directly a distance of one. Thus, it is possible for the Second message
through that found portal computer. Conceptually, the graph to reach the receiving computer before the first message.
becomes elongated in the direction of where the new nodes When the broadcast channel is in a steady state (i.e., no
are added. FIGS. 4A-4C illustrate that possible problem. computers connecting or disconnecting from the broadcast
FIG. 4A illustrates the broadcast channel of FIG. 1 with an channel), out-of-order messages are not a problem because
added computer. Computer J was connected to the broadcast each computer will eventually receive both messages and
channel by edge pinning edges C-D and E-H to computer can queue messages until all earlier ordered messages are
J. The diameter of this broadcast channel is still two. FIG. 35 received. If, however, the broadcast channel is not in a
4B illustrates the broadcast channel of FIG. 4A with an Steady State, then problems can occur. In particular, a com
added computer. Computer K was connected to the broad puter may connect to the broadcast channel after the Second
cast channel by edge pinning edges E-J and B-C to com message has already been received and forwarded on by its
puter K. The diameter of this broadcast channel is three, new neighbors. When a new neighbor eventually receives
because the Shortest path from computer G to computer Kis 40 the first message, it sends the message to the newly con
through edges G-A, A-E, and E-K. FIG. 4C also illustrates nected computer. Thus, the newly connected computer will
the broadcast channel of FIG. 4A with an added computer. receive the first message, but will not receive the Second
Computer K was connected to the broadcast channel by edge message. If the newly connected computer needs to process
pinning edges D-G and E-J to computer K. The diameter of the messages in order, it would wait indefinitely for the
this broadcast channel is, however, still two. Thus, the 45 Second message.
Selection of neighbors impacts the diameter of the broadcast One Solution to this problem is to have each computer
channel. To help minimize the diameter, the broadcast queue all the messages that it receives until it can Send them
technique uses a random Selection technique to identify the in their proper order to its neighbors. This Solution, however,
four neighbors of a computer in the Seeking connection State. may tend to Slow down the propagation of messages through
The random Selection technique tends to distribute the 50 the computers of the broadcast channel. Another Solution
connections to new Seeking computers throughout the com that may have leSS impact on the propagation Speed is to
puters of the broadcast channel which may result in Smaller queue messages only at computers who are neighbors of the
overall diameters. newly connected computers. Each already connected neigh
Broadcasting Through the Graph bor would forward messages as it receives them to its other
55 neighbors who are not newly connected, but not to the newly
AS described above, each computer that is connected to connected neighbor. The already connected neighbor would
the broadcast channel can broadcast messages onto the only forward messages from each originating computer to
broadcast channel and does receive all messages that are the newly connected computer when it can ensure that no
broadcast on the broadcast channel. The computer that gaps in the messages from that originating computer will
originates a message to be broadcast sends that message to 60 occur. In one embodiment, the already connected neighbor
each of its four neighbors using the internal connections. may track the highest Sequence number of the messages
When a computer receives a broadcast message from a already received and forwarded on from each originating
neighbor, it sends the message to its three other neighbors. computer. The already connected computer will Send only
Each computer on the broadcast channel, except the origi higher numbered messages from the originating computers
nating computer, will thus receive a copy of each broadcast 65 to the newly connected computer. Once all lower numbered
message from each of its four neighbors. Each computer, messages have been received from all originating
however, only sends the first copy of the message that it computers, then the already connected computer can treat
US 6,829,634 B1
10
the newly connected computer as its other neighbors and the connection request, it communicates with the requesting
Simply forward each message as it is received. In another computer through its external port to establish a connection
embodiment, each computer may queue messages and only between the two computers. FIG. 5B illustrates the discon
forwards to the newly connected computer those messages necting of a computer from the broadcast channel in an
as the gaps are filled in. For example, a computer might unplanned manner. In this illustration, computer H has
receive messages 4 and 5 and then receive message 3. In disconnected in an unplanned manner. When each of its
Such a case, the already connected computer would forward neighbors, computers A, E, F, and I, recognizes the
queue messages 4 and 5. When message 3 is finally received, disconnection, each neighbor broadcasts a port connection
the already connected computer will Send messages 3, 4, and request indicating that it needs to fill an empty port. AS
5 to the newly connected computer. If messages 4 and 5 were shown by the dashed lines, computers F and I and computers
Sent to the newly connected computer before message 3, A and E respond to each other's requests and establish a
then the newly connected computer would process messages connection.
4 and 5 and disregard message 3. Because the already It is possible that a planned or unplanned disconnection
connected computer queues messages 4 and 5, the newly may result in two neighbors each having an empty internal
connected computer will be able to proceSS message 3. It is 15 port. In Such a case, Since they are neighbors, they are
possible that a newly connected computer will receive a Set already connected and cannot fill their empty ports by
of messages from an originating computer through one connecting to each other. Such a condition is referred to as
neighbor and then receive another Set of message from the the “neighbors with empty ports' condition. Each neighbor
Same originating computer through another neighbor. If the broadcasts a port connection request when it detects that it
Second Set of messages contains a message that is ordered has an empty port as described above. When a neighbor
earlier than the messages of the first Set received, then the receives the port connection request from the other neighbor,
newly connected computer may ignore that earlier ordered it will recognize the condition that its neighbor also has an
message if the computer already processed those later empty port. Such a condition may also occur when the
ordered messages. broadcast channel is in the Small regime. The condition can
Decomposing the Graph 25 only be corrected when in the large regime. When in the
Small regime, each computer will have less than four neigh
A connected computer disconnects from the broadcast bors. To detect this condition in the large regime, which
channel either in a planned or unplanned manner. When a would be a problem if not repaired, the first neighbor to
computer disconnects in a planned manner, it sends a receive the port connection request recognizes the condition
disconnect message to each of its four neighbors. The and sends a condition check message to the other neighbor.
neighbors of the disconnecting computer. When a neighbor of the sending computer. When the receiving computer
computer in the list will try to connect to the Second 35 occurred in the large regime and repair is needed. To repair
in various other combinations. If connections cannot be 40 request, it disconnects from one of its neighbors (other than
When a computer with an available internal port receives the Thus, one of the original neighbors involved in the condition
puter that broadcast the message. FIGS.5A-5D illustrate the 45 in need of a connection, the other original neighbor and the
its neighbors (computers A, E, F and I) and then disconnects 50 they receive the requests. If, however, the two computers are
indicated by the dashed line, and Similarly for computers E It is possible that the two original neighbors with the
When a computer disconnects in an unplanned manner, 55 neighbor that receives the condition check message deter
disconnected computer recognizes that it is short one con 60 it determines whether it has the same Set of neighbors as the
disconnected, it broadcasts a port connection request on the neighbors are different, then the computer that received the
port that needs a connection. The port connection request 65 message to the original neighbors with the condition. The
US 6,829,634 B1
11 12
neighbors with the condition by Sending a condition repair embodiment, the computers connected to the broadcast
message. Thus, one of the original neighbors with the channel dynamically allocate their port numbers. Each com
condition will have its port filled. puter could simply try to locate the lowest number unused
FIG. 5C illustrates the neighbors with empty ports con port on that computer and use that port as the call-in port. A
dition. In this illustration, computer H disconnected in an Seeking computer, however, does not know in advance the
unplanned manner, but computers F and I responded to the call-in port number of the portal computers when the port
port connection request of the other and are now connected numbers are dynamically allocated. Thus, a Seeking com
together. The other former neighbors of computer H, com puter needs to dial ports of a portal computer Starting with
puters A and E, are already neighbors, which gives rise to the the lowest port number when locating the call-in port of a
neighbors with empty ports condition. In this example, portal computer. If the portal computer is connected to (or
computer E received the port connection request from attempting to connect to) the broadcast channel, then the
computer A, recognized the possible condition, and Sent Seeking computer would eventually find the call-in port. If
(since they are neighbors via the internal connection) a the portal computer is not connected, then the Seeking
condition check message with a list of its neighbors to computer would eventually dial every user port. In addition,
computer A. When computer A received the list, it recog 15 if each application program on a computer tried to allocate
nized that computer E has a different set of neighbor (i.e., the low-ordered port numbers, then a portal computer may end
broadcast channel is in the large regime). Computer A up with a high-numbered port for its call-in port because
Selected computer D, which is a neighbor of computer E and many of the low-ordered port numbers would be used by
Sent it a condition repair request. When computer D received other application programs. Since the dialing of a port is a
the condition repair request, it disconnected from one of its relatively slow process, it would take the Seeking computer
neighbors (other than computer E), which is computer G in a long time to locate the call-in port of a portal computer. To
this example. Computer D then connected to computer A. minimize this time, the broadcast technique uses a port
FIG. 5D illustrates two computers that are not neighbors ordering algorithm to identify the port number order that a
who now have empty ports. Computers E and G now have portal computer should use when finding an available port
empty ports and are not currently neighbors. Therefore, 25 for its call-in port. In one embodiment, the broadcast tech
computers E and G can connect to each other. nique uses a hashing algorithm to identify the port order. The
FIGS. 5E and 5F further illustrate the neighbors with algorithm preferably distributes the ordering of the port
empty ports condition. FIG.5E illustrates the neighbors with numbers randomly through out the user port number Space
empty ports condition in the Small regime. In this example, and only Selects each port number once. In addition, every
if computer E disconnected in an unplanned manner, then time the algorithm is executed on any computer for a given
each computer broadcasts a port connection request when it channel type and channel instance, it generates the same port
detects the disconnect. When computer A receives the port ordering. As described below, it is possible for a computer
connection request form computer B, it detects the neigh to be connected to multiple broadcast channels that are
bors with empty ports condition and Sends a condition check uniquely identified by channel type and channel instance.
message to computer B. Computer B recognizes that it has 35 The algorithm may be “seeded with channel type and
the same set of neighbors (computer C and D) as computer channel instance in order to generate a unique ordering of
A and then Sends a condition double check message to port numbers for each broadcast channel. Thus, a Seeking
computer C. Computer C recognizes that the broadcast computer will dial the ports of a portal computer in the same
channel is in the Small regime because is also has the same order as the portal computer used when allocating its call-in
Set of neighbors as computers A and B, computer C may then 40 port.
broadcast a message indicating that the broadcast channel is If many computers are at the same time Seeking connec
in the Small regime. tion to a broadcast channel through a single portal computer,
FIG. 5F illustrates the situation of FIG. 5E when in the then the ports of the portal computer may be busy when
large regime. AS discussed above, computer C receives the called by Seeking computers. The Seeking computers would
condition double check message from computer B. In this
45 typically need to keep on redialing a busy port. The process
case, computer C recognizes that the broadcast channel is in of locating a call-in port may be significantly slowed by Such
the large regime because it has a set of neighbors that is redialing. In one embodiment, each Seeking computer may
different from computer B. The edges extending up from each reorder the first few port numbers generated by the
computer C and D indicate connections to other computers. hashing algorithm. For example, each Seeking computer
Computer C then Sends a condition check message to
50 could randomly reorder the first eight port numbers gener
computer B. When computer B receives the condition check ated by the hashing algorithm. The random ordering could
message, it sends a condition repair message to one of the also be weighted where the first port number generated by
neighbors of computer C. The computer that receives the the hashing algorithm would have a 50% chance of being
condition repair message disconnects from one of its first in the reordering, the Second port number would have
neighbors, other than computer C, and tries to connect to
55 a 25% chance of being first in the reordering, and So on.
computer B and the neighbor from which it disconnected Because the Seeking computers would use different
tries to connect to computer A. orderings, the likelihood of finding a busy port is reduced.
For example, if the first eight port numbers are randomly
Port Selection Selected, then it is possible that eight Seeking computers
60 could be simultaneously dialing ports in different Sequences
uses five user port numbers on each computer: one external
port and four internal ports. Generally, user ports cannot be Each computer that can connect to the broadcast channel
Statically allocated to an application program because other 65 has a list of one or more portal computers through which it
applications programs executing on the same computer may can connect to the broadcast channel. In one embodiment,
use conflicting port numbers. As a result, in one each computer has the same Set of portal computers. A
US 6,829,634 B1
13 14
Seeking computer locates a portal computer that is connected To Select the four computers, a portal computer Sends an
to the broadcast channel by Successively dialing the ports of edge connection request message through one of its internal
each portal computer in the order Specified by an algorithm. connections that is randomly Selected. The receiving com
A Seeking computer could Select the first portal computer puter again sends the edge connection request message
and then dial all its ports until a call-in port of a computer through one of its internal connections that is randomly
that is fully connected to the broadcast channel is found. If Selected. This Sending of the message corresponds to a
no call-in port is found, then the Seeking computer would random walk through the graph that represents the broadcast
Select the next portal computer and repeat the proceSS until channel. Eventually, a receiving computer will decide that
a portal computer with Such a call-in port is found. A the message has traveled far enough to represent a randomly
problem with Such a Seeking technique is that all user ports Selected computer. That receiving computer will offer the
of each portal computer are dialed until a portal computer internal connection upon which it received the edge con
fully connected to the broadcast channel is found. In an nection request message to the Seeking computer for edge
alternate embodiment, the Seeking computer Selects a port pinning. Of course, if either of the computers at the end of
number according to the algorithm and then dials each portal the offered internal connection are already neighbors of the
computer at that port number. If no acceptable call-in port to 15 Seeking computer, then the Seeking computer cannot connect
the broadcast channel is found, then the Seeking computer through that internal connection. The computer that decided
Selects the next port number and repeats the process. Since that the message has traveled far enough will detect this
the call-in ports are likely allocated at lower-ordered port condition of already being a neighbor and Send the message
numbers, the Seeking computer first dials the port numbers to a randomly Selected neighbor.
that are most likely to be call-in ports of the broadcast In one embodiment, the distance that the edge connection
channel. The Seeking computers may have a maximum request message travels is established by the portal computer
search depth, that is the number of ports that it will dial when to be approximately twice the estimated diameter of the
Seeking a portal computer that is fully connected. If the broadcast channel. The message includes an indication of
Seeking computer exhausts its Search depth, then either the the distance that it is to travel. Each receiving computer
broadcast channel has not yet been established or, if the 25 decrements that distance to travel before Sending the mes
Seeking computer is also a portal computer, it can then Sage on. The computer that receives a message with a
establish the broadcast channel with itself as the first fully distance to travel that is Zero is considered to be the
connected computer. randomly Selected computer. If that randomly Selected com
When a Seeking computer locates a portal computer that puter cannot connect to the Seeking computer (e.g., because
is itself not fully connected, the two computers do not it is already connected to it), then that randomly selected
connect when they first locate each other because the computer forwards the edge connection request to one of its
broadcast channel may already be established and accessible neighbors with a new distance to travel. In one embodiment,
through a higher-ordered port number on another portal the forwarding computer toggles the new distance to travel
computer. If the two seeking computers were to connect to between Zero and one to help prevent two computers from
each other, then two disjoint broadcast channels would be 35 Sending the message back and forth between each other.
formed. Each Seeking computer can share its experience in Because of the local nature of the information maintained
trying to locate a portal computer with the other Seeking by each computer connected to the broadcast channel, the
computer. In particular, if one Seeking computer has computers need not generally be aware of the diameter of the
Searched all the portal computers to a depth of eight, then the broadcast channel. In one embodiment, each message Sent
one Seeking computer can share that it has Searched to a 40
through the broadcast channel has a distance traveled field.
depth of eight with another Seeking computer. If that other Each computer that forwards a message increments the
Seeking computer has Searched to a depth of, for example, distance traveled field. Each computer also maintains an
only four, it can Skip Searching through depths five through estimated diameter of the broadcast channel. When a com
eight and that other Seeking computer can advance its puter receives a message that has traveled a distance that
Searching to a depth of nine. 45
indicates that the estimated diameter is too Small, it updates
In one embodiment, each computer may have a different its estimated diameter and broadcasts an estimated diameter
Set of portal computers and a different maximum Search message. When a computer receives an estimated diameter
depth. In Such a situation, it may be possible that two disjoint message that indicates a diameter that is larger than its own
broadcast channels are formed because a Seeking computer 50
estimated diameter, it updates its own estimated diameter.
cannot locate a fully connected port computer at a higher This estimated diameter is used to establish the distance that
depth. Similarly, if the Set of portal computers are disjoint, an edge connection request message should travel.
then two separate broadcast channels would be formed.
Identifying Neighbors for a Seeking Computer External Data Representation
AS described above, the neighbors of a newly connecting 55 The computers connected to the broadcast channel may
computer are preferably Selected randomly from the Set of internally Store their data in different formats. For example,
currently connected computers. One advantage of the broad one computer may use 32-bit integers, and another computer
cast channel, however, is that no computer has global may use 64-bit integers. AS another example, one computer
knowledge of the broadcast channel. Rather, each computer may use ASCII to represent text and another computer may
has local knowledge of itself and its neighbors. This limited 60 use Unicode. To allow communications between heteroge
local knowledge has the advantage that all the connected neous computers, the messages Sent over the broadcast
computers are peers (as far as the broadcasting is concerned) channel may use the XDR (“external Data Representation”)
and the failure of any one computer (actually any three format.
computers when in the 4-regular and 4-connect form) will The underlying peer-to-peer communications protocol
not cause the broadcast channel to fail. This local knowledge 65 may send multiple messages in a Single message Stream. The
makes it difficult for a portal computer to randomly Select traditional technique for retrieving messages from a stream
four neighbors for a Seeking computer. has been to repeatedly invoke an operating System routine to
US 6,829,634 B1
15 16
retrieve the next message in the Stream. The retrieval of each broadcast channel to which it is connected. The broadcaster
message may require two calls to the operating System: one component may be implement as an object that is instanti
to retrieve the Size of the next message and the other to ated within the proceSS Space of the application program.
retrieve the number of bytes indicated by the retrieved size. Alternatively, the broadcaster component may execute as a
Such calls to the operating System can, however, be very Separate proceSS or thread from the application program. In
Slow in comparison to the invocations of local routines. To one embodiment, the broadcaster component provides func
overcome the inefficiencies of Such repeated calls, the broad tions (e.g., methods of class) that can be invoked by the
cast technique in one embodiment, uses XDR to identify the application programs. The primary functions provided may
message boundaries in a stream of messages. The broadcast include a connect function that an application program
technique may request the operating System to provide the invokes passing an indication of the broadcast channel to
next, for example, 1,024 bytes from the stream. The broad which the application program wants to connect. The appli
cast technique can then repeatedly invoke the XDR routines cation program may provide a callback routine that the
to retrieve the messages and use the Success or failure of broadcaster component invokes to notify the application
each invocation to determine whether another block of 1,024 program that the connection has been completed, that is the
bytes needs to be retrieved from the operating System. The 15 process enters the fully connected State. The broadcaster
invocation of XDR routines do not involve system calls and component may also provide an acquire message function
are thus more efficient than repeated System calls. that the application program can invoke to retrieve the next
message that is broadcast on the broadcast channel.
M-Regular Alternatively, the application program may provide a call
In the embodiment described above, each fully connected back routine (which may be a virtual function provided by
computer has four internal connections. The broadcast tech the application program) that the broadcaster component
nique can be used with other numbers of internal connec invokes to notify the application program that a broadcast
tions. For example, each computer could have 6, 8, or any message has been received. Each broadcaster component
even number of internal connections. AS the number of allocates a call-in port using the hashing algorithm. When
internal connections increase, the diameter of the broadcast 25 calls are answered at the call-in port, they are transferred to
channel tends to decrease, and thus propagation time for a other ports that Serve as the external and internal ports.
message tends to decrease. The time that it takes to connect The computers connecting to the broadcast channel may
a Seeking computer to the broadcast channel may, however, include a central processing unit, memory, input devices
increase as the number of internal connections increases. (e.g., keyboard and pointing device), output devices (e.g.,
When the number of internal connectors is even, then the display devices), and storage devices (e.g., disk drives). The
broadcast channel can be maintained as m-regular and memory and Storage devices are computer-readable medium
m-connected (in the steady State). If the number of internal that may contain computer instructions that implement the
connections is odd, then when the broadcast channel has an broadcaster component. In addition, the data structures and
odd number of computers connected, one of the computers 35
message Structures may be Stored or transmitted via a signal
will have less than that odd number of internal connections. transmitted on a computer-readable media, Such as a com
In Such a situation, the broadcast network is neither munications link.
m-regular nor m-connected. When the next computer con FIG. 7 is a block diagram illustrating the Sub-components
nects to the broadcast channel, it can again become of the broadcaster component in one embodiment. The
m-regular and m-connected. Thus, with an odd number of 40 broadcaster component includes a connect component 701,
internal connections, the broadcast channel toggles between an external dispatcher 702, an internal dispatcher 703 for
being and not being m-regular and m-connected. each internal connection, an acquire message component
Components 704 and a broadcast component 712. The application pro
gram may provide a connect callback component 710 and a
FIG. 6 is a block diagram illustrating components of a 45 receive response component 711 that are invoked by the
computer that is connected to a broadcast channel. The broadcaster component. The application program invokes
above description generally assumed that there was only one the connect component to establish a connection to a des
broadcast channel and that each computer had only one ignated broadcast channel. The connect component identi
connection to that broadcast channel. More generally, a fies the external port and installs the external dispatcher for
network of computers may have multiple broadcast 50 handling messages that are received on the external port.
channels, each computer may be connected to more than one The connect component invokes the Seek portal computer
broadcast channel, and each computer can have multiple component 705 to identify a portal computer that is con
connections to the same broadcast channel. The broadcast nected to the broadcast channel and invokes the connect
channel is well Suited for computer processes (e.g., appli request component 706 to ask the portal computer (if fully
cation programs) that execute collaboratively, Such as net 55 connected) to select neighbor processes for the newly con
work meeting programs. Each computer process can connect necting process. The external dispatcher receives external
to one or more broadcast channels. The broadcast channels messages, identifies the type of message, and invokes the
can be identified by channel type (e.g., application program appropriate handling routine 707. The internal dispatcher
name) and channel instance that represents separate broad receives the internal messages, identifies the type of
cast channels for that channel type. When a proceSS attempts 60 message, and invokes the appropriate handling routine 708.
to connect to a broadcast channel, it seeks a process cur The received broadcast messages are Stored in the broadcast
rently connected to that broadcast channel that is executing message queue 709. The acquire message component is
on a portal computer. The Seeking process identifies the invoked to retrieve messages from the broadcast queue. The
broadcast channel by channel type and channel instance. broadcast component is invoked by the application program
Computer 600 includes multiple application programs 65 to broadcast messages in the broadcast channel.
601 executing as Separate processes. Each application pro The following tables list messages Sent by the broadcaster
gram interfaces with a broadcaster component 602 for each components.
US 6,829,634 B1
17 18
External Messages process enters the partially connected State, and when the
process eventually connects to four neighbors, it enters the
fully connected state. When in the small regime, a fully
connected process may have less than four neighbors. In
EXTERNAL MESSAGES block 801, the routine opens the call-in port through which
Message Type Description the proceSS is to communicate with other processes when
establishing external and internal connections. The port is
seeking Indicates that a seeking process would like to know Selected as the first available port using the hashing algo
connection call whether the receiving process is fully connected to the rithm described above. In block 802, the routine sets the
broadcast channel
connection Indicates that the sending process would like the connect time to the current time. The connect time is used to
request call receiving process to initiate a connection of the identify the instance of the process that is connected through
sending process to the broadcast channel
edge proposal Indicates that the sending process is proposing an edge
this external port. One proceSS may connect to a broadcast
call through which the receiving process can connect to the channel of a certain channel type and channel instance using
broadcast channel (i.e., edge pinning) one call-in port and then disconnects, and another process
port Indicates that the sending process is proposing a port 15 may then connect to that same broadcast channel using the
connection call through which the receiving process can connect to the
broadcast channel
Same call-in port. Before the other process becomes fully
connected stimt Indicates that the sending process is connected to the connected, another proceSS may try to communicate with it
broadcast channel thinking it is the fully connected old process. In Such a case,
condition Indicates that the receiving process should disconnect the connect time can be used to identify this situation. In
repair stimt from one of its neighbors and connect to one of the block 803, the routine invokes the seek portal computer
processes involved in the neighbors with empty port routine passing the channel type and channel instance. The
condition
Seek portal computer routine attempts to locate a portal
computer through which this process can connect to the
broadcast channel for the passed type and instance. In
Internal Messages 25 decision block 804, if the seek portal computer routine is
Successful in locating a fully connected process on that
portal computer, then the routine continues at block 805, else
the routine returns an unsuccessful indication. In decision
INTERNAL MESSAGES
block 805, if no portal computer other than the portal
Message Type Description computer on which the process is executing was located,
broadcast stimt Indicates a message that is being broadcast through
then this is the first process to fully connect to broadcast
the broadcast channel for the application programs channel and the routine continues at block 806, else the
connection port Indicates that the designated process is looking for a routine continues at block 808. In block 806, the routine
search stimt port through which it can connect to the broadcast
channel
invokes the achieve connection routine to change the State of
connection edge Indicates that the requesting process is looking for
35 this process to fully connected. In block 807, the routine
search call an edge through which it can connect to the installs the external dispatcher for processing messages
broadcast channel received through this process external port for the passed
connection edge Indicates whether the edge between this process and channel type and channel instance. When a message is
search resp the sending neighbor has been accepted by the received through that external port, the external dispatcher is
requesting party
diameter estimate Indicates an estimated diameter of the broadcast 40 invoked. The routine then returns. In block 808, the routine
Stmt channel installs an external dispatcher. In block 809, the routine
diameter reset Indicates to reset the estimated diameter to invokes the connect request routine to initiate the process of
Stmt
disconnect stimt
indicated diameter
Indicates that the sending neighbor is disconnecting
identifying neighbors for the Seeking computer. The routine
from the broadcast channel
then returns.
condition check Indicates that neighbors with empty port condition 45 FIG. 9 is a flow diagram illustrating the processing of the
Stmt have been detected Seek portal computer routine in one embodiment. This
condition double Indicates that the neighbors with empty ports have routine is passed the channel type and channel instance of
check stimt the same set of neighbors
shutdown stimt Indicates that the broadcast channel is being the broadcast channel to which this proceSS wishes to
shutdown connect. This routine, for each Search depth (e.g., port
50 number), checks the portal computers at that search depth. If
a portal computer is located at that Search depth with a
Flow Diagrams process that is fully connected to the broadcast channel, then
the routine returns an indication of Success. In blockS
FIGS. 8-34 are flow diagrams illustrating the processing 902-911, the routine loops selecting each search depth until
of the broadcaster component in one embodiment. FIG. 8 is 55 a process is located. In block 902, the routine selects the next
a flow diagram illustrating the processing of the connect Search depth using a port number ordering algorithm. In
routine in one embodiment. This routine is passed a channel decision block 903, if all the search depths have already
type (e.g., application name) and channel instance (e.g., been Selected during this execution of the loop, that is for the
Session identifier), that identifies the broadcast channel to currently Selected depth, then the routine returns a failure
which this proceSS wants to connect. The routine is also 60 indication, else the routine continues at block 904. In blocks
passed auxiliary information that includes the list of portal 904-911, the routine loops selecting each portal computer
computers and a connection callback routine. When the and determining whether a process of that portal computer
connection is established, the connection callback routine is is connected to (or attempting to connect to) the broadcast
invoked to notify the application program. When this pro channel with the passed channel type and channel instance.
ceSS invokes this routine, it is in the Seeking connection 65 In block 904, the routine selects the next portal computer. In
State. When a portal computer is located that is connected decision block 905, if all the portal computers have already
and this routine connects to at least one neighbor, this been selected, then the routine loops to block 902 to select
US 6,829,634 B1
19 20
the next Search depth, else the routine continues at block restarts the process of connecting to the broadcast channel
computer through the port represented by the Search depth. one of the found portal computers through the call-in port.
In decision block 30 907, if the dialing was successful, then In decision block 1104, if the dialing is successful, then the
the routine continues at block 908, else the routine loops to routine continues at block 1105, else the routine continues at
block 904 to select the next portal computer. The dialing will block 1113. The dialing may be unsuccessful if, for example,
which contacts the answering process of the portal computer 1106, the routine receives the response message (i.e.,
is fully connected to the broadcast channel. In block 909, the response message is Successfully received, then the routine
routine hangs up on the Selected portal computer. In decision continues at block 1108, else the routine continues at block
block 910, if the answering process is fully connected to the 15 1113. In block 1108, the routine sets the expected number of
broadcast channel, then the routine returns a Success holes (i.e., empty internal connections) for this process
indicator, else the routine continues at block 911. In block based on the received response. When in the large regime,
911, the routine invokes the check for external call routine the expected number of holes is zero. When in the small
to determine whether an external call has been made to this regime, the expected number of holes varies from one to
proceSS as a portal computer and processes that call. The three. In block 1109, the routine sets the estimated diameter
routine then loops to block 904 to select the next portal of the broadcast channel based on the received response. In
computer. decision block 1111, if the dialed process is ready to connect
FIG. 10 is a flow diagram illustrating the processing of the to this process as indicated by the response message, then
contact process routine in one embodiment. This routine the routine continues at block 1112, else the routine contin
determines whether the process of the Selected portal com 25 ues at block 1113. In block 1112, the routine invokes the add
puter that answered the call-in to the Selected port is fully neighbor routine to add the answering proceSS as a neighbor
connected to the broadcast channel. In block 1001, the to this process. This adding of the answering process typi
routine sends an external message (i.e., seeking cally occurs when the broadcast channel is in the Small
connection call) to the answering process indicating that a regime. When in the large regime, the random walk Search
Seeking proceSS wants to know whether the answering for a neighbor is performed. In block 1113, the routine hangs
proceSS is fully connected to the broadcast channel. In block up the external connection with the answering proceSS
1002, the routine receives the external response message computer and then returns.
from the answering process. In decision block 1003, if the FIG. 12 is a flow diagram of the processing of the check
external response message is Successfully received (i.e., for external call routine in one embodiment. This routine is
Seeking connection resp), then the routine continues at 35 invoked to identify whether a fellow Seeking process is
block 1004, else the routine returns. Wherever the broadcast attempting to establish a connection to the broadcast channel
component requests to receive an external message, it Sets a through this process. In block 1201, the routine attempts to
time out period. If the external message is not received answer a call on the call-in port. In decision block 1202, if
within that time out period, the broadcaster component the answer is Successful, then the routine continues at block
checks its own call-in port to see if another process is calling 40 1203, else the routine returns. In block 1203, the routine
it. In particular, the dialed proceSS may be calling the dialing receives the external message from the external port. In
process, which may result in a deadlock situation. The decision block 1204, if the type of the message indicates that
broadcaster component may repeat the receive request Sev a Seeking process is calling (i.e., Seeking connection call),
eral times. If the expected message is not received, then the then the routine continues at block 1205, else the routine
broadcaster component handles the error as appropriate. In 45 returns. In block 1205, the routine sends an external message
decision block 1004, if the answering process indicates in its (i.e., seeking connection resp) to the other seeking pro
response message that it is fully connected to the broadcast ceSS indicating that this process is also is Seeking a connec
channel, then the routine continues at block 1005, else the tion. In decision block 1206, if the sending of the external
routine continues at block 1006. In block 1005, the routine message is Successful, then the routine continues at block
adds the Selected portal computer to a list of connected 50 1207, else the routine returns. In block 1207, the routine
portal computers and then returns. In block 1006, the routine adds the other Seeking process to a list of fellow Seeking
adds the answering process to a list of fellow Seeking processes and then returns. This list may be used if this
processes and then returns. process can find no process that is fully connected to the
FIG. 11 is a flow diagram illustrating the processing of the broadcast channel. In which case, this process may check to
connect request routine in one embodiment. This routine 55 See if any fellow Seeking process were Successful in con
requests a process of a portal computer that was identified as necting to the broadcast channel. For example, a fellow
being fully connected to the broadcast channel to initiate the Seeking process may become the first process fully con
connection of this process to the broadcast channel. In nected to the broadcast channel.
decision block 1101, if at least one process of a portal FIG. 13 is a flow diagram of the processing of the achieve
computer was located that is fully connected to the broadcast 60 connection routine in one embodiment. This routine Sets the
channel, then the routine continues at block 1103, else the State of this process to fully connected to the broadcast
routine continues at block 1102. A process of the portal channel and invokes a callback routine to notify the appli
computer may no longer be in the list if it recently discon cation program that the process is now fully connected to the
nected from the broadcast channel. In one embodiment, a requested broadcast channel. In block 1301, the routine sets
Seeking computer may always Search its entire Search depth 65 the connection State of this process to fully connected. In
and find multiple portal computers through which it can block 1302, the routine notifies fellow seeking processes
connect to the broadcast channel. In block 1102, the routine that it is fully connected by Sending a connected external
US 6,829,634 B1
21 22
message to them (i.e., connected stmt). In block 1303, the decision block 1601, if this process is currently fully con
application program and then returns. at block 1603, else the routine hangs up on the external port
is invoked when the external port receives a message. This response message. In block 1604, the routine Sets the
the routine answers (e.g., picks up) the external port and process is not a neighbor of this process. In block 1606, the
message was retrieved, then the routine continues at block is responsive to the connection request call (i.e.,
1403, else the routine hangs up on the external port in block connection request resp). In block 1607, the routine notes
1415 and returns. In decision block 1403, if the message 15 the number of holes that the calling process needs to fill as
type is for a process Seeking a connection (i.e., Seeking indicated in the request message. In decision block 1608, if
connection call), then the routine invokes the handle Seek this process is ready to connect to the calling process, then
ing connection call routine in block 1404, else the routine the routine continues at block 1609, else the routine contin
continues at block 1405. In decision block 1405, if the ues at block 1611. In block 1609, the routine invokes the add
message type is for a connection request call (i.e., neighbor routine to add the calling process as a neighbor. In
connection request call), then the routine invokes the block 1610, the routine decrements the number of holes that
handle connection request call routine in block 1406, else the calling proceSS needs to fill and continues at block 1611.
the routine continues at block 1407. In decision block 1407, In block 1611, the routine hangs up on the external port. In
if the message type is edge proposal call (i.e., edge decision block 1612, if this process has no holes or the
proposal call), then the routine invokes the handle edge 25 estimated diameter is greater than one (i.e., in the large
proposal call routine in block 1408, else the routine contin regime), then the routine continues at block 1613, else the
ues at block 1409. In decision block 1409, if the message routine continues at block 1616. In blocks 1613-1615, the
type is port connect call (i.e., port connect call), then the routine loops forwarding a request for an edge through
routine invokes the handle port connection call routine in which to connect to the calling process to the broadcast
block 1410, else the routine continues at block 1411. In channel. One request is forwarded for each pair of holes of
decision block 1411, if the message type is a connected the calling process that needs to be filled. In decision block
Statement (i.e., connected Stmt), the routine invokes the 1613, if the number of holes of the calling process to be
handle connected Statement in block 1112, else the routine filled is greater than or equal to two, then the routine
continues at block 1212. In decision block 1412, if the continues at block 1614, else the routine continues at block
message type is a condition repair Statement (i.e., 35 1616. In block 1614, the routine invokes the forward con
condition repair Stmt), then the routine invokes the handle nection edge Search routine. The invoked routine is passed
condition repair routine in block 1413, else the routine loops to an indication of the calling process and the random walk
to block 1414 to process the next message. After each distance. In one embodiment, the distance is twice in the
handling routine is invoked, the routine loops to block 1414. estimated diameter of the broadcast channel. In block 1614,
In block 1414, the routine hangs up on the external port and 40 the routine decrements the holes left to fill by two and loops
continues at block 1401 to receive the next message. to block 1613. In decision block 1616, if there is still a hole
FIG. 15 is a flow diagram illustrating the processing of the to fill, then the routine continues at block 1617, else the
handle Seeking connection call routine in one embodiment. routine returns. In block 1617, the routine invokes the fill
This routine is invoked when a Seeking proceSS is calling to hole routine passing the identification of the calling process.
identify a portal computer through which it can connect to 45 The fill hole routine broadcasts a connection port Search
the broadcast channel. In decision block 1501, if this process Statement (i.e., connection port Search Stnt) for a hole of
is currently fully connected to the broadcast channel iden a connected process through which the calling process can
tified in the message, then the routine continues at block connect to the broadcast channel. The routine then returns.
1502, else the routine continues at block 1503. In block FIG. 17 is a flow diagram illustrating the processing of the
1502, the routine Sets a message to indicate that this proceSS 50 add neighbor routine in one embodiment. This routine adds
is fully connected to the broadcast channel and continues at the process calling on the external port as a neighbor to this
block 1505. In block 1503, the routine sets a message to process. In block 1701, the routine identifies the calling
indicate that this proceSS is not fully connected. In block process on the external port. In block 1702, the routine sets
1504, the routine adds the identification of the seeking a flag to indicate that the neighbor has not yet received the
process to a list of fellow Seeking processes. If this proceSS 55 broadcast messages from this process. This flag is used to
is not fully connected, then it is attempting to connect to the ensure that there are no gaps in the messages initially Sent to
broadcast channel. In block 1505, the routine sends the the new neighbor. The external port becomes the internal
external message response (i.e., Seeking connection resp) port for this connection. In decision block 1703, if this
to the Seeking process and then returns. process is in the Seeking connection State, then this proceSS
FIG. 16 is a flow diagram illustrating processing of the 60 is connecting to its first neighbor and the routine continues
handle connection request call routine in one embodiment. at block 1704, else the routine continues at block 1705. In
This routine is invoked when the calling process wants this block 1704, the routine sets the connection state of this
process to initiate the connection of the process to the process to partially connected. In block 1705, the routine
broadcast channel. This routine either allows the calling adds the calling process to the list of neighbors of this
process to establish an internal connection with this proceSS 65 process. In block 1706, the routine installs an internal
(e.g., if in the Small regime) or starts the process of identi dispatcher for the new neighbor. The internal dispatcher is
fying a process to which the calling process can connect. In invoked when a message is received from that new neighbor
US 6,829,634 B1
23 24
fully connected, then the routine continues at block 1708, routine continues at block 1902, else the routine continues at
a process that is partially connected may buffer the messages or its neighbor is a neighbor of this process, then the routine
1709, if the number of holes of this process equals the pending as a proposed neighbor, then the routine continues
routine continues at block 1711. In block 1710, the routine an external message to the proposing process (i.e., edge
invokes the achieve connected routine to indicate that this proposal resp) indicating that the proposed edge is
process is fully connected. In decision block 1711, if the 15
accepted. In decision block 1908, if the sending of the
number of holes for this process is Zero, then the routine message was Successful, then the routine continues at block
1909, else the routine returns. In block 1909, the routine
continues at block 1712, else the routine returns. In block adds the edge as a pending edge. In block 1910, the routine
1712, the routine deletes any pending edges and then invokes the add neighbor routine to add the proposing
returns. A pending edge is an edge that has been proposed to process on the external port as a neighbor. The routine then
this process for edge pinning, which in this case is no longer returns. In block 1911, the routine Sends an external message
needed. (i.e., edge proposal resp) indicating that this proposed
FIG. 18 is a flow diagram illustrating the processing of the edge is not accepted. In decision block 1912, if the number
forward connection edge Search routine in one embodiment. of holes is odd, then the routine continues at block 1913, else
This routine is responsible for passing along a request to the routine returns. In block 1913, the routine invokes the fill
connect a requesting process to a randomly Selected neigh 25 hole routine and then returns.
bor of this process through the internal port of the Selected FIG.20 is a flow diagram illustrating the processing of the
neighbor, that is part of the random walk. In decision block handle port connection call routine in one embodiment. This
1801, if the forwarding distance remaining is greater than routine is invoked when an external message is received
Zero, then the routine continues at block 1804, else the then indicates that the Sending proceSS wants to connect to
routine continues at block 1802. In decision block 1802, if one hole of this process. In decision block 2001, if the
the number of neighbors of this proceSS is greater than one, number of holes of this proceSS is greater than Zero, then the
then the routine continues at block 1804, else this broadcast routine continues at block 2002, else the routine continues at
channel is in the Small regime and the routine continues at block 2003. In decision block 2002, if the sending process
block 1803. In decision block 1803, if the requesting process is not a neighbor, then the routine continues at block 2004,
is a neighbor of this process, then the routine returns, else the 35 else the routine continues to block 2003. In block 2003, the
routine continues at block 1804. In blocks 1804–1807, the routine Sends a port connection response external message
routine loops attempting to Send a connection edge Search (i.e., port connection resp) to the Sending process that
call internal message (i.e., connection edge Search call) indicates that it is not okay to connect to this process. The
to a randomly selected neighbor. In block 1804, the routine routine then returns. In block 2004, the routine sends a port
randomly Selects a neighbor of this proceSS. In decision 40 connection response external message to the Sending process
block 1805, if all the neighbors of this process have already that indicates that is okay to connect this process. In decision
been Selected, then the routine cannot forward the message block 2005, if the sending of the message was successful,
and the routine returns, else the routine continues at block then the routine continues at block 2006, else the routine
1806. In block 1806, the routine sends a connection edge continues at block 2007. In block 2006, the routine invokes
Search call internal message to the Selected neighbor. In 45 the add neighbor routine to add the Sending proceSS as a
decision block 1807, if the sending of the message is neighbor of this process and then returns. In block 2007, the
Successful, then the routine continues at block 1808, else the routine hangs up the external connection. In block 2008, the
routine loops to block 1804 to select the next neighbor. routine invokes the connect request routine to request that a
When the Sending of an internal message is unsuccessful, process connect to one of the holes of this process. The
then the neighbor may have disconnected from the broadcast 50 routine then returns.
channel in an unplanned manner. Whenever Such a situation FIG. 21 is a flow diagram illustrating the processing of the
is detected by the broadcaster component, it attempts to find fill hole routine in one embodiment. This routine is passed
another neighbor by invoking the fill holes routine to fill a an indication of the requesting process. If this proceSS is
Single hole or the forward connecting edge Search routine to requesting to fill a hole, then this routine Sends an internal
fill two holes. In block 1808, the routine notes that the 55 message to other processes. If another proceSS is requesting
recently Sent connection edge Search call has not yet been to fill a hole, then this routine invokes the routine to handle
acknowledged and indicates that the edge to this neighbor is a connection port Search request. In block 2101, the routine
reserved if the remaining forwarding distance is less than or initializes a connection port Search Statement internal mes
equal to one. It is reserved because the Selected neighbor Sage (i.e., connection port Search Stnt). In decision
may offer this edge to the requesting process for edge 60 block 2102, if this process is the requesting process, then the
pinning. The routine then returns. routine continues at block 2103, else the routine continues at
FIG. 19 is a flow diagram illustrating the processing of the block 2104. In block 2103, the routine distributes the
handle edge proposal call routine. This routine is invoked message to the neighbors of this process through the internal
when a message is received from a proposing process that ports and then returns. In block 2104, the routine invokes the
proposes to connect an edge between the proposing proceSS 65 handle connection port Search routine and then returns.
and one of its neighbors to this proceSS for edge pinning. In FIG.22 is a flow diagram illustrating the processing of the
decision block 1901, if the number of holes of this process internal dispatcher routine in one embodiment. This routine
US 6,829,634 B1
25 26
message. This routine identifies the message type and block 2601, the routine invokes the distribute internal mes
invokes the appropriate routine to handle the message. In Sage which sends the message to each of its neighbors other
the message has already been received (i.e., a duplicate), then the routine continues at block 2605, else the routine
2208, else the routine continues at block 2203A. In decision the court neighbor routine and then returns. The court
block 2203A, if the process is partially connected, then the neighbor routine connects this process to the requesting
routine continues at block 2203B, else the routine continues process if possible. In block 2605, if this process has one
at block 2204. In block 2203B, the routine adds the message hole, then the neighbors with empty ports condition exists
to the pending connection buffer and continues at block 15 and the routine continues at block 2606, else the routine
2204. In decision blocks 2204-2207, the routine decodes the returns. In block 2606, the routine generates a condition
message type and invokes the appropriate routine to handle check message (i.e., condition check) that includes a list of
the message. For example, in decision block 2204, if the this process neighbors. In block 2607, the routine sends the
type of the message is broadcast Statement (i.e., broadcast message to the requesting neighbor.
Stmt), then the routine invokes the handle broadcast message FIG. 27 is a flow diagram illustrating the processing of the
routine in block 2205. After invoking the appropriate han court neighbor routine in one embodiment. This routine is
dling routine, the routine continues at block 2208. In deci passed an indication of the prospective neighbor for this
sion block 2208, if the partially connected buffer is full, then process. If this proceSS can connect to the prospective
the routine continues at block 2209, else the routine contin neighbor, then it sends a port connection call external
ues at block 2210. The broadcaster component collects all its 25 message to the prospective neighbor and adds the prospec
internal messages in a buffer while partially connected So tive neighbor as a neighbor. In decision block 2701, if the
that it can forward the messages as it connects to new prospective neighbor is already a neighbor, then the routine
neighbors. If, however, that buffer becomes full, then the returns, else the routine continues at block 2702. In block
proceSS assumes that it is now fully connected and that the 2702, the routine dials the prospective neighbor. In decision
expected number of connections was too high, because the block 2703, if the number of holes of this process is greater
broadcast channel is now in the small regime. In block 2209, than Zero, then the routine continues at block 2704, else the
the routine invokes the achieve connection routine and then routine continues at block 2706. In block 2704, the routine
continues in block 2210. In decision block 2210, if the sends a port connection call external message (i.e., port
application program message queue is empty, then the connection call) to the prospective neighbor and receives
routine returns, else the routine continues at block 2212. In 35 its response (i.e., port connection resp). ASSuming the
block 2212, the routine invokes the receive response routine response is successfully received, in block 2705, the routine
passing the acquired message and then returns. The received adds the prospective neighbor as a neighbor of this process
response routine is a callback routine of the application by invoking the add neighbor routine. In block 2706, the
program. routine hangs up with the prospect and then returns.
FIG.23 is a flow diagram illustrating the processing of the 40 FIG. 28 is a flow diagram illustrating the processing of the
handle broadcast message routine in one embodiment. This handle connection edge Search call routine in one embodi
routine is passed an indication of the originating process, an ment. This routine is passed a indication of the neighbor who
indication of the neighbor who sent the broadcast message, Sent the message and the message itself. This routine either
and the broadcast message itself In block 2301, the routine forwards the message to a neighbor or proposes the edge
performs the out of order processing for this message. The 45 between this process and the Sending neighbor to the
broadcaster component queues messages from each origi requesting process for edge pinning. In decision block 2801,
nating process until it can Send them in Sequence number if this proceSS is not the requesting proceSS or the number of
order to the application program. In block 2302, the routine holes of the requesting process is still greater than or equal
invokes the distribute broadcast message routine to forward to two, then the routine continues at block 2802, else the
the message to the neighbors of this process. In decision 50 routine continues at block 2813. In decision block 2802, if
block 2303, if a newly connected neighbor is waiting to the forwarding distance is greater than Zero, then the random
receive messages, then the routine continues at block 2304, walk is not complete and the routine continues at block
else the routine returns. In block 2304, the routine sends the 2803, else the routine continues at block 2804. In block
messages in the correct order if possible for each originating 2803, the routine invokes the forward connection edge
proceSS and then returns. 55 Search routine passing the identification of the requesting
FIG.24 is a flow diagram illustrating the processing of the process and the decremented forwarding distance. The rou
distribute broadcast message routine in one embodiment. tine then continues at block 2815. In decision block 2804, if
This routine Sends the broadcast message to each of the the requesting proceSS is a neighbor or the edge between this
neighbors of this process, except for the neighbor who sent process and the Sending neighbor is reserved because it has
the message to this process. In block 2401, the routine 60 already been offered to a process, then the routine continues
Selects the next neighbor other than the neighbor who sent at block 2805, else the routine continues at block 2806. In
the message. In decision block 2402, if all such neighbors block 2805, the routine invokes the forward connection edge
have already been Selected, then the routine returns. In block Search routine passing an indication of the requesting party
2403, the routine sends the message to the selected neighbor and a toggle indicator that alternatively indicates to continue
and then loops to block 2401 to select the next neighbor. 65 the random walk for one or two more computers. The routine
FIG. 26 is a flow diagram illustrating the processing of the then continues at block 2815. In block 2806, the routine dials
handle connection port Search Statement routine in one the requesting process via the call-in port. In block 2807, the
US 6,829,634 B1
27 28
proposal resp). ASSuming that the response is Successfully broadcast channel. In decision block 3102, if a message was
received, the routine continues at block 2808. In decision retrieved, then the routine returns an indication of Success,
this process and the sending neighbor. In block 2810, the of the handle condition check message in one embodiment.
removes the Sending neighbor as a neighbor. In block 2812, process. In decision block 3201, if the number of holes of
the routine hangs up the external port and continues at block this process is equal to one, then the routine continues at
2815. In decision block 2813, if this process is the requesting block 3202, else the neighbors with empty ports condition
proceSS and the number of holes of this proceSS equals one, 15
does not exist any more and the routine returns. In decision
then the routine continues at block 2814, else the routine block 3202, if the sending neighbor and this process have the
continues at block 2815. In block 2814, the routine invokes same set of neighbors, the routine continues at block 3203,
the fill hole routine. In block 2815, the routine sends an else the routine continues at block 3205. In block 3203, the
connection edge Search response message (i.e., connection routine initializes a condition double check message (i.e.,
edge Search response) to the Sending neighbor indicating condition double check) with the list of neighbors of this
acknowledgement and then returns. The graphs are Sensitive process. In block 3204, the routine Sends the message
internally to a neighbor other than Sending neighbor. The
to parity. That is, all possible paths starting from a node and routine then returns. In block 3205, the routine selects a
ending at that node will have an even length unless the graph neighbor of the Sending process that is not also a neighbor
has a cycle whose length is odd. The broadcaster component of this process. In block 3206, the routine sends a condition
uses a toggle indicator to vary the random walk distance 25 repair message (i.e., condition repair Stmt) externally to
between even and odd distances. the selected process. In block 3207, the routine invokes the
FIG.29 is a flow diagram illustrating the processing of the add neighbor routine to add the Selected neighbor as a
handle connection edge Search response routine in one neighbor of this process and then returns.
embodiment. This routine is passed as indication of the FIG. 33 is a flow diagram illustrating processing of the
requesting process, the Sending neighbor, and the message. handle condition repair Statement routine in one embodi
In block 2901, the routine notes that the connection edge ment. This routine removes an existing neighbor and con
Search response (i.e., connection edge Search resp) has nects to the process that sent the message. In decision block
been received and if the forwarding distance is less than or 3301, if this process has no holes, then the routine continues
equal to one unreserves the edge between this proceSS and at block 3302, else the routine continues at block 3304. In
the sending neighbor. In decision block 2902, if the request 35 block 3302, the routine sects a neighbor that is not involved
ing process indicates that the edge is acceptable as indicated the neighbors with empty ports condition. In block 3303, the
in the message, then the routine continues at block 2903, else routine removes the Selected neighbor as a neighbor of this
the routine returns. In block 2903, the routine reserves the process. Thus, this process that is executing the routine now
edge between this proceSS and the Sending neighbor. In has at least one hole. In block 3304, the routine invokes the
block 2904, the routine removes the sending neighbor as a 40 add neighbor routine to add the process that Sent the message
neighbor. In block 2905, the routine invokes the court as a neighbor of this process. The routine then returns.
neighbor routine to connect to the requesting process. In FIG. 34 is a flow diagram illustrating the processing of the
decision block 2906, if the invoked routine was handle condition double check routine. This routine deter
unsuccessful, then the routine continues at block 2907, else mines whether the neighbors with empty ports condition
the routine returns. In decision block 2907, if the number of 45 really is a problem or whether the broadcast channel is in the
holes of this proceSS is greater than Zero, then the routine small regime. In decision block 3401, if this process has one
continues at block 2908, else the routine returns. In block hole, then the routine continues at block 3402, else the
2908, the routine invokes the fill hole routine and then routine continues at block 3403. If this process does not have
returns. one hole, then the Set of neighbors of this proceSS is not the
FIG.30 is a flow diagram illustrating the processing of the 50 Same as the Set of neighbors of the Sending process. In
broadcast routine in one embodiment. This routine is decision block 3402, if this proceSS and the Sending process
invoked by the application program to broadcast a message have the same Set of neighbors, then the broadcast channel
on the broadcast channel. This routine is passed the message is not in the Small regime and the routine continues at block
to be broadcast. In decision block 3001, if this process has 3403, else the routine continues at block 3406. In decision
at least one neighbor, then the routine continues at block 55 block 3403, if this process has no holes, then the routine
3002, else the routine returns since it is the only process returns, else the routine continues at block 3404. In block
connected to be broadcast channel. In block 3002, the 3404, the routine sets the estimated diameter for this process
routine generates an internal message of the broadcast to one. In block 3405, the routine broadcasts a diameter reset
statement type (i.e., broadcast stmt). In block 3003, the internal message (i.e., diameter reset) indicating that the
routine Sets the Sequence number of the message. In block 60 estimated diameter is one and then returns. In block 3406,
3004, the routine invokes the distribute internal message the routine creates a list of neighbors of this process. In
routine to broadcast the message on the broadcast channel. block 3407, the routine sends the condition check message
The routine returns. (i.e., condition check Stmt) with the list of neighbors to
FIG.31 is a flow diagram illustrating the processing of the the neighbor who sent the condition double check message
acquire message routine in one embodiment. The acquire 65 and then returns.
message routine may be invoked by the application program From the above description, it will be appreciated that
or by a callback routine provided by the application pro although specific embodiments of the technology have been
US 6,829,634 B1
29 30
described, various modifications may be made without devi 11. The broadcast channel of claim 10 wherein the broad
ating from the Spirit and Scope of the invention. For cast component disregards received data that it has already
example, the communications on the broadcast channel may Sent to its neighbor participants.
be encrypted. Also, the channel instance or Session identifier 12. The broadcast channel of claim 10 wherein a partici
may be a very large number (e.g., 128 bits) to help prevent 5 pant connects to the broadcast channel by contacting a
an unauthorized user to maliciously tap into a broadcast participant already connected to the broadcast channel.
channel. The portal computer may also enforce Security and 13. The broadcast channel of claim 10 wherein each
not allow an unauthorized user to connect to the broadcast participant is a computer process.
channel. Accordingly, the invention is not limited except by 14. The broadcast channel of claim 10 wherein each
the claims.
What is claim is:
participant is a computer thread.
15. The broadcast channel of claim 10 wherein each
1. A non-routing table based computer network having a participant is a computer.
plurality of participants, each participant having connections 16. The broadcast channel of claim 10 wherein the
to at least three neighbor participants, wherein an originating
participant sends data to the other participants by Sending 15 communications network uses TCP/IP protocol.
the data through each of its connections to its neighbor 17. The broadcast channel of claim 10 wherein the
participants, wherein each participant Sends data that it communications network is the Internet.
receives from a neighbor participant to its other neighbor 18. The broadcast channel of claim 10 wherein the
participants, wherein data is numbered Sequentially So that participants are peers.
data received out of order can be queued and rearranged, 19. A non-routing table based computer-readable medium
further wherein the network is m-regular and m-connected, containing instructions for controlling communications of a
where m is the number of neighbor participants of each participant of a broadcast channel within a network, by a
participant, and further wherein the number of participants method comprising:
is at least two greater than m thus resulting in a non locating a portal computer;
complete graph. 25
requesting the located portal computer to provide an
2. The computer network of claim 1 wherein each par indication of neighbor participants to which the par
ticipant is connected to 4 other participants.
3. The computer network of claim 1 wherein each par ticipant can be connected;
ticipant is connected to an even number of other participants. receiving the indications of the neighbor participants, and
4. The computer network of claim 1 wherein all the establishing a connection between the participant and
participants are peers. each of the indicated neighbor participants, wherein a
5. The computer network of claim 1 wherein the connec connection between the portal computer and the par
tions are peer-to-peer connections. ticipant is not established, wherein a connection
6. The computer network of claim 1 wherein the connec between the portal computer and the neighbor partici
tions are TCP/IP connections. 35
pants is not established, further wherein the network is
7. The computer network of claim 1 wherein each par m-regular and m-connected, where m is the number of
ticipant is a proceSS eXecuting on a computer. neighbor participants of each participant, and further
8. The computer network of claim 1 wherein a computer wherein the number of participants is at least two
hosts more than one participant. greater than m thus resulting in a non-complete graph.
9. The computer network of claim 1 wherein each par 40
20. The computer-readable medium of claim 19 wherein
ticipant Sends to each of its neighbors only one copy of the each participant is a computer process.
data.
10. A non-routing table based broadcast channel for 21. The computer-readable medium of claim 19 wherein
participants, comprising: the indicated participants are computer processes executing
45 on different computer Systems.
a communications network that provides peer-to-peer 22. The computer-readable medium of claim 19 includ
communications between the participants connected to ing:
the broadcast channel; and receiving data from a neighbor participant of the partici
for each participant connected to the broadcast channel, pant; and
an indication of four neighbor participants of that 50 transmitting the received data to the other neighbor par
participant; and ticipants.
a broadcast component that receives data from a neighbor 23. The computer-readable medium of claim 19 includ
participant using the communications network and that ing:
Sends the received data to its other neighbor partici receiving a request to connect to another participant;
pants to effect the broadcasting of the data to each 55
participant of the to broadcast channel, wherein the disconnecting from a neighbor participant; and
network is m-regular and m-connected, where m is the connecting to the other participant.
number of neighbor participants of each participant, 24. The computer-readable medium of claim 19 wherein
and further wherein the number of participants is at the connections are established using the TCP/IP protocol.
least two greater than m thus resulting in a non
complete graph. k k k k k
EXHIBIT 4
USOO6732147B1

Holt et al. (45) Date of Patent: May 4, 2004
(54) LEAVING A BROADCAST CHANNEL OTHER PUBLICATIONS

(75) Inventors: Fred B. Holt, Seattle, WA (US); Virgil Bondy et al. “Graph Theory With Applications' American
E. Bourassa, Bellevue, WA (US) Elsevier Publishing Co. Inc. pp. 47–50 Secion 3.3.*
Yavatkar et al. “A Reliable Dissemination Protocol for
(73) Assignee: The Boeing Company, Seattle, WA Interactive Collaborative Applications” Proc. ACM Multi
(US) media, 1995 p.333-344 http:/citeseer.nj.nec.com/article/
yavatkar95reliable.html.*
(*) Notice: Subject to any disclaimer, the term of this Alagar, S. and Venkatesan, S., “Reliable Broadcast in
patent is extended or adjusted under 35 Mobile Wireless Networks,” Department of Computer Sci
U.S.C. 154(b) by 719 days. ence, University of Texas at Dallas, Military Communica
tions Conference, 1995, MILCOM 95 Conference Record,
(21) Appl. No.: 09/629,577 IEEE San Diego, California, Nov. 5–8, 1995 (pp. 236-240).
1-1. International Search Report for The Boeing Company, Inter
(22) Filed: Jul. 31, 2000 national Patent Application No. PCT/US01/24240, Jun. 5,
(51) Int. Cl." ................................................ G06F 15/16 2002 (7 pages).
(52) U.S. Cl. ........................................ 709/204; 709/227 U.S. patent application Ser. No. 09/629,570, Bourassa et al.,
(58) Field of Search ................................. 709/204, 227, filed Jul. 31, 2000.
709/217 U.S. patent application Ser. No. 09/629,576, Bourassa et al.,
filed Jul. 31, 2000.
(56) References Cited U.S. patent application Ser. No. 09/629,575, Bourassa et al.,
U.S. PATENT DOCUMENTS filed Jul. 31, 2000.
U.S. patent application Ser. No.09/629,572, Bourassa et al.,
4.912,656 A 3/1990 Cain et al. filed Jul. 31, 2000.
5,056,085. A 10/1991 Vu s
5,309.437 A 5/1994 Perlman et al.
5,426,637 A 6/1995 Derby et al. (List continued on next page.)
55. A 16:3.
5636,371 A
St.
6/1997 Yu
Primary Examiner-Patrice Winder
ASSistant Examiner-David Lazaro
5,673,265 A 9/1997 Gupta et al. (74) Attorney, Agent, or Firm-Perkins Coie LLP
5,696,903 A 12/1997 Mahan
5,732,074 A 3/1998 Spaur e al. (57) ABSTRACT
5,732.219 A 3/1998 Blumer et al. A method for leaving a multicast computer network is
5,734,865 A 3/1998 Yu disclosed. The method allows for the disconnection of a first
5,754,830 A 5/1998 Butts et al. computer from a Second computer. When the first computer
5,761,425 A 6/1998 Miller decides to disconnect from the Second computer, the first
5,764,756 A 6/1998 Onweller computer Sends a disconnect message to the Second com
5,790,548 A 8/1998 Sistanizadeh et al. puter. Then, when the Second computer receives the discon
5,790,553 A 8/1998 Deaton, Jr. et al. nect message from the first computer, the Second computer
5,799,016 A 8/1998 Onweller broadcasts a connection port Search message to find a third
5,802.285 A 9/1998 Hirviniemi computer to which it can connect.
5,864,711 A 1/1999 Mairs et al.
(List continued on next page.) 16 Claims, 39 Drawing Sheets
US 6,732,147 B1
Page 2
U.S. PATENT DOCUMENTS The Gamer's Guide, “First-Person Shooters,” Oct. 20, 1998
5,867,660 A 2/1999 Schmidt et al.
(4 pages).
5,867,667 A 2/1999 Butman et al. The O'Reilly Network, “Gnutella: Alive, Well, and Chang
5,870,605 A 2/1999 Bracho et al. ing Fast,” Jan. 25, 2001 (5 pages) http://www.open2p.com/
5,874,960 A 2/1999 Mairs et al. lpt/ . . . Accessed Jan. 29, 2002).
5,899.980 A 5/1999 Wilf et al. Oram, Andy, “Gnutella and Freenet Represents True Tech
5,907,610 A 5/1999 Onweller nological Innovation,” May 12, 2000 (7 pages) The O'Reilly
5,928,335 A 7/1999 Morita Network http://www.oreillynet.com/1pt . . . Accessed Jan.
5,935,215 A 8/1999 Bell et al.
5,946,316 A * 8/1999 Chen et al. ................. 370/408 29, 2002).
5.948,054 A 9/1999 Nielsen Internetworking Technologies Handbook, Chapter 43 (pp.
5,949.975 A 9/1999 Batty et al. 43-1-43-16).
5,956.484 A 9/1999 Rosenberg et al. Oram, Andy, “Peer-to-Peer Makes the Internet Interesting
5.974.043 A 10/1999 Solomon Again,” Sep. 22, 2000 (7 pages) The O'Reilly Network
5.987,506 A 11/1999 Carter et al.
6,003,088 A 12/1999 Houston et al. http://linux.oreillynet.com/1pt. . . Accessed Jan. 29, 2002).
6,013,107 A 1/2000 Blackshear et al. Monte, Richard, “The Random Walk for Dummies, MIT
6,023,734. A 2/2000 Ratcliff et al. Undergraduate Journal of Mathematics (pp. 143-148).
6,029,171 A 2/2000 Smiga et al. Srinivasan, R., “XDR: External Data Representation Stan
6,032,188 A 2/2000 Mairs et al.
6,038,602 A 3/2000 Ishikawa dard.” Sun Microsystems, Aug. 1995 (20 pages) Internet
6,047.289 A 4/2000 Thorne et al. RFC/STD/FYI/BCP Archives http://www.faqs.org/rfcs/
6,073,177 A * 6/2000 Hebel et al. ................ 709/228 rfc1832.html Accessed Jan. 29, 2002).
6,094,676 A 7/2000 Gray et al. A Databeam Corporate White Paper, “A Primer on the T120
6,199,116 B1 3/2001 May et al. Series Standards,” Copyright 1995 (pp. 1-16).
6,216,177 B1 4/2001 Mairs et al.
6,223,212 B1 4/2001 Batty et al. Kessler, Gary, C., “An Overview of TCP/IP Protocols and
6.243,691 B1 6/2001 Fisher et al. the Internet,” Apr. 23, 1999 (23 pages) Hill Associates, Inc.
6,252,884 B1 6/2001 Hunter ....................... 370/443 http://www.hill.com/library/publications/t . . . Accessed
6,268.855 B1 7/2001 Mairs et al. Jan. 29, 2002).
6,271,839 B1 8/2001 Mairs et al. Bondy, J.A., and Murty, U.S.R., “Graph Theory with Appli
6,285,363 B1 9/2001 Mairs et al. cations,” Chapters 1-3 (pp. 1-47), 1976 American Elsevier
6,304,928 B1 10/2001 Mairs et al. Publishing Co., Inc., New York, New York.
6,353,599 B1 * 3/2002 Bi et al. ..................... 370/328
6,618,752 B1 * 9/2003 Moore et al. ............... 709/217 Cormen, Thomas H. et al., Introduction to Algorithms,
OTHER PUBLICATIONS
Chapter 5.3 (pp. 84-91), Chapter 12 (pp. 218-243), Chapter
13 (p. 245), 1990, The MIT Press, Cambridge, Massachu
U.S. patent application Ser. No. 09/629,023, Bourassa et al., setts, McGraw-Hill Book Company, New York.
filed Jul. 31, 2000. The Common Object Request Broker: Architecture and
U.S. patent application Ser. No. 09/629,043, Bourassa et al., Specification, Revision 2.6, Dec. 2001, Chapter 12 (pp.
filed Jul. 31, 2000. 12-1-12-10), Chapter 13 (pp. 13-1-13-56) Chapter 16 (pp.
U.S. patent application Ser. No. 09/629,024, Bourassa et al., 16-1-16-26), Chapter 18 (pp. 18-1-18-52), Chapter 20
filed Jul. 31, 2000. (pp. 20-1-20–22).
U.S. patent application Ser. No. 09/629,042, Bourassa et al., The University of Warwick, Computer Science Open Days,
filed Jul. 31, 2000. “Demonstration on the Problems of Distributed Systems.”
Murphy, Patricia, A., “The Next Generation Networking http://www.dcs. Warwick.ac.u. . . . Accessed Jan. 29, 2002).
Paradigm: Producer/Consumer Model.” Dedicated Systems
Magazine-2000 (pp. 26-28). * cited by examiner
U.S. Patent MaV 4, 2004 Sheet 1 of 39

U.S. Patent May 4, 2004 Sheet 2 of 39 US 6,732,147 B1



U.S. Patent May 4, 2004 Sheet 7 of 39


U.S. Patent May 4, 2004 Sheet 9 of 39 US 6, 732,147 B1

U.S. Patent May 4, 2004 Sheet 10 Of 39 US 6,732,147 B1

O<
S
s
O
/NO m N
Z09
(81),
9
00/
0
4.
ºff,
Z
?sep org 8stI an b

19 u O g 901
1sep ouq ar?nboy 38esau
QA190YI
Channel Instance,
801
Connect Aux Info)
Open call in port
802 Fig. 8
Set connect-time
803
instance)
804
<C N Return (false)

Y
805 806
Y e
N 807
808
809
ceae

902
Select next depth
All depths selected Return (failure)
Select next portal computer

Fig. 9
905
selected
N 906
Y 908
conic proces 909

computer
910
Return (success)
Contact process
1 OO1
10O2
1005
Add as connected portal Y Answering process
computer connected
Connect request
Fig. II 1101
Was a fully 1102
connected portal found

computer
1104
N
Y 1105
1112
Y Add neighbor
Ready to connect
N 1113
Check for externa

call
Fig. 12
ype = F seeking
connection call

Achieve connection
1301
Fig. 13
connected
1302
1303

1401
Pick up and receive
external message
Message
1404
Handle seeking
connection call
1406
Handle connection
request call
1408
Handle edge proposal
call
1410
Port connect call Handle port
connection call
Handle connected
Connected statement Statement

statement repair statement
Handle seeking Fig. I 5

connection call
1502 1503
connected connected
1504
Add other as fellow
seeking process
1505
andle connection
s request call
Connecteddin
Set newcomer's
holes to expect
1601
1 6O3
1 6 O4
SO2

response
1 6O 5 Fig. 16

Set newcomer's
holes to fill
608 SOS
v Add neighbor
610
Holes to fill
> = Z.
Holes to fill - F Z
1616 617
ewcomer's Y u
holes to fill > 9 > Fill hole (requestor)
Add neighbor
1701
Identifies calling party Fig. I 7
1702
Sets neighbor to
messages pending
703 17O4
g d Y Connection state =
<ssingess-
N
partially connected
1705
Add as neighbor
17O6
for new neighbor
1707 1708
N
1709 1C
expected hole Y Achieve connected

1711 1712
1

Fig. 18
Distance Y
remaining > 0
N
1802
# of Y
neighbors
>
1804
requestor
All neighbors
selected
N

Fig. 19
N 1903
proposed
neighbors
1911 pending
1907
1912
N 1908
Holes odd N
Y
1913 Y 1909
Fill hole
Add edge as pending
1910
Add neighbor
Handle port
connection call
2OO3
not ok)

2006
Add neighbor
2008
Connect request
Fill hole
2101
Fig. 21 Initialize internal
message
2102
party the request
ing part
2103
Internal
dispatcher
22O1
Fig. 22 Received internal message
22O2
Assess diameter
This
process = -
originating
Insert message into

Type 2005
Statement message
Type 22O6 2007

== shutdown Y Handle shutdown
Statement Statement
Y Pending
connection buffer
as N. N
210
message queue
N
Handle broadcast origin

Fig. 23 2301 message
message
23O2
message
2304
Distribute message
Fig. 24 from neighbor
24O1
All neighbor
selected
2403
Send internal
message

for search message
26O1
Distribute internal d
Fig. 26
602
N
603 2604
Is requestor
a neighbor
605
enerate
condition check
message w/neighbors
2607
to requestor
Prospect
Fig. 27 701
Is prospect
a neighbor
N
2702
Dial prospect
Send and receive

external message
Add neighbor
27O6
Hang up prospect

edge search call IIIeSSage
Fig. 28
Not
my message ll
holes >= Z.
2813
Message
N<1rom this pt, && Forward
Remaining connection second
distance > 0 edge (requestor
Y 2814 remaining dist -l)
Fill hole (self)
A> 2815
Requestor
a
Forward
connection edge
eld litela is neighbor or edge search (requestor,
reserved
message (from O)
neighbor, ack
N 2806 Wy
2807
Send and receive
external message

neighbor
281O
U.S. Patent May 4, 2004 Sheet 34 0f 39 US 6,732,147 B1

resp. from neighbor
search response
Y 2908
Fill hole (self)
message
Fig. 30
Generate internal
message
3003
number
3004
Distribute internal
message
Acquire message message
31 O1
Fig. 31
Pop message queue
Message Return false

retrieved
Same set of
neighbors
32O5
Set up message with list Select a neighbor
of neighbors of sending process
not my neighbor
32O6
32O7
Add neighbor
Handle condition
repair statement
Fig. 33

Remove selected
neighbor
3304
Add neighbor
Same set of
neighbors
N
3405
US 6,732,147 B1
1 2
LEAVING A BROADCAST CHANNEL ticularly well Suited to sharing of information among many
participants. In particular, when a client Stores information
CROSS-REFERENCE TO RELATED to be shared at the Server, each other client would need to
APPLICATIONS poll the Server to determine that new information is being
5 shared. Such polling places a very high overhead on the
No. 09/629,576, entitled “BROAD CASTING
communications network. Alternatively, each client may
NETWORK,” filed on Jul. 31, 2000; U.S. patent application register a callback with the Server, which the Server then
invokes when new information is available to be shared.
Ser. No. 09/629,570, entitled “JOINING A BROADCAST Such a callback technique presents a performance bottleneck
CHANNEL,” filed on Jul. 31, 2000; U.S. patent application because a Single Server needs to call back to each client
Ser. No. 09/629,577, “LEAVING A BROADCAST whenever new information is to be shared. In addition, the
CHANNEL,” filed on Jul. 31, 2000; U.S. patent application reliability of the entire Sharing of information depends upon
Ser. No. 09/629,575, entitled “BROADCASTING ON A the reliability of the Single Server. Thus, a failure at a single
BROADCAST CHANNEL, filed on Jul. 31, 2000; U.S. computer (i.e., the server) would prevent communications
patent application Ser. No. 09/629,572, entitled “CON 15 between any of the clients.
TACTING A BROADCAST CHANNEL, filed on Jul. 31, The multicasting network protocols allow the Sending of
2000; U.S. patent application Ser. No. 09/629,023, entitled broadcast messages to multiple recipients of a network. The
“DISTRIBUTED AUCTION SYSTEM,” filed on Jul. 31, current implementations of Such multicasting network pro
2000; U.S. patent application Ser. No. 09/629,043, entitled tocols tend to place an unacceptable overhead on the under
“AN INFORMATION DELIVERY SERVICE,” filed on Jul. lying network. For example, UDP multicasting would
31, 2000; U.S. patent application Ser. No. 09/629,024, Swamp the Internet when trying to locate all possible par
entitled “DISTRIBUTED CONFERENCING SYSTEM,” ticipants. IP multicasting has other problems that include
filed on Jul. 31, 2000; and U.S. patent application Ser. No. needing special-purpose infrastructure (e.g., routers) to Sup
09/629,042, entitled “DISTRIBUTED GAME port the Sharing of information efficiently.
ENVIRONMENT, filed on Jul. 31, 2000, the disclosures of 25
which are incorporated herein by reference. The peer-to-peer middleware communications Systems
TECHNICAL FIELD point-to-point network protocols. Such peer-to-peer middle
BACKGROUND of the connections used for Sharing the information. Thus, it
There are a wide variety of computer network communi 35
ware Systems when more than a Small number of partici
client/server middleware, multicasting network protocols, the T120 Internet standard is a tree structure, which relies on
none is particularly well Suited to the Simultaneous sharing 40
order to be received by all participants.
For example, collaborative processing applications, Such as network that is Suitable for the Simultaneous Sharing of
information in a timely manner to all participants who may widely distributed.
be geographically distributed. 45 BRIEF DESCRIPTION OF THE DRAWINGS
The point-to-point network protocols, such as UNIX
pipes, TCP/IP, and UDP, allow processes on different com FIG. 1 illustrates a graph that is 4-regular and 4-connected
puters to communicate via point-to-point connections. The which represents a broadcast channel.
interconnection of all participants using point-to-point FIG. 2 illustrates a graph representing 20 computers
connections, while theoretically possible, does not Scale well 50 connected to a broadcast channel.
as a number of participants grows. For example, each FIGS. 3A and 3B illustrate the process of connecting a
participating process would need to manage its direct con new computer Z to the broadcast channel.
nections to all other participating processes. Programmers, FIG. 4A illustrates the broadcast channel of FIG. 1 with
however, find it very difficult to manage Single connections, an added computer.
and management of multiple connections is much more 55 FIG. 4B illustrates the broadcast channel of FIG. 4A with
to the number of direct connections that they can Support. FIG. 4C also illustrates the broadcast channel of FIG. 4A
This limits the number of possible participants in the Sharing
of information. with an added computer.
The client/server middleware systems provide a server 60 FIG. 5A illustrates the disconnecting of a computer from
that coordinates the communications between the various the broadcast channel in a planned manner.
clients who are sharing the information. The Server functions FIG. 5B illustrates the disconnecting of a computer from
as a central authority for controlling access to shared the broadcast channel in an unplanned manner.
resources. Examples of client/server middleware Systems FIG. 5C illustrates the neighbors with empty ports con
include remote procedure calls (“RPC"), database servers, 65 dition.
and the common object request broker architecture FIG. 5D illustrates two computers that are not neighbors
(“CORBA). Client/server middleware systems are not par who now have empty ports.
US 6,732,147 B1
3 4
FIG. 5E illustrates the neighbors with empty ports con FIG. 34 is a flow diagram illustrating the processing of the
large regime.
FIG. 8 is a flow diagram illustrating the processing of the 1O nel. In one embodiment, the broadcast technique provides a
contact process routine in one embodiment. 15 channel. Each computer that is connected to the broadcast
connected. The logical broadcast channel is implemented
FIG. 15 is a flow diagram illustrating the processing of the 25 The broadcast technique overlays the underlying network
each computer is connected to four other computers, referred
FIG. 17 is a flow diagram illustrating the processing of the to as neighbors. (Actually, a process executing on a com
forward connection edge Search routine in one embodiment. originating computer sends the message to each of its
handle edge proposal call routine. 35 puter that receives the message then sends the message to its
FIG.22 is a flow diagram illustrating the processing of the 40 channel. A graph in which each node is connected to four
4-regular graph means that a computer would become
FIG.24 is a flow diagram illustrating the processing of the 45 broadcast technique also has the property that it would take
FIG. 26 is a flow diagram illustrating the processing of the Sub-graphs, that is two Separate broadcast channels. This
FIG.27 is a flow diagram illustrating the processing of the 50 FIG. 1 illustrates a graph that is 4-regular and 4-connected
FIG. 28 is a flow diagram illustrating the processing of the nodes A-I represents a computer that is connected to the
FIG.29 is a flow diagram illustrating the processing of the 55 channel. The time it takes to broadcast a message to each
embodiment. the connections between the computers and the number of
60 number of connections that a message would need to
FIG.31 is a flow diagram illustrating the processing of the traverse between each pair of computerS is the “distance'
acquire message routine in one embodiment. between the computers (i.e., the shortest path between the
FIG. 32 is a flow diagram illustrating processing of the two nodes of the graph). For example, the distance between
FIG. 33 is a flow diagram illustrating processing of the 65 connected to computer F. The distance between computerSA
ment. computers A and B, but computer F is directly connected to
US 6,732,147 B1
S 6
computer B. Thus, a message originating at computer A Dare the two pairs that are identified as the neighbors for the
would be sent directly to computer F, and then sent from new computer Z. The connections between each of these
nel. The diameter of the broadcast channel represented by 5 by FIG.3B. The process of breaking the connection between
computers connected to a-broadcast channel. The diameter Stretched and pinned to a new node.
graph), (2) the broadcasting of messages over the broadcast 15 tions between internal ports of neighbors are referred to as
disconnecting of computers from the broadcast channel (i.e., broadcast channel form the 4-regular and 4-connected
small regime is described below in detail. When five or more is shared among all the processes that may execute on-that
computer is to connect. Each of these four computers then 45 computers until a portal computer "answers,” a call on its
ing of the Seeking computer to the broadcast channel. A to or attempting to connect to the broadcast channel and its
“Seeking connection State.” A computer that is connected to 50 receives a call on its call-in port, it transferS the call to
"partially connected State.” A computer that is currently, or nicates through that transfer-to port, which is the external
Since the broadcast channel is a 4-regular graph, each of 55 computer then communicates via that external port to
puters. Thus, Some connections between computers need to Seeking computer to the broadcast channel. The Seeking
identifies two pairs of computers that are currently con 60 order. AS discussed below in detail, the broadcast technique
the connection between them, and then each of the four which may result in improved performance.
computer Z connecting to the broadcast channel. FIG. 3A 65 the found portal computer or directly connected to one of its
nected. The pairs of computers B and E and computers C and identifying the neighbors for the Seeking computer is that the
US 6,732,147 B1
7 8
diameter of the broadcast channel may increase when each receiving computer. The Second message only has to travel
Seeking computer uses the same found portal computer and a distance of one. Thus, it is possible for the Second message
establishes a connection to the broadcast channel directly to reach the receiving computer before the first message.
through that found portal computer. Conceptually, the graph When the broadcast channel is in a steady state (i.e., no
becomes elongated in the direction of where the new nodes computers connecting or disconnecting from the broadcast
are added. FIGS. 4A-4C illustrate that possible problem. channel), out-of-order messages are not a problem because
FIG. 4A illustrates the broadcast channel of FIG. 1 with an each computer will eventually receive both messages and
added computer. Computer J was connected to the broadcast can queue messages until all earlier ordered messages are
channel by edge pinning edges C-D and E-H to computer received. If, however, the broadcast channel is not in a
J. The diameter of this broadcast channel is still two. FIG. Steady State, then problems can occur. In particular, a com
new neighbors. When a new neighbor eventually receives
cast channel by edge pinning edges E-J and B-C to com the first message, it sends the message to the newly con
puter K. The diameter of this broadcast channel is three, nected computer. Thus, the newly connected computer will
because the Shortest path from computer G to computer Kis 15
receive the first message, but will not receive the Second
through edges G-A, A-E, and E-K. FIG. 4C also illustrates message. If the newly connected computer needs to process
the broadcast channel of FIG. 4A with an added computer. the messages in order, it would wait indefinitely for the
Computer K was connected to the broadcast channel by edge Second message.
pinning edges D-G and E-J to computer K. The diameter of One Solution to this problem is to have each computer
this broadcast channel is, however, still two. Thus, the queue all the messages that it receives until it can Send them
Selection of neighbors impacts the diameter of the broadcast in their proper order to its neighbors. This Solution, however,
channel. To help minimize the diameter, the broadcast may tend to Slow down the propagation of messages through
technique uses a random Selection technique to identify the the computers of the broadcast channel. Another Solution
four neighbors of a computer in the Seeking connection State. that may have leSS impact on the propagation Speed is to
The random Selection technique tends to distribute the 25 queue messages only at computers who are neighbors of the
connections to new Seeking computers throughout the com newly, connected computers. Each already connected neigh
puters of the broadcast channel which may result in Smaller bor would forward messages as it receives them to its other
overall diameters. neighbors who are not newly connected, but not to the newly
Broadcasting Through the Graph connected neighbor. The already connected neighbor would
AS described above, each computer that is connected to only forward messages from each originating computer to
the broadcast channel can broadcast messages onto the the newly connected computer when it can ensure that no
broadcast channel and does receive all messages that are gaps in the messages from that originating computer will
broadcast on the broadcast channel. The computer that occur. In one embodiment, the already connected neighbor
originates a message to be broadcast sends that message to may track the highest Sequence number of the messages
each of its four neighbors using the internal connections. 35 already received and forwarded on from each originating
When a computer receives a broadcast message from a computer. The already connected computer will Send only
neighbor, it sends the message to its three other neighbors. higher numbered messages from the originating computers
Each computer on the broadcast channel, except the origi to the newly connected computer. Once all lower numbered
nating computer, will thus receive a copy of each broadcast messages have been received from all originating
message from each of its four neighbors. Each computer, 40 computers, then the already connected computer can treat
however, only sends the first copy of the message that it the newly connected computer as its other neighbors and
receives to its neighbors and disregards Subsequently Simply forward each message as it is received. In another
received copies. Thus, the total number of copies of a embodiment, each computer may queue messages and only
message that is Sent between the computerS is 3N+1, where forwards to the newly connected computer those messages
N is the number of computers connected to the broadcast 45 as the gaps are filled in. For example, a computer might
channel. Each computer Sends three copies of the message, receive messages 4 and 5 and then receive message 3. In
except for the originating computer, which Sends four copies Such a case, the already connected computer would forward
of the message. queue messages 4 and 5. When message 3 is finally received,
The redundancy of the message Sending helps to ensure the already connected computer will Send messages 3, 4, and
the overall reliability of the broadcast channel. Since each 50 5 to the newly connected computer. If messages 4 and 5 were
computer has four connections to the broadcast channel, if Sent to the newly connected computer before message 3,
one computer fails during the broadcast of a message, its then the newly connected computer would process messages
neighbors have three other connections through which they 4 and 5 and disregard message 3. Because the already
will receive copies of the broadcast message. Also, if the connected computer queues messages 4 and 5, the newly
internal connection between two computers is slow, each 55 connected computer will be able to proceSS message 3. It is
computer has three other connections through which it may possible that a newly connected computer will receive a Set
receive a copy of each message Sooner. of messages from an originating computer through one
Each computer that originates a message numbers its own neighbor and then receive another Set of message from the
messages Sequentially. Because of the dynamic nature of the Same originating computer through another neighbor. If the
broadcast channel and because there are many possible 60 Second Set of messages contains a message that is ordered
connection paths between computers, the messages may be earlier than the messages of the first Set received, then the
received out of order. For example, the distance between an newly connected computer may ignore that earlier ordered
originating computer and a certain receiving computer may message if the computer already processed those later
be four. After Sending the first message, the originating ordered messages.
computer and receiving computer may become neighbors 65 Decomposing the Graph
and thus the distance between them changes to one. The first A connected computer disconnects from the broadcast
message may have to travel a distance of four to reach the channel either in a planned or unplanned manner. When a
US 6,732,147 B1
9 10
receives the disconnect message, it tries to connect to one of 5 receives the list, it compares the list to its own list of
When a computer with an available internal port receives the 15 Thus, one of the original neighbors involved in the condition
puter that broadcast the message. FIGS.5A-5D illustrate the in need of a connection, the other original neighbor and the
indicated by the dashed line, and Similarly for computers E 25 It is possible that the two original neighbors with the
When a computer disconnects in an unplanned manner, neighbor that receives the condition check message deter
disconnected computer recognizes that it is short one con it determines whether it has the same Set of neighbors as the
disconnected, it broadcasts a port connection request on the 35 neighbors are different, then the computer that received the
port that needs a connection. The port connection request message to the original neighbors with the condition. The
the connection request, it communicates with the requesting 40 neighbors with the condition by Sending a condition repair
between the two computers. FIG. 5B illustrates the discon condition will have its port filled.
disconnected in an unplanned manner. When each of its 45 unplanned manner, but computers F and I responded to the
disconnection, each neighbor broadcasts a port connection together. The other former neighbors of computer H, com
A and E respond to each other's requests and establish a 50 computer E received the port connection request from
It is possible that a planned or unplanned disconnection (since they are neighbors via the internal connection) a
already connected and cannot fill their empty ports by 55 nized that computer E has a different set of neighbor (i.e., the
the “neighbors with empty ports' condition. Each neighbor Selected computer D, which is a neighbor of computer E and
receives the port connection request from the other neighbor, 60 neighbors (other than computer E), which is computer G in
empty port. Such a condition may also occur when the FIG. 5D illustrates two computers that are not neighbors
Small regime, each computer will have less than four neigh 65 computers E and G can connect to each other.
would be a problem if not repaired, the first neighbor to empty ports condition. FIG.5E illustrates the neighbors with
US 6,732,147 B1
11 12
empty ports condition in the Small regime. In this example, channel type and channel instance, it generates the same port
if computer E disconnected in an unplanned manner, then ordering. AS described below, it is possible for a computer
FIG. 5F illustrates the situation of FIG. 5E when in the 15 typically need to keep on redialing a busy port. The process
condition double check message from computer B. In this redialing. In one embodiment, each Seeking computer may
neighbors of computer C. The computer that receives the 25 a 25% chance of being first in the reordering, and So on.
neighbors, other than computer C, and tries to connect to orderings, the likelihood of finding a busy port is reduced.
Statically allocated to an application program because other 35 can connect to the broadcast channel. In one embodiment,
use conflicting port numbers. As a result, in one Seeking computer locates a portal computer that is connected
puter could simply try to locate the lowest number unused 40 A Seeking computer could Select the first portal computer
Seeking computer, however, does not know in advance the that is fully connected to the broadcast channel is found. If
puter needs to dial ports of a portal computer Starting with 45 a portal computer with Such a call-in port is found. A
portal computer. If the portal computer is connected to (or of each portal computer are dialed until a portal computer
the portal computer is not connected, then the Seeking 50 number according to the algorithm and then dials each portal
if each application program on a computer tried to allocate the broadcast channel is found, then the Seeking computer
many of the low-ordered port numbers would be used by 55 numbers, the Seeking computer first dials the port numbers
relatively slow process, it would take the Seeking computer channel. The Seeking computers may have a maximum
ordering algorithm to identify the port-number order that a 60 Seeking computer exhausts its Search depth, then either the
for its call-in port. In one embodiment, the broadcast tech Seeking computer is also a portal computer, it can then
nique uses a hashing algorithm to identify the port-order. establish the broadcast channel with itself as the first fully
The algorithm preferably distributes the ordering of the port connected computer.
numbers randomly through out the user port number Space 65 When a Seeking computer locates a portal computer that
time the algorithm is executed on any computer for a given connect when they first locate each other because the
US 6,732,147 B1
13 14
broadcast channel may already be established and accessible the forwarding computer toggles the new distance to travel
through a higher-ordered port number on another portal between Zero and one to help prevent two computers from
computer. If the two seeking computers were to connect to Sending the message back and forth between each other.
each other, then two disjoint broadcast channels would be Because of the local nature of the information maintained
formed. Each Seeking computer can share its experience in by each computer connected to the broadcast channel, the
trying to locate a portal computer with the other Seeking computers need not generally be aware of the diameter of the
computer. In particular, if one Seeking computer has broadcast channel. In one embodiment, each message Sent
Searched all the portal computers to a depth of eight, then the through the broadcast channel has a distance traveled field.
one Seeking computer can share that it has Searched to a Each computer that forwards a message increments the
depth of eight with another Seeking computer. If that other distance traveled field. Each computer also maintains an
Seeking computer has Searched to a depth of, for example, estimated diameter of the broadcast channel. When a com
only four, it can Skip Searching through depths five through puter receives a message that has traveled a distance that
eight and that other Seeking computer can advance its indicates that the estimated diameter is too Small, it updates
Searching to a depth of nine. its estimated diameter and broadcasts an estimated diameter
In one embodiment, each computer may have a different 15 message. When a computer receives an estimated diameter
Set of portal computers and a different maximum Search message that indicates a diameter that is larger than its own
broadcast channels are formed because a Seeking computer This estimated diameter is used to establish the distance that
currently connected computers. One advantage of the broad 25 may use ASCII to represent text and another computer may
has local knowledge of itself and its neighbors. This limited channel may use the XDR (“external Data Representation”)
four neighbors for a Seeking computer. 35 message may require two calls to the operating System: one
To Select the four computers, a portal computer Sends an to retrieve the Size of the next message and the other to
connections that is randomly Selected. The receiving com Such calls to the operating System can, however, be very
through one of its internal connections that is randomly 40 overcome the inefficiencies of Such repeated calls, the broad
channel. Eventually, a receiving computer will decide that technique may request the operating System to provide the
Selected computer. That receiving computer will offer the 45 cast technique can then repeatedly invoke the XDR routines
pinning. Of course, if either of the computers at the end of bytes needs to be retrieved from the operating System. The
Seeking computer, then the Seeking computer-cannot con 50 are thus more efficient than repeated System calls.
nect through that internal connection. The computer that M-Regular.
decided that the message has traveled far enough will detect In the embodiment described above, each fully connected
this condition of already being a neighbor and Send the computer has four internal connections. The broadcast tech
message to a randomly Selected neighbor. nique can be used with other numbers of internal connec
In one embodiment, the distance that the edge connection 55 tions. For example, each computer could have 6, 8, or any
broadcast channel. The message includes an indication of channel tends to decrease, and thus propagation time for a
decrements that distance to travel before Sending the mes 60 a Seeking computer to the broadcast channel may, however,
randomly Selected computer. If that randomly Selected com broadcast channel can be maintained as m-regular and
it is already connected to it), then that randomly selected 65 connections is odd, then when the broadcast channel has an
US 6,732,147 B1
15 16
In Such a situation, the broadcast network is neither broadcaster component includes a connect component 701,
m-regular nor m-connected. When the next computer con an external dispatcher 702, an internal dispatcher 703 for
nects to the broadcast channel, it can again become each internal connection, an acquire message component
m-regular and m-connected. Thus, with an odd number of 704 and a broadcast component 712. The application pro
internal connections, the broadcast channel toggles between gram may provide a connect callback component 710 and a
being and not being m-regular and m-connected. receive response component 711 that are invoked by the
Components
broadcast channel, and each computer can have multiple 15
connections to the same broadcast channel. The broadcast request component 706 to ask the portal computer (if fully
channel is well Suited for computer processes (e.g., appli connected) to select neighbor processes for the newly con
cation programs) that execute collaboratively, Such as net necting process. The external dispatcher receives external
work meeting programs. Each computer process can connect messages, identifies the type of message, and invokes the
to one or more broadcast channels. The broadcast channels appropriate handling routine 707. The internal dispatcher
can be identified by channel type (e.g., application program receives the internal messages, identifies the type of
name) and channel instance that represents separate broad message, and invokes the appropriate handling routine 708.
cast channels for that channel type. When a proceSS attempts The received broadcast messages are Stored in the broadcast
to connect to a broadcast channel, it seeks a process cur message queue 709. The acquire message component is
rently connected to that broadcast channel that is executing invoked to retrieve messages from the broadcast queue. The
on a portal computer. The Seeking process identifies the 25
broadcast component is invoked by the application program
broadcast channel by channel type and channel instance. to broadcast messages in the broadcast channel.
Computer 600 includes multiple application programs
broadcast channel to which it is connected. The broadcaster
component may be implement as an object that is instanti
ated within the proceSS Space of the application program. EXTERNAL MESSAGES
Alternatively, the broadcaster component may execute as a
Separate process or thread from the application program. In Message Type Description
one embodiment, the broadcaster component provides func 35
seeking connection call Indicates that a seeking process would like to
tions (e.g., methods of class) that can be invoked by the her the receiving process is fully
application programs. The primary functions provided may connected to the broadcast channel
include a connect function that an application program connection request call Indicates that the sending process would like
invokes passing an indication of the broadcast channel to the receiving process to initiate a connection
he sending process to the broadcast
which the application program wants to connect. The appli 40
cation program may provide a callback routine that the edge proposal call Indicates that the sending process is
broadcaster component invokes to notify the application proposing an edge through which the
program that the connection has been completed, that is the receiving process can connect to the broad
cast channel (i.e., edge pinning)
proceSS enters the fully connected State. The broadcaster port connection call Indicates that the sending process is
component may also provide an acquire message function 45 proposing a port through which the receiving
that the application program can invoke to retrieve the next process can connect to the broadcast channel
message that is broadcast on the broadcast channel. connected stimt Indicates that the sending process is
connected to the broadcast channel
Alternatively, the application program may provide a call condition repair stimt Indicates that the receiving process should
back routine (which may be a virtual function provided by disconnect from one of its neighbors and
the application program) that the broadcaster component 50 connect to one of the processes involved in
invokes to notify the application program that a broadcast
message has been received. Each broadcaster component
allocates a call-in port using the hashing algorithm. When
calls are answered at the call-in port, they are transferred to
other ports that Serve as the external and internal ports. 55
INTERNAL MESSAGES
The computers connecting to the broadcast channel may
include a central processing unit, memory, input devices Message Type Description
(e.g., keyboard and pointing device), output devices (e.g.,
display devices), and storage devices (e.g., disk drives). The broadcast stimt Indicates a message that is being broad
memory and Storage devices are computer-readable medium 60 cast through the broadcast channel for
the application programs
that may contain computer instructions that implement the connection port search stimt Indicates that the designated process is
broadcaster component. In addition, the data structures and looking for a port through which it can
connect to the broadcast channel
message Structures may be Stored or transmitted via a signal connection edge search call Indicates that the requesting process is
transmitted on a computer-readable media, Such as a com looking for an edge through which it
munications link. 65 can connect to the broadcast
FIG. 7 is a block diagram illustrating the Sub-components channel
of the broadcaster component in one embodiment. The
US 6,732,147 B1
17 18
routine continues at block 808. In block 806, the routine
-continued invokes the achieve connection routine to change the State of
INTERNAL MESSAGES
this process to fully connected. In block 807, the routine
installs the external dispatcher for processing messages
Message Type Description received through this process external port for the passed
channel type and channel instance. When a message is
connection edge search resp Indicates whether the edge between this received through that external port, the external dispatcher is
process and the sending neighbor has
been accepted by the requesting party invoked. The routine then returns. In block 808, the routine
diameter estimate stimt Indicates an estimated diameter of the installs an external dispatcher. In block 809, the routine
broadcast channel invokes the connect request routine to initiate the process of
diameter reset stimt Indicates to reset the estimated diameter
to indicated diameter
then returns.
disconnect stimt Indicates that the sending neighbor is
disconnecting from the broadcast FIG. 9 is a flow diagram illustrating the processing of the
channel Seek portal computer routine in one embodiment. This
condition check stimt Indicates that neighbors with empty port 15 routine is passed the channel type and channel instance of
condition have been detected
condition double check stimt Indicates that the neighbors with empty the broadcast channel to which this proceSS wishes to
ports have the same set of neighbors connect. This routine, for each Search depth (e.g., port
shutdown stimt Indicates that the broadcast channel is number), checks the portal computers at that search depth. If
being shutdown a portal computer is located at that Search depth with a
process that is fully connected to the broadcast channel, then
Flow Diagrams the routine returns an indication of Success. In blockS
of the broadcaster component in one embodiment. FIG. 8 is a process is located. In block 902, the routine selects the next
routine in one embodiment. This routine is passed a channel 25 decision block 903, if all the search depths have already
which this proceSS wants to connect. The routine is also indication, else the routine continues at block 904. In blocks
ceSS invokes this routine, it is in the Seeking connection In block 904, the routine Selects the next portal computer. In
and this routine connects to at least one neighbor, this 35 been selected, then the routine loops to block 902 to select
proceSS enters the partially connected State, and when the the next Search depth, else the routine continues at block
process eventually connects to four neighbors, it enters the 906. In block 906, the routine dials the selected portal
fully connected state. When in the small regime, a fully computer through the port represented by the Search depth.
connected process may have leSS than four neighbors. In In decision block 907, if the dialing was successful, then the
block 801, the routine opens the call-in port through which 40 routine continues at block 908, else the routine loops to
the process is to communicate with other processes when block 904 to select the next portal computer. The dialing will
establishing external and internal connections. The port is be successful if the dialed port is the call-in port of the
Selected as the first available port using the hashing algo broadcast channel of the passed channel type and channel
rithm described above. In block 802, the routine sets the instance of a proceSS executing on that portal computer. In
connect time to the current time. The connect time is used to 45 block 908, the routine invokes a contact process routine,
identify the instance of the process that is connected through which contacts the answering process of the portal computer
this external port. One proceSS may connect to a broadcast through the dialed port and determines whether that process
channel of a certain channel type and channel instance using is fully connected to the broadcast channel. In block 909, the
may then connect to that same broadcast channel using the 50 block 910, if the answering process is fully connected to the
block 803, the routine invokes the seek portal computer 55 process as a portal computer and processes that call. The
decision block 804, if the seek portal computer routine is 60 determines whether the process of the Selected portal com
the routine returns an unsuccessful indication. In decision routine sends an external message (i.e., seeking
computer on which the proceSS is executing was located, 65 Seeking process wants to know whether the answering
US 6,732,147 B1
19 20
Seeking connection resp), then the routine continues at invoked to identify whether a fellow Seeking process is
checks its own call-in port to see if another process is calling 1203, else the routine returns. In block 1203, the routine
broadcaster component handles the error as appropriate. In returns. In block 1205, the routine sends an external message
response message that it is fully connected to the broadcast 15 ceSS indicating that this process is also is Seeking a connec
adds the Selected portal computer to a list of connected 1207, else the routine returns. In block 1207, the routine
connect request routine in one embodiment. This routine See if any fellow Seeking process were Successful in con
being fully connected to the broadcast channel to initiate the 25 Seeking process may become the first process fully con
computer was located that is fully connected to the broadcast connection routine in one embodiment. This routine Sets the
Seeking computer may always Search its entire Search depth the connection State of this process to fully connected. In
connect to the broadcast channel. In block 1102, the routine 35 that it is fully connected by Sending a connected external
restarts the process of connecting to the broadcast channel message to them (i.e., connected stmt). In block 1303, the
and returns. In block 1103, the routine dials the process of routine invokes the connect callback routine to notify the
one of the found portal computers through the call-in port. application program and then returns.
In decision block 1104, if the dialing is successful, then the FIG. 14 is a flow diagram illustrating the processing of the
routine continues at block 1105, else the routine continues at 40 external dispatcher routine in one embodiment. This routine
block 1113. The dialing may be unsuccessful if, for example, is invoked when the external port receives a message. This
the dialed process recently disconnected from the broadcast routine retrieves the message, identifies the external mes
channel. In block 1105, the routine sends an external mes Sage type, and invokes the appropriate routine to handle that
Sage to the dialed process requesting a connection to the message. This routine loops processing each message until
broadcast channel (i.e., connection request call). In block 45 all the received messages have been handled. In block 1401,
1106, the routine receives the response message (i.e., the routine answers (e.g., picks up) the external port and
connection request resp). In decision block 1107, if the retrieves an external message. In decision block 1402, if a
response message is Successfully received, then the routine message was retrieved, then the routine continues at block
1113. In block 1108, the routine sets the expected number of 50 1415 and returns. In decision block 1403, if the message
three. In block 1109, the routine sets the estimated diameter 55 message type is for a connection request call (i.e.,
ues at block 1113. In block 1112, the routine invokes the add 60 proposal call), then the routine invokes the handle edge
cally occurs when the broadcast channel is in the Small type is port connect call (i.e., port connect call), then the
for a neighbor is performed. In block 1113, the routine hangs 65 block 1410, else the routine continues at block 1411. In
computer and then returns. Statement (i.e., connected Stmt), the routine invokes the
US 6,732,147 B1
21 22
message type is a condition repair Statement (i.e., 1616. In block 1614, the routine invokes the forward con
In block 1414, the routine hangs up on the external port and the routine decrements the holes left to fill by two and loops
identify a portal computer through which it can connect to The fill hole routine broadcasts a connection port Search
is currently fully connected to the broadcast channel iden 15 a connected process through which the calling process can
1502, the routine Sets a message to indicate that this proceSS add neighbor routine in one embodiment. This routine adds
process to a list of fellow Seeking processes. If this proceSS broadcast messages from this process. This flag is used to
broadcast channel. In block 1505, the routine sends the 25 the new neighbor. The external port becomes the internal
FIG. 16 is a flow diagram illustrating processing of the is connecting to its first neighbor and the routine continues
process to establish an internal connection with this process process. In block 1706, the routine installs an internal
fying a process to which the calling process can connect. In 35 invoked when a message is received from that new neighbor
decision block 1601, if this process is currently fully con through the internal port of that new neighbor. In decision
nected to the broadcast channel, then the routine continues block 1707, if this process buffered up messages while not
at block 1603, else the routine hangs up on the external port fully connected, then the routine continues at block 1708,
in block 1602 and returns. In block 1603, the routine sets the else the routine continues at block 1709. In one embodiment,
number of holes that the calling process should expect in the 40 a process that is partially connected may buffer the messages
response message. In block 1604, the routine Sets the that it receives is through an internal connection So that it
estimated diameter in the response message. In block 1605, can Send these messages as it connects to new neighbors. In
the routine indicates whether this proceSS is ready to connect block 1708, the routine sends the buffered messages to the
to the calling process. This process is ready to connect when new neighbor through the internal port. In decision block
the number of its holes is greater than Zero and the calling 45 1709, if the number of holes of this process equals the
process is not a neighbor of this process. In block 1606, the expected number of holes, then this process is fully con
routine Sends to the calling process an external message that nected and the routine continues at block 1710, else the
is responsive to the connection request call (i.e., routine continues at block 1711. In block 1710, the routine
connection request resp). In block 1607, the routine notes invokes the achieve connected routine to indicate that this
the number of holes that the calling process needs to fill as 50 process is fully connected. In decision block 1711, if the
indicated in the request message. In decision block 1608, if number of holes for this process is Zero, then the routine
this process is ready to connect to the calling process, then continues at block 1712, else the routine returns. In block
the routine continues at block 1609, else the routine contin 1712, the routine deletes any pending edges and then
ues at block 1611. In block 1609, the routine invokes the add returns. A pending edge is an edge that has been proposed to
neighbor routine to add the calling proceSS as a neighbor. In 55 this process for edge pinning, which in this case is no longer
block 1610, the routine decrements the number of holes that needed.
the calling proceSS needs to fill and continues at block 1611. FIG. 18 is a flow diagram illustrating the processing of the
In block 1611, the routine hangs up on the external port. In forward connection edge Search routine in one embodiment.
decision block 1612, if this process has no holes or the This routine is responsible for passing along a request to
estimated diameter is greater than one (i.e., in the large 60 connect a requesting process to a randomly Selected neigh
regime), then the routine continues at block 1613, else the bor of this process through the internal port of the Selected
routine continues at block 1616. In blocks 1613-1615, the neighbor, that is part of the random walk. In decision block
routine loops forwarding a request for an edge through 1801, if the forwarding distance remaining is greater than
which to connect to the calling process to the broadcast Zero, then the routine continues at block 1804, else the
channel. One request is forwarded for each pair of holes of 65 routine continues at block 1802. In decision block 1802, if
the calling process that needs to be filled. In decision block the number of neighbors of this proceSS is greater than one,
1613, if the number of holes of the calling process to be then the routine continues at block 1804, else this broadcast
US 6,732,147 B1
23 24
channel is in the Small regime and the routine continues at routine continues at block 2002, else the routine continues at
block 1803. In decision block 1803, if the requesting process block 2003. In decision block 2002, if the sending process
is a neighbor of this process, then the routine returns, else the is not a neighbor, then the routine continues at block 2004,
routine continues at block 1804. In blocks 1804–1807, the else the routine continues to block 2003. In block 2003, the
routine loops attempting to Send a connection edge Search routine Sends a port connection response external message
call internal message (i.e., connection edge Search call) (i.e., port connection resp) to the Sending process that
to a randomly selected neighbor. In block 1804, the routine indicates that it is not okay to connect to this process. The
randomly Selects a neighbor of this proceSS. In decision routine then returns. In block 2004, the routine sends a port
block 1805, if all the neighbors of this process have already connection response external message to the Sending process
been Selected, then the routine cannot forward the message that indicates that is okay to connect this process. In decision
and the routine returns, else the routine continues at block block 2005, if the sending of the message was successful,
1806. In block 1806, the routine sends a connection edge then the routine continues at block 2006, else the routine
Search call internal message to the Selected neighbor. In continues at block 2007. In block 2006, the routine invokes
decision block 1807, if the sending of the message is the add neighbor routine to add the Sending proceSS as a
Successful, then the routine continues at block 1808, else the 15 neighbor of this process and then returns. In block 2007, the
routine loops to block 1804 to select the next neighbor. routine hangs up the external connection. In block 2008, the
When the Sending of an internal message is unsuccessful, routine invokes the connect request routine to request that a
then the neighbor may have disconnected from the broadcast process connect to one of the holes of this process. The
channel in an unplanned manner. Whenever Such a situation routine then returns.
is detected by the broadcaster component, it attempts to find FIG. 21 is a flow diagram illustrating the processing of the
another neighbor by invoking the fill holes routine to fill a fill hole routine in one embodiment. This routine is passed
Single hole or the forward connecting edge Search routine to an indication of the requesting process. If this proceSS is
fill two holes. In is block 1808, the routine notes that the requesting to fill a hole, then this routine Sends an internal
recently Sent connection edge Search call has not yet been message to other processes. If another proceSS is requesting
acknowledged and indicates that the edge to this neighbor is 25 to fill a hole, then this routine invokes the routine to handle
reserved if the remaining forwarding distance is less than or a connection port Search request. In block 2101, the routine
equal to one. It is reserved because the Selected neighbor initializes a connection port Search Statement internal, mes
may offer this edge to the requesting process for edge Sage (i.e., connection port Search Stnt). In decision
pinning. The routine then returns. block 2102, if this process is the requesting process, then the
FIG. 19 is a flow diagram illustrating the processing of the routine continues at block 2103, else the routine continues at
handle edge proposal call routine. This routine is invoked block 2104. In block 2103, the routine distributes the
when a message is received from a proposing process that message to the neighbors of this process through the internal
proposes to connect an edge between the proposing proceSS ports and then returns. In block 2104, the routine invokes the
and one of its neighbors to this proceSS for edge pinning. In handle connection port Search routine and then returns.
decision block 1901, if the number of holes of this process 35 FIG.22 is a flow diagram illustrating the processing of the
minus the number of pending edges is greater than or equal internal dispatcher routine in one embodiment. This routine
to one, then this process still has holes to be filled and the is passed an indication of the neighbor who sent the internal
routine continues at block 1902, else the routine continues at message. In block 2201, the routine receives the internal
block 1911. In decision block 1902, if the proposing process message. This routine identifies the message type and
or its neighbor is a neighbor of this process, then the routine 40 invokes the appropriate routine to handle the message. In
continues at block 1911, else the routine continues at block block 2202, the routine assesses whether to change the
1903. In block 1903, the routine indicates that the edge is estimated diameter of the broadcast channel based on the
pending between this proceSS and the proposing process. In information in the received message. In decision block 2203,
decision block 1904, if a proposed neighbor is already if this proceSS is the originating process of the message or
pending as a proposed neighbor, then the routine continues 45 the message has already been received (i.e., a duplicate),
at block 1911, else the routine continues at block 1907. In then the routine ignores the message and continues at block
block 1907, the routine sends an edge proposal response as 2208, else the routine continues at block 2203A. In decision
an external message to the proposing process (i.e., edge block 2203A, if the process is partially connected, then the
proposal resp) indicating that the proposed edge is routine continues at block 2203B, else the routine continues
accepted. In decision block 1908, if the sending of the 50 at block 2204. In block 2203B, the routine adds the message
message was Successful, then the routine continues at block to the pending connection buffer and continues at block
proceSS on the external port as a neighbor. The routine then 55 type of the message is broadcast Statement (i.e., broadcast
the routine returns. In block 1913, the routine invokes the fill 60 the routine continues at block 2209, else the routine contin
FIG.20 is a flow diagram illustrating the processing of the internal messages in a buffer while partially connected So
then indicates that the Sending proceSS wants to connect to 65 process assumes that it is now fully connected and that the
US 6,732,147 B1
25 26
routine returns, else the routine continues at block 2212. In its response (i.e., port connection resp). ASSuming the
FIG.23 is a flow diagram illustrating the processing of the FIG. 28 is a flow diagram illustrating the processing of the
and the broadcast message itself. In block 2301, the routine forwards the message to a neighbor or proposes the edge
performs the out of order processing for this message. The between this process and the Sending neighbor to the
broadcaster component queues messages from each origi 15 requesting process for edge pinning. In decision block 2801,
the message to the neighbors of this process. In decision routine continues at block 2813. In decision block 2802, if
proceSS and then returns. Search routine passing the identification of the requesting
FIG.24 is a flow diagram illustrating the processing of the 25 process and the decremented forwarding distance. The rou
the message to this process. In block 2401, the routine already been offered to a process, then the routine continues
have already been Selected, then the routine returns. In block Search routine passing an indication of the requesting party
2403, the routine sends the message to the Selected neighbor and a toggle indicator that alternatively indicates to continue
and then loops to block 2401 to select the next neighbor. the random walk for one or two more computers. The routine
FIG. 26 is a flow diagram illustrating the processing of the 35 then continues at block 2815. In block 2806, the routine dials
embodiment. This routine is passed an indication of the routine sends an edge proposal call external message (i.e.,
neighbor that Sent the message and the message itself. In edge proposal call) and receives the response (i.e., edge
block 2601, the routine invokes the distribute internal mes proposal resp). ASSuming that the response is Successfully
Sage which sends the message to each of its neighbors other 40 received, the routine continues at block 2808. In decision
than the sending neighbor. In decision block 2602, if the block 2808, if the response indicates that the edge is
number of holes of this proceSS is greater than Zero, then the acceptable to the requesting process, then the routine con
routine continues at block 2603, else the routine returns. In tinues at block 2809, else the routine continues at block
decision block 2603, if the requesting process is a neighbor, 2812. In block 2809, the routine reserves the edge between
then the routine continues at block 2605 else the routine 45 this process and the sending neighbor. In block 2810, the
continues at block 2604, In block 2604, the routine invokes routine adds the requesting process as a neighbor by invok
the court neighbor routine and then returns. The court ing the add neighbor routine. In block 2811, the routine
hole, then the neighbors with empty ports condition exists 50 2815. In decision block 2813, if this process is the requesting
message to the requesting neighbor. 55 connection edge Search response message (i.e., connection
neighbor, then it sends a port connection call external 60 has a cycle whose length is odd. The broadcaster component
tive neighbor as a neighbor. In decision block 2701, if the between even and odd distances.
2702, the routine dials the prospective neighbor. In decision 65 embodiment. This routine is passed as indication of the
US 6,732,147 B1
27 28
Search response (i.e., connection edge Search resp) has ment. This routine removes an existing neighbor and con
been received and if the forwarding distance is less than or nects to the process that Sent the message. In decision block
equal to one unreserves the edge between this proceSS and 3301, if this process has no holes, then the routine continues
the sending neighbor. In decision block 2902, if the request at block 3302, else the routine continues at block 3304. In
ing process indicates that the edge is acceptable as indicated block 3302, the routine selects a neighbor that is not
in the message, then the routine continues at block 2903, else involved in the neighbors with empty ports condition. In
the routine returns. In block 2903, the routine reserves the block 3303, the routine removes the selected neighbor as a
edge between this proceSS and the Sending neighbor. In neighbor of this process. Thus, this process that is executing
block 2904, the routine removes the sending neighbor as a the routine now has at least one hole. In block 3304, the
neighbor. In block 2905, the routine invokes the court routine invokes the add neighbor routine to add the process
neighbor routine to connect to the requesting process. In that Sent the message as a neighbor of this process. The
decision block 2906, if the invoked routine was routine then returns.
unsuccessful, then the routine continues at block 2907, else FIG. 34 is a flow diagram illustrating the processing of the
the routine returns. In decision block 2907, if the number of handle condition double check routine. This routine deter
holes of this proceSS is greater than Zero, then the routine 15 mines whether the neighbors with empty ports condition
continues at block 2908, else the routine returns. In block really is a problem or whether the broadcast channel is in the
2908, the routine invokes the fill hole routine and then small regime. In decision block 3401, if this process has one
returns. hole, then the routine continues at block 3402, else the
FIG.30 is a flow diagram illustrating the processing of the routine continues at block 3403. If this process does not have
broadcast routine in one embodiment. This routine is one hole, then the Set of neighbors of this proceSS is not the
invoked by the application program to broadcast a message Same as the Set of neighbors of the Sending process. In
on the broadcast channel. This routine is passed the message decision block 3402, if this proceSS and the Sending process
to be broadcast. In decision block 3001, if this process has have the same Set of neighbors, then the broadcast channel
at least one neighbor, then the routine continues at block is not in the Small regime and the routine continues at block
3002, else the routine returns since it is the only process 25 3403, else the routine continues at block 3406. In decision
connected to be broadcast channel. In block 3002, the block 3403, if this process has no holes, then the routine
routine generates an internal message of the broadcast returns, else the routine continues at block 3404. In block
statement type (i.e., broadcast stmt). In block 3003, the 3404, the routine sets the estimated diameter for this process
routine Sets the Sequence number of the message. In block to one. In block 3405, the routine broadcasts a diameter reset
3004, the routine invokes the distribute internal message internal message (i.e., diameter reset) indicating that the
routine to broadcast the message on the broadcast channel. estimated diameter is one and then returns. In block 3406,
The routine returns. the routine creates a list of neighbors of this process. In
FIG.31 is a flow diagram illustrating the processing of the block 3407, the routine sends the condition check message
acquire message routine in one embodiment. The acquire (i.e., condition check Stmt) with the list of neighbors to
message routine may be invoked by the application program 35 the neighbor who sent the condition double check message
or by a callback routine provided by the application pro and then returns.
gram. This routine returns a message. In block 3101, the From the above description, it will be appreciated that
routine pops the message from the message queue of the although specific embodiments of the technology have been
broadcast channel. In decision block 3102, if a message was described, various modifications may be made without devi
retrieved, then the routine returns an indication of Success, 40 ating from the Spirit and Scope of the invention. For
else the routine returns indication of failure. example, the communications on the broadcast channel may
FIGS. 32-34 are flow diagrams illustrating the processing be encrypted. Also, the channel instance or Session identifier
of messages associated with the neighbors with empty ports may be a very large number (e.g., 128 bits) to help prevent
condition. FIG. 32 is a flow diagram illustrating processing an unauthorized user to maliciously tap into a broadcast
of the handle condition check message in one embodiment. 45 channel. The portal computer may also enforce Security and
This message is sent by a neighbor process that has one hole not allow an unauthorized user to connect to the broadcast
and has received a request to connect to a hole of this channel.
process. In decision block 3201, if the number of holes of Accordingly, the invention is not limited except by the
this process is equal to one, then the routine continues at claims.
block 3202, else the neighbors with empty ports condition 50 We claim:
does not exist any more and the routine returns. In decision 1. A method of disconnecting a first computer from a
block 3202, if the sending neighbor and this process have the Second computer, the first computer and the Second com
same set of neighbors, the routine continues at block 3203, puter being connected to a broadcast channel, Said broadcast
else the routine continues at block 3205. In block 3203, the channel forming an m-regular graph where m is at least 3,
routine initializes a condition double check message (i.e., 55 the method comprising:
condition double check) with the list of neighbors of this when the first computer decides to disconnect from the
process. In block 3204, the routine Sends the message Second computer, the first computer Sends a disconnect
internally to a neighbor other than Sending neighbor. The message to the Second computer, Said disconnect mes
routine then returns. In block 3205, the routine selects a Sage including a list of neighbors of the first computer;
neighbor of the Sending process that is not also a neighbor 60 and
of this process. In block 3206, the routine sends a condition when the Second computer receives the disconnect mes
repair message (i.e., condition repair stmt) externally to Sage from the first computer, the Second computer
the selected process. In block 3207, the routine invokes the broadcasts a connection port Search message on the
add neighbor routine to add the Selected neighbor as a broadcast channel to find a third computer to which it
neighbor of this process and then returns. 65 can connect in order to maintain an m-regular graph,
FIG. 33 is a flow diagram illustrating processing of the Said third computer being one of the neighbors on Said
handle condition repair Statement routine in one embodi list of neighbors.
US 6,732,147 B1
29 30
2. The method of claim 1 wherein the second computer 10. The method of claim 6 wherein the broadcasting
receives a port connection message indicating that the third includes Sending the message to each computer to which the
computer is proposing that the third computer and the first computer is connected.
Second computer connect. 11. A computer-readable medium containing instructions
3. The method of claim 1 wherein the first computer for controlling disconnecting of a computer from another
disconnects from the Second computer after Sending the computer, the computer and the other computer being con
disconnect message. nected to a broadcast channel, Said broadcast channel being
4. The method of claim 1 wherein the broadcast channel an m-regular graph where m is at least 3, comprising:
is implemented using the Internet. a component that, when the computer decides to discon
5. The method of claim 1 wherein the first computer and nect from the other computer, the computer Sends a
second computer are connected via a TCP/IP connection. disconnect message to the other computer, Said discon
6. A method for healing a disconnection of a first com nect message including a list of neighbors of the
puter from a Second computer, the computers being con computer, and
nected to a broadcast channel, Said broadcast channel being a component that, when the computer receives a discon
an m-regular graph where m is at least 3, the method 15 nect message from another computer, the computer
comprising: broadcasts a connection port Search message on the
attempting to Send a message from the first computer to broadcast channel to find a computer to which it can
the Second computer; and connect in order to maintain an m-regular graph, Said
when the attempt to Send the message is unsuccessful, computer to which it can connect being one of the
neighbors on Said list of neighbors.
broadcasting from the first computer a connection port 12. The computer-readable medium of claim 11 including:
Search message indicating that the first computer needs a component that, when the computer receives a connec
a connection; and tion port Search message and the computer needs to
having a third computer not already connected to Said first connect to another computer, Sends to the computer that
computer respond to Said connection port Search mes 25 Sent the connection port Search message a port connec
Sage in a manner as to maintain an m-regular graph. tion message indicating that the computer is proposing
7. The method of claim 6 including: that the computer that Sent the connection port Search
when a third computer receives the connection port Search message connect to the computer.
message and the third computer also needs a 13. The computer-readable medium of claim 12 includ
connection, Sending a message from the third computer ing:
to the first computer proposing that the first computer a component that, when the computer receives a port
and third computer connect. connection message, connecting to the computer that
8. The method of claim 7 including: Sent the port connection message.
when the first computer receives the message proposing 14. The computer-readable medium of claim 11 wherein
that the first computer and third computer connect, 35 the computers are connected via a TCP/IP connection.
Sending from the first computer to the third computer a 15. The computer-readable medium of claim 11 wherein
message indicating that the first computer accepts the the computers that are connected to the broadcast channel
proposal to connect the first computer to the third are peerS.
computer. 16. The computer-readable medium of claim 11 wherein
9. The method of claim 6 wherein each computer con 40 the broadcast channel is implemented using the Internet.
nected to the broadcast channel is connected to at least three
other computers. k k k k k

DATED : May 4, 2004
INVENTOR(S) : Fred B. Holt
Column 5,
Line 9, “a-broadcast Should be -- a broadcast --
Column 6,
Line 30, “on-that Should be -- on that --
Column 8,
Line 26, delete comma between “newly’;
Column 11,
Line 60, "port-number” should be -- port number --;
Line 63, "port-order” should be -- port order --;
Column 13,
Line 50, "computer-cannot' should be -- computer cannot --;
Column 14,
Line 51, delete period after “Regular”;
Column 22,
Line 41, delete “is” between “receives” and “through';
Column 23,
Line 23, delete “is” between “In” and “block':
Column 25,
Line 45, insert comma between “2605” and "else':

DATED : May 4, 2004
INVENTOR(S) : Fred B. Holt
Column 25 (Cont'd),
Line 46, delete comma between “2604” and “In';

Twenty-seventh Day of July, 2004
WDJ
JON W. DUDAS
Acting Director of the United States Patent and Trademark Office
EXHIBIT 5
USOO691 OO69B1

Holt et al. (45) Date of Patent: Jun. 21, 2005
(54) JOINING A BROADCAST CHANNEL 5,696,903 A 12/1997 Mahany

5,732,074 A 3/1998 Spaur et al.
(75) Inventors: Fred B. Holt, Seattle, WA (US); Virgil 5,732,086 A * 3/1998 Liang et al. ................ 370/410
E. Bourassa, Bellevue, WA (US) 5,732.219 A 3/1998 Blumer et al.
s s 5,734,865 A 3/1998 Yu
(73) ASSignee: Its posing Company, Seattle, WA 5,754,830 A 5/1998 Butts et al.
(Continued)
(*) Notice: Subject to any disclaimer, the term of this
patent is extended or adjusted under 35 OTHER PUBLICATIONS
U.S.C. 154(b) by 708 days. Cho et al., “A Flood Routing Method for Data Networks,”
Sep.1997, Proceedings of 1997 International Conference on
(21) Appl. No.: 09/629,570 Information, Communications and Signal Processing, Vol. 3,
22) Filled: 31, 2000 pp. 1418-1422.*
(22) File Ju 9 Bandyopadhyay et al., “A Flexible Architecture for Multi
(51) Int. Cl." .............................................. G06F 15/177 Hop Optical Networks.” Oct. 1998, 7th International Con
(52) U.S. Cl. ....................... 709/221; 709/252; 709/243; ference on Computer Communications and Networks, 1998,
709/227 pp. 472–478.*
(58) Field of Search ................................. 709/221, 220, Continued
709/252, 243, 227, 223, 204, 238; 370/225, (Continued)
260, 400; 455/428 Primary Examiner-Glenton B. Burgess
Assistant Examiner Bradley Edelman
(56) References Cited (74) Attorney, Agent, or Firm-Perkins Coie LLP
U.S. PATENT DOCUMENTS (57) ABSTRACT
4.912,656 A 3/1990 Cain et al. A technique for adding a participant to a network is pro
5,056,085 A 10/1991 Vu Vided. This technique allows for the Simultaneous Sharing of
5,058,105 A 10/1991. Mansour et al. information among many participants in a network without
5,079,767 A 1/1992 Perlman the placement of a high overhead on the underlying com
5,099.235 A * 3/1992 Crookshanks .............. 455/13.1
5,101,480 A 3/1992 Shin et al. .................. 710/317 munication network. To connect to the broadcast channel, a
5,117,422 A * 5/1992 Hauptschein et al. ....... 370/255 Seeking computer first locates a computer that is fully
5,309.437 A 5/1994 Perlman et al. connected to the broadcast channel. The Seeking computer
5,345,558 A 9/1994 Opher et al. then establishes a connection with a number of the comput
5,426,637 A 6/1995 Derby et al. ers that are already connected to the broadcast channel. The
5,459,725 A 10/1995 Bodner et al. technique for adding a participant to a network includes
5,471,623 A * 11/1995 Napolitano, Jr. ............ 709/243 identifying a pair of participants that are connected to the
5,511,168 A 4/1996 Perlman et al. network, disconnecting the participants of the identified pair
5.535,199 A 7/1996 Amri et al.
5,568.487 A 10/1996 Sitbon et al. from each other, and connecting each participant of the
5,636,371 A 6/1997 Yu identified pair of participants to the added participant.
5,644,714 A 7/1997 Kikinis
5,673,265 A 9/1997 Gupta et al. 17 Claims, 39 Drawing Sheets
US 6,910,069 B1
Page 2
U.S. PATENT DOCUMENTS OTHER PUBLICATIONS

5.757.795 5/1998 Schnell Hsu, “On Four-Connecting a Triconnected Graph,” Oct.
5,761,425 6/1998 Miller 1992, Annual Symposium on Foundations of Computer
5,764,756 6/1998 Onweller Science, 1992, pp. 70-79.*
5,790,548 8/1998 Sistanizadeh et al. Shiokawa et al., “Performance Analysis of Network Con
5,790,553 8/1998 Deaton, Jr. et al. nective Probability of Multihop Network under Correlated
5,799,016 8/1998 Onweller Breakage,” Jun. 1996, 1996 IEEE International Conference
5,802.285 9/1998 Hirviniemi
5,850,592 12/1998 Ramanathan
on Communications, vol. 3, pp. 1581-1585.*
5,864,711 1/1999 Mairs et al. Komine et al., “A Distributed Restoration Algorithm for
5,867,660 2/1999 Schmidt et al. Multiple-Link and Node Failures of Transport Networks,”
5,867,667 2/1999 Butman et al. Dec. 199 IEEE Globecom '90, “Communications: Connect
5,870,605 2/1999 Bracho et al. ing the Future, vol. 1, pp. 459-463.*
5,874,960 2/1999 Mairs et al. U.S. Appl. No. 09/629,576, filed Jul. 31, 2000, Bourassa et
5/1999 Wilf et al. 1
5,899.980
5,907,610 5/1999 Onweller S. Appl. No. 09/629,577, filed Jul. 31, 2000, Bourassa et
5,925,097 7/1999 Gopinath et al.
5,928,335 7/1999 Morita S. Appl. No. 09/629,575, filed Jul. 31, 2000, Bourassa et
5,935,215 8/1999 Bell et al.
5,946,316 8/1999 Chen et al. S. Appl. No. 09/629,572, filed Jul. 31, 2000, Bourassa et
5.948,054 9/1999 Nielsen
5,949,975 9/1999 Batty et al. S. Appl. No. 09/629,023, filed Jul. 31, 2000, Bourassa et
5,953,318 9/1999 Nattkemper et al.
5,956,484 9/1999 Rosenberg et al. S. Appl. No. 09/629,043, filed Jul. 31, 2000, Bourassa et
5,970.232 10/1999 Passint et al.
5,974,043 10/1999 Solomon S. Appl. No. 09/629,024, filed Jul. 31, 2000, Bourassa et
5.987,506 11/1999 Carter et al.
6,003,088 12/1999 Houston et al.
6,013,107 1/2000 Blackshear et al.
S. Appl. No. 09/629,042, filed Jul. 31, 2000, Bourassa et
1.
6,023,734 2/2000 Ratcliff et al.
6,029,171 2/2000 Smiga et al. Murphy, Patricia, A., “The Next Generation Networking
6,032,188 2/2000 Mairs et al. Paradigm: Producer/Consumer Model.” Dedicated Systems
6,038,602 3/2000 Ishikawa Magazine-2000 (pp. 26-28).
6,047.289 4/2000 Thorne et al. The Gamer's Guide, “First-Person Shooters,” Oct. 20, 1998
6,065,063 5/2000 Abali ......................... 709/242 (4 pages).
6,073,177 6/2000 Hebel et al. The O'Reilly Network, “Gnutella: Alive, Well, and Chang
6,094,676 7/2000 Gray et al. ing Fast,” Jan. 25, 2001 (5 pages) http://www.open2p.com/
6,115,580 9/2000 Chuprun et al. 1.pt/ . . . Accessed Jan. 29, 2002).
6,151,633 11/2000 Hurst Oram, Andy, “Gnutella and Freenet Represents True Tech
6,167.432 12/2000 Jiang nological Innovation,” May 12, 2000 (7 pages) The O'Reilly
6,173,314 1/2001 Kurashima et al. Network http://www.oreillynet.com/1pt . . . Accessed Jan.
6,195,366 2/2001 Kayashima 29, 2003).
6,199,116 3/2001 May et al. Internetworking Technologies Handbook, Chapter 43 (pp.
6,216,177 4/2001 Mairs et al. 43-1-43-16).
6,223,212 4/2001 Batty et al. Oram, Andy, “Peer-to-Peer Makes the Internet Interesting
6.243,691 6/2001 Fisher et al. Again,” Sep. 22, 2000 (7 pages) The O'Reilly Network
6.252.884 6/2001 Hunter http://linux.oreillynet.com/1pt. . . Accessed Jan. 29, 2002).
6,268.855 7/2001 Mairs et al.
7/2001 Kumar
Monte, Richard, “The Random Walk for Dummies,'MIT
6,269,080 Undergraduate Journal of Mathematics (pp. 143-148).
6,271,839 8/2001 Mairs et al.
6,272.548 8/2001 Cotter et al. Srinivasan, R., “XDR: External Data Representation Stan
6,285,363 9/2001 Mairs et al. dard.” Sun Microsystems, Aug. 1995 (20 pages) Internet
6,304,928 10/2001 Mairs et al. RFC/STD/FYI/BCP Archives http://www.faqs.org/rfcs/
6,321,270 11/2001 Crawley rfc1832.html Accessed Jan. 29, 2002).
6,353,599 3/2002 Bi et al. A Databeam Corporate White Paper, “A Primer on the T120
6,415,270 7/2002 Rackson et al. Series Standards,” Copyright 1995 (pp. 1-16).
6,434622 8/2002 Monteiro et al. Kessler, Gary, C., “An Overview of TCP/IP Protocols and
6,463,078 10/2002 Engstrom et al. the Internet,” Apr. 23, 1999 (23 pages) Hill Associates, Inc.
6,490.247 :
12/2002 Gilbert et al. .............. 370/222 http://www.hill.com/library/publications/t . . . Accessed
6,499,251 12/2002 Weder Jan. 29, 2002).
6,505.289 1/2003 Han et al. ..................... 712/11 Bondy, J.A., and Murty, U.S.R., “Graph Theory with Appli
6,524,189 2/2003 Rautila
6,553,020 4/2003 Hughes et al. .............. 370/347 cations,” Chapters 1-3 (pp. 1-47), 1976 American Elsevier
6,603,742 8/2003 Steele et al. ................ 370/254 Publishing Co., Inc., New York, New York.
6,611,872 8/2003 McCanne Cormen, Thomas, H. et al., Introduction to Algorithms,
6,618,752 9/2003 Moore et al. Chapter 5.3 (pp. 84-91), Chapter 12 (pp. 218-243), Chapter
6,701,344 3/2004 Holt et al. 13 (p. 245), 1990, The MIT Press, Cambridge, Massachu
2002/0027896 3/2002 Hughes et al. setts, McGraw-Hill Book Company, New York.
US 6,910,069 B1
Page 3
The Common Object Request Broker: Architecture and Business Wire, “Boeing Panthesis Complete SWAN Trans
Specification, Review 2.6, Dec. 2001, Chapter 12 (pp. action,” Jul. 22, 2002, pp 1 ff.
12-1-12-10), Chapter 13 (pp. 13-1-13-56), Chapter 16
(pp. 16-1-16-26), Chapter 18 (pp. 18-1-18-52), Chapter PR NewSwire, “Microsoft Annouces Launch Date for
20 (pp. 20-1-20–22). UltraCrops, Its Second Premium Title for the Internet Gam
The University of Warwick, Computer Science Open Days, ing Zone,” Mar. 27, 1998, pp 1 ff.
“Demonstration on the Problems of Distributed Systems.” PR Newswire, “Microsoft Boosts Accessibility to Internet
http://www.dcs. Warwick.ac.u. . . . Accessed Jan. 29, 2002). Gaming Zone with Latest Release, Apr. 27, 1998, pp 1 ff.
Alagar, S. and Venkatesan, S., “Reliable Broadcast in
Mobile Wireless Networks,” Department of Computer Sci Peercy et al., “Distributed Algorithms for Shortest-Path,
ence, University of Texas at Dallas, Military Communica Deadlock-Free Routing and Broadcasting in Arbitrarily
tions Conference, 1995, MILCOM 95 Conference Record, Faulty Hypercubes,” Jun. 1990, 20th International Sympo
IEEE San Diego, California, Nov. 5–8, 1995 (pp. 236-240). sium on Fault-Tolerant Computing, 1990, pp-218-225.
International Search Report for The Boeing Company, Inter
national Patent Application No. PCT/US01/24240, Jun. 5, Azar et al., “Routing Strategies for Fast Networks,” May
2002 (7 pages). 1992, INFOCOM 92 Eleventh Annual Joint Conference of
Yavatkar et al., “A reliable Dissemination Protocol for the IEEE Computer Communications Societies, vol. 1,
Interactive Collaborative Applications.” Proc. ACM Multi 170-179ifiti.
media, 1995, p. 333-344; http://citeseer.nj.nec.com/article/
yavatkar95reliable.htm. * cited by examiner
U.S. Patent Jun. 21, 2005 Sheet 1 of 39

U.S. Patent Jun. 21, 2005 Sheet 2 of 39 US 6,910,069 B1

U.S. Patent Jun. 21, 2005 Sheet 3 Of 39

U.S. Patent Jun. 21, 2005 Sheet 6 of 39




U.S. Patent Jun. 21, 2005 US 6,910,069 B1

Oc
(Y).
Na
O
U.S. Patent Jun. 21, 2005 Sheet 12 Of 39 US 6,910,069 B1
(814
9
20/.
90/
109uO
||
04
(Channel Type,
Channel Instance,
801
Connect Aux Info)
Open call in port
802 Fig. 8
803
instance)
804.
N
Y
806
N
e 807
808
809
Connect request

902
Select next depth
All depths selected Return (failure)
Fig. 9
selected

1001
1OO2
1005
computer computer
connected
Connect request
Fig. 11
Was a fully Restat
connected portal foung

computer
1 104
N
Y 1105
1106

response
1112
heck for externe

call
Fig. 12
ype = = seeking
connection call

Achieve connection
1301
Fig. 13
Connection - state as fully
connected
1302
1303

Pick up and receive
external message
Handle seeking
connection call
Handle connection
request call
4.08
o edge
call
proposal
1410
Port connect call
Handle port
connection call
Handle connected
1413 1414
Condition repair Y Handle condition
Statement - repair statement
Handle seeking Fig. 15

connection call
1503
Add other as fellow

seeking process
1505
andle connection
s request call
1601
Connecteddin
Set newcomer's
holes to expect
S4
Return
SC

response SC Fig. I6
S5
Set newcomer's
holes to fill
608 SS
Add neighbor
Holes to fill - =
Y Fill hole (requestor)

oles to fill > -
Add neighbor
Fig. 17
Sets neighbor to
messages pending
703 P 4.
ses N
1705
first
C Cit tate a
Add as neighbor
OS
for new neighbor
Connecting buffer
1709
Holes = = Achieve connected
expected hole
N
1711
Purge pending edges
-1

Fig. 18
it of
neighbors Y
>
3O4
requestor
Return
.
selected


Fig. 19
N 190
proposed
neighbors
pending
907
1912 1908
1913 Y 1909
Fill hole Add edge as pending
1910
Our D adopter
Handle port
connection call
Fig. 20
2003
Caller is not (point-connect-resp
neighbor not ok)

2006
Add neighbor
2008
Connect request
Fill hole
Fig. 21 Initialize internal

message
2103
Handle Connection Distribute internal
internal
dispatcher
22O1
22O2
Assess diameter
This
process = =
originating
Partially connected
Insert In S
Type 2005
= = broadcast Handle broadcast
stateet message
2007
= = shutdown Handle shutdown
Stateet
Statenent
Y Pending
connection buffer
a st
Is
message queue
5-1 list
Receive response ( )
andle broadcast origin

Fig. 23 message
message
23O2
message
3O3 2304
S at W Y
neighbor received Clear out of order info
messages -
O Distribute Enessage
F3. 24 from neighbor
24O1
All neighbor
selected
2403
Send internal
message

for search message
26C
Distribute internal
Fig. 26
602
N
603 2604
Is requestor
a neighbor N - cour neighbor
SOS
efeate
condition check
message wineighbors
26O7
to requestor
Fig. 27
N
27O2
Dial prospect
Send and receive

external message
Add neighbor
27O6
Hang up prospect

Fig. 28 8O1
Not
my message ll
holes >= Z
2813
N Message
from this pt. &&.
oles = = K
connection second
edge (requestor
Y 2814 remaining dist-l
Fill hole (self)
Reguestor Forward
A> 2815 connection edge
end interna is neighbor or edge search (requestor,
message (from reserved 0)
neighbor, ack
2806 VA
R
2807
Send and receive
external message

resp. from neighbor
Fig. 29 901 message
search response
Edge selected
Y 9.
904
2905
Courtneighbor
U.S. Patent Jun. 21, 2005 Sheet 35 0f 39 US 6,910,069 B1
Fig. 30
Y
30O2
Generate internal
message
3003
number
3004
Distribute internal
message
message
Acquire message
31 O1
Fig. 31
Pop message queue

Fig. 32 201
N
Y
32O2
Y Same set of N
neighbors
32O3 3205
elect a neighbor
Set "Fish list ofnotsending process
my neighbor
3204 3206
32O7
Add neighbor
Handle condition
repair statement

Remove selected
neighbor
33O4.
Add neighbor
Handle condition
double check
Same set of
neighbors
Create list of neighbors

A4

to-from neighbor
US 6,910,069 B1
1 2
JOINING A BROADCAST CHANNEL (“CORBA). Client/server middleware systems are not par
ticularly well Suited to sharing of information among many
CROSS-REFERENCE TO RELATED participants. In particular, when a client Stores information
APPLICATIONS to be shared at the Server, each other client would need to
5 poll the Server to determine that new information is being
shared. Such polling places a very high overhead on the
No. 09/629,576, entitled “BROAD CASTING communications network. Alternatively, each client may
NETWORK,” filed on Jul. 31, 2000; U.S. patent application register a callback with the Server, which the Server then
Ser. No. 09/629,570, entitled “JOINING A BROADCAST invokes when new information is available to be shared.
CHANNEL,” filed on Jul. 31, 2000; U.S. patent application Such a callback technique presents a performance bottleneck
Ser. No. 09/629,577, “LEAVING A BROADCAST because a single Server needs to callback to each client
CHANNEL,” filed on Jul. 31, 2000; U.S. patent application whenever new information is to be shared. In addition, the
Ser. No. 09/629,575, entitled “BROADCASTING ON A reliability of the entire Sharing of information depends upon
BROADCAST CHANNEL, filed on Jul. 31, 2000; U.S. the reliability of the Single Server. Thus, a failure at a single
patent application Ser. No. 09/629,572, entitled “CON 15 computer (i.e., the server) would prevent communications
TACTING A BROADCAST CHANNEL, filed on Jul. 31, between any of the clients.
2000; U.S. patent application Ser. No. 09/629,023, entitled The multicasting network protocols allow the Sending of
“DISTRIBUTED AUCTION SYSTEM,” filed on Jul. 31, broadcast messages to multiple recipients of a network. The
2000; U.S. patent application Ser. No. 09/629,043, entitled current implementations of Such multicasting network pro
“AN INFORMATION DELIVERY SERVICE,” filed on Jul. tocols tend to place an unacceptable overhead on the under
31, 2000, now U.S. Pat. No. 6,714,966; U.S. patent appli lying network. For example, UDP multicasting would
cation Ser. No. 09/629,024, entitled “DISTRIBUTED CON Swamp the Internet when trying to locate all possible par
FERENCING SYSTEM, filed on Jul. 31, 2000; and U.S. ticipants. IP multicasting has other problems that include
patent application Ser. No. 09/629,042, entitled “DISTRIB needing special-purpose infrastructure (e.g., routers) to Sup
UTED GAME ENVIRONMENT' filed on Jul. 31, 2000, 25 port the Sharing of information efficiently.
the disclosures of which are incorporated herein by refer
CCC. The peer-to-peer middleware communications Systems
BACKGROUND
35
client/server middleware, multicasting network protocols, the T120 Internet standard is a tree structure, which relies on
40
none is particularly well Suited to the Simultaneous sharing order to be received by all participants.
information in a timely manner to all participants who may 45
widely distributed.
The point-to-point network protocols, such as UNIX FIG. 1 illustrates a graph that is 4-regular and 4-connected
pipes, TCP/IP, and UDP, allow processes on different com which represents a broadcast channel.
puters to communicate via point-to-point connections. The FIG. 2 illustrates a graph representing 20 computers
interconnection of all participants using point-to-point 50 connected to a broadcast channel.
connections, while theoretically possible, does not Scale well FIGS. 3A and 3B illustrate the process of connecting a
as a number of participants grows. For example, each new computer Z to the broadcast channel.
participating process would need to manage its direct con FIG. 4A illustrates the broadcast channel of FIG. 1 with
nections to all other participating processes. Programmers,
however, find it very difficult to manage Single connections, 55
an added computer.
of information. 60 FIG. 5A illustrates the disconnecting of a computer from
The client/server middleware systems provide a server the broadcast channel in a planned manner.
that coordinates the communications between the various FIG. 5B illustrates the disconnecting of a computer from
clients who are sharing the information. The Server functions the broadcast channel in an unplanned manner.
as a central authority for controlling access to shared FIG. 5C illustrates the neighbors with empty ports con
resources. Examples of client/server middleware Systems 65 dition.
include remote procedure calls (“RPC"), database servers, FIG. 5D illustrates two computers that are not neighbors
and the common object request broker architecture who now have empty ports.
US 6,910,069 B1
3 4
large regime.
US 6,910,069 B1
S 6
ered to be in the "large regime.” This description assumes 65.535. The first 2056 ports are reserved for specific appli
US 6,910,069 B1
7 8
diameter of the broadcast channel may increase when each receiving computer. The Second message only has to travel
Seeking computer uses the same found portal computer and a distance of one. Thus, it is possible for the Second message
establishes a connection to the broadcast channel directly to reach the receiving computer before the first message.
through that found portal computer. Conceptually, the graph When the broadcast channel is in a steady state (i.e., no
becomes elongated in the direction of where the new nodes computers connecting or disconnecting from the broadcast
are added. FIGS. 4A-4C illustrate that possible problem. channel), out-of-order messages are not a problem because
FIG. 4A illustrates the broadcast channel of FIG. 1 with an each computer will eventually receive both messages and
added computer. Computer J was connected to the broadcast can queue messages until all earlier ordered messages are
channel by edge pinning edges C-D and E-H to computer received. If, however, the broadcast channel is not in a
J. The diameter of this broadcast channel is still two. FIG. Steady State, then problems can occur. In particular, a com
new neighbors. When a new neighbor eventually receives
cast channel by edge pinning edges E-J and B-C to com the first message, it sends the message to the newly con
puter K. The diameter of this broadcast channel is three, nected computer. Thus, the newly connected computer will
because the Shortest path from computer G to computer Kis 15
receive the first message, but will not receive the Second
through edges G-A, A-E, and E-K. FIG. 4C also illustrates message. If the newly connected computer needs to process
the broadcast channel of FIG. 4A with an added computer. the messages in order, it would wait indefinitely for the
Computer K was connected to the broadcast channel by edge Second message.
pinning edges D-G and E-J to computer K. The diameter of One Solution to this problem is to have each computer
this broadcast channel is, however, still two. Thus, the queue all the messages that it receives until it can Send them
Selection of neighbors impacts the diameter of the broadcast in their proper order to its neighbors. This Solution, however,
channel. To help minimize the diameter, the broadcast may tend to Slow down the propagation of messages through
technique uses a random Selection technique to identify the the computers of the broadcast channel. Another Solution
four neighbors of a computer in the Seeking connection State. that may have leSS impact on the propagation Speed is to
The random Selection technique tends to distribute the 25 queue messages only at computers who are neighbors of the
connections to new Seeking computers throughout the com newly connected computers. Each already connected neigh
puters of the broadcast channel which may result in Smaller bor would forward messages as it receives them to its other
overall diameters. neighbors who are not newly connected, but not to the newly
AS described above, each computer that is connected to only forward messages from each originating computer to
the broadcast channel can broadcast messages onto the the newly connected computer when it can ensure that no
broadcast channel and does receive all messages that are gaps in the messages from that originating computer will
broadcast on the broadcast channel. The computer that occur. In one embodiment, the already connected neighbor
originates a message to be broadcast sends that message to may track the highest Sequence number of the messages
each of its four neighbors using the internal connections. 35 already received and forwarded on from each originating
When a computer receives a broadcast message from a computer. The already connected computer will Send only
neighbor, it sends the message to its three other neighbors. higher numbered messages from the originating computers
Each computer on the broadcast channel, except the origi to the newly connected computer. Once all lower numbered
nating computer, will thus receive a copy of each broadcast messages have been received from all originating
message from each of its four neighbors. Each computer, 40 computers, then the already connected computer can treat
however, only sends the first copy of the message that it the newly connected computer as its other neighbors and
receives to its neighbors and disregards Subsequently Simply forward each message as it is received. In another
received copies. Thus, the total number of copies of a embodiment, each computer may queue messages and only
message that is Sent between the computerS is 3N+1, where forwards to the newly connected computer those messages
N is the number of computers connected to the broadcast 45 as the gaps are filled in. For example, a computer might
channel. Each computer Sends three copies of the message, receive messages 4 and 5 and then receive message 3. In
except for the originating computer, which Sends four copies Such a case, the already connected computer would forward
of the message. queue messages 4 and 5. When message 3 is finally received,
The redundancy of the message Sending helps to ensure the already connected computer will Send messages 3, 4, and
the overall reliability of the broadcast channel. Since each 50 5 to the newly connected computer. If messages 4 and 5 were
computer has four connections to the broadcast channel, if Sent to the newly connected computer before message 3,
one computer fails during the broadcast of a message, its then the newly connected computer would process messages
neighbors have three other connections through which they 4 and 5 and disregard message 3. Because the already
will receive copies of the broadcast message. Also, if the connected computer queues messages 4 and 5, the newly
internal connection between two computers is slow, each 55 connected computer will be able to proceSS message 3. It is
computer has three other connections through which it may possible that a newly connected computer will receive a Set
receive a copy of each message Sooner. of messages from an originating computer through one
Each computer that originates a message numbers its own neighbor and then receive another Set of message from the
messages Sequentially. Because of the dynamic nature of the Same originating computer through another neighbor. If the
broadcast channel and because there are many possible 60 Second Set of messages contains a message that is ordered
connection paths between computers, the messages may be earlier than the messages of the first Set received, then the
received out of order. For example, the distance between an newly connected computer may ignore that earlier ordered
originating computer and a certain receiving computer may message if the computer already processed those later
be four. After Sending the first message, the originating ordered messages.
computer and receiving computer may become neighbors 65 Decomposing the Graph
and thus the distance between them changes to one. The first A connected computer disconnects from the broadcast
US 6,910,069 B1
9 10
receives the disconnect message, it tries to connect to one of 5 receives the list, it compares the list to its own list of
When a computer with an available internal port receives the 15 Thus, one of the original neighbors involved in the condition
puter that broadcast the message. FIGS.5A-5D illustrate the in need of a connection, the other original neighbor and the
indicated by the dashed line, and Similarly for computers E 25 It is possible that the two original neighbors with the
When a computer disconnects in an unplanned manner, neighbor that receives the condition check message deter
disconnected computer recognizes that it is short one con it determines whether it has the same set of neighbors as the
disconnected, it broadcasts a port connection request on the 35 neighbors are different, then the computer that received the
port that needs a connection. The port connection request message to the original neighbors with the condition. The
the connection request, it communicates with the requesting 40 neighbors with the condition by Sending a condition repair
between the two computers. FIG. 5B illustrates the discon condition will have its port filled.
disconnected in an unplanned manner. When each of its 45 unplanned manner, but computers F and I responded to the
disconnection, each neighbor broadcasts a port connection together. The other former neighbors of computer H, com
A and E respond to each other's requests and establish a 50 computer E received the port connection request from
It is possible that a planned or unplanned disconnection (since they are neighbors via the internal connection) a
already connected and cannot fill their empty ports by 55 nized that computer E has a different set of neighbor (i.e., the
the “neighbors with empty ports' condition. Each neighbor Selected computer D, which is a neighbor of computer E and
receives the port connection request from the other neighbor, 60 neighbors (other than computer E), which is computer G in
empty port. Such a condition may also occur when the FIG. 5D illustrates two computers that are not neighbors
Small regime, each computer will have less than four neigh 65 computers E and G can connect to each other.
would be a problem if not repaired, the first neighbor to empty ports condition. FIG.5E illustrates the neighbors with
US 6,910,069 B1
11 12
FIG. 5F illustrates the situation of FIG. 5E when in the 15 typically need to keep on redialing a busy port. The process
condition double check message from computer B. In this redialing. In one embodiment, each Seeking computer may
neighbors of computer C. The computer that receives the 25 a 25% chance of being first in the reordering, and So on.
neighbors, other than computer C, and tries to connect to orderings, the likelihood of finding a busy port is reduced.
Statically allocated to an application program because other 35 can connect to the broadcast channel. In one embodiment,
use conflicting port numbers. As a result, in one Seeking computer locates a portal computer that is connected
puter could simply try to locate the lowest number unused 40 A Seeking computer could Select the first portal computer
Seeking computer, however, does not know in advance the that is fully connected to the broadcast channel is found. If
puter needs to dial ports of a portal computer Starting with 45 a portal computer with Such a call-in port is found. A
portal computer. If the portal computer is connected to (or of each portal computer are dialed until a portal computer
the portal computer is not connected, then the Seeking 50 number according to the algorithm and then dials each portal
if each application program on a computer tried to allocate the broadcast channel is found, then the Seeking computer
many of the low-ordered port numbers would be used by 55 numbers, the Seeking computer first dials the port numbers
relatively slow process, it would take the Seeking computer channel. The Seeking computers may have a maximum
ordering algorithm to identify the port number order that a 60 Seeking computer exhausts its Search depth, then either the
for its call-in port. In one embodiment, the broadcast tech Seeking computer is also a portal computer, it can then
nique uses a hashing algorithm to identify the port order. The establish the broadcast channel with itself as the first fully
algorithm preferably distributes the ordering of the port connected computer.
numbers randomly through out the user port number Space 65 When a Seeking computer locates a portal computer that
time the algorithm is executed on any computer for a given connect when they first locate each other because the
US 6,910,069 B1
13 14
Searching to a depth of nine. its estimated diameter and broadcasts an estimated diameter
In one embodiment, each computer may have a different 15 message. When a computer receives an estimated diameter
currently connected computers. One advantage of the broad 25 may use ASCII to represent text and another computer may
four neighbors for a Seeking computer. 35 message may require two calls to the operating System: one
through one of its internal connections that is randomly 40 overcome the inefficiencies of Such repeated calls, the broad
channel. Eventually, a receiving computer will decide that technique may request the operating System to provide the
Selected computer. That receiving computer will offer the 45 cast technique can then repeatedly invoke the XDR routines
pinning. Of course, if either of the computers at the end of bytes needs to be retrieved from the operating System. The
Seeking computer, then the Seeking computer cannot connect 50 are thus more efficient than repeated System calls.
through that internal connection. The computer that decided M-Regular
that the message has traveled far enough will detect this In the embodiment described above, each fully connected
condition of already being a neighbor and Send the message computer has four internal connections. The broadcast tech
to a randomly Selected neighbor. nique can be used with other numbers of internal connec
In one embodiment, the distance that the edge connection 55 tions. For example, each computer could have 6, 8, or any
broadcast channel. The message includes an indication of channel tends to decrease, and thus propagation time for a
decrements that distance to travel before Sending the mes 60 a Seeking computer to the broadcast channel may, however,
randomly Selected computer. If that randomly Selected com broadcast channel can be maintained as m-regular and
it is already connected to it), then that randomly selected 65 connections is odd, then when the broadcast channel has an
US 6,910,069 B1
15 16
In Such a situation, the broadcast network is neither broadcaster component includes a connect component 701,
m-regular nor m-connected. When the next computer con an external dispatcher 702, an internal dispatcher 703 for
nects to the broadcast channel, it can again become each internal connection, an acquire message component
m-regular and m-connected. Thus, with an odd number of 704 and a broadcast component 712. The application pro
internal connections, the broadcast channel toggles between gram may provide a connect callback component 710 and a
being and not being m-regular and m-connected. receive response component 711 that are invoked by the
Components
broadcast channel, and each computer can have multiple 15
connections to the same broadcast channel. The broadcast request component 706 to ask the portal computer (if fully
channel is well Suited for computer processes (e.g., appli connected) to select neighbor processes for the newly con
cation programs) that execute collaboratively, Such as net necting process. The external dispatcher receives external
work meeting programs. Each computer process can connect messages, identifies the type of message, and invokes the
to one or more broadcast channels. The broadcast channels appropriate handling routine 707. The internal dispatcher
can be identified by channel type (e.g., application program receives the internal messages, identifies the type of
name) and channel instance that represents separate broad message, and invokes the appropriate handling routine 708.
cast channels for that channel type. When a proceSS attempts The received broadcast messages are Stored in the broadcast
to connect to a broadcast channel, it seeks a process cur message queue 709. The acquire message component is
rently connected to that broadcast channel that is executing invoked to retrieve messages from the broadcast queue. The
on a portal computer. The Seeking process identifies the 25
broadcast component is invoked by the application program
broadcast channel by channel type and channel instance. to broadcast messages in the broadcast channel.
Computer 600 includes multiple application programs
broadcast channel to which it is connected. The broadcaster
component may be implement as an object that is instanti
ated within the process space of the application program. EXTERNAL MESSAGES
Alternatively, the broadcaster component may execute as a
Separate process or thread from the application program. In Message Type Description
one embodiment, the broadcaster component provides func 35
seeking connection call Indicates that a seeking process would like to
tions (e.g., methods of class) that can be invoked by the her the receiving process is fully
application programs. The primary functions provided may the broadcast channel
include a connect function that an application program connection request call he sending process would like
invokes passing an indication of the broadcast channel to the receiving process to initiate a connection
he sending process to the broadcast
which the application program wants to connect. The appli 40
cation program may provide a callback routine that the edge proposal call Indicates that the sending process is
broadcaster component invokes to notify the application proposing an edge through which the
process can connect to the
program that the connection has been completed, that is the broadcast channel (i.e., edge pinning)
proceSS enters the fully connected State. The broadcaster port connection call Indicates that the sending process is
component may also provide an acquire message function 45 proposing a port through which the
that the application program can invoke to retrieve the next receiving process can connect to the
broadcast channel
message that is broadcast on the broadcast channel. connected stimt Indicates that the sending process is
Alternatively, the application program may provide a call connected to the broadcast channel
back routine (which may be a virtual function provided by condition repair stimt Indicates that the receiving process should
the application program) that the broadcaster component 50 disconnect from one of its neighbors and
connect to one of the processes involved in
invokes to notify the application program that a broadcast the neighbors with empty port condition
message has been received. Each broadcaster component
allocates a call-in port using the hashing algorithm. When
calls are answered at the call-in port, they are transferred to
other ports that Serve as the external and internal ports. 55
The computers connecting to the broadcast channel may INTERNAL MESSAGES
include a central processing unit, memory, input devices
(e.g., keyboard and pointing device), output devices (e.g., Message Type Description
display devices), and storage devices (e.g., disk drives). The broadcast stimt Indicates a message that is being
memory and Storage devices are computer-readable medium 60
broadcast through the broadcast channel
that may contain computer instructions that implement the for the application programs
broadcaster component. In addition, the data structures and connection port search stimt Indicates that the designated process is
message Structures may be Stored or transmitted via a signal looking for a port through which it can
connect to the broadcast channel
transmitted on a computer-readable media, Such as a com connection edge search call Indicates that the requesting process is
munications link. 65 looking for an edge through which it
FIG. 7 is a block diagram illustrating the Sub-components can connect to the broadcast channel
of the broadcaster component in one embodiment. The
US 6,910,069 B1
17 18
routine continues at block 808. In block 806, the routine
-continued invokes the achieve connection routine to change the State of
INTERNAL MESSAGES
this process to fully connected. In block 807, the routine
installs the external dispatcher for processing messages
Message Type Description received through this process external port for the passed
channel type and channel instance. When a message is
connection edge search resp Indicates whether the edge between this received through that external port, the external dispatcher is
process and the sending neighbor has
been accepted by the requesting party invoked. The routine then returns. In block 808, the routine
diameter estimate stimt Indicates an estimated diameter of the installs an external dispatcher. In block 809, the routine
broadcast channel invokes the connect request routine to initiate the process of
diameter reset stimt Indicates to reset the estimated diameter
to indicated diameter
then returns.
disconnect stimt Indicates that the sending neighbor is
disconnecting from the broadcast FIG. 9 is a flow diagram illustrating the processing of the
channel Seek portal computer routine in one embodiment. This
condition check stimt Indicates that neighbors with empty port 15 routine is passed the channel type and channel instance of
condition have been detected
condition double check stimt Indicates that the neighbors with empty the broadcast channel to which this proceSS wishes to
ports have the same set of neighbors connect. This routine, for each Search depth (e.g., port
shutdown stimt Indicates that the broadcast channel is number), checks the portal computers at that search depth. If
being shutdown a portal computer is located at that Search depth with a
process that is fully connected to the broadcast channel, then
Flow Diagrams the routine returns an indication of Success. In blockS
of the broadcaster component in one embodiment. FIG. 8 is a process is located. In block 902, the routine selects the next
routine in one embodiment. This routine is passed a channel 25 decision block 903, if all the search depths have already
which this proceSS wants to connect. The routine is also indication, else the routine continues at block 904. In blocks
ceSS invokes this routine, it is in the Seeking connection In block 904, the routine selects the next portal computer. In
and this routine connects to at least one neighbor, this 35 been selected, then the routine loops to block 902 to select
proceSS enters the partially connected State, and when the the next Search depth, else the routine continues at block
process eventually connects to four neighbors, it enters the 906. In block 906, the routine dials the selected portal
fully connected state. When in the small regime, a fully computer through the port represented by the Search depth.
connected process may have leSS than four neighbors. In In decision block 907, if the dialing was successful, then the
block 801, the routine opens the call-in port through which 40 routine continues at block 908, else the routine loops to
the process is to communicate with other processes when block 904 to select the next portal computer. The dialing will
establishing external and internal connections. The port is be successful if the dialed port is the call-in port of the
Selected as the first available port using the hashing algo broadcast channel of the passed channel type and channel
rithm described above. In block 802, the routine sets the instance of a proceSS executing on that portal computer. In
connect time to the current time. The connect time is used to 45 block 908, the routine invokes a contact process routine,
identify the instance of the process that is connected through which contacts the answering process of the portal computer
this external port. One proceSS may connect to a broadcast through the dialed port and determines whether that process
channel of a certain channel type and channel instance using is fully connected to the broadcast channel. In block 909, the
may then connect to that same broadcast channel using the 50 block 910, if the answering process is fully connected to the
block 803, the routine invokes the seek portal computer 55 process as a portal computer and processes that call. The
decision block 804, if the seek portal computer routine is 60 determines whether the process of the Selected portal com
the routine returns an unsuccessful indication. In decision routine sends an external message (i.e., seeking
computer on which the proceSS is executing was located, 65 Seeking process wants to know whether the answering
US 6,910,069 B1
19 20
response message that it is fully connected to the broadcast 15 ceSS indicating that this process is also is Seeking a connec
being fully connected to the broadcast channel to initiate the 25 Seeking process may become the first process fully con
computer was located that is fully connected to the broadcast connection routine in one embodiment. This routine Sets the
Seeking computer may always Search its entire Search depth the connection State of this process to fully connected. In
connect to the broadcast channel. In block 1102, the routine 35 that it is fully connected by Sending a connected external
restarts the process of connecting to the broadcast channel message to them (i.e., connected stmt). In block 1303, the
and returns. In block 1103, the routine dials the process of routine invokes the connect callback routine to notify the
one of the found portal computers through the call-in port. application program and then returns.
In decision block 1104, if the dialing is successful, then the FIG. 14 is a flow diagram illustrating the processing of the
routine continues at block 1105, else the routine continues at 40 external dispatcher routine in one embodiment. This routine
block 1113. The dialing may be unsuccessful if, for example, is invoked when the external port receives a message. This
the dialed process recently disconnected from the broadcast routine retrieves the message, identifies the external mes
channel. In block 1105, the routine sends an external mes Sage type, and invokes the appropriate routine to handle that
Sage to the dialed process requesting a connection to the message. This routine loops processing each message until
broadcast channel (i.e., connection request call). In block 45 all the received messages have been handled. In block 1401,
1106, the routine receives the response message (i.e., the routine answers (e.g., picks up) the external port and
connection request resp). In decision block 1107, if the retrieves an external message. In decision block 1402, if a
response message is Successfully received, then the routine message was retrieved, then the routine continues at block
1113. In block 1108, the routine sets the expected number of 50 1415 and returns. In decision block 1403, if the message
three. In block 1109, the routine sets the estimated diameter 55 message type is for a connection request call (i.e.,
ues at block 1113. In block 1112, the routine invokes the add 60 proposal call), then the routine invokes the handle edge
cally occurs when the broadcast channel is in the Small type is port connect call (i.e., port connect call), then the
for a neighbor is performed. In block 1113, the routine hangs 65 block 1410, else the routine continues at block 1411. In
computer and then returns. Statement (i.e., connected Stmt), the routine invokes the
US 6,910,069 B1
21 22
is currently fully connected to the broadcast channel iden 15 a connected process through which the calling process can
broadcast channel. In block 1505, the routine sends the 25 the new neighbor. The external port becomes the internal
FIG. 16 is a flow diagram illustrating processing of the is connecting to its first neighbor and the routine continues
process to establish an internal connection with this proceSS process. In block 1706, the routine installs an internal
fying a process to which the calling process can connect. In 35 invoked when a message is received from that new neighbor
decision block 1601, if this process is currently fully con through the internal port of that new neighbor. In decision
nected to the broadcast channel, then the routine continues block 1707, if this process buffered up messages while not
at block 1603, else the routine hangs up on the external port fully connected, then the routine continues at block 1708,
in block 1602 and returns. In block 1603, the routine sets the else the routine continues at block 1709. In one embodiment,
number of holes that the calling process should expect in the 40 a process that is partially connected may buffer the messages
response message. In block 1604, the routine Sets the that it receives through an internal connection So that it can
estimated diameter in the response message. In block 1605, Send these messages as it connects to new neighbors. In
the routine indicates whether this proceSS is ready to connect block 1708, the routine sends the buffered messages to the
to the calling process. This process is ready to connect when new neighbor through the internal port. In decision block
the number of its holes is greater than Zero and the calling 45 1709, if the number of holes of this process equals the
process is not a neighbor of this process. In block 1606, the expected number of holes, then this process is fully con
routine Sends to the calling process an external message that nected and the routine continues at block 1710, else the
is responsive to the connection request call (i.e., routine continues at block 1711. In block 1710, the routine
connection request resp). In block 1607, the routine notes invokes the achieve connected routine to indicate that this
the number of holes that the calling process needs to fill as 50 process is fully connected. In decision block 1711, if the
indicated in the request message. In decision block 1608, if number of holes for this process is Zero, then the routine
this process is ready to connect to the calling process, then continues at block 1712, else the routine returns. In block
the routine continues at block 1609, else the routine contin 1712, the routine deletes any pending edges and then
ues at block 1611. In block 1609, the routine invokes the add returns. A pending edge is an edge that has been proposed to
neighbor routine to add the calling proceSS as a neighbor. In 55 this process for edge pinning, which in this case is no longer
block 1610, the routine decrements the number of holes that needed.
the calling proceSS needs to fill and continues at block 1611. FIG. 18 is a flow diagram illustrating the processing of the
In block 1611, the routine hangs up on the external port. In forward connection edge Search routine in one embodiment.
decision block 1612, if this process has no holes or the This routine is responsible for passing along a request to
estimated diameter is greater than one (i.e., in the large 60 connect a requesting process to a randomly Selected neigh
regime), then the routine continues at block 1613, else the bor of this process through the internal port of the Selected
routine continues at block 1616. In blocks 1613-1615, the neighbor, that is part of the random walk. In decision block
routine loops forwarding a request for an edge through 1801, if the forwarding distance remaining is greater than
which to connect to the calling process to the broadcast Zero, then the routine continues at block 1804, else the
channel. One request is forwarded for each pair of holes of 65 routine continues at block 1802. In decision block 1802, if
the calling process that needs to be filled. In decision block the number of neighbors of this proceSS is greater than one,
1613, if the number of holes of the calling process to be then the routine continues at block 1804, else this broadcast
US 6,910,069 B1
23 24
channel is in the Small regime and the routine continues at routine continues at block 2002, else the routine continues at
call internal message (i.e., connection edge Search call) (i.e., port connection resp) to the Sending process that
to a randomly selected neighbor. In block 1804, the routine indicates that it is not okay to connect to this process. The
and the routine returns, else the routine continues at block block 2005, if the sending of the message was successful,
Successful, then the routine continues at block 1808, else the 15 neighbor of this process and then returns. In block 2007, the
routine loops to block 1804 to select the next neighbor. routine hangs up the external connection. In block 2008, the
another neighbor by invoking the fill holes routine to fill a fill hole routine in one embodiment. This routine is passed
fill two holes. In block 1808, the routine notes that the requesting to fill a hole, then this routine Sends an internal
recently Sent connection edge Search call has not yet been message to other processes. If another proceSS is requesting
acknowledged and indicates that the edge to this neighbor is 25 to fill a hole, then this routine invokes the routine to handle
reserved if the remaining forwarding distance is less than or a connection port Search request. In block 2101, the routine
equal to one. It is reserved because the Selected neighbor initializes a connection port Search Statement internal mes
may offer this edge to the requesting process for edge Sage (i.e., connection port Search Stnt). In decision
pinning. The routine then returns. block 2102, if this process is the requesting process, then the
FIG. 19 is a flow diagram illustrating the processing of the routine continues at block 2103, else the routine continues at
handle edge proposal call routine. This routine is invoked block 2104. In block 2103, the routine distributes the
when a message is received from a proposing process that message to the neighbors of this process through the internal
proposes to connect an edge between the proposing proceSS ports and then returns. In block 2104, the routine invokes the
and one of its neighbors to this proceSS for edge pinning. In handle connection port Search routine and then returns.
decision block 1901, if the number of holes of this process 35 FIG.22 is a flow diagram illustrating the processing of the
minus the number of pending edges is greater than or equal internal dispatcher routine in one embodiment. This routine
to one, then this process still has holes to be filled and the is passed an indication of the neighbor who sent the internal
routine continues at block 1902, else the routine continues at message. In block 2201, the routine receives the internal
block 1911. In decision block 1902, if the proposing process message. This routine identifies the message type and
or its neighbor is a neighbor of this process, then the routine 40 invokes the appropriate routine to handle the message. In
continues at block 1911, else the routine continues at block block 2202, the routine assesses whether to change the
1903. In block 1903, the routine indicates that the edge is estimated diameter of the broadcast channel based on the
pending between this proceSS and the proposing process. In information in the received message. In decision block 2203,
decision block 1904, if a proposed neighbor is already if this proceSS is the originating process of the message or
pending as a proposed neighbor, then the routine continues 45 the message has already been received (i.e., a duplicate),
at block 1911, else the routine continues at block 1907. In then the routine ignores the message and continues at block
block 1907, the routine sends an edge proposal response as 2208, else the routine continues at block 2203A. In decision
an external message to the proposing process (i.e., edge block 2203A, if the process is partially connected, then the
proposal resp) indicating that the proposed edge is routine continues at block 2203B, else the routine continues
accepted. In decision block 1908, if the sending of the 50 at block 2204. In block 2203B, the routine adds the message
message was Successful then the routine continues at block to the pending connection buffer and continues at block
proceSS on the external port as a neighbor. The routine then 55 type of the message is broadcast Statement (i.e., broadcast
the routine returns. In block 1913, the routine invokes the fill 60 the routine continues at block 2209, else the routine contin
FIG.20 is a flow diagram illustrating the processing of the internal messages in a buffer while partially connected So
then indicates that the Sending proceSS wants to connect to 65 process assumes that it is now fully connected and that the
US 6,910,069 B1
25 26
broadcaster component queues messages from each origi 15 requesting process for edge pinning. In decision block 2801,
FIG.24 is a flow diagram illustrating the processing of the 25 process and the decremented forwarding distance. The rou
the message to this process. In block 2401, the routine already been offered to a process, then the routine continues
have already been selected, then the routine returns. In block Search routine passing an indication of the requesting party
2403, the routine sends the message to the selected neighbor and a toggle indicator that alternatively indicates to continue
and then loops to block 2401 to select the next neighbor. the random walk for one or two more computers. The routine
FIG. 26 is a flow diagram illustrating the processing of the 35 then continues at block 2815. In block 2806, the routine dials
embodiment. This routine is passed an indication of the routine sends an edge proposal call external message (i.e.,
neighbor that Sent the message and the message itself. In edge proposal call) and receives the response (i.e., edge
block 2601, the routine invokes the distribute internal mes proposal resp). ASSuming that the response is Successfully
Sage which sends the message to each of its neighbors other 40 received, the routine continues at block 2808. In decision
than the sending neighbor. In decision block 2602, if the block 2808, if the response indicates that the edge is
number of holes of this proceSS is greater than Zero, then the acceptable to the requesting process, then the routine con
routine continues at block 2603, else the routine returns. In tinues at block 2809, else the routine continues at block
decision block 2603, if the requesting process is a neighbor, 2812. In block 2809, the routine reserves the edge between
then the routine continues at block 2605, else the routine 45 this process and the sending neighbor. In block 2810, the
continues at block 2604. In block 2604, the routine invokes routine adds the requesting process as a neighbor by invok
the court neighbor routine and then returns. The court ing the add neighbor routine. In block 2811, the routine
hole, then the neighbors with empty ports condition exists 50 2815. In decision block 2813, if this process is the requesting
message to the requesting neighbor. 55 connection edge Search response message (i.e., connection
neighbor, then it sends a port connection call external 60 has a cycle whose length is odd. The broadcaster component
tive neighbor as a neighbor. In decision block 2701, if the between even and odd distances.
2702, the routine dials the prospective neighbor. In decision 65 embodiment. This routine is passed as indication of the
US 6,910,069 B1
27 28
been received and if the forwarding distance is less than or nects to the process that Sent the message. In decision block
the routine returns. In block 2903, the routine reserves the block 3303, the routine removes the selected neighbor as a
routine then returns.
decision block 2906, if the invoked routine was FIG. 34 is a flow diagram illustrating the processing of the
unsuccessful, then the routine continues at block 2907, else handle condition double check routine. This routine deter
the routine returns. In decision block 2907, if the number of mines whether the neighbors with empty ports condition
holes of this proceSS is greater than Zero, then the routine 15
really is a problem or whether the broadcast channel is in the
continues at block 2908, else the routine returns. In block small regime. In decision block 3401, if this process has one
2908, the routine invokes the fill hole routine and then hole, then the routine continues at block 3402, else the
returns. routine continues at block 3403. If this process does not have
FIG.30 is a flow diagram illustrating the processing of the one hole, then the Set of neighbors of this proceSS is not the
broadcast routine in one embodiment. This routine is Same as the Set of neighbors of the Sending process. In
invoked by the application program to broadcast a message decision block 3402, if this proceSS and the Sending process
on the broadcast channel. This routine is passed the message have the same Set of neighbors, then the broadcast channel
to be broadcast. In decision block 3001, if this process has is not in the Small regime and the routine continues at block
at least one neighbor, then the routine continues at block 3403, else the routine continues at block 3406. In decision
3002, else the routine returns since it is the only process 25 block 3403, if this process has no holes, is then the routine
connected to be broadcast channel. In block 3002, the returns, else the routine continues at block 3404. In block
routine generates an internal message of the broadcast 3404, the routine sets the estimated diameter for this process
statement type (i.e., broadcast stmt). In block 3003, the to one. In block 3405, the routine broadcasts a diameter reset
routine Sets the Sequence number of the message. In block internal message (i.e., diameter reset) indicating that the
3004, the routine invokes the distribute internal message estimated diameter is one and then returns. In block 3406,
routine to broadcast the message on the broadcast channel. the routine creates a list of neighbors of this process. In
The routine returns. block 3407, the routine sends the condition check message
FIG.31 is a flow diagram illustrating the processing of the (i.e., condition check Stmt) with the list of neighbors to
acquire message routine in one embodiment. The acquire the neighbor who sent the condition double check message
message routine may be invoked by the application program 35 and then returns.
or by a callback routine provided by the application pro From the above description, it will be appreciated that
gram. This routine returns a message. In block 3101, the although specific embodiments of the technology have been
routine pops the message from the message queue of the described, various modifications may be made without devi
broadcast channel. In decision block 3102, if a message was ating from the Spirit and Scope of the invention. For
retrieved, then the routine returns an indication of Success, 40 example, the communications on the broadcast channel may
else the routine returns indication of failure. be encrypted. Also, the channel instance or Session identifier
FIGS. 32-34 are flow diagrams illustrating the processing may be a very large number (e.g., 128 bits) to help prevent
of messages associated with the neighbors with empty ports an unauthorized user to maliciously tap into a broadcast
condition. FIG. 32 is a flow diagram illustrating processing channel. The portal computer may also enforce Security and
of the handle condition check message in one embodiment. 45 not allow an unauthorized user to connect to the broadcast
This message is sent by a neighbor process that has one hole channel. Accordingly, the invention is not limited except by
and has received a request to connect to a hole of this the claims.
process. In decision block 3201, if the number of holes of What is claimed is:
this process is equal to one, then the routine continues at 1. A computer-based, non-routing table based, non-Switch
block 3202, else the neighbors with empty ports condition 50 based method for adding a participant to a network of
does not exist any more and the routine returns. In decision participants, each participant being connected to three or
block 3202, if the sending neighbor and this process have the more other participants, the method comprising:
same set of neighbors, the routine continues at block 3203, identifying a pair of participants of the network that are
else the routine continues at block 3205. In block 3203, the connected wherein a Seeking participant contacts a
routine initializes a condition double check message (i.e., 55 fully connected portal computer, which in turn Sends an
condition double check) with the list of neighbors of this edge connection request to a number of randomly
process. In block 3204, the routine Sends the message Selected neighboring participants to which the Seeking
internally to a neighbor other than Sending neighbor. The participant is to connect;
routine then returns. In block 3205, the routine selects a disconnecting the participants of the identified pair from
neighbor of the Sending process that is not also a neighbor 60 each other; and
of this process. In block 3206, the routine sends a condition connecting each participant of the identified pair of par
repair message (i.e., condition repair stmt) externally to ticipants to the Seeking participant.
the selected process. In block 3207, the routine invokes the 2. The method of claim 1 wherein each participant is
add neighbor routine to add the Selected neighbor as a connected to 4 participants.
neighbor of this process and then returns. 65 3. The method of claim 1 wherein the identifying of a pair
FIG. 33 is a flow diagram illustrating processing of the includes randomly Selecting a pair of participants that are
handle condition repair Statement routine in one embodi connected.
US 6,910,069 B1
29 30
4. The method of claim 3 wherein the randomly selecting 13. The method of claim 1 wherein the participants are
of a pair includes Sending a message through the network on computer processes.
a randomly Selected path. 14. A computer-based, non-Switch based method for add
5. The method of claim 4 wherein when a participant ing nodes to a graph that is m-regular and m-connected to
receives the message, the participant Sends the message to a maintain the graph as m-regular, where m is four or greater,
randomly Selected participant to which it is connected. the method comprising:
6. The method of claim 4 wherein the randomly selected
path is proportional to the diameter of the network. identifying p pairs of nodes of the graph that are
7. The method of claim 1 wherein the participant to be connected, where p is one half of m, wherein a Seeking
added requests a portal computer to initiate the identifying node contacts a fully connected portal node, which in
of the pair of participants. turn Sends an edge connection request to a number of
8. The method of claim 7 wherein the initiating of the randomly Selected neighboring nodes to which the
identifying of the pair of participants includes the portal Seeking node is to connect;
computer Sending a message to a connected participant disconnecting the nodes of each identified pair from each
requesting an edge connection. 15
other; and
9. The method of claim 8 wherein the portal computer connecting each node of the identified pairs of nodes to
indicates that the message is to travel a distance proportional the Seeking node.
to the diameter of the network and wherein the participant
that receives the message after the message has traveled that 15. The method of claim 14 wherein identifying of the p
distance is one of the participants of the identified pair of pairs of nodes includes randomly Selecting a pair of con
participants. nected nodes.
10. The method of claim 9 wherein the certain distance is 16. The method of claim 14 wherein the nodes are
twice the diameter of the network. computers and the connections are point-to-point commu
11. The method of claim 1 wherein the participants are nications connections.
connected via the Internet. 25 17. The method of claim 14 wherein m is even.
12. The method of claim 1 wherein the participants are
connected via TCP/IP connections.
EXHIBIT 6
USOO6920497B1

Bourassa et al. (45) Date of Patent: Jul.19, 2005
(54) CONTACTING A BROADCAST CHANNEL 5,737,526 A 4/1998 Periasamy et al.

5,754,830 A 5/1998 Butts et al.
(75) Inventors: Virgil E. Bourassa, Bellevue, WA 5,757,795 A * 5/1998 Schnell - - - - - - - - - - - - - - - - - - - - - - - 370/392
(US); Fred B. Holt, Seattle, WA (US) 5,761,425 A 6/1998 Miller
5,764,756 A 6/1998 Onweller
5,790,548 A 8/1998 Sistanizadeh et al.
(73) ASSignee: Its posing Company, Seattle, WA 5,790,553 A 8/1998 Deaton, Jr. et al.
5,799,016 A 8/1998 Onweller
patent is extended or adjusted under 35
... A
5,864,711 A
3.E. RNin
1/1999 Mairs et al.
U.S.C. 154(b) by 750 days. 5,867,660 A 2/1999 Schmidt et al.
5,867,667 A 2/1999 Butman et al.
(21) Appl. No.: 09/629,572 5,870,605 A 2/1999 Bracho et al.
5,874,960 A 2/1999 Mairs et al.
(22) Filed: Jul. 31, 2000 5,883,894. A * 3/1999 Patel et al. ................. 370/438
5,899.980 A 5/1999 Wilf et al.
(51) Int. Cl." ................................................ G06F 15/16 39A 5. SR et al
(52) U.S. C. - - - - - 709/227; 709/250 5928,335 A 7/1999 Morita
(58) Field of Search ................................. 709/249, 250,
709/227; 370/389, 392, 463; 710/62 (Continued)
(56) References Cited OTHER PUBLICATIONS
U.S. PATENT DOCUMENTS Bandyopadhyay et al., “A Flexible Architecture for Multi
4.912,656 A 3/1990 Cain et al. Hop Optical Networks.” Oct. 1998, 7th International Con
5,056,085. A 10/1991 Vu ference on Computer Communications and Networks, 1998,
5,058,105. A 10/1991. Mansour et al. pp. 472–478.
5,079,767 A 1/1992 Perlman
5,099.235 A 3/1992 Crookshanks (Continued)
5,101,480 A 3/1992 Shin
5,117,422 A 5/1992 Hauptschein Primary Examiner Bradley Edelman
5,309.437 A 5/1994 Perlman et al. (74) Attorney, Agent, or Firm-Perkins Coie LLP
5,345,558 A 9/1994 Opher
5,426,637 A 6/1995 Derby et al. (57) ABSTRACT
SE A 1919. its still A method of connecting tO a network through a portal
5,511,168 A 4/1996 Perlman computer. A Seeking computer dials the communications
5.535,199 A 7/1996 Amri et al. ports of a portal computer until it locates a call-in port. A
5,568,487. A 10/1996 Sitbon et al. port ordering algorithm is used to identify the call-in port.
5,636,371 A 6/1997 Yu Communications ports Selected by the port ordering algo
5,644,714 A 7/1997 Kikinis rithm may be re-ordered. The Seeking computer uses the
5,673,265 A 9/1997 Gupta et al. Selected call-in port to request that the portal computer
5,696,903 A 12/1997 Mahany coordinate the connection of the Seeking computer to the
5,732,074 A 3/1998 Spaur et al. network
5,732,086 A 3/1998 Liang
5,732.219 A 3/1998 Blumer et al.
5,734,865 A 3/1998 Yu 16 Claims, 39 Drawing Sheets
US 6,920.497 B1
Page 2
U.S. PATENT DOCUMENTS Peercy et al., “Distributed Algorithms for Shortest-Path,

5,935,215 A 8/1999 Bell et al.
Deadlock-Free Routing and Broadcasting in Arbitrarily
5,946,316 A 8/1999 Chen et al. Faulty Hypercubes,” Jun. 1990, 20th International Sympo
5.948,054 A 9/1999 Nielsen sium on Fault-Tolerant Computing, 1990, pp-218-225.
5,949.975 A 9/1999 Batty Yavatkar et al., “A reliable Dissemination Protocol for
5,956.484 A 9/1999 Rosenberg et al. Interactive Collaborative Applications.” Proc. ACM Multi
5,970.232 A 10/1999 Passint et al. media, 1995, p. 333-344; http://citeseer.nj.nec.com/article/
5.974.043 A 10/1999 Solomon yavatkar95reliable.htlm.
5.987,506 A 11/1999 Carter et al.
6,003,088 A 12/1999 Houston et al. PR Newswire, “Microsoft Boosts Accessibility to Internet
6,013,107 A 1/2000 Blackshear et al. Gaming Zone with Latest Release, Apr. 27, 1998, pp 1 ff.
6,023,734. A 2/2000 Ratcliff et al. PR Newswire, “Microsoft Annouces Launch Date for Ultra
6,029,171 A 2/2000 Smiga et al. Corps, Its Second Premium Title for the Internet Gaming
6,032,188 A 2/2000 Mairs et al. Zone,” Mar. 27, 1998, pp 1 ff.
6,038,602 A 3/2000 Ishikawa
6,047.289 A 4/2000 Thorne et al. Business Wire, “Boeing Panthesis Complete SWAN Trans
6,065,063 A 5/2000 Abali action,” Jul. 22, 2002, pp 1 ff.
6,073,177 A 6/2000 Hebel et al. Azar, et al., “Routing Strategies for Fast Networks,” May
6,094,676 A 7/2000 Gray et al. 1992 INFOCOM 92, Eleventh Annual Joint Conference of
6,115,580 A 9/2000 Chuprun et al. the IEEE Computer and Communications Societies, vol. 1...,
6,151,633 A 11/2000 Hurst pp. 170–179.
6,167.432 A 12/2000 Jiang
6,173,314 B1 1/2001 Kurashima et al. U.S. Appl. No. 09/629,570, filed Jul. 31, 2000, Bourassa et
6,195,366 B1 * 2/2001 Kayashima et al. ........ 370/475 al
6,199,116 B1 3/2001 May et al. U.S. Appl. No. 09/629,577, filed Jul. 31, 2000, Bourassa et
6,216,177 B1 4/2001 Mairs et al. 1
6,223,212 B1 4/2001 Batty et al.
6,243,691 B1 6/2001 Fisher et al. U.S. Appl. No. 09/629,575, filed Jul. 31, 2000, Bourassa et
6.252,884 B1 6/2001 Hunter 1.
6,268.855 B1 7/2001 Mairs et al. U.S. Appl. No. 09/629,576, filed Jul. 31, 2000, Bourassa et
6,269,080 B1 7/2001 Kumar 1
all.
6,271,839 B1 8/2001 Mairs et al.
6,272.548 B1 8/2001 Cotter et al. ls. Appl. No. 09/629,023, filed Jul. 31, 2000, Bourassa et
6,285,363 B1 9/2001 Mairs et al.
6,304,928 B1 10/2001 Mairs et al. U.S. Appl. No. 09/629,043, filed Jul. 31, 2000, Bourassa et
6,321,270 B1 11/2001 Crawley 1
6,353,599 B1 3/2002 Bi et al. U.S. Appl. No. 09/629,024, filed Jul. 31, 2000, Bourassa et
6,415,270 B1 7/2002 Rackson 1
6,421,735 B1 * 7/2002 Jung et al. .................. 709/250
6,434622 B1 8/2002 Monteiro U.S. Appl. No. 09/629,042, filed Jul. 31, 2000, Bourassa et
6,449.251 B1 * 9/2002 Awadallah et al. ......... 370/229 al.
6,449,601 B1 9/2002 Friedland Murphy, Patricia, A., “The Next Generation Networking
6,463,078 B1 10/2002 Engstrom et al. Paradigm: Producer/Consumer Model.” Dedicated Systems
6,490,247 B1 12/2002 Gilbert Magazine-2000 (pp. 26-28).
6,505,289 B1 1/2003 Han
6,524,189 B1 2/2003 Rautila The Gamer's Guide, “First-Person Shooters,” Oct. 20, 1998
6,553,020 B1 4/2003 Hughes (4 pages).
6,603,742 B1 8/2003 Steele The O'Reilly Network, “Gnutella: Alive, Well, and Chang
6,618,752 B1 9/2003 Moore et al. ing Fast,” Jan. 25, 2001 (5 pages) http://www.open2p.com/
6,701,344 B1 3/2004 Holt
2002/0027896 A1 3/2002 Hughes et al. 1.pt/ . . . Accessed Jan. 29, 2002).
Oram, Andy, “Gnutella and Freenet Represents True Tech
OTHER PUBLICATIONS nological Innovation,” May 12, 2000 (7 pages) The O'Reilly
Network http://www.oreillynet.com/1pt . . . Accessed Jan.
Hsu, “On-Four-Connecting a Triconnected Graph,” Oct. 29, 2002).
1992, Annual Symposium on Foundations of Computer Internetworking Technologies Handbook, Chapter 43 (pp.
Science, 1992, pp. 70-79. 43-1-43-16).
Cho, et al., “A Flood Routing Method for Data Networks,” Oram, Andy, “Peer-to-Peer Makes the Internet Interesting
Sep.1997, Proceedings of 1997 International Conference on Again,” Sep. 22, 2000 (7 pages) The O'Reilly Network
Information, Communications and Signal Processing, Vol. 3, http://linux.oreillynet.com/1pt. . . Accessed Jan. 29, 2002).
pp. 1418-1422.
Shiokawa et al., “Performance Analysis on Network Con Monte, Richard, “The Random Walk for Dummies,'MIT
nective Probability of Multihop Network Under Correlated Undergraduate Journal of Mathematics (pp. 143-148).
Breakage,” Jun. 1996, 1996 IEEE International Conference Srinivasan, R., “XDR: External Data Representation Stan
on Communications, vol. 3, pp. 1581-1585. dard.” Sun Microsystems, Aug. 1995 (20 pages) Internet
Komine et al., “A Distributed Restoration Algorithm for RFC/STD/FYI/BCP Archives http://www.faqs.org/rfcs/
Multiple-Link and Node Failures of Transport Networks,” rfc1832.html Accessed Jan. 29, 2002).
Dec. 1999, IEEE Globecom 90, Communicaitons: Com A Databeam Corporate White Paper, “A Primer on the T120
mecting the Future, vol. 1, pp. 459-463. Series Standards,” Copyright 1995 (pp. 1-16).
US 6,920.497 B1
Page 3
Kessler, Gary, C., “An Overview of TCP/IP Protocols and The University of Warwick, Computer Science Open Days,
the Internet,” Apr. 23, 1999 (23 pages) Hill Associates, Inc. “Demonstration on the Problems of Distributed Systems.”
http://www.hill.com/library/publications/t . . . Accessed http://www.dcs. Warwick.ac.u. . . . Accessed Jan. 29, 2002).
Jan. 29, 2002).
Bondy, J.A., and Murty, U.S.R., “Graph Theory with Appli Alagar, S. and Venkatesan, S., “Reliable Broadcast in
cations,” Chapters 1-3 (pp. 1-47), 1976 American Elsevier Mobile Wireless Networks,” Department of Computer Sci
Publishing Co., Inc., New York, New York. ence, University of Texas at Dallas, Military Communica
Cormen, Thomas H. et al., Introduction to Algorithms, tions Conference, 1995, MILCOM 95 Conference Record,
Chapter 5.3 (pp. 84-91), Chapter 12 (pp. 218-243), Chapter
13 (p. 245), 1990, The MIT Press, Cambridge, Massachu IEEE San Diego, California, Nov. 5–8, 1995 (pp. 236-240).
setts, McGraw-Hill Book Company, New York. International Search Report for The Boeing Company, Inter
The Common Object Request Broker: Architecture and national Patent Application No. PCT/US01/24240, Jun. 5,
Specification, Revision 2.6, Dec. 2001, Chapter 12 (pp.
12-1-12-10), Chapter 13 (pp. 13-1-13-56), Chapter 16 2002 (7 pages).
(pp. 16-1-16-26), Chapter 18 (pp. 18-1-18-52), Chapter
20 (pp. 20-1-20–22). * cited by examiner
U.S. Patent Jul.19, 2005 Sheet 1 of 39

U.S. Patent Jul.19, 2005 Sheet 2 of 39 US 6,920,497 B1

U.S. Patent Jul.19, 2005 Sheet 6 of 39



U.S. Patent Jul.19, 2005 Sheet 10 Of 39 US 6,920,497 B1

?a
OC
R
le
SO
(9
Om
(81,1
9

Channel Instance,
801 Connect Aux Info)
Open call in port
802 Fig. 8
Set connect-time
803
instance)
804
<G> Y
805 806
Contacts
ess Y Achieve connection
O
N
808 807
Install external dispatcher Install external dispatcher
809
Connect request

w 902
Select next depth
select
Select next portal computer
next portal computer Fig. 9
A portal computers
selected
Check for external

cal
Contact process
1OO
1002
1005 1006
Add as connected portal Y Answering process Add as fellow seeking
Fig. 11
Was a fully
Y 1103 Return
computer
104
Y 1105
1106
Y
response
1109
111 1112
Ready to connect add neighbor
N 1113
Check for externa

call
Fig. 12
12O2
N
ype = F seeking
connection call

1301 Fig. 13
connected
1302
1303
External dispatcher
Fig. 14
14O1
Pick up and receive
external message
14O4.
In Handle seekin
1406
Handle connection
request call
1408
call
1410
Handie port
connection call
Connected statement Handle connected

Staternet

Stateet - repair statement
Handle seeking
connection call Fig. 15
1501 1503
Set message to indicate Fully connected Set message to not
connected connected
1504
Add other as fellow
seeking process
1505
indle connection
request call
1601 St
Set newcomer's
holes to expect
S4
response
S Fig. I6
SS
SO
Set newcomer's
holes to fill
608 SS
Ready Add neighbor
612
ole = = 0 or
diameter >
oles to fil
Add neighbor
Identifies calling party F.3. 17

Sets neighbor to
messages pending
ses if N
703 Y
O5
Connection state = 4.
installfor interal dispatch

new neighbor
7O7
se-- N
9.
Holes = a
expected holes
N
Y-Achieve connected
1711
Hole as a 0 Y Purge pending edges
N

Fig. 18
C.- 1804
neighbors
requestor Select random neighbor
All neighbors
selected
N
1806
1807
18O8
search call

Fig. 19

190
1912
N 908
Holes odd N Re
Y
Y
re. I
ill hole
u. 1909
1910
O Return ) Add neighbor
Handle port
connection call
Fig. 20 2001
N
Caller isb not Send external message

(point-connect-res
Y

(point-connect-resp, ok). C Return D
O05
N Y
2006
2008
a nighbor
come reques

dispatcher
2201.
220
Partially co

2005
is a broadcast Handle broadcast
Stateet message
TypeN2206 2007
shutdown Handle shutdown
statement - Stateer
N
Y Pending
connection buffer
on S. Is
N
2O
message queue N

O eSS
Fig. 23 2301 3.
message
23O2
message
Distribute message
24O1
All neighbor
selected
2403
Send internal
message

for search message
2601
Distribute internal O
Fig. 26
SO2
N-O Return D
Y
Is requestor
a neighbor
N
coungbor e
elerate
condition check
message wineighbors
26O7
to requestor
Fig. 27 701
Is prospect Y Return
a neighbor
N
2702
Dial prospect
C 703
4.
Send and receive
external message
Add neighbor
27O6
Hang up prospect

Fig. 28 801
Not
my message ll
holes >= Z.
28.13
ag
from this pt. &&
otes =
connection second
Y 2814 edge (requestor
remaining dist -
Fill hole (self)
A. 2815 Requestor
end interna is neighbor or edge connection edge
message (from reserved search (requestor,
neighbor, ack O)
806
2807
Send and receive
external message

resp. from neighbor
Fig. 29 2SO message
search response
Edge selected
Y S3
neighbor
4
Remove from neighbor
2905
court neighbor
Y 2908
Fill hole (self)
message
Fig. 30 OO1
Any neighbors
Y
3OO2
Generate internal
message
3
number
3004
Distribute internal
message
message
3101
Fig. 31
Pop message queue
Handle condition check

ch
201
O
Same set of
neighbors
Set up message with list elect a neighbor

of neighbors ofnotsending process
my neighbor
32O6
Send internal Send external message
32O7
Add neighbor
Handle condition
repair statement
Fig. 33
Holes s = 0

Remove selected
neighbor
3304
Add neighbor
Handle condition
double check
Fig. 34
34O1
Holes = s. N
Same
neighbors
Holes s as 0
O

to-from neighbor
US 6,920,497 B1
1 2
CONTACTING A BROADCAST CHANNEL (“CORBA). Client/server middleware systems are not par
ticularly well Suited to sharing of information among many
CROSS-REFERENCE TO RELATED participants. In particular, when a client Stores information
APPLICATIONS to be shared at the Server, each other client would need to
5 poll the Server to determine that new information is being
shared. Such polling places a very high overhead on the
No. 09/629,576, entitled “BROAD CASTING communications network. Alternatively, each client may
NETWORK, filed on Jul. 31, 2000, now U.S. Pat. No. register a callback with the Server, which the Server then
6.829,634; U.S. patent application Ser. No. 09/629,570, invokes when new information is available to be shared.
entitled “JOINING A BROADCASTING CHANNEL.” Such a callback technique presents a performance bottleneck
filed on Jul. 31, 2000; U.S. patent application Ser. No. because a Single Server needs to call back to each client
09/629,577, entitled “LEAVING A BROADCAST whenever new information is to be shared. In addition, the
CHANNEL, filed on Jul. 31, 2000, now U.S. Pat. No. reliability of the entire Sharing of information depends upon
6,732,147; U.S. patent application Ser. No. 09/629,575, the reliability of the Single Server. Thus, a failure at a single
entitled “BROADCASTING ON A BROADCAST
CHANNEL,” filed on Jul. 31, 2000; U.S. patent application
15 computer (i.e., the server) would prevent communications
Ser. No. 09/629,023, entitled “DISTRIBUTED AUCTION between any of the clients.
SYSTEM," filed on Jul. 31, 2000; U.S. patent application The multicasting network protocols allow the Sending of
Ser. No. 09/629,043, entitled “AN INFORMATION DELIV broadcast messages to multiple recipients of a network. The
ERY SERVICE, filed on Jul. 31, 2000, now U.S. Pat. No. current implementations of Such multicasting network pro
6,714,966; U.S. patent application Ser. No. 09/629,024, tocols tend to place an unacceptable overhead on the under
entitled “DISTRIBUTED CONFERENCING SYSTEM,” lying network. For example, UDP multicasting would
filed on Jul. 31, 2000; and U.S. patent application Ser. No. Swamp the Internet when trying to locate all possible par
09/629,042, entitled “DISTRIBUTED GAME ticipants. IP multicasting has other problems that include
ENVIRONMENT, filed on Jul. 31, 2000, now U.S. Pat. No. needing special-purpose infrastructure (e.g., routers) to Sup
6,701,344, the disclosures of which are incorporated herein
25 port the Sharing of information efficiently.
by reference. The peer-to-peer middleware communications Systems
BACKGROUND
35
client/server middleware, multicasting network protocols, the T. 120 Internet standard is a tree structure, which relies
and peer-to-peer middleware. Each of these communications on the root node of the tree for reliability of the entire
techniques have their advantages and disadvantages, but network. That is, each message must pass through the root
40
none is particularly well Suited to the Simultaneous sharing node in order to be received by all participants.
information in a timely manner to all participants who may 45
widely distributed.
The point-to-point network protocols, such as UNIX FIG. 1 illustrates a graph that is 4-regular and 4-connected
pipes, TCP/IP, and UDP, allow processes on different com which represents a broadcast channel.
puters to communicate via point-to-point connections. The FIG. 2 illustrates a graph representing 20 computers
interconnection of all participants using point-to-point 50 connected to a broadcast channel.
connections, while theoretically possible, does not Scale well FIGS. 3A and 3B illustrate the process of connecting a
as a number of participants grows. For example, each new computer Z to the broadcast channel.
participating process would need to manage its direct con FIG. 4A illustrates the broadcast channel of FIG. 1 with
nections to all other participating processes. Programmers,
however, find it very difficult to manage Single connections, 55
an added computer.
of information. 60 FIG. 5A illustrates the disconnecting of a computer from
The client/server middleware systems provide a server the broadcast channel in a planned manner.
that coordinates the communications between the various FIG. 5B illustrates the disconnecting of a computer from
clients who are sharing the information. The Server functions the broadcast channel in an unplanned manner.
as a central authority for controlling access to shared FIG. 5C illustrates the neighbors with empty ports con
resources. Examples of client/server middleware Systems 65 dition.
include remote procedure calls (“RPC"), database servers, FIG. 5D illustrates two computers that are not neighbors
and the common object request broker architecture who now have empty ports.
US 6,920,497 B1
3 4
large regime.
US 6,920,497 B1
S 6
12-15, 15-18, and 18-3). other computers. Four of the ports are referred to as “inter
the connection first locates a computer that is currently ally messages either through their internal ports of their connec
ered to be in the "large regime.” This description assumes 65.535. The first 2056 ports are reserved for specific appli
US 6,920,497 B1
7 8
diameter of the broadcast channel may increase when each message may have to travel a distance of four to reach the
Seeking computer uses the same found portal computer and receiving computer. The Second message only has to travel
establishes a connection to the broadcast channel directly a distance of one. Thus, it is possible for the Second message
through that found portal computer. Conceptually, the graph to reach the receiving computer before the first message.
becomes elongated in the direction of where the new nodes When the broadcast channel is in a steady state (i.e., no
are added. FIGS. 4A-4C illustrate that possible problem. computers connecting or disconnecting from the broadcast
FIG. 4A illustrates the broadcast channel of FIG. 1 with an
added computer. Computer J was connected to the broadcast channel), out-of-order messages are not a problem because
channel by edge pinning edges C-D and E-H to computer each computer will eventually receive both messages and
J. The diameter of this broadcast channel is still two. FIG. can queue messages until all earlier ordered messages are
4B illustrates the broadcast channel of FIG. 4A with an received. If, however, the broadcast channel is not in a
added computer. Steady State, then problem can occur. In particular, a com
Computer K was connected to the broadcast channel by puter may connect to the broadcast channel after the Second
edge pinning edges E-J and B-C to a computer K. The message has already been received and forwarded on by its
diameter of this broadcast channel is three, because the 15 new neighbors. When a new neighbor eventually receives
Shortest path from computer G to computer K is through the first message, it sends the message to the newly con
edges G-A, A-E, and E-K. FIG. 4C also illustrates the nected computer. Thus, the newly connected computer will
broadcast channel of FIG. 4A with an added computer. receive the first message, but will not receive the Second
Computer K was connected to the broadcast channel by edge message. If the newly connected computer needs to process
pinning edges D-G and E-J to computer K. The diameter of the messages in order, it would wait indefinitely for the
this broadcast channel is, however, still two. Thus, the Second message.
Selection of neighbors impacts the diameter of the broadcast One Solution to this problem is to have each computer
channel. To help minimize the diameter, the broadcast queue all the messages that it receives until it can Send them
technique uses a random Selection technique to identify the in their proper order to its neighbors. This Solution, however,
four neighbors of a computer in the Seeking connection State. 25 may tend to Slow down the propagation of messages through
The random Selection technique tends to distribute the the computers of the broadcast channel. Another Solution
connections to new Seeking computers throughout the com that may have leSS impact on the propagation Speed is to
puters of the broadcast channel which may result in Smaller queue messages only at computers who are neighbors of the
overall diameters. newly connected computers. Each already connected neigh
Broadcasting through the Graph bor would forward messages as it receives them to its other
AS described above, each computer that is connected to neighbors who are not newly connected, but not to the newly
the broadcast channel can broadcast messages onto the connected neighbor. The already connected neighbor would
broadcast channel and does receive all messages that are only forward messages from each originating computer to
broadcast on the broadcast channel. The computer that the newly connected computer when it can ensure that no
originates a message to be broadcast sends that message to 35 gaps in the messages from that originating computer will
each of its four neighbors using the internal connections. occur. In one embodiment, the already connected neighbor
When a computer receives a broadcast message from a may track the highest Sequence number of the messages
neighbor, it sends the message to its three other neighbors. already received and forwarded on from each originating
Each computer on the broadcast channel, except the origi computer. The already connected computer will Send only
nating computer, will thus receive a copy of each broadcast 40 higher numbered messages from the originating computers
message from each of its four neighbors. Each computer, to the newly connected computer. Once all lower numbered
however, only sends the first copy of the message that it messages have been received from all originating
receives to its neighbors and disregards Subsequently computers, then the already connected computer can treat
received copies. Thus, the total number of copies of a the newly connected computer as its other neighbors and
message that is Sent between the computerS is 3N+1, where 45 Simply forward each message as it is received. In another
N is the number of computers connected to the broadcast embodiment, each computer may queue messages and only
channel. Each computer Sends three copies of the message, forwards to the newly connected computer those messages
except for the originating computer, which Sends four copies as the gaps are filled in. For example, a computer might
of the message. receive messages 4 and 5 and then receive message 3. In
The redundancy of the message Sending helps to ensure 50 Such a case, the already connected computer would forward
the overall reliability of the broadcast channel. Since each queue messages 4 and 5. When message 3 is finally received,
computer has four connections to the broadcast channel, if the already connected computer will Send messages 3, 4, and
one computer fails during the broadcast of a message, its 5 to the newly connected computer. If messages 4 and 5 were
neighbors have three other connections through which they Sent to the newly connected computer before message 3,
will receive copies of the broadcast message. Also, if the 55 then the newly connected computer would process messages
internal connection between two computers is slow, each 4 and 5 and disregard message 3. Because the already
computer has three other connections through which it may connected computer queues messages 4 and 5, the newly
receive a copy of each message Sooner. connected computer will be able to proceSS message 3. It is
Each computer that originates a message numbers its own possible that a newly connected computer will receive a Set
messages Sequentially. Because of the dynamic nature of the 60 of messages from an originating computer through one
broadcast channel and because there are many possible neighbor and then receive another Set of message from the
connection paths between computers, the messages may be Same originating computer through another neighbor. If the
received out of order. For example, the distance between an Second Set of messages contains a message that is ordered
originating computer and a certain receiving computer may earlier than the messages of the first Set received, then the
be four. After Sending the first message, the originating 65 newly connected computer may ignore that earlier ordered
computer and receiving computer may become neighbors message if the computer already processed those later
and thus the distance between them changes to one. The first ordered messages.
US 6,920,497 B1
9 10
Decomposing the Graph Small regime, each computer will have less than four neigh
A connected computer disconnects from the broadcast bors. To detect this condition in the large regime, which
channel either in a planned or unplanned manner. When a would be a problem if not repaired, the first neighbor to
puter which is not already a neighbor of the receiving
cannot connect (e.g., the first and Second computers are computer. When the computer receives the condition repair
already connected), then the computers may try connecting request, it disconnects from one of its neighbors (other than
in various other combinations. If connections cannot be 15
the neighbor that is involved with the condition) and con
established, each computer broadcasts a message that it nects to the computer that Sent the condition repair request.
needs to establish a connection with another computer. Thus, one of the original neighbors involved in the condition
When a computer with an available internal port receives the will have had a port filled. However, two computers are still
message, it can then establish a connection with the com in need of a connection, the other original neighbor and the
puter that broadcast the message. FIGS.5A-5D illustrate the computer that is now disconnected from the computer that
disconnecting of a computer from the broadcast channel. received the condition repair request. Those two computers
FIG. 5A illustrates the disconnecting of a computer from the Send out port connection requests. If those two computers
broadcast channel in a planned manner. When computer H are not neighbors, then they will connect to each other when
decides to disconnect, it sends its list of neighbors to each of they receive the requests. If, however, the two computers are
its neighbors (computers A, E, F and I) and then disconnects 25 neighbors, then they repeat the condition repair proceSS until
from each of its neighbors. When computers A and I receive two non-neighbors are in need of connections.
the message they establish a connection between them as It is possible that the two original neighbors with the
indicated by the dashed line, and Similarly for computers E condition may have the same set of neighbors. When the
and F. neighbor that receives the condition check message deter
When a computer disconnects in an unplanned manner, mines that the Sets of neighbors are the Same, it sends a
Such as resulting from a power failure, the neighbors con condition double check message to one of its neighbors
nected to the disconnected computer recognize the discon other than the neighbor who also has the condition. When
nection when each attempts to Send its next message to the the computer receives the condition double check message,
now disconnected computer. Each former neighbor of the it determines whether it has the same Set of neighbors as the
disconnected computer recognizes that it is short one con 35 Sending computer. If So, the broadcast channel is in the Small
nection (i.e., it has a hole or empty port). When a connected regime and the condition is not a problem. If the Set of
computer detects that one of its neighbors is now neighbors are different, then the computer that received the
disconnected, it broadcasts a port connection request on the condition double check message Sends a condition check
broadcast channel, which indicates that it has one internal message to the original neighbors with the condition. The
port that needs a connection. The port connection request 40 computer that receives that condition check message directs
identifies the call-in port of the requesting computer. When one of it neighbors to connect to one of the original
a connected computer that is also short connection receives neighbors with the condition by Sending a condition repair
the connection request, it communicates with the requesting message. Thus, one of the original neighbors with the
computer through its external port to establish a connection condition will have its port filled.
between the two computers. FIG. 5B illustrates the discon 45 FIG. 5C illustrates the neighbors with empty ports con
necting of a computer from the broadcast channel in an dition. In this illustration, computer H disconnected in an
unplanned manner. In this illustration, computer H has unplanned manner, but computers F and I responded to the
disconnected in an unplanned manner. When each of its port connection request of the other and are now connected
neighbors, computers A, E, F, and I, recognizes the together. The other former neighbors of computer H, com
disconnection, each neighbor broadcasts a port connection 50 puters A and E, are already neighbors, which gives rise to the
request indicating that it needs to fill an empty port. AS neighbors with empty ports condition. In this example,
shown by the dashed lines, computers F and I and computers computer E received the port connection request from
A and E respond to each other's requests and establish a computer A, recognized the possible condition, and Sent
connection. (since they are neighbors via the internal connection) a
It is possible that a planned or unplanned disconnection 55 condition check message with a list of its neighbors to
may result in two neighbors each having an empty internal computer A. When computer A received the list, it recog
port. In Such a case, Since they are neighbors, they are nized that computer E has a different set of neighbor (i.e., the
already connected and cannot fill their empty ports by broadcast channel is in the large regime). Computer A
connecting to each other. Such a condition is referred to as Selected computer D, which is a neighbor of computer E and
the “neighbors with empty ports' condition. Each neighbor 60 Sent it a condition repair request. When computer D received
broadcasts a port connection request when it detects that it the condition repair request, it disconnected from one of its
has an empty port as described above. When a neighbor neighbors (other than computer E), which is computer G in
receives the port connection request from the other neighbor, this example. Computer D then connected to computer A.
it will recognize the condition that its neighbor also has an FIG. 5D illustrates two computers that are not neighbors
empty port. Such a condition may also occur when the 65 who now have empty ports. Computers E and G now have
broadcast channel is in the Small regime. The condition can empty ports and are not currently neighbors. Therefore,
only be corrected when in the large regime. When in the computers E and G can connect to each other.
US 6,920,497 B1
11 12
FIGS. 5E and 5F further illustrate the neighbors with and only Selects each port number once. In addition, every
empty ports condition. FIG.5E illustrates the neighbors with time the algorithm is executed on any computer for a given
port.
computer C. Computer C recognizes that the broadcast If many computers are at the same time Seeking connec
channel is in the Small regime because is also has the same tion to a broadcast channel through a single portal computer,
Set of neighbors as computers A and B, computer C may then then the ports of the portal computer may be busy when
broadcast a message indicating that the broadcast channel is 15 called by Seeking computers. The Seeking computers would
in the Small regime. typically need to keep on redialing a busy port. The process
FIG. 5F illustrates the situation of FIG. 5E when in the of locating a call-in port may be significantly slowed by Such
large regime. AS discussed above, computer C receives the redialing. In one embodiment, each Seeking computer may
condition double check message from computer B. In this each reorder the first few port numbers generated by the
case, computer C recognizes that the broadcast channel is in hashing algorithm. For example, each Seeking computer
the large regime because it has a set of neighbors that is could randomly reorder the first eight port numbers gener
different from computer B. The edges extending up from ated by the hashing algorithm. The random ordering could
computer C and D indicate connections to other computers. also be weighted where the first port number generated by
Computer C then Sends a condition check message to the hashing algorithm would have a 50% chance of being
computer B. When computer B receives the condition check 25 first in the reordering, the Second port number would have
message, it sends a condition repair message to one of the a 25% chance of being first in the reordering, and So on.
neighbors of computer C. The computer that receives the Because the Seeking computers would use different
condition repair message disconnects from one of its orderings, the likelihood of finding a busy port is reduced.
neighbors, other than computer C, and tries to connect to For example, if the first eight port numbers are randomly
computer B and the neighbor from which it disconnected Selected, then it is possible that eight Seeking computers
tries to connect to computer A. could be simultaneously dialing ports in different Sequences
Port Selection which would reduce the chances of dialing a busy port.
As described above, the TCP/IP protocol designates ports Locating a Portal Computer
above number 2056 as user ports. The broadcast technique Each computer that can connect to the broadcast channel
uses five user port numbers on each computer: one external 35 has a list of one or more portal computers through which it
port and four internal ports. Generally, user ports cannot be can connect to the broadcast channel. In one embodiment,
Statically allocated to an application program because other each computer has the same Set of portal computers. A
applications programs executing on the same computer may Seeking computer locates a portal computer that is connected
use conflicting port numbers. As a result, in one to the broadcast channel by Successively dialing the ports of
embodiment, the computers connected to the broadcast 40 each portal computer in the order Specified by an algorithm.
channel dynamically allocate their port numbers. Each com A Seeking computer could Select the first portal computer
puter could simply try to locate the lowest number unused and then dial all its ports until a call-in port of a computer
port on that computer and use that port as the call-in port. A that is fully connected to the broadcast channel is found. If
Seeking computer, however, does not know in advance the no call-in port is found, then the Seeking computer would
call-in port number of the portal computers when the port 45 Select the next portal computer and repeat the proceSS until
numbers are dynamically allocated. Thus, a Seeking com a portal computer with Such a call-in port is found. A
puter needs to dial ports of a portal computer Starting with problem with Such a Seeking technique is that all user ports
the lowest port number when locating the call-in port of a of each portal computer are dialed until a portal computer
portal computer. If the portal computer is connected to (or fully connected to the broadcast channel is found. In an
attempting to connect to) the broadcast channel, then the 50 alternate embodiment, the Seeking computer Selects a port
Seeking computer would eventually find the call-in port. If number according to the algorithm and then dials each portal
the portal computer is not connected, then the Seeking computer at that port number. If no acceptable call-in port to
computer would eventually dial every user port. In addition, the broadcast channel is found, then the Seeking computer
if each application program on a computer tried to allocate Selects the next port number and repeats the process. Since
low-ordered port numbers, then a portal computer may end 55 the call-in ports are likely allocated at lower-ordered port
up with a high-numbered port for its call-in port because numbers, the Seeking computer first dials the port numbers
many of the low-ordered port numbers would be used by that are most likely to be call-in ports of the broadcast
other application programs. Since the dialing of a port is a channel. The Seeking computers may have a maximum
relatively slow process, it would take the Seeking computer search depth, that is the number of ports that it will dial when
a long time to locate the call-in port of a portal computer. To 60 Seeking a portal computer that is fully connected. If the
minimize this time, the broadcast technique uses a port Seeking computer exhausts its Search depth, then either the
ordering algorithm to identify the port number order that a broadcast channel has not yet been established or, if the
portal computer should use when finding an available port Seeking computer is also a portal computer, it can then
for its call-in port. In one embodiment, the broadcast tech establish the broadcast channel with itself as the first fully
nique uses a hashing algorithm to identify the port order. The 65 connected computer.
algorithm preferably distributes the ordering of the port When a Seeking computer locates a portal computer that
numbers randomly through out the user port number Space is itself not fully connected, the two computers do not
US 6,920,497 B1
13 14
connect when they first locate each other because the neighbors with a new distance to travel. In one embodiment,
Searching to a depth of nine. 15 its estimated diameter and broadcasts an estimated diameter
In one embodiment, each computer may have a different message. When a computer receives an estimated diameter
computer are preferably Selected randomly from the Set of 25 may use 64-bit integers. AS another example, one computer
currently connected computers. One advantage of the broad may use ASCII to represent text and another computer may
and the failure of any one computer (actually any three may send multiple messages in a single message Stream. The
makes it difficult for a portal computer to randomly Select 35 retrieve the next message in the Stream. The retrieval of each
four neighbors for a Seeking computer. message may require two calls to the operating System: one
puter again Sends the edge connection request message 40 Slow in comparison to the invocations of local routines. To
through one of its internal connections that is randomly overcome the inefficiencies of Such repeated calls, the broad
Selected. This Sending of the message corresponds to a cast technique in once embodiment, uses XDR to identify
random walk through the graph that represents the broadcast the message boundaries in a Stream of messages. The
channel. Eventually, a receiving computer will decide that broadcast technique may request the operating System to
the message has traveled far enough to represent a randomly 45 provide the next, for example, 1,024 bytes from the Stream.
Selected computer. That receiving computer will offer the The broadcast technique can then repeatedly invoke the
internal connection upon which it received the edge con XDR routines to retrieve the messages and use the Success
nection request message to the Seeking computer for edge or failure of each invocation to determine whether another
pinning. Of course, if either of the computers at the end of block of 1,024 bytes needs to be retrieved from the operating
the offered internal connection are already neighbors of the 50 system. The invocation of XDR routines do not involve
Seeking computer, then the Seeking computer cannot connect System calls and are thus more efficient than repeated System
through that internal connection. The computer that decided calls.
that the message has traveled far enough will detect this M-Regular
condition of already being a neighbor and Send the message In the embodiment described above, each fully connected
to a randomly Selected neighbor. 55 computer has four internal connections. The broadcast tech
In one embodiment, the distance that the edge connection nique can be used with other numbers of internal connec
request message travels is established by the portal computer tions. For example, each computer could have 6, 8, or any
to be approximately twice the estimated diameter of the even number of internal connections. AS the number of
broadcast channel. The message includes an indication of internal connections increase, the diameter of the broadcast
the distance that it is to travel. Each receiving computer 60 channel tends to decrease, and thus propagation time for a
decrements that distance to travel before Sending the mes message tends to decrease. The time that it takes to connect
Sage on. The computer that receives a message with a a Seeking computer to the broadcast channel may, however,
distance to travel that is Zero is considered to be the increase as the number of internal connections increases.
randomly Selected computer. If that randomly Selected com When the number of internal connectors is even, then the
puter cannot connect to the Seeking computer (e.g., because 65 broadcast channel can be maintained as m-regular and
it is already connected to it), then that randomly selected m-connected (in the steady State). If the number of internal
computer forwards the edge connection request to one of its connections is odd, then when the broadcast channel has an
US 6,920,497 B1
15 16
odd number of computers connected, one of the computers message that is broadcast on the broadcast channel.
will have less than that odd number of internal connections. Alternatively, the application program may provide a call
In Such a situation, the broadcast network is neither back routine (which may be a virtual function provided by
m-regular nor m-connected. When the next computer con the application program) that the broadcaster component
nects to the broadcast channel, it can again become invokes to notify the application program that a broadcast
m-regular and m-connected. Thus, with an odd number of message has been received. Each broadcaster component
internal connections, the broadcast channel toggles between allocates a call-in port using the hashing algorithm. When
being and not being m-regular and m-connected. calls are answered at the call-in port, they are transferred to
Components other ports that Serve as the external and internal ports.
FIG. 6 is a block diagram illustrating components of a The computers connecting to the broadcast channel may
computer that is connected to a broadcast channel. The include a central processing unit, memory, input devices
above description generally assumed that there was only one (e.g., keyboard and pointing device), output devices (e.g.,
broadcast channel and that each computer had only one display devices), and storage devices (e.g., disk drives). The
connection to that broadcast channel. More generally, a memory and Storage devices are computer-readable medium
network of computers may have multiple broadcast 15 that may contain computer instructions that implement the
channels, each computer may be connected to more than one broadcaster component. In addition, the data structures and
broadcast channel, and each computer can have multiple message Structures may be Stored or transmitted via a signal
connections to the same broadcast channel. The broadcast transmitted on a computer-readable media, Such as a com
channel is well Suited for computer processes (e.g., appli munications link.
cation programs) that execute collaboratively, Such as net FIG. 7 is a block diagram illustrating the Sub-components
work meeting programs. Each computer process can connect of the broadcaster component in one embodiment. The
to one or more broadcast channels. The broadcast channels broadcaster component includes a connect component 701,
can be identified by channel type (e.g., application program an external dispatcher 702, an internal dispatcher 703 for
name) and channel instance that represents separate broad each internal connection, an acquire message component
cast channels for that channel type. When a proceSS attempts 25 704 and a broadcast component 712. The application pro
to connect to a broadcast channel, it seeks a process cur gram may provide a connect callback component 710 and a
rently connected to that broadcast channel that is executing receive response component 711 that are invoked by the
on a portal computer. The Seeking process identifies the broadcaster component. The application program invokes
broadcast channel by channel type and channel instance. the connect component to establish a connection to a des
Computer 600 includes multiple application programs ignated broadcast channel. The connect component identi
601 executing as Separate processes. Each application pro fies the external port and installs the external dispatcher for
gram interfaces with a broadcaster component 602 for each handling messages that are received on the external port.
broadcast channel to which it is connected. The broadcaster The connect component invokes the Seek portal computer
component may be implement as an object that is instanti component 705 to identify a portal computer that is con
ated within the proceSS Space of the application program. 35 nected to the broadcast channel and invokes the connect
Alternatively, the broadcaster component may execute as a request component 706 to ask the portal computer (if fully
Separate process or thread from the application program. In connected) to select neighbor processes for the newly con
one embodiment, the broadcaster component provides func necting process. The external dispatcher receives external
tions (e.g., methods of class) that can be invoked by the messages, identifies the type of message, and invokes the
application programs. The primary functions provided may 40 appropriate handling routine 707. The internal dispatcher
include a connect function that an application program receives the internal messages, identifies the type of
invokes passing an indication of the broadcast channel to message, and invokes the appropriate handling routine 708.
which the application program wants to connect. The appli The received broadcast messages are Stored in the broadcast
cation program may provide a callback routine that the message queue 709. The acquire message component is
broadcaster component invokes to notify the application 45 invoked to retrieve messages from the broadcast queue. The
program that the connection has been completed, that is the broadcast component is invoked by the application program
proceSS enters the fully connected State. The broadcaster to broadcast messages in the broadcast channel.
component may also provide an acquire message function The following tables list messages Sent by the broadcaster
that the application program can invoke to retrieve the next components.
EXTERNAL MESSAGES

seeking connection call Indicates that a seeking process would like to know whether the
receiving process is fully connected to the broadcast channel
connection request call Indicates that the sending process would like the receiving
process to initiate a connection of the sending process to the
broadcast channel
edge proposal call Indicates that the sending process is proposing an edge through
channel (i.e., edge pinning)
port connection call Indicates that the sending process is proposing a port through
channel
connected stimt Indicates that the sending process is connected to the broadcast
channel
US 6,920,497 B1
17 18
-continued
EXTERNAL MESSAGES

condition repair stimt Indicates that the receiving process should disconnect from one
of its neighbors and connect to one of the processes involved in
INTERNAL MESSAGES

broadcast stimt Indicates a message that is being broadcast through the
broadcast channel for the application programs
connection port search stimt Indicates that the designated process is looking for a port
connection edge search call Indicates that the requesting process is looking for an edge
connection edge search resp Indicates whether the edge between this process and the
sending neighbor has been accepted by the requesting
party
diameter estimate stimt Indicates an estimated diameter of the broadcast channel
diameter reset stimt Indicates to reset the estimated diameter to indicated
diameter
disconnect stimt Indicates that the sending neighbor is disconnecting from
the broadcast channel
condition check stimt Indicates that neighbors with empty port condition have
been detected
condition double check stimt Indicates that the neighbors with empty ports have the
same set of neighbors
shutdown stimt Indicates that the broadcast channel is being shutdown
Flow Diagrams block 803, the routine invokes the seek portal computer
FIGS. 8-34 are flow diagrams illustrating the processing 35
routine passing the channel type and channel instance. The
of the broadcaster component in one embodiment. FIG. 8 is Seek portal computer routine attempts to locate a portal
a flow diagram illustrating the processing of the connect computer through which this process can connect to the
routine in one embodiment. This routine is passed a channel broadcast channel for the passed type and instance. In
type (e.g., application name) and channel instance (e.g., decision block 804, if the seek portal computer routine is
Session identifier), that identifies the broadcast channel to 40 Successful in locating fully connected process on that portal
which this proceSS wants to connect. The routine is also computer, then the routine continues at block 805, else the
passed auxiliary information that includes the list of portal routine returns an unsuccessful indication. In decision block
computers and a connection callback routine. When the 805, if no portal computer other than the portal computer on
connection is established, the connection callback routine is which the process is executing was located, then this is the
invoked to notify the application program. When this pro 45 first process to fully connect to broadcast channel and the
ceSS invokes this routine, it is in the Seeking connection routine continues at block 806, else the routine continues at
State. When a portal computer is located that is connected block 808. In block 806, the routine invokes the achieve
and this routine connects to at least one neighbor, this connection routine to change the State of this process to fully
proceSS enters the partially connected State, and when the connected. In block 807, the routine installs the external
process eventually connects to four neighbors, it enters the 50 dispatcher for processing messages received through this
fully connected state. When in the small regime, a fully process external port for the passed channel type and
connected process may have leSS than four neighbors. In channel instance. When a message is received through that
block 801, the routine opens the call-in port through which external port, the external dispatcher is invoked. The routine
the process is to communicate with other processes when then returns. In block 808, the routine installs an external
establishing external and internal connections. The port is 55 dispatcher. In block 809, the routine invokes the connect
Selected as the first available port using the hashing algo request routine to initiate the process of identifying neigh
rithm described above. In block 802, the routine sets the bors for the Seeking computer. The routine then returns.
connect time to the current time. The connect time is used to FIG. 9 is a flow diagram illustrating the processing of the
identify the instance of the process that is connected through Seek portal computer routine in one embodiment. This
this external port. One proceSS may connect to a broadcast 60 routine is passed the channel type and channel instance of
channel of a certain channel type and channel instance using the broadcast channel to which this proceSS wishes to
one call-in port and then disconnects, and another proceSS connect. This routine, for each Search depth (e.g., port
may then connect to that same broadcast channel using the number), checks the portal computers at that search depth. If
Same call-in port. Before the other proceSS becomes fully a portal computer is located at that Search depth with a
connected, another process may try to communicate with it 65 process that is filly connected to the broadcast channel, then
thinking it is the fully connected old process. In Such a case, the routine returns an indication of Success. In blockS
the connect time can be used to identify this situation. In 902-911, the routine loops selecting each search depth until
US 6,920,497 B1
19 20
a process is located. In block 902, the routine selects the next requests a process of a portal computer that was identified as
Search depth using a port number ordering algorithm. In being fully connected to the broadcast channel to initiate the
decision block 903, if all the search depths have already connection of this process to the broadcast channel. In
been Selected during this execution of the loop, that is for the decision block 1101, if at least one process of a portal
currently Selected depth, then the routine returns a failure computer was located that is fully connected to the broadcast
indication, else the routine continues at block 904. In blocks channel, then the routine continues at block 1103, else the
904-911, the routine loops selecting each portal computer routine continues at block 1102. A process of the portal
and determining whether a process of that portal computer computer may no longer be in the list if it recently discon
is connected to (or attempting to connect to) the broadcast nected from the broadcast channel. In one embodiment, a
channel with the passed channel type and channel instance. Seeking computer may always Search its entire Search depth
In block 904, the routine selects the next portal computer. In and find multiple portal computers through which it can
decision block 905, if all the portal computers have already connect to the broadcast channel. In block 1102, the routine
been selected, then the routine loops to block 902 to select restarts the process of connecting to the broadcast channel
the next Search depth, else the routine continues at block
computer through the port represented by the Search depth. 15 one of the found portal computers through the call-in port.
In decision block 907, if the dialing was successful, then the In decision block 1104, if the dialing is successful, then the
routine continues at block 908, else the routine loops to routine continues at block 1105, else the routine continues at
block 904 to select the next portal computer. The dialing will block 1113. The dialing may be unsuccessful if, for example,
which contacts the answering process of the portal computer 1106, the routine receives the response message (i.e.,
is fully connected to the broadcast channel. In block 909, the 25 response message is Successfully received, then the routine
routine hangs up on the Selected portal computer. In decision continues at block 1108, else the routine continues at block
block 910, if the answering process is fully connected to the 1113. In block 1108, the routine sets the expected number of
broadcast channel, then the routine returns a Success holes (i.e., empty internal connections) for this process
indicator, else the routine continues at block 911. In block based on the received response. When in the large regime,
911, the routine invokes the check for external call routine the expected number of holes is zero. When in the small
to determine whether an external call has been made to this regime, the expected number of holes varies from one to
process as a portal computer and processes that call. The three. In block 1109, the routine sets the estimated diameter
routine then loops to block 904 to select the next portal of the broadcast channel based on the received response. In
computer. decision block 11111, if the dialed process is ready to
FIG. 10 is a flow diagram illustrating the processing of the 35 connect to this proceSS as indicated by the response message,
contact process routine in one embodiment. This routine then the routine continues at block 1112, else the routine
determines whether the process of the Selected portal com continues at block 1113. In block 1112, the routine invokes
puter that answered the call-in to the Selected port is fully the add neighbor routine to add the answering process as a
connected to the broadcast channel. In block 1001, the neighbor to this process. This adding of the answering
routine sends an external message (i.e., seeking 40 process typically occurs when the broadcast channel is in the
connection call) to the answering process indicating that a Small regime. When in the large regime, the random walk
Seeking proceSS wants to know whether the answering search for a neighbor is performed. In block 1113, the
proceSS is fully connected to the broadcast channel. In block routine hangs up the external connection with the answering
1002, the routine receives the external response message process computer and then returns.
from the answering process. In decision block 1003, if the 45 FIG. 12 is a flow diagram of the processing of the check
time out period. If the external message is not received 50 answer a call on the call-in port. In decision block 1202, if
broadcaster component may repeat the receive request Sev 55 a Seeking process is calling (i.e., Seeking connection call),
response message that it is fully connected to the broadcast ceSS indicating that this process is also is Seeking a connec
channel, then the routine continues at block 1005, else the 60 tion. In decision block 1206, if the sending of the external
process eS and then returns. 65 process can find no process that is fully connected to the
US 6,920,497 B1
21 22
necting to the broadcast channel. For example, a fellow is not fully connected, then it is attempting to connect to the
Seeking process may become the first process fully con broadcast channel. In block 1505, the routine sends the
nected to the broadcast channel. external message response (i.e., Seeking connection resp)
FIG. 13 is a flow diagram of the processing of the achieve to the Seeking process and then returns.
connection routine in one embodiment. This routine Sets the FIG. 16 is a flow diagram illustrating processing of the
State of this process to fully connected to the broadcast handle connection request call routine in one embodiment.
channel and invokes a callback routine to notify the appli This routine is invoked when the calling process wants this
cation program that the process is now fully connected to the process to initiate the connection of the process to the
requested broadcast channel. In block 1301, the routine sets broadcast channel. This routine either allows the calling
the connection State of this process to fully connected. In process to establish an internal connection with this proceSS
block 1302, the routine notifies fellow seeking processes (e.g., if in the Small regime) or starts the process of identi
that it is fully connected by Sending a connected external fying a process to which the calling process can connect. In
message to them (i.e., connected stmt). In block 1303, the decision block 1601, if this process is currently fully con
application program and then returns. 15 at block 1603, else the routine hangs up on the external port
is invoked when the external port receives a message. This response message. In block 1604, the routine Sets the
the routine answers (e.g., picks up) the external port and process is not a neighbor of this process. In block 1606, the
message was retrieved, then the routine continues at block 25 is responsive to the connection request call (i.e.,
1403, else the routine hangs up on the external port in block connection request resp). In block 1607, the routine notes
1415 and returns. In decision block 1403, if the message the number of holes that the calling process needs to fill as
type is for a process Seeking a connection (i.e., Seeking indicated in the request message. In decision block 1608, if
connection call), then the routine invokes the handle Seek this process is ready to connect to the calling process, then
ing connection call routine in block 1404, else the routine the routine continues at block 1609, else the routine contin
continues at block 1405. In decision block 1405, if the ues at block 1611. In block 1609, the routine invokes the add
message type is for a connection request call (i.e., neighbor routine to add the calling process as a neighbor. In
connection request call), then the routine invokes the block 1610, the routine decrements the number of holes that
handle connection request call routine in block 1406, else the calling proceSS needs to fill and continues at block 1611.
the routine continues at block 1407. In decision block 1407, 35 In block 1611, the routine hangs up on the external port. In
if the message type is edge proposal call (i.e., edge decision block 1612, if this process has no holes or the
proposal call), then the routine invokes the handle edge estimated diameter is greater than one (i.e., in the large
proposal call routine in block 1408, else the routine contin regime), then the routine continues at block 1613, else the
ues at block 1409. In decision block 1409, if the message routine continues at block 1616. In blocks 1613-1615, the
type is port connect call (i.e., port connect call), then the 40 routine loops forwarding a request for an edge through
routine invokes the handle port connection call routine in which to connect to the calling process to the broadcast
block 1410, else the routine continues at block 1411. In channel. One request is forwarded for each pair of holes of
decision block 1411, if the message type is a connected the calling process that needs to be filled. In decision block
Statement (i.e., connected Stmt), the routine invokes the 1613, if the number of holes of the calling process to be
handle connected Statement in block 1412, else the routine 45 filled is greater than or equal to two, then the routine
continues at block 1412.a. In decision block 1412a, if the continues at block 1614, else the routine continues at block
to block 1414 to process the next message. After each 50 distance. In one embodiment, the distance is twice in the
handle Seeking connection call routine in one embodiment. 55 routine returns. In block 1617, the routine invokes the fill
is currently fully connected to the broadcast channel iden a connected process through which the calling process can
tified in the message, then the routine continues at block 60 connect to the broadcast channel. The routine then returns.
indicate that this proceSS is not fully connected. In block 65 process on the external port. In block 1702, the routine sets
US 6,920,497 B1
23 24
ensure that there are no gaps in the messages initially Sent to recently Sent connection edge Search call has not yet been
the new neighbor. The external port becomes the internal acknowledged and indicates that the edge to this neighbor is
port for this connection. In decision block 1703, if this reserved if the remaining forwarding distance is less than or
proceSS is in the Seeking connection State, then this proceSS equal to one. It is reserved because the Selected neighbor
is connecting to its first neighbor and the routine continues may offer this edge to the requesting process for edge
at block 1704, else the routine continues at block 1705. In pinning. The routine then returns.
block 1704, the routine sets the connection state of this FIG. 19 is a flow diagram illustrating the processing of the
process to partially connected. In block 1705, the routine handle edge proposal call routine. This routine is invoked
adds the calling process to the list of neighbors of this when a message is received from a proposing process that
process. In block 1706, the routine installs an internal proposes to connect an edge between the proposing proceSS
dispatcher for the new neighbor. The internal dispatcher is and one of its neighbors to this proceSS for edge pinning. In
invoked when a message is received from that new neighbor decision block 1901, if the number of holes of this process
fully connected, then the routine continues at block 1708, 15 routine continues at block 1902, else the routine continues at
a process that is partially connected may buffer the messages or its neighbor is a neighbor of this process, then the routine
1709, if the number of holes of this process equals the pending as a proposed neighbor, then the routine continues
routine continues at block 1711. In block 1710, the routine 25 an external message to the proposing process (i.e., edge
invokes the achieve connected routine to indicate that this proposal resp) indicating that the proposed edge is
process is fully connected. In decision block 1711, if the accepted. In decision block 1908, if the sending of the
number of holes for this process is Zero, then the routine message was Successful, then the routine continues at block
continues at block 1712, else the routine returns. In block 1909, else the routine returns. In block 1909, the routine
1712, the routine deletes any pending edges and then adds the edge as a pending edge. In block 1910, the routine
returns. A pending edge is an edge that has been proposed to invokes the add neighbor routine to add the proposing
this process for edge pinning, which in this case is no longer process on the external port as a neighbor. The routine then
needed. returns. In block 1911, the routine Sends an external message
FIG. 18 is a flow diagram illustrating the processing of the (i.e., edge proposal resp) indicating that this proposed
forward connection edge Search routine in one embodiment. 35 edge is not accepted. In decision block 1912, if the number
This routine is responsible for passing along a request to of holes is odd, then the routine continues at block 1913, else
connect a requesting process to a randomly Selected neigh the routine returns. In block 1913, the routine invokes the fill
bor of this process through the internal port of the Selected hole routine and then returns.
neighbor, that is part of the random walk. In decision block FIG.20 is a flow diagram illustrating the processing of the
1801, if the forwarding distance remaining is greater than 40 handle port connection call routine in one embodiment. This
Zero, then the routine continues at block 1804, else the routine is invoked when an external message is received
routine continues at block 1802. In decision block 1802, if then indicates that the Sending proceSS wants to connect to
the number of neighbors of this proceSS is greater than one, one hole of this process. In decision block 2001, if the
then the routine continues at block 1804, else this broadcast number of holes of this proceSS is greater than Zero, then the
channel is in the Small regime and the routine continues at 45 routine continues at block 2002, else the routine continues at
call internal message (i.e., connection edge Search call) to 50 (i.e., port connection resp) to the Sending process that
a randomly selected neighbor. In block 1804, the routine indicates that it is not okay to connect to this process. The
and the routine returns, else the routine continues at block 55 block 2005, if the sending of the message was successful,
Successful, then the routine continues at block 1808, else the neighbor of this process and then returns. In block 2007, the
routine loops to block 1804 to select the next neighbor. 60 routine hangs up the external connection. In block 2008, the
another neighbor by invoking the fill holes routine to fill a 65 fill hole routine in one embodiment. This routine is passed
fill two holes. In block 1808, the routine notes that the requesting to fill a hole, then this routine Sends an internal
US 6,920,497 B1
25 26
message to other processes. If another proceSS is requesting FIG.24 is a flow diagram illustrating the processing of the
to fill a hole, then this routine invokes the routine to handle distribute broadcast message routine in one embodiment.
a connection port Search request. In block 2101, the routine This routine Sends the broadcast message to each of the
initializes a connection port Search Statement internal mes neighbors of this process, except for the neighbor who sent
Sage (i.e., connection port Search Stnt). In decision the message to this process. In block 2401, the routine
block 2102, if this process is the requesting process, then the Selects the next neighbor other than the neighbor who sent
routine continues at block 2103, else the routine continues at the message. In decision block 2402, if all such neighbors
block 2104. In block 2103, the routine distributes the have already been Selected, then the routine returns. In block
message to the neighbors of this process through the internal 2403, the routine sends the message to the selected neighbor
ports and then returns. In block 2104, the routine invokes the and then loops to block 2401 to select the next neighbor.
handle connection port Search routine and then returns. FIG. 26 is a flow diagram illustrating the processing of the
FIG.22 is a flow diagram illustrating the processing of the
internal dispatcher routine in one embodiment. This routine handle connection port Search Statement routine in one
message. This routine identifies the message type and 15 block 2601, the routine invokes the distribute internal mes
invokes the appropriate routine to handle the message. In Sage which sends the message to each of its neighbors other
the message has already been received (i.e., a duplicate), then the routine continues at block 2605, else the routine
2208, else the routine continues at block 2203A. In decision the court neighbor routine and then returns. The court
block 2203A, if the process is partially connected, then the neighbor routine connects this process to the requesting
routine continues at block 2203B, else the routine continues 25 process if possible. In block 2605, if this process has one
at block 2204. In block 2203B, the routine adds the message hole, then the neighbors with empty ports condition exists
to the pending connection buffer and continues at block and the routine continues at block 2606, else the routine
2204. In decision blocks 2204-2207, the routine decodes the returns. In block 2606, the routine generates a condition
message type and invokes the appropriate routine to handle check message (i.e., condition check) that includes a list of
the message. For example, in decision block 2204, if the this process neighbors. In block 2607, the routine sends the
type of the message is broadcast Statement (i.e., broadcast message to the requesting neighbor.
stmt), then the routine invokes the handle broadcast message FIG.27 is a flow diagram illustrating the processing of the
routine in block 2205. After invoking the appropriate han court neighbor routine in one embodiment. This routine is
dling routine, the routine continues at block 2208. In deci passed an indication of the prospective neighbor for this
sion block 2208, if the partially connected buffer is full, then 35 process. If this proceSS can connect to the prospective
the routine continues at block 2209, else the routine contin neighbor, then it sends a port connection call external
ues at block 2210. The broadcaster component collects all its message to the prospective neighbor and adds the prospec
internal messages in a buffer while partially connected So tive neighbor as a neighbor. In decision block 2701, if the
that it can forward the messages as it connects to new prospective neighbor is already a neighbor, then the routine
neighbors. If, however, that buffer becomes full, then the 40 returns, else the routine continues at block 2702. In block
proceSS assumes that it is now fully connected and that the 2702, the routine dials the prospective neighbor. In decision
expected number of connections was too high, because the block 2703, if the number of holes of this process is greater
broadcast channel is now in the small regime. In block 2209, than Zero, then the routine continues at block 2704, else the
continues in block 2210. In decision block 2210, if the 45 sends a port connection call external message (i.e., port
response routine is a callback routine of the application 50 by invoking the add neighbor routine. In block 2706, the
indication of the neighbor who sent the broadcast message, 55 Sent the message and the message itself. This routine either
broadcaster component queues messages from each origi requesting process for edge pinning. In decision block 2801,
order to the application program. In block 2302, the routine 60 holes of the requesting process is still greater than or equal
else the routine returns. In block 2304, the routine sends the 65 2803, else the routine continues at block 2804. In block
US 6,920,497 B1
27 28
proceSS and the decremented forwarding distance. The rou 3002, else the routine returns since it is the only process
tine then continues at block 2815. In decision block 2804, if connected to be broadcast channel. In block 3002, the
the requesting process is a neighbor or the edge between this routine generates an internal message of the broadcast
proceSS and the Sending neighbor is reserved because it has statement type (i.e., broadcast stmt). In block 3003, the
already been offered to a process, then the routine continues routine Sets the Sequence number of the message. In block
at block 2805, else the routine continues at block 2806. In 3004, the routine invokes the distribute internal message
block 2805, the routine invokes the forward connection edge routine to broadcast the message on the broadcast channel.
Search routine passing an indication of the requesting party The routine returns.
and a toggle indicator that alternatively indicates to continue FIG.31 is a flow diagram illustrating the processing of the
the random walk for one or two more computers. The routine acquire message routine in one embodiment. The acquire
then continues at block 2815. In block 2806, the routine dials message routine may be invoked by the application program
the requesting process via the call-in port. In block 2807, the or by a callback routine provided by the application pro
proposal resp). ASSuming that the response is Successfully 15 broadcast channel. In decision block 3102, if a message was
received, the routine continues at block 2808. In decision retrieved, then the routine returns an indication of Success,
this process and the sending neighbor. In block 2810, the of the handle condition check message in one embodiment.
removes the Sending neighbor as a neighbor. In block 2812, process. In decision block 3201, if the number of holes of
the routine hangs up the external port and continues at block 25 this process is equal to one, then the routine continues at
2815. In decision block 2813, if this process is the requesting block 3202, else the neighbors with empty ports condition
proceSS and the number of holes of this proceSS equals one, does not exist any more and the routine returns. In decision
then the routine continues at block 2814, else the routine block 3202, if the sending neighbor and this process have the
continues at block 2815. In block 2814, the routine invokes same set of neighbors, the routine continues at block 3203,
the fill hole routine. In block 2815, the routine sends an else the routine continues at block 3205. In block 3203, the
connection edge Search response message (i.e., connection routine initializes a condition double check message (i.e.,
edge Search response) to the sending neighbor indicating condition double check) with the list of neighbors of this
acknowledgement and then returns. The graphs are Sensitive process. In block 3204, the routine Sends the message
to parity. That is, all possible paths starting from a node and internally to a neighbor other than Sending neighbor. The
ending at that node will have an even length unless the graph 35 routine then returns. In block 3205, the routine selects a
has a cycle whose length is odd. The broadcaster component neighbor of the Sending process that is not also a neighbor
uses a toggle indicator to vary the random walk distance of this process. In block 3206, the routine sends a condition
between even and odd distances. repair message (i.e., condition repair Stmt) externally to
FIG.29 is a flow diagram illustrating the processing of the the selected process. In block 3207, the routine invokes the
handle connection edge Search response routine in one 40 add neighbor routine to add the Selected neighbor as a
embodiment. This routine is passed as indication of the neighbor of this process and then returns.
requesting process, the Sending neighbor, and the message. FIG. 33 is a flow diagram illustrating processing of the
In block 2901, the routine notes that the connection edge handle condition repair Statement routine in one embodi
been received and if the forwarding distance is less than or 45 nects to the process that Sent the message. In decision block
the routine returns. In block 2903, the routine reserves the 50 block 3303, the routine removes the selected neighbor as a
decision block 2906, if the invoked routine was 55 routine then returns.
unsuccessful, then the routine continues at block 2907, else FIG. 34 is a flow diagram illustrating the processing of the
the routine returns. In decision block 2907, if the number of handle condition double check routine. This routine deter
holes of this proceSS is greater than Zero, then the routine mines whether the neighbors with empty ports condition
continues at block 2908, else the routine returns. In block really is a problem or whether the broadcast channel is in the
2908, the routine invokes the fill hole routine and then 60 small regime. In decision block 3401, if this process has one
returns. hole, then the routine continues at block 3402, else the
FIG.30 is a flow diagram illustrating the processing of the routine continues at block 3403. If this process does not have
broadcast routine in one embodiment. This routine is one hole, then the Set of neighbors of this proceSS is not the
invoked by the application program to broadcast a message Same as the Set of neighbors of the Sending process. In
on the broadcast channel. This routine is passed the message 65 decision block 3402, if this proceSS and the Sending process
to be broadcast. In decision block 3001, if this process has have the same Set of neighbors, then the broadcast channel
at least one neighbor, then the routine continues at block is not in the Small regime and the routine continues at block
US 6,920,497 B1
29 30
3403, else the routine continues at block 3406. In decision 5. The method of claim 3 wherein the hashing algorithm
block 3403, if this process has no holes, then the routine ordering is modified to reduce conflicts with other Seeking
returns, else the routine continues at block 3404. In block computers that use the same hashing algorithm.
3404, the routine sets the estimated diameter for this process 6. The method of claim 5 wherein a number of the first
to one. In block 3405, the routine broadcasts a diameter reset communications ports ordered by the hashing algorithm are
internal message (i.e., diameter reset) indicating that the reordered.
estimated diameter is one and then returns. In block 3406, 7. The method of claim 1 wherein the identification of a
the routine creates a list of neighbors of this process. In
block 3407, the routine sends the condition check message plurality of portal computerS is provided and when a com
(i.e., condition check Stmt) with the list of neighbors to 1O
munications port is Selected, attempting to communicate
the neighbor who sent the condition double check message with each of the identified portal computers through the
and then returns. Selected communications port before Selecting the next
From the above description, it will be appreciated that communications port.
although specific embodiments of the technology have been 8. The method of claim 1 wherein the communications
described, various modifications may be made without devi 15 ports are TCP/IP ports.
ating from the Spirit and Scope of the invention. For 9. A component in a computer System for locating a call-in
example, the communications on the broadcast channel may port of a portal computer, comprising:
be encrypted. Also, the channel instance or Session identifier
may be a very large number (e.g., 128 bits) to help prevent means for identifying the portal computer, the portal
an unauthorized user to maliciously tap into a broadcast computer having a dynamically Selected call-in port for
channel. The portal computer may also enforce Security and communicating with other computers,
not allow an unauthorized user to connect to the broadcast
channel. Accordingly, the invention is not limited except by means for identifying the call-in port of the identified
the claims. portal computer by repeatedly trying to establish a
What is claimed is: connection with the identified portal computer through
1. A method in a computer for locating a computer 25 contacting a communications port or communications
through which to connect to a network, the method com ports until a connection is Successfully established;
prising: means for Selecting the call-in port of the identified portal
providing an identification of a portal computer or a computer using a port ordering algorithm; and
plurality of portal computers, the portal computer or the means for re-ordering the communications ports Selected
plurality of portal computers having a communications by the port ordering algorithm.
port or communications ports with a call-in port being
enabled for communications when the portal computer 10. The component of claim 9 wherein the communica
or the plurality of portal computerS is in a State to tions ports are contacted in an order that is the same as that
coordinate the connection of a Seeking computer to the used by the portal computer when it dynamically Selects a
network, wherein the call-in port is a type of commu 35
communications port.
nications port, 11. The component of claim 9 wherein the communica
Selecting the communications port or communications tions ports are contacted based on a hashing algorithm
ports of the portal computer or the plurality of portal ordering.
computers and attempting to communicate with the 12. The component of claim 11 wherein the hashing
Selected communications port or communications ports 40 algorithm ordering provides an ordering in which each
until communications with the call-in port is communications port is tried without contacting a commu
Successful, wherein a port ordering algorithm is used to nications port more than once.
identify the call-in port, and wherein the communica 13. The component of claim 11 wherein the hashing
tions ports Selected by the port ordering algorithm may algorithm ordering is modified to reduce conflicts with other
be re-ordered; and 45 computers that use the same hashing algorithm.
using the call-in port to request that the portal computer 14. The component of claim 13 wherein a number of the
or the plurality of portal computers coordinate the first communications ports ordered by the hashing algorithm
connecting of the Seeking computer to the network. are reordered.
2. The method of claim 1 wherein the portal computer or 15. The component of claim 9 including:
the plurality of portal computerS Selects a call-in port, and 50
further wherein the communications ports are Selected in an means for identifying a plurality of portal computers, and
order that is the Same as used by the portal computer when means for trying to establish a connection with each of the
it Selected a call-in port. identified portal computers through a single communi
3. The method of claim 1 wherein the communications cations port before trying the next communications
ports are Selected based on a hashing algorithm ordering. 55
port.
4. The method of claim 3 wherein the hashing algorithm 16. The component of claim 9 wherein the communica
ordering provides an ordering in which each communica tions ports are TCP/IP ports.
tions port is Selected without re-Selecting a communications
port.

DATED : July 19, 2005
INVENTOR(S) : Bourassa et al.
Column 5
Line 21, “ally” should be -- fully --;
Column 8
Line 12, “problem” should be -- problems --.

Thirteenth Day of September, 2005
WDJ
JON W. DUDAS
Director of the United States Patent and Trademark Office
EXHIBIT 7
USOO7412537B2
(12) United States Patent (10) Patent No.: US 7.412,537 B2

Holt et al. (45) Date of Patent: Aug. 12, 2008
(54) METHOD FOR DETERMINING AN 5, 101480 A 3, 1992 Shin
ESTMATED DIAMETER OF A BROADCAST 5,117,422 A 5/1992 Hauptschein
CHANNEL 5,309.437 A 5/1994 Perlman et al.
5,345,558 A 9/1994 Opher
(75) Inventors: Fred B. Holt, Seattle, WA (US); Virgil 5.426,637 A 6/1995 Derby et al.
E. Bourassa, Bellevue, WA (US)
5,459,725 A 10, 1995 Bodner et al.
(73) Assignee: The Boeing Company, Chicago, IL 5,471,623 A 1 1/1995 Napolitano
(US) 5,511,168 A 4/1996 Perlman
5,535,199 A 7/1996 Amri et al.
patent is extended or adjusted under 35
U.S.C. 154(b) by 970 days.
(21) Appl. No.: 10/733,669 (Continued)
OTHER PUBLICATIONS
(22) Filed: Dec. 11, 2003
Jaekel et al. "A Flexible Architecture for Multi-Hop Optical Net
(65) Prior Publication Data works' Oct. 1998, 7" Internation Conference on Computer Commu
US 2004/O148348A1 Jul. 29, 2004 nications and Networks, pp. 472-478 provided by applicant's Aug.9,
2004 IDS.*
Related U.S. Application Data (Continued)
(62) Division of application No. 09/629,577, filed on Jul. Primary Examiner Saleh Naijar
31, 2000, now Pat. No. 6,732,147. Assistant Examiner—David Lazaro
(51) Int. Cl. (74) Attorney, Agent, or Firm Perkins Coie LLP
G06F 5/16 (2006.01)
HD4L 2/56 (2006.01) (57) ABSTRACT
(52) U.S. Cl. ....................... 709/242; 709/226; 709/228;
37Of 406
(58) Field of Classification Search ................. 709/238, A method for a node in a network to determine an estimated
709/241, 230, 236, 242, 226, 228: 370/351, diameter of a broadcast channel is disclosed. In particular
37Of 406 embodiments, when a node receives a message from a neigh
See application file for complete search history. bor computer, it can determine the received message’s dis
tance traveled and set that distance as an estimated diameter.
(56) References Cited It can then increment the distance traveled and forward the
U.S. PATENT DOCUMENTS message to a neighbor computer. The node may be configured
to set a new estimated diameter only if the new estimate is
4,742,511 A 5, 1988 Johnson ...................... 370/406 greater than the prior estimate. It may also be configured to
4,912,656 A 3, 1990 Cain et al.
5,056,085 A 10, 1991 Vu
send out a message broadcasting the new diameterestimate to
5,058,105 A 10, 1991 Mansour et al.
its neighbors.
5,079,767 A 1/1992 Perlman
5,099,235 A 3, 1992 CrookShanks 7 Claims, 39 Drawing Sheets
A
US 7412,537 B2
Page 2
U.S. PATENT DOCUMENTS 6,285,363 B1 9, 2001 Mairs et al.

6,304,928 B1 10/2001 Mairs et al.
5,568.487 10, 1996 Sitbon et al. 6,321,270 B1 1 1/2001 Crawley
5,636,371 6, 1997 Yu 6,353,599 B1 3/2002 Bi et al.
5,644,714 7, 1997 Kikinis 6,415,270 B1 7/2002 Rackson
5,673,265 9, 1997 Gupta et al. 6,434,622 B1 8, 2002 Monteiro
5,696,903 12, 1997 Mahany 6,449,601 B1 9, 2002 Friedland
5,732,074 3, 1998 Spauer et al. 6.463,078 B1 10/2002 Engstrom et al.
5,732,086 3, 1998 Liang 6,490,247 B1 12/2002 Gilbert
5,732,219 3, 1998 Blumer et al. 6,499,251 B2 12/2002 Weder
5,734,865 3, 1998 Yu 6,505,289 B1 1/2003 Han
5,737,526 4, 1998 Periasamy et al. 6,524, 189 B1 2/2003 Rautila
5,754,830 5, 1998 Butts et al. 6,553,020 B1 4/2003 Hughes
5,757,795 5, 1998 Schnell 6,603,742 B1 8, 2003 Steele
5,761.425 6, 1998 Miller 6,611,872 B1 8, 2003 McCanne
5,764,756 6, 1998 Onweller 6,618,752 B1 9, 2003 Moore et al.
5,790,548 8, 1998 Sistanizadeh et al. 6,701,344 B1 3, 2004 Holt
5,790,553 8, 1998 Deaton, Jr. et al. 2002fOO27896 A1 3/2002 Hughes et al.
5,799,016 8, 1998 Onweller
5,802,285 9, 1998 Nirviniemi OTHER PUBLICATIONS
5,850,592 12, 1998 Ramanathan
5,864,711 1, 1999 Mairs et al. Baker, F. “RFC 1812: Requirements for IP Version 4 Routers' Jun.
5,867,660 2, 1999 Schmidt et al. 1995, only Table of Contents and section 4.2.2.9. pp. 1-6 and 46-47.*
5,867,667 2, 1999 Butman et al. U.S. Appl. No. 09/629,570, filed Jul. 31, 2000, Bourassa et al.
5,870,605 2, 1999 Bracho et al. U.S. Appl. No. 09/629,576, filed Jul. 31, 2000, Bourassa et al.
5,874,960 2, 1999 Mairs et al. U.S. Appl. No. 09/629,575, filed Jul. 31, 2000, Bourassa et al.
5,899,980 5, 1999 Wilfetal. U.S. Appl. No. 09/629,572, filed Jul. 31, 2000, Bourassa et al.
5,907,610 5, 1999 Onweller U.S. Appl. No. 09/629,023, filed Jul. 31, 2000, Bourassa et al.
5,925,097 7, 1999 Gopinath et al. U.S. Appl. No. 09/629,043, filed Jul. 31, 2000, Bourassa et al.
5,928,335 7, 1999 Morita U.S. Appl. No. 09/629,024, filed Jul. 31, 2000, Bourassa et al.
5,935,215 8, 1999 Bell et al. U.S. Appl. No. 09/629,042, filed Jul. 31, 2000, Bourassa et al.
5,946,316 8, 1999 Chen et al. Azar et al., “Routing Strategies for Fast Networks.” May 1992,
9, 1999 Nielsen INFOCOM 92 Eleventh Annual Joint Conference of the IEEE Com
5.948,054
5,949,975 9/1999 Batty et al. puter Communications Societies, vol. 1, 170-179iiii.
5,953,318 9, 1999 Nattkemper et al. Bandyopadhyay et al., “A Flexible Architecture for Multi-Hop Opti
5,956,484 9, 1999 Rosenberg et al. cal Networks.” Oct. 1998, 7th International Conference on Computer
5,970,232 10, 1999 Passintet al. Communications and Networks, 1998, pp. 472-478.
5,974,043 10, 1999 Solomon Business Wire, “Boeing Panthesis Complete SWAN Transaction.”
5,987,506 11, 1999 Carter et al. Jul. 22, 2002, pp. 1 ff.
6,003,088 12, 1999 Houston et al. Cho, et al., “A Flood Routing Method for DataNetworks.” Sep.1997.
6,023,734 2, 2000 Ratcliff et al. Proceedings of 1997 International Conference on Information, Com
6,029,171 2, 2000 Smiga et al. munications and Signal Processing, vol. 3, pp. 1418-1422.
6,032,188 2, 2000 Mairs et al. Hsu, "On-Four-Connecting a Triconnected Graph.” Oct. 1992,
6,038,602 3, 2000 Ishikawa Annual Symposium on Foundations of Computer Science, 1992, pp.
4, 2000 Thorne et al. 70-79.
6,047.289 Komine et al., “A Distributed Restoration Algorithm for Multiple
6,065,063 5, 2000 Abali
6,073,177 6, 2000 Hebel et al. Link and Node Failures of Transport Networks.” Dec. 1999, IEEE
6,094,676 T/2000 Gray et al. GLOBECOM’90, Communicaitons: Commecting the Future, vol. 1,
6,115,580 9, 2000 Chuprun et al. pp. 459-463.
6,151,633 11, 2000 Hurst Peercy et al., “Distributed Algorithms for Shortest-Path, Deadlock
6,167.432 12, 2000 Jiang Free Routing and Broadcasting in Arbitrarily Faulty Hypercubes.”
6,173,314 1, 2001 Kurashima et al. Jun. 1990, 20th International Symposium on Fault-Tolerant Comput
6, 195,366 2, 2001 Kayashima ing, 1990, pp. 218-225.
6,199,116 3, 2001 May et al. PRNewswire, “Microsoft Annouces Launch Date for UltraCorps. Its
6,205,146 3, 2001 Rochberger et al. .... 370,395.53 Second Premium Title for the Internet Gaming Zone.” Mar. 27, 1998,
6,216,177 4, 2001 Mairs et al. pp. 1 ff.
6,223,212 4, 2001 Batty et al. PR Newswire, “Microsoft Boosts Accessibility to Internet Gaming
6,243,691 6, 2001 Fisher et al. Zone with Latest Release.” Apr. 27, 1998, pp. 1 ff.
6,252,884 6, 2001 Hunter Shiokawa et al., “Performance Analysis on Network Connective
6,268,855 T/2001 Mairs et al. Probability of Multihop Network Under Correlated Breakage.” Jun.
6,269,080 T/2001 Kumar 1996, 1996 IEEE International Conference on Communications, vol.
6,271,839 8, 2001 Mairs et al. 3, pp. 1581-1585.
6,272.548 8, 2001 Cotter et al. * cited by examiner
U.S. Patent Aug. 12, 2008 Sheet 2 of 39 US 7.412,537 B2

U.S. Patent US 7412,537 B2

U.S. Patent Aug. 12, 2008 Sheet 8 Of 39 US 7412,537 B2

U.S. Patent Aug. 12, 2008 Sheet 9 of39 US 7412,537 B2

U.S. Patent Aug. 12, 2008 Sheet 10 Of 39


9
(81,1
U.S. Patent Aug. 12, 2008 Sheet 13 of 39 US 7412,537 B2
00/
1
Z!
?sep Oug
14
!
109uoO
Channel Instance,
Connect Aux Info)
801
Open call in port
802 Fig. 8
Set connect-time
803
instance)
804
<C- Y
805 806
Contacts
Y Achieve connection
N 807
808
809
Connect request

902
Select next depth
Select
Select next
next portal
portal computer
computer Fig.e 9
905
selected
N 906
907
Y 908
conte process 909

computer
91O
Y
Return (success)
Contact process
1001
10O2
1005 1006
Add as connected portal Answering process Add as fellow seeking
Connect request
Fig. II 1101 1102
Was a fully N

computer
1104
N
Y 1105
1106
1112
Y Add neighbor
heck for externa

call
Fig. 12
ype = F seeking
connection call

Achieve connection
1301
Fig. 13
connected
1302
1303

1401
Pick up and receive
external message
1402
Message
1404
Handle seeking
connection call
1406
Handle connection
request call
1408
call
1410
Port connect call
Handle port
connection call
Handle connected

State ent - repair statement
Handle seeking
connection call
Fl9. 15
1502 1503
Add other as fellow

seeking process
1505
andle connection
s request call
1601
Connected din
Set newcomer's
holes to expect
160
1604
Return

response O
1605 F9. I6

Set newconer's
holes to fill
608 1609
Add neighbor
1610
Holes to fill - = Z.
1616 S1
ewcomer's Y
Qoles to fill > 9 > - Fill hole (requestor)
Add neighbor
17O
Identifies calling party F9. 17
170
Sets neighbor to
G.
messages pending
singes
<Seeking connectio
N
703 Y Connection state F O4
partially connected
1705
Add as neighbor
17O6
for new neighbor
11709
Holes = =
expected hole y-Achieve connected
1711 1.
1 if

Fig. 18 1801
Distance Y
remaining > 0
N
it of 18O
802 Y
neighbors
1804
requestor
selected
Return
1808
search call

proposal call Out message
Fig. 19
N 1903
proposed
neighbors
1911 pending
1907
N 1912 1 908
Y
1913 Y 1909
Fill hol
Finhole Add edge as pending
1910
| Add neighbor
Handle port
connection call
2003
neighbor
(point-connect-resp
not ok)

2006
Add neighbor
2008
Connect request
Fill hole
2101
Initialize internal
message
2103
Internal
dispatcher
22O1
Fig 22 Received internal message
22O2
Assess diameter
process ==
originating
Insert message into

pending connection buffer Type 2005
= = broadcast Handle broadcast
Statement message
fypeN2206 2007
= = shutdown Y Handle shutdown
Y Pending
connection buffer
aca, N. N
message queue
S-1 Y it
Receive response ( )

O eSSaC
F.9. 23 2301 g
message
23O2
message
2304
Distribute message
24O1
402
All neighbor
selected
2403
Send internal
message

for search message
2601
Distribute internal
Fig. 26
602
N
603 2604
Is requestor
a neighbor
Return
enerate
condition check
message w/neighbors
26O7
to requestor
Fig. 27 701
Is prospect
a neighbor
N
27O2
Dial prospect
703
N
Send and receive

external message
Add neighbor
27O6
Hang up prospect

Fig. 28 801
Not
N my message 11
holes >= Z.
M 2813
essage Y
N(from thispi && Forward 2803
Remaining connection second
distance20 edge (requestor
Y 2814 remaining dist-l
Fill hole (self)
Forward
A> 2815 8 Requestor connection edge
end interna is neighbor or edge search (requestor,
message (from reserved O)
neighbor, ack
2806
2807
Send and receive
external message

resp. from neighbor
search response
Edge selected
Y 2903
neighbor
2904
Remove from neighbor
2905
Court neighbor
message
Fig. 30
Generate internal
message
3003
number
3004
Distribute internal
message
message
Acquire message
3101
Fig. 31 Pop message queue
Message
retrieved
Same set of
neighbors
3205
elect a neighbor
3206
to selected neighbor
32O7
Add neighbor
Handle condition
repair statement
Fig. 33

Remove selected
neighbor
3304
Add neighbor
Handle condition
double check
Same set of
neighbors
N 3406
3407 3404

to-from neighbor
3405
US 7,412,537 B2
1. 2
METHOD FOR DETERMINING AN Examples of client/server middleware systems include
ESTMATED DIAMETER OF A BROADCAST remote procedure calls (“RPC), database servers, and the
CHANNEL common object requestbroker architecture (“CORBA). Cli
ent/server middleware systems are not particularly well
CROSS-REFERENCE TO RELATED Suited to sharing of information among many participants. In
APPLICATIONS particular, when a client stores information to be shared at the
server, each other client would need to poll the server to
This application is a divisional of U.S. patent application determine that new information is being shared. Such polling
Ser. No. 09/629,577, entitled “LEAVING A BROADCAST places a very high overhead on the communications network.
CHANNEL filed on Jul. 31, 2000 now U.S. Pat. No. 6,732, 10 Alternatively, each client may register a callback with the
147 and is related to U.S. patent application Ser. No. 09/629, server, which the server then invokes when new information
576, entitled “BROADCASTING NETWORK filed on Jul. is available to be shared. Such a callback technique presents a
31, 2000; U.S. patent application Ser. No. 09/629,570, performance bottleneck because a single server needs to call
entitled “JOINING A BROADCAST CHANNEL filed on back to each client whenever new information is to be shared.
Jul. 31, 2000; U.S. patent application Ser. No. 09/629,577, 15 In addition, the reliability of the entire sharing of information
“LEAVING A BROADCAST CHANNEL filed on Jul. 31, depends upon the reliability of the single server. Thus, a
2000; U.S. patent application Ser. No. 09/629,575, entitled failure at a single computer (i.e., the server) would prevent
“BROADCASTING ON A BROADCAST CHANNEL communications between any of the clients.
filed on Jul.31, 2000; U.S. patent application Ser. No. 09/629, The multicasting network protocols allow the sending of
572, entitled “CONTACTING A BROADCAST CHAN broadcast messages to multiple recipients of a network. The
NEL filed on Jul. 31, 2000; U.S. patent application Ser. No. current implementations of such multicasting network proto
09/629,023 entitled “DISTRIBUTED AUCTION SYS cols tend to place an unacceptable overhead on the underlying
TEM filed on Jul. 31, 2000; U.S. patent application Ser. No. network. For example, UDP multicasting would swamp the
09/629,043, entitled “AN INFORMATION DELIVERY Internet when trying to locate all possible participants. IP
SERVICE, filedon Jul.31, 2000; U.S. patent application Ser. 25 multicasting has other problems that include needing special
No. 09/629,024, entitled “DISTRIBUTED CONFERENC purpose infrastructure (e.g., routers) to Support the sharing of
ING SYSTEM filed on Jul. 31, 2000; and U.S. patent appli information efficiently.
cation Ser. No. 09/629,042, entitled “DISTRIBUTED The peer-to-peer middleware communications systems
GAME ENVIRONMENT filed on Jul. 31, 2000, the disclo rely on a multicasting network protocol or a graph of point
sures of which are incorporated herein by reference. 30 to-point network protocols. Such peer-to-peer middleware is
provided by the T. 120 Internet standard, which is used in such
TECHNICAL FIELD products as Data Connection’s D.C.-share and Microsoft’s
NetMeeting. These peer-to-peer middleware systems rely
The described technology relates generally to a computer upon a user to assemble a point-to-point graph of the connec
network and more particularly, to a broadcast channel for a 35 tions used for sharing the information. Thus, it is neither
Subset of a computers of an underlying network. suitable nor desirable to use peer-to-peer middleware systems
when more than a small number of participants is desired. In
BACKGROUND addition, the underlying architecture of the T. 120 Internet
standard is a tree structure, which relies on the root node of the
There are a wide variety of computer network communi 40 tree for reliability of the entire network. That is, each message
cations techniques such as point-to-point network protocols, must pass through the root node in order to be received by all
client/server middleware, multicasting network protocols, participants.
and peer-to-peer middleware. Each of these communications It would be desirable to have a reliable communications
techniques have their advantages and disadvantages, but none network that is suitable for the simultaneous sharing of infor
is particularly well Suited to the simultaneous sharing of 45 mation among a large number of the processes that are widely
information among computers that are widely distributed. For distributed.
example, collaborative processing applications, such as a net
work meeting programs, have a need to distribute information BRIEF DESCRIPTION OF THE DRAWINGS
in a timely manner to all participants who may be geographi
cally distributed. 50 FIG. 1 illustrates a graph that is 4-regular and 4-connected
The point-to-point network protocols, such as UNIX pipes, which represents a broadcast channel.
TCP/IP, and UDP, allow processes on different computers to FIG. 2 illustrates a graph representing 20 computers con
communicate via point-to-point connections. The intercon nected to a broadcast channel.
nection of all participants using point-to-point connections, FIGS. 3A and 3B illustrate the process of connecting a new
while theoretically possible, does not scale well as a number 55 computer Z to the broadcast channel.
of participants grows. For example, each participating pro FIG. 4A illustrates the broadcast channel of FIG. 1 with an
cess would need to manage its direct connections to all other added computer.
participating processes. Programmers, however, find it very FIG. 4B illustrates the broadcast channel of FIG. 4A with
difficult to manage single connections, and management of an added computer.
multiple connections is much more complex. In addition, 60 FIG. 4C also illustrates the broadcast channel of FIG. 4A
participating processes may be limited to the number of direct with an added computer.
connections that they can Support. This limits the number of FIG. 5A illustrates the disconnecting of a computer from
possible participants in the sharing of information. the broadcast channel in a planned manner.
The client/server middleware systems provide a server that FIG. 5B illustrates the disconnecting of a computer from
coordinates the communications between the various clients 65 the broadcast channel in an unplanned manner.
who are sharing the information. The server functions as a FIG. 5C illustrates the neighbors with empty ports condi
central authority for controlling access to shared resources. tion.
US 7,412,537 B2
3 4
FIG.5D illustrates two computers that are not neighbors DETAILED DESCRIPTION
who now have empty ports.
FIG. 5E illustrates the neighbors with empty ports condi A broadcast technique in which a broadcast channel over
tion in the Small regime. lays a point-to-point communications network is provided.
FIG. SF illustrates the situation of FIG. SE when in the The broadcasting of a message over the broadcast channel is
large regime. effectively a multicast to those computers of the network that
FIG. 6 is a block diagram illustrating components of a are currently connected to the broadcast channel. In one
computer that is connected to a broadcast channel. embodiment, the broadcast technique provides a logical
FIG. 7 is a block diagram illustrating the sub-components broadcast channel to which host computers through their
of the broadcaster component in one embodiment. 10 executing processes can be connected. Each computer that is
FIG. 8 is a flow diagram illustrating the processing of the connected to the broadcast channel can broadcast messages
connect routine in one embodiment. onto and receive messages off of the broadcast channel. Each
FIG. 9 is a flow diagram illustrating the processing of the computer that is connected to the broadcast channel receives
seek portal computer routine in one embodiment. all messages that are broadcast while it is connected. The
FIG. 10 is a flow diagram illustrating the processing of the 15 logical broadcast channel is implemented using an underly
contact process routine in one embodiment. ing network system (e.g., the Internet) that allows each com
FIG. 11 is a flow diagram illustrating the processing of the puter connected to the underlying network system to send
connect request routine in one embodiment. messages to each other connected computer using each com
FIG. 12 is a flow diagram of the processing of the check for puter's address. Thus, the broadcast technique effectively
external call routine in one embodiment. provides a broadcast channel using an underlying network
FIG. 13 is a flow diagram of the processing of the achieve system that sends messages on a point-to-point basis.
connection routine in one embodiment. The broadcast technique overlays the underlying network
FIG. 14 is a flow diagram illustrating the processing of the system with a graph of point-to-point connections (i.e.,
external dispatcher routine in one embodiment. edges) between host computers (i.e., nodes) through which
FIG. 15 is a flow diagram illustrating the processing of the
25 the broadcast channel is implemented. In one embodiment,
handle seeking connection call routine in one embodiment. to as neighbors. (Actually, a process executing on a computer
FIG. 16 is a flow diagram illustrating processing of the is connected to four other processes executing on this or four
handle connection request call routine in one embodiment. other computers.) To broadcast a message, the originating
FIG. 17 is a flow diagram illustrating the processing of the 30 computer sends the message to each of its neighbors using its
add neighbor routine in one embodiment. point-to-point connections. Each computer that receives the
FIG. 18 is a flow diagram illustrating the processing of the message then sends the message to its three other neighbors
forward connection edge search routine in one embodiment. using the point-to-point connections. In this way, the message
FIG. 19 is a flow diagram illustrating the processing of the is propagated to each computer using the underlying network
handle edge proposal call routine. 35 to effect the broadcasting of the message to each computer
FIG. 20 is a flow diagram illustrating the processing of the over a logical broadcast channel. A graph in which each node
handle port connection call routine in one embodiment. is connected to four other nodes is referred to as a 4-regular
FIG. 21 is a flow diagram illustrating the processing of the graph. The use of a 4-regular graph means that a computer
fill hole routine in one embodiment. would become disconnected from the broadcast channel only
FIG.22 is a flow diagram illustrating the processing of the 40 if all four of the connections to its neighbors fail. The graph
internal dispatcher routine in one embodiment. used by the broadcast technique also has the property that it
FIG. 23 is a flow diagram illustrating the processing of the would take a failure of four computers to divide the graph into
handle broadcast message routine in one embodiment. disjoint Sub-graphs, that is two separate broadcast channels.
FIG. 24 is a flow diagram illustrating the processing of the This property is referred to as being 4-connected. Thus, the
distribute broadcast message routine in one embodiment. 45 graph is both 4-regular and 4-connected.
FIG. 26 is a flow diagram illustrating the processing of the FIG. 1 illustrates a graph that is 4-regular and 4-connected
handle connection port search statement routine in one which represents the broadcast channel. Each of the nine
embodiment. nodes A-I represents a computer that is connected to the
FIG. 27 is a flow diagram illustrating the processing of the broadcast channel, and each of the edges represents an "edge'
court neighbor routine in one embodiment. 50 connection between two computers of the broadcast channel.
FIG. 28 is a flow diagram illustrating the processing of the The time it takes to broadcast a message to each computer on
handle connection edge search call routine in one embodi the broadcast channel depends on the speed of the connec
ment. tions between the computers and the number of connections
FIG. 29 is a flow diagram illustrating the processing of the between the originating computer and each other computer
55 on the broadcast channel. The minimum number of connec
handle connection edge search response routine in one tions that a message would need to traverse between each pair
embodiment.
FIG. 30 is a flow diagram illustrating the processing of the of computers is the “distance' between the computers (i.e.,
broadcast routine in one embodiment. the shortest path between the two nodes of the graph). For
example, the distance between computers A and F is one
FIG.31 is a flow diagram illustrating the processing of the 60 because computer A is directly connected to computer F. The
acquire message routine in one embodiment. distance between computers A and B is two because there is
FIG. 32 is a flow diagram illustrating processing of the no direct connection between computers A and B, but com
handle condition check message in one embodiment. puter F is directly connected to computer B. Thus, a message
FIG. 33 is a flow diagram illustrating processing of the originating at computer A would be sent directly to computer
handle condition repair statement routine in one embodiment. 65 F, and then sent from computer F to computer B. The maxi
FIG. 34 is a flow diagram illustrating the processing of the mum of the distances between the computers is the “diam
handle condition double check routine. eter of broadcast channel. The diameter of the broadcast
US 7,412,537 B2
5 6
channel represented by FIG. 1 is two. That is, a message sent necting each of the former neighbors to another computer is
by any computer would traverse no more than two connec referred to as "edge pinning as the edge between two nodes
tions to reach every other computer. FIG. 2 illustrates a graph may be considered to be stretched and pinned to a new node.
representing 20 computers connected to a broadcast channel. Each computer connected to the broadcast channel allo
The diameter of this broadcast channel is 4. In particular, the cates five communications ports for communicating with
shortest path between computers 1 and 3 contains four con other computers. Four of the ports are referred to as “internal
nections (1-12, 12-15, 15-18, and 18-3). ports because they are the ports through which the messages
The broadcast technique includes (1) the connecting of of the broadcast channels are sent. The connections between
computers to the broadcast channel (i.e., composing the internal ports of neighbors are referred to as “internal con
graph), (2) the broadcasting of messages over the broadcast 10 nections. Thus, the internal connections of the broadcast
channel (i.e., broadcasting through the graph), and (3) the channel form the 4-regular and 4-connected graph. The fifth
disconnecting of computers from the broadcast channel (i.e., port is referred to as an “external' port because it is used for
decomposing the graph) composing the graph. sending non-broadcast messages between two computers.
Neighbors can send non-broadcast messages either through
Composing the Graph 15 their internal ports of their connection or through their exter
To connect to the broadcast channel, the computer seeking nal ports. A seeking computeruses external ports when locat
the connection first locates a computer that is currently fully ing a portal computer.
connected to the broadcast channel and then establishes a In one embodiment, the broadcast technique establishes
connection with four of the computers that are already con the computer connections using the TCP/IP communications
nected to the broadcast channel. (This assumes that there are protocol, which is a point-to-point protocol, as the underlying
at least four computers already connected to the broadcast network. The TCP/IP protocol provides for reliable and
channel. When there are fewer than five computers con ordered delivery of messages between computers. The TCP/
nected, the broadcast channel cannot be a 4-regular graph. In IP protocol provides each computer with a "port space' that is
Such a case, the broadcast channel is considered to be in a shared among all the processes that may execute on that
“small regime.” The broadcast technique for the small regime 25 computer. The ports are identified by numbers from 0 to
is described below in detail. When five or more computers are 65.535. The first 2056 ports are reserved for specific applica
connected, the broadcast channel is considered to be in the tions (e.g., port 80 for HTTP messages). The remainder of the
“large regime.” This description assumes that the broadcast ports are user ports that are available to any process. In one
channel is in the large regime, unless specified otherwise.) embodiment, a set of port numbers can be reserved for use by
Thus, the process of connecting to the broadcast channel 30 the computer connected to the broadcast channel. In an alter
includes locating the broadcast channel, identifying the native embodiment, the port numbers used are dynamically
neighbors for the connecting computer, and then connecting identified by each computer. Each computer dynamically
to each identified neighbor. Each computer is aware of one or identifies an available port to be used as its call-in port. This
more "portal computers' through which that computer may call-in port is used to establish connections with the external
locate the broadcast channel. A seeking computer locates the 35 port and the internal ports. Each computer that is connected to
broadcast channel by contacting the portal computers until it the broadcast channel can receive non-broadcast messages
finds one that is currently fully connected to the broadcast through its external port. A seeking computer tries “dialing
channel. The found portal computer then directs the identify the port numbers of the portal computers until a portal com
ing of four computers (i.e., to be the seeking computers puter "answers, a call on its call-in port. A portal computer
neighbors) to which the seeking computer is to connect. Each 40 answers when it is connected to or attempting to connect to
of these four computers then cooperates with the seeking the broadcast channel and its call-in port is dialed. (In this
computer to effect the connecting of the seeking computer to description, a telephone metaphor is used to describe the
the broadcast channel. A computer that has started the process connections.) When a computer receives a call on its call-in
of locating a portal computer, but does not yet have a neigh port, it transfers the call to another port. Thus, the seeking
bor, is in the “seeking connection state. A computer that is 45 computer actually communicates through that transfer-to
connected to at least one neighbor, but not yet four neighbors, port, which is the external port. The call is transferred so that
is in the “partially connected state.” A computer that is cur other computers can place calls to that computer via the
rently, or has been, previously connected to four neighbors is call-in port. The seeking computer then communicates via
in the “fully connected state.” that external port to request the portal computer to assist in
Since the broadcast channel is a 4-regular graph, each of 50 connecting the seeking computer to the broadcast channel.
the identified computers is already connected to four comput The seeking computer could identify the call-in port number
ers. Thus, Some connections between computers need to be of a portal computer by Successively dialing each port in port
broken so that the seeking computer can connect to four number order. As discussed below in detail, the broadcast
computers. In one embodiment, the broadcast technique iden technique uses a hashing algorithm to select the port number
tifies two pairs of computers that are currently connected to 55 order, which may result in improved performance.
each other. Each of these pairs of computers breaks the con A seeking computer could connect to the broadcast chan
nection between them, and then each of the four computers nel by connecting to computers either directly connected to
(two from each pair) connects to the seeking computer. FIGS. the found portal computer or directly connected to one of its
3A and 3B illustrate the process of a new computer Z con neighbors. A possible problem with such a scheme for iden
necting to the broadcast channel. FIG. 3A illustrates the 60 tifying the neighbors for the seeking computer is that the
broadcast channel before computer Z is connected. The pairs diameter of the broadcast channel may increase when each
of computers Band E and computers C and Dare the two pairs seeking computer uses the same found portal computer and
that are identified as the neighbors for the new computer Z. establishes a connection to the broadcast channel directly
The connections between each of these pairs is broken, and a through that found portal computer. Conceptually, the graph
connection between computer Zand each of computers B, C, 65 becomes elongated in the direction of where the new nodes
D, and E is established as indicated by FIG. 3B. The process are added. FIGS. 4A-4C illustrate that possible problem. FIG.
of breaking the connection between two neighbors and recon 4A illustrates the broadcast channel of FIG. 1 with an added
US 7,412,537 B2
7 8
computer. Computer J was connected to the broadcast chan received. If, however, the broadcast channel is not in a steady
nel by edge pinning edges C-D and E-H to computer J. The state, then problems can occur. In particular, a computer may
diameter of this broadcast channel is still two. FIG. 4B illus connect to the broadcast channel after the second message has
trates the broadcast channel of FIG. 4A with an added com already been received and forwarded on by its new neighbors.
puter. Computer K was connected to the broadcast channel by When a new neighbor eventually receives the first message, it
edge pinning edges E-J and B-C to computer K. The diameter sends the message to the newly connected computer. Thus,
of this broadcast channel is three, because the shortest path the newly connected computer will receive the first message,
from computer G to computer K is through edges G-A. A-E, but will not receive the second message. If the newly con
and E-K. FIG. 4C also illustrates the broadcast channel of nected computer needs to process the messages in order, it
FIG. 4A with an added computer. Computer Kwas connected 10 would wait indefinitely for the second message.
to the broadcast channel by edge pinning edges D-G and E-J One solution to this problem is to have each computer
to computer K. The diameter of this broadcast channel is, queue all the messages that it receives until it can send them
however, still two. Thus, the selection of neighbors impacts in their proper order to its neighbors. This solution, however,
the diameter of the broadcast channel. To help minimize the may tend to slow down the propagation of messages through
diameter, the broadcast technique uses a random selection 15 the computers of the broadcast channel. Another solution that
technique to identify the four neighbors of a computer in the may have less impact on the propagation speed is to queue
seeking connection state. The random selection technique messages only at computers who are neighbors of the newly
tends to distribute the connections to new seeking computers connected computers. Each already connected neighbor
throughout the computers of the broadcast channel which would forward messages as it receives them to its other neigh
may result in Smaller overall diameters. bors who are not newly connected, but not to the newly
only forward messages from each originating computer to the
As described above, each computer that is connected to the newly connected computer when it can ensure that no gaps in
broadcast channel can broadcast messages onto the broadcast the messages from that originating computer will occur. In
channel and does receive all messages that are broadcast on 25 one embodiment, the already connected neighbor may track
the broadcast channel. The computer that originates a mes the highest sequence number of the messages already
sage to be broadcast sends that message to each of its four received and forwarded on from each originating computer.
neighbors using the internal connections. When a computer The already connected computer will send only higher num
receives a broadcast message from a neighbor, it sends the bered messages from the originating computers to the newly
message to its three other neighbors. Each computer on the 30 connected computer. Once all lower numbered messages
broadcast channel, except the originating computer, will thus have been received from all originating computers, then the
receive a copy of each broadcast message from each of its four already connected computer can treat the newly connected
neighbors. Each computer, however, only sends the first copy computer as its other neighbors and simply forward each
of the message that it receives to its neighbors and disregards message as it is received. In another embodiment, each com
Subsequently received copies. Thus, the total number of cop 35 puter may queue messages and only forwards to the newly
ies of a message that is sent between the computers is 3N+1. connected computer those messages as the gaps are filled in.
where N is the number of computers connected to the broad For example, a computer might receive messages 4 and 5 and
cast channel. Each computer sends three copies of the mes then receive message 3. In such a case, the already connected
sage, except for the originating computer, which sends four computer would forward queue messages 4 and 5. When
copies of the message. 40 message 3 is finally received, the already connected computer
The redundancy of the message sending helps to ensure the will send messages 3, 4, and 5 to the newly connected com
overall reliability of the broadcast channel. Since each com puter. If messages 4 and 5 were sent to the newly connected
puter has four connections to the broadcast channel, if one computer before message 3, then the newly connected com
computer fails during the broadcast of a message, its neigh puter would process messages 4 and 5 and disregard message
bors have three other connections through which they will 45 3. Because the already connected computer queues messages
receive copies of the broadcast message. Also, if the internal 4 and 5, the newly connected computer will be able to process
connection between two computers is slow, each computer message 3. It is possible that a newly connected computer will
has three other connections through which it may receive a receive a set of messages from an originating computer
copy of each message Sooner. through one neighbor and then receive another set of message
Each computer that originates a message numbers its own 50 from the same originating computer through another neigh
messages sequentially. Because of the dynamic nature of the bor. If the second set of messages contains a message that is
broadcast channel and because there are many possible con ordered earlier than the messages of the first set received, then
nection paths between computers, the messages may be the newly connected computer may ignore that earlier
received out of order. For example, the distance between an ordered message if the computer already processed those
originating computer and a certain receiving computer may 55 later ordered messages.
be four. After sending the first message, the originating com
puter and receiving computer may become neighbors and Decomposing the Graph
thus the distance between them changes to one. The first A connected computer disconnects from the broadcast
receiving computer. The second message only has to travel a 60 computer disconnects in a planned manner, it sends a discon
distance of one. Thus, it is possible for the second message to nect message to each of its four neighbors. The disconnect
reach the receiving computer before the first message. message includes a list that identifies the four neighbors of the
When the broadcast channel is in a steady state (i.e., no disconnecting computer. When a neighbor receives the dis
computers connecting or disconnecting from the broadcast connect message, it tries to connect to one of the computers on
channel), out-of-order messages are not a problem because 65 the list. In one embodiment, the first computer in the list will
each computer will eventually receive both messages and can try to connect to the second computer in the list, and the third
queue messages until all earlier ordered messages are computer in the list will try to connect to the fourth computer
US 7,412,537 B2
10
in the list. If a computer cannot connect (e.g., the first and request, it disconnects from one of its neighbors (other than
second computers are already connected), then the computers the neighbor that is involved with the condition) and connects
may try connecting in various other combinations. If connec to the computer that sent the condition repair request. Thus,
tions cannot be established, each computer broadcasts a mes one of the original neighbors involved in the condition will
sage that it needs to establish a connection with another have had a port filled. However, two computers are still in
computer. When a computer with an available internal port need of a connection, the other original neighbor and the
receives the message, it can then establish a connection with computer that is now disconnected from the computer that
the computer that broadcast the message. FIGS.5A-5D illus received the condition repair request. Those two computers
trate the disconnecting of a computer from the broadcast send out port connection requests. If those two computers are
channel. FIG. 5A illustrates the disconnecting of a computer 10
not neighbors, then they will connect to each other when they
from the broadcast channel in a planned manner. When com receive the requests. If, however, the two computers are
puter H decides to disconnect, it sends its list of neighbors to neighbors, then they repeat the condition repair process until
each of its neighbors (computers A, E, F and I) and then two non-neighbors are in need of connections.
disconnects from each of its neighbors. When computers A
and I receive the message they establisha connection between 15 It is possible that the two original neighbors with the con
them as indicated by the dashed line, and similarly for com dition may have the same set of neighbors. When the neighbor
puters E and F. that receives the condition check message determines that the
When a computer disconnects in an unplanned manner, sets of neighbors are the same, it sends a condition double
Such as resulting from a power failure, the neighbors con check message to one of its neighbors other than the neighbor
nected to the disconnected computer recognize the discon who also has the condition. When the computer receives the
nection when each attempts to send its next message to the condition double check message, it determines whether it has
now disconnected computer. Each former neighbor of the the same set of neighbors as the sending computer. If so, the
disconnected computer recognizes that it is short one connec broadcast channel is in the Small regime and the condition is
tion (i.e., it has a hole or empty port). When a connected not a problem. If the set of neighbors are different, then the
computer detects that one of its neighbors is now discon 25
computer that received the condition double check message
nected, it broadcasts a port connection request on the broad sends a condition check message to the original neighbors
cast channel, which indicates that it has one internal port that with the condition. The computer that receives that condition
needs a connection. The port connection request identifies the check message directs one of it neighbors to connect to one of
call-in port of the requesting computer. When a connected the original neighbors with the condition by sending a con
computer that is also short a connection receives the connec 30
dition repair message. Thus, one of the original neighbors
tion request, it communicates with the requesting computer
through its external port to establish a connection between the with the condition will have its port filled.
two computers. FIG. 5B illustrates the disconnecting of a FIG. 5C illustrates the neighbors with empty ports condi
computer from the broadcast channel in an unplanned man tion. In this illustration, computer H disconnected in an
ner. In this illustration, computer H has disconnected in an 35 unplanned manner, but computers F and I responded to the
unplanned manner. When each of its neighbors, computers A, port connection request of the other and are now connected
E, F, and I, recognizes the disconnection, each neighbor together. The other former neighbors of computer H. com
broadcasts a port connection request indicating that it needs puters A and E, are already neighbors, which gives rise to the
to fill an empty port. As shown by the dashed lines, computers neighbors with empty ports condition. In this example, com
F and I and computers A and E respond to each other's 40 puter E received the port connection request from computer
requests and establish a connection. A, recognized the possible condition, and sent (since they are
It is possible that a planned or unplanned disconnection neighbors via the internal connection) a condition check mes
may result in two neighbors each having an empty internal sage with a list of its neighbors to computer A. When com
port. In Such a case, since they are neighbors, they are already puter A received the list, it recognized that computer E has a
connected and cannot fill their empty ports by connecting to 45 different set of neighbor (i.e., the broadcast channel is in the
each other. Such a condition is referred to as the “neighbors large regime). Computer A selected computer D, which is a
with empty ports' condition. Each neighbor broadcasts a port neighbor of computer E and sent it a condition repair request.
connection request when it detects that it has an empty port as When computer D received the condition repair request, it
described above. When a neighbor receives the port connec disconnected from one of its neighbors (other than computer
tion request from the other neighbor, it will recognize the 50 E), which is computer G in this example. Computer D then
condition that its neighbor also has an empty port. Such a connected to computer A. FIG.5D illustrates two computers
condition may also occur when the broadcast channel is in the that are not neighbors who now have empty ports. Computers
Small regime. The condition can only be corrected when in the E and G now have empty ports and are not currently neigh
large regime. When in the Small regime, each computer will bors. Therefore, computers E and G can connect to each other.
have less than four neighbors. To detect this condition in the 55 FIGS. 5E and 5F further illustrate the neighbors with
large regime, which would be a problem if not repaired, the empty ports condition. FIG.5E illustrates the neighbors with
first neighbor to receive the port connection request recog empty ports condition in the Small regime. In this example, if
nizes the condition and sends a condition check message to computer Edisconnected in an unplanned manner, then each
the other neighbor. The condition check message includes a computer broadcasts a port connection request when it
list of the neighbors of the sending computer. When the 60 detects the disconnect. When computer A receives the port
receiving computer receives the list, it compares the list to its connection request form computer B, it detects the neighbors
own list of neighbors. If the lists are different, then this con with empty ports condition and sends a condition check mes
dition has occurred in the large regime and repair is needed. sage to computer B. Computer B recognizes that it has the
To repair this condition, the receiving computer will send a same set of neighbors (computer C and D) as computer A and
condition repair request to one of the neighbors of the sending 65 then sends a condition double check message to computer C.
computer which is not already a neighbor of the receiving Computer C recognizes that the broadcast channel is in the
computer. When the computer receives the condition repair Small regime because is also has the same set of neighbors as
US 7,412,537 B2
11 12
computers A and B, computer C may then broadcast a mes the ports of the portal computer may be busy when called by
Sage indicating that the broadcast channel is in the small seeking computers. The seeking computers would typically
regime. need to keep on redialing a busy port. The process of locating
FIG. 5F illustrates the situation of FIG. 5E when in the a call-in port may be significantly slowed by such redialing. In
large regime. As discussed above, computer C receives the 5 one embodiment, each seeking computer may each reorder
condition double check message from computer B. In this the first few port numbers generated by the hashing algo
case, computer C recognizes that the broadcast channel is in rithm. For example, each seeking computer could randomly
the large regime because it has a set of neighbors that is reorder the first eight port numbers generated by the hashing
different from computer B. The edges extending up from algorithm. The random ordering could also be weighted
computer C and D indicate connections to other computers. 10 where the first port number generated by the hashing algo
Computer C then sends a condition check message to com rithm would have a 50% chance of being first in the reorder
puter B. When computer B receives the condition check mes ing, the second port number would have a 25% chance of
Sage, it sends a condition repair message to one of the neigh being first in the reordering, and so on. Because the seeking
bors of computer C. The computer that receives the condition computers would use different orderings, the likelihood of
repair message disconnects from one of its neighbors, other 15 finding a busy port is reduced. For example, if the first eight
than computer C, and tries to connect to computer Band the port numbers are randomly selected, then it is possible that
neighbor from which it disconnected tries to connect to com eight seeking computers could be simultaneously dialing
puter A.
ports in different sequences which would reduce the chances
Port Selection of dialing a busy port.
As described above, the TCP/IP protocol designates ports
port and four internal ports. Generally, user ports cannot be has a list of one or more portal computers through which it can
Statically allocated to an application program because other 25 connect to the broadcast channel. In one embodiment, each
applications programs executing on the same computer may computer has the same set of portal computers. A seeking
use conflicting port numbers. As a result, in one embodiment, computer locates a portal computer that is connected to the
the computers connected to the broadcast channel dynami broadcast channel by successively dialing the ports of each
cally allocate their port numbers. Each computer could sim portal computer in the order specified by an algorithm. A
ply try to locate the lowest number unused port on that com- 30 seeking computer could select the first portal computer and
puter and use that port as the call-in port. A seeking computer, then dial all its ports until a call-in port of a computer that is
however, does not know in advance the call-in port number of fully connected to the broadcast channel is found. If no call-in
the portal computers when the port numbers are dynamically port is found, then the seeking computer would select the next
allocated. Thus, a seeking computer needs to dial ports of a portal computer and repeat the process until a portal computer
portal computer starting with the lowest port number when 35 with such a call-in port is found. A problem with such a
locating the call-in port of a portal computer. If the portal seeking technique is that all user ports of each portal com
computer is connected to (or attempting to connect to) the puter are dialed until a portal computer fully connected to the
broadcast channel, then the seeking computer would eventu broadcast channel is found. In an alternate embodiment, the
ally find the call-in port. If the portal computer is not con seeking computer selects a port number according to the
nected, then the seeking computer would eventually dial 40 algorithm and then dials each portal computer at that port
every user port. In addition, if each application program on a number. If no acceptable call-in port to the broadcast channel
computer tried to allocate low-ordered port numbers, then a is found, then the seeking computer selects the next port
portal computer may end up with a high-numbered port for its number and repeats the process. Since the call-in ports are
call-in port because many of the low-ordered port numbers likely allocated at lower-ordered port numbers, the seeking
would be used by other application programs. Since the dial- 4s computer first dials the port numbers that are most likely to be
ing of a port is a relatively slow process, it would take the call-in ports of the broadcast channel. The seeking computers
seeking computer a long time to locate the call-in port of a may have a maximum search depth, that is the number of
portal computer. To minimize this time, the broadcast tech ports that it will dial when seeking a portal computer that is
nique uses a port ordering algorithm to identify the port fully connected. If the seeking computer exhausts its search
number order that a portal computer should use when finding so depth, then either the broadcast channel has not yet been
an available port for its call-in port. In one embodiment, the established or, if the seeking computer is also a portal com
broadcast technique uses a hashing algorithm to identify the puter, it can then establish the broadcast channel with itself as
port order. The algorithm preferably distributes the ordering the first fully connected computer.
of the port numbers randomly through out the user port num When a seeking computer locates a portal computer that is
ber space and only selects each port number once. In addition, ss itself not fully connected, the two computers do not connect
every time the algorithm is executed on any computer for a when they first locate each other because the broadcast chan
given channel type and channel instance, it generates the nel may already be established and accessible through a
same port ordering. As described below, it is possible for a higher-ordered port number on another portal computer. If the
computer to be connected to multiple broadcast channels that two seeking computers were to connect to each other, then
are uniquely identified by channel type and channel instance. 60 two disjoint broadcast channels would be formed. Each seek
The algorithm may be "seeded with channel type and chan ing computer can share its experience in trying to locate a
nel instance in order to generate a unique ordering of port portal computer with the other seeking computer. In particu
numbers for each broadcast channel. Thus, a seeking com lar, if one seeking computer has searched all the portal com
puter will dial the ports of a portal computer in the same order puters to a depth of eight, then the one seeking computer can
as the portal computer used when allocating its call-in port. 65 share that it has searched to a depth of eight with another
If many computers are at the same time seeking connection seeking computer. If that other seeking computer has
to a broadcast channel through a single portal computer, then searched to a depth of for example, only four, it can skip
US 7,412,537 B2
13 14
searching through depths five through eight and that other receives a message that has traveled a distance that indicates
seeking computer can advance its searching to a depth of nine. that the estimated diameter is too small, it updates its esti
In one embodiment, each computer may have a different mated diameter and broadcasts an estimated diameter mes
set of portal computers and a different maximum search sage. When a computer receives an estimated diameter mes
depth. In such a situation, it may be possible that two disjoint sage that indicates a diameter that is larger than its own
broadcast channels are formed because a seeking computer estimated diameter, it updates its own estimated diameter.
cannot locate a fully connected port computer at a higher This estimated diameter is used to establish the distance that
depth. Similarly, if the set of portal computers are disjoint, an edge connection request message should travel.
then two separate broadcast channels would be formed. External Data Representation
10
Identifying Neighbors for a Seeking Computer The computers connected to the broadcast channel may
As described above, the neighbors of a newly connecting internally store their data in different formats. For example,
computer are preferably selected randomly from the set of one computer may use 32-bit integers, and another computer
currently connected computers. One advantage of the broad may use 64-bit integers. As another example, one computer
cast channel, however, is that no computer has global knowl 15 may use ASCII to represent text and another computer may
edge of the broadcast channel. Rather, each computer has use Unicode. To allow communications between heteroge
local knowledge of itself and its neighbors. This limited local neous computers, the messages sent over the broadcast chan
knowledge has the advantage that all the connected comput nel may use the XDR (eXternal Data Representation') for
ers are peers (as far as the broadcasting is concerned) and the mat.
failure of any one computer (actually any three computers The underlying peer-to-peer communications protocol
when in the 4-regular and 4-connect form) will not cause the may send multiple messages in a single message stream. The
broadcast channel to fail. This local knowledge makes it traditional technique for retrieving messages from a stream
difficult for a portal computer to randomly select four neigh has been to repeatedly invoke an operating system routine to
bors for a seeking computer. retrieve the next message in the stream. The retrieval of each
To select the four computers, a portal computer sends an 25 message may require two calls to the operating system: one to
edge connection request message through one of its internal retrieve the size of the next message and the other to retrieve
connections that is randomly selected. The receiving com the number of bytes indicated by the retrieved size. Such calls
puter again sends the edge connection request message to the operating system can, however, be very slow in com
through one of its internal connections that is randomly parison to the invocations of local routines. To overcome the
selected. This sending of the message corresponds to a ran 30 inefficiencies of Such repeated calls, the broadcast technique
dom walk through the graph that represents the broadcast in one embodiment, uses XDR to identify the messagebound
channel. Eventually, a receiving computer will decide that the aries in a stream of messages. The broadcast technique may
message has traveled far enough to represent a randomly request the operating system to provide the next, for example,
selected computer. That receiving computer will offer the 1,024 bytes from the stream. The broadcast technique can
internal connection upon which it received the edge connec 35 then repeatedly invoke the XDR routines to retrieve the mes
tion request message to the seeking computer for edge pin sages and use the Success or failure of each invocation to
ning. Of course, if either of the computers at the end of the determine whether another block of 1,024 bytes needs to be
offered internal connection are already neighbors of the seek retrieved from the operating system. The invocation of XDR
ing computer, then the seeking computer cannot connect routines do not involve system calls and are thus more effi
through that internal connection. The computer that decided 40 cient than repeated system calls.
that the message has traveled far enough will detect this M-Regular
condition of already being a neighbor and send the message to In the embodiment described above, each fully connected
a randomly selected neighbor. computer has four internal connections. The broadcast tech
In one embodiment, the distance that the edge connection nique can be used with other numbers of internal connections.
request message travels is established by the portal computer 45
For example, each computer could have 6, 8, or any even
to be approximately twice the estimated diameter of the number of internal connections. As the number of internal
broadcast channel. The message includes an indication of the connections increase, the diameter of the broadcast channel
distance that it is to travel. Each receiving computer decre tends to decrease, and thus propagation time for a message
ments that distance to travel before sending the message on. tends to decrease. The time that it takes to connect a seeking
The computer that receives a message with a distance to travel 50
computer to the broadcast channel may, however, increase as
that is zero is considered to be the randomly selected com the number of internal connections increases. When the num
puter. If that randomly selected computer cannot connect to ber of internal connectors is even, then the broadcast channel
the seeking computer (e.g., because it is already connected to can be maintained as m-regular and m-connected (in the
it), then that randomly selected computer forwards the edge steady state). If the number of internal connections is odd,
connection request to one of its neighbors with a new distance 55 then when the broadcast channel has an odd number of com
to travel. In one embodiment, the forwarding computer puters connected, one of the computers will have less than
toggles the new distance to travel between Zero and one to that odd number of internal connections. In such a situation,
help prevent two computers from sending the message back the broadcast network is neither m-regular norm-connected.
and forth between each other.
Because of the local nature of the information maintained 60
When the next computer connects to the broadcast channel, it
can again become m-regular and m-connected. Thus, with an
by each computer connected to the broadcast channel, the odd number of internal connections, the broadcast channel
computers need not generally be aware of the diameter of the toggles between being and not being m-regular and m-con
broadcast channel. In one embodiment, each message sent nected.
through the broadcast channel has a distance traveled field.
Each computer that forwards a message increments the dis 65 Components
tance traveled field. Each computer also maintains an esti FIG. 6 is a block diagram illustrating components of a
mated diameter of the broadcast channel. When a computer computer that is connected to a broadcast channel. The above
US 7,412,537 B2
15 16
description generally assumed that there was only one broad cast channel. The connect component identifies the external
cast channel and that each computer had only one connection port and installs the external dispatcher for handling mes
to that broadcast channel. More generally, a network of com sages that are received on the external port. The connect
puters may have multiple broadcast channels, each computer component invokes the seek portal computer component 705
may be connected to more than one broadcast channel, and to identify a portal computer that is connected to the broadcast
each computer can have multiple connections to the same channel and invokes the connect request component 706 to
broadcast channel. The broadcast channel is well suited for ask the portal computer (if fully connected) to select neighbor
computer processes (e.g., application programs) that execute processes for the newly connecting process. The external
collaboratively, Such as network meeting programs. Each dispatcher receives external messages, identifies the type of
computer process can connect to one or more broadcast chan 10
message, and invokes the appropriate handling routine 707.
nels. The broadcast channels can be identified by channel The internal dispatcher receives the internal messages, iden
type (e.g., application program name) and channel instance tifies the type of message, and invokes the appropriate han
that represents separate broadcast channels for that channel dling routine 708. The received broadcast messages are stored
type. When a process attempts to connect to a broadcast in the broadcast message queue 709. The acquire message
channel, it seeks a process currently connected to that broad 15
component is invoked to retrieve messages from the broad
cast channel that is executing on a portal computer. The cast queue. The broadcast component is invoked by the appli
seeking process identifies the broadcast channel by channel cation program to broadcast messages in the broadcast chan
type and channel instance. nel.
Computer 600 includes multiple application programs 601
executing as separate processes. Each application program The following tables list messages sent by the broadcaster
interfaces with a broadcaster component 602 for each broad components.
cast channel to which it is connected. The broadcaster com
ponent may be implement as an object that is instantiated
within the process space of the application program. Alterna EXTERNAL MESSAGES
tively, the broadcaster component may execute as a separate 25
process or thread from the application program. In one Message Type Description
embodiment, the broadcaster component provides functions Seeking connection call indicates that a seeking process would
(e.g., methods of class) that can be invoked by the application ike to know whether the receiving process is
programs. The primary functions provided may include a ully connected to the broadcast channel
connect function that an application program invokes passing 30 connection request call indicates that the sending process would like
he receiving process to initiate a connection
an indication of the broadcast channel to which the applica of the sending process to the broadcast
tion program wants to connect. The application program may channel
provide a callback routine that the broadcaster component edge proposal call indicates that the sending process is pro
invokes to notify the application program that the connection posing an edge through which the receiving
process can connect to the broadcast chan
has been completed, that is the process enters the fully con 35
nel (i.e., edge pinning)
nected State. The broadcaster component may also provide an port connection call indicates that the sending process is pro
acquire message function that the application program can posing a port through which the receiving
process can connect to the broadcast
invoke to retrieve the next message that is broadcast on the channel
broadcast channel. Alternatively, the application program connected stimt indicates that the sending process is con
may provide a callback routine (which may be a virtual func 40 nected to the broadcast channel
tion provided by the application program) that the broad condition repair stimt indicates that the receiving process should
caster component invokes to notify the application program disconnect from one of its neighbors and
connect to one of the processes involved in
that a broadcast message has been received. Each broadcaster he neighbors with empty port condition
component allocates a call-in port using the hashing algo
rithm. When calls are answered at the call-in port, they are 45
transferred to other ports that serve as the external and inter
nal ports.
The computers connecting to the broadcast channel may INTERNAL MESSAGES
include a central processing unit, memory, input devices (e.g.,
keyboard and pointing device), output devices (e.g., display 50 Message Type Description
devices), and storage devices (e.g., disk drives). The memory broadcast stimt indicates a message that is being
and storage devices are computer-readable medium that may broadcast through the broadcast
contain computer instructions that implement the broadcaster channel for the application programs
component. In addition, the data structures and message connection port search stmt Indicates that the designated pro
structures may be stored or transmitted via a signal transmit 55 cess is looking for a port through
which it can connect to the broad
ted on a computer-readable media, Such as a communications cast channel
link. connection edge search call Indicates that the requesting pro
FIG. 7 is a block diagram illustrating the sub-components cess is looking for an edge through
of the broadcaster component in one embodiment. The broad which it can connect to the broad
cast channel
caster component includes a connect component 701, an 60
connection edge search resp Indicates whether the edge between
external dispatcher 702, an internal dispatcher 703 for each his process and the sending neigh
internal connection, an acquire message component 704 and bor has been accepted by the re
a broadcast component 712. The application program may questing party
diameter estimate stimt indicates an estimated diameter of
provide a connect callback component 710 and a receive he broadcast channel
response component 711 that are invoked by the broadcaster 65 diameter reset stimt indicates to reset the estimated dia
component. The application program invokes the connect meter to indicated diameter
component to establish a connection to a designated broad
US 7,412,537 B2
17 18
received through that external port, the external dispatcher is
-continued invoked. The routine then returns. In block 808, the routine
INTERNAL MESSAGES
installs an external dispatcher. In block 809, the routine
invokes the connect request routine to initiate the process of
Message Type Description identifying neighbors for the seeking computer. The routine
then returns.
disconnect stimt Indicates that the sending neighbor is FIG. 9 is a flow diagram illustrating the processing of the
disconnecting from the broad
cast channel seek portal computer routine in one embodiment. This routine
condition check stimt Indicates that neighbors with empty is passed the channel type and channel instance of the broad
port condition have been detected 10 cast channel to which this process wishes to connect. This
condition double check stimt Indicates that the neighbors with routine, for each search depth (e.g., port number), checks the
empty ports have the same set of portal computers at that search depth. If a portal computer is
neighbors
shutdown stimt Indicates that the broadcast channel is located at that search depth with a process that is fully con
being shutdown nected to the broadcast channel, then the routine returns an
15 indication of success. In blocks 902-911, the routine loops
selecting each search depth until a process is located. In block
Flow Diagrams 902, the routine selects the next search depth using a port
FIGS. 8-34 are flow diagrams illustrating the processing of number ordering algorithm. In decision block 903, if all the
the broadcaster component in one embodiment. FIG. 8 is a search depths have already been selected during this execu
flow diagram illustrating the processing of the connect rou tion of the loop, that is for the currently selected depth, then
tine in one embodiment. This routine is passed a channel type the routine returns a failure indication, else the routine con
(e.g., application name) and channel instance (e.g., Session tinues at block 904. In blocks 904-911, the routine loops
identifier), that identifies the broadcast channel to which this selecting each portal computer and determining whether a
process wants to connect. The routine is also passed auxiliary process of that portal computer is connected to (or attempting
information that includes the list of portal computers and a 25 to connect to) the broadcast channel with the passed channel
connection callback routine. When the connection is estab type and channel instance. In block 904, the routine selects
lished, the connection callback routine is invoked to notify the the next portal computer. In decision block 905, if all the
application program. When this process invokes this routine, portal computers have already been selected, then the routine
it is in the seeking connection state. When a portal computer loops to block 902 to select the next search depth, else the
is located that is connected and this routine connects to at least 30 routine continues at block 906. In block 906, the routine dials
one neighbor, this process enters the partially connected State, the selected portal computer through the port represented by
and when the process eventually connects to four neighbors, the search depth. In decision block 907, if the dialing was
it enters the fully connected state. When in the small regime, successful, then the routine continues at block 908, else the
a fully connected process may have less than four neighbors. routine loops to block 904 to select the next portal computer.
In block 801, the routine opens the call-in port through which 35 The dialing will be successful if the dialed port is the call-in
the process is to communicate with other processes when port of the broadcast channel of the passed channel type and
establishing external and internal connections. The port is channel instance of a process executing on that portal com
selected as the first available port using the hashing algorithm puter. In block 908, the routine invokes a contact process
described above. In block 802, the routine sets the connect routine, which contacts the answering process of the portal
time to the current time. The connect time is used to identify 40 computer through the dialed port and determines whether that
the instance of the process that is connected through this process is fully connected to the broadcast channel. In block
external port. One process may connect to a broadcast chan 909, the routine hangs up on the selected portal computer. In
nel of a certain channel type and channel instance using one decision block 910, if the answering process is fully con
call-in port and then disconnects, and another process may nected to the broadcast channel, then the routine returns a
then connect to that same broadcast channel using the same 45 success indicator, else the routine continues at block 911. In
call-in port. Before the other process becomes fully con block 911, the routine invokes the check for external call
nected, another process may try to communicate with it think routine to determine whether an external call has been made
ing it is the fully connected old process. In Such a case, the to this process as a portal computer and processes that call.
connect time can be used to identify this situation. In block The routine then loops to block 904 to select the next portal
803, the routine invokes the seek portal computer routine 50 computer.
passing the channel type and channel instance. The seek FIG. 10 is a flow diagram illustrating the processing of the
portal computer routine attempts to locate a portal computer contact process routine in one embodiment. This routine
through which this process can connect to the broadcast chan determines whether the process of the selected portal com
nel for the passed type and instance. In decision block 804, if puter that answered the call-in to the selected port is fully
the seek portal computer routine is successful in locating a 55 connected to the broadcast channel. In block 1001, the routine
fully connected process on that portal computer, then the sends an external message (i.e., seeking connection call) to
routine continues at block 805, else the routine returns an the answering process indicating that a seeking process wants
unsuccessful indication. In decision block 805, if no portal to know whether the answering process is fully connected to
computer other than the portal computer on which the process the broadcast channel. In block 1002, the routine receives the
is executing was located, then this is the first process to fully 60 external response message from the answering process. In
connect to broadcast channel and the routine continues at decision block 1003, if the external response message is suc
block 806, else the routine continues at block 808. In block cessfully received (i.e., seeking connection resp), then the
806, the routine invokes the achieve connection routine to routine continues at block 1004, else the routine returns.
change the state of this process to fully connected. In block Wherever the broadcast component requests to receive an
807, the routine installs the external dispatcher for processing 65 external message, it sets a time out period. If the external
messages received through this process external port for the message is not received within that time out period, the broad
passed channel type and channel instance. When a message is caster component checks its own call-in port to see if another
US 7,412,537 B2
19 20
process is calling it. In particular, the dialed process may be sion block 1204, if the type of the message indicates that a
calling the dialing process, which may result in a deadlock seeking process is calling (i.e., seeking connection call),
situation. The broadcaster component may repeat the receive then the routine continues at block 1205, else the routine
request several times. If the expected message is not received, returns. In block 1205, the routine sends an external message
then the broadcaster component handles the error as appro- 5 (i.e., seeking connection resp) to the other seeking process
priate. In decision block 1004, if the answering process indi indicating that this process is also is seeking a connection. In
cates in its response message that it is fully connected to the decision block 1206, if the sending of the external message is
broadcast channel, then the routine continues at block 1005, successful, then the routine continues at block 1207, else the
else the routine continues at block 1006. In block 1005, the routine returns. In block 1207, the routine adds the other
routine adds the selected portal computer to a list of con- 10 seeking process to a list of fellow seeking processes and then
nected portal computers and then returns. In block 1006, the returns. This list may be used if this process can find no
routine adds the answering process to a list of fellow seeking process that is fully connected to the broadcast channel. In
processes and then returns.
FIG. 11 is a flow diagram illustrating the processing of the which case, this process may check to see if any fellow
connect request routine in one embodiment. This routine 15 cast seeking process were Successful in connecting to the broad
requests a process of a portal computer that was identified as channel. For example, a fellow seeking process may
being fully connected to the broadcast channel to initiate the channel. the first process fully connected to the broadcast
become
connection of this process to the broadcast channel. In deci
sion block 1101, if at least one process of a portal computer FIG. 13 is a flow diagram of the processing of the achieve
was located that is fully connected to the broadcast channel, 20 connection routine in one embodiment. This routine sets the
then the routine continues at block 1103, else the routine state of this process to fully connected to the broadcast chan
continues at block 1102. A process of the portal computer nel and invokes a callback routine to notify the application
may no longer be in the list if it recently disconnected from program that the process is now fully connected to the
the broadcast channel. In one embodiment, a seeking com requested broadcast channel. In block 1301, the routine sets
puter may always search its entire search depth and find 25 the connection state of this process to fully connected. In
multiple portal computers through which it can connect to the block 1302, the routine notifies fellow seeking processes that
broadcast channel. In block 1102, the routine restarts the it is fully connected by sending a connected external message
process of connecting to the broadcast channel and returns. In to them (i.e., connected stimt). In block 1303, the routine
block 1103, the routine dials the process of one of the found invokes the connect callback routine to notify the application
portal computers through the call-in port. In decision block 30 program and then returns.
1104, if the dialing is successful, then the routine continues at FIG. 14 is a flow diagram illustrating the processing of the
block 1105, else the routine continues at block 1113. The external dispatcher routine in one embodiment. This routine
dialing may be unsuccessful if, for example, the dialed pro is invoked when the external port receives a message. This
cess recently disconnected from the broadcast channel. In routine retrieves the message, identifies the external message
block 1105, the routine sends an external message to the 35 type, and invokes the appropriate routine to handle that mes
dialed process requesting a connection to the broadcast chan sage. This routine loops processing each message until all the
nel (i.e., connection request call). In block 1106, the routine received messages have been handled. In block 1401, the
receives the response message (i.e., connection request routine answers (e.g., picks up) the external port and retrieves
resp). In decision block 1107, if the response message is an external message. In decision block 1402, ifa message was
Successfully received, then the routine continues at block 40 retrieved, then the routine continues at block 1403, else the
1108, else the routine continues at block 1113. In block 1108, routine hangs up on the external port in block 1415 and
the routine sets the expected number of holes (i.e., empty returns. In decision block 1403, if the message type is for a
internal connections) for this process based on the received process seeking a connection (i.e., seeking connection call).
response. When in the large regime, the expected number of then the routine invokes the handle seeking connection call
holes is Zero. When in the small regime, the expected number 45 routine in block 1404, else the routine continues at block
of holes varies from one to three. In block 1109, the routine 1405. In decision block 1405, if the message type is for a
sets the estimated diameter of the broadcast channel based on connection request call (i.e., connection request call), then
the received response. In decision block 1111, if the dialed the routine invokes the handle connection request call routine
process is ready to connect to this process as indicated by the in block 1406, else the routine continues at block 1407. In
response message, then the routine continues at block 1112, 50 decision block 1407, if the message type is edge proposal call
else the routine continues at block 1113. In block 1112, the (i.e., edge proposal call), then the routine invokes the handle
routine invokes the add neighbor routine to add the answering edge proposal call routine in block 1408, else the routine
process as a neighbor to this process. This adding of the continues at block 1409. In decision block 1409, if the mes
answering process typically occurs when the broadcast chan sage type is port connect call (i.e., port connect call), then
nel is in the Small regime. When in the large regime, the 55 the routine invokes the handle port connection call routine in
random walk search for a neighbor is performed. In block block 1410, else the routine continues at block 1411. In deci
1113, the routine hangs up the external connection with the sion block 1411, if the message type is a connected Statement
answering process computer and then returns. (i.e., connected Stmt), the routine invokes the handle con
FIG. 12 is a flow diagram of the processing of the check for nected statement in block 1112, else the routine continues at
external call routine in one embodiment. This routine is 60 block 1212. In decision block 1412, if the message type is a
invoked to identify whether a fellow seeking process is condition repair statement (i.e., condition repair stimt), then
attempting to establish a connection to the broadcast channel the routine invokes the handle condition repair routine in
through this process. In block 1201, the routine attempts to block 1413, else the routine loops to block 1414 to process the
answer a call on the call-in port. In decision block 1202, if the next message. After each handling routine is invoked, the
answer is Successful, then the routine continues at block 65 routine loops to block 1414. In block 1414, the routine hangs
1203, else the routine returns. In block 1203, the routine up on the external port and continues at block 1401 to receive
receives the external message from the external port. In deci the next message.
US 7,412,537 B2
21 22
FIG. 15 is a flow diagram illustrating the processing of the invokes the fill hole routine passing the identification of the
handle seeking connection call routine in one embodiment. calling process. The fill hole routine broadcasts a connection
This routine is invoked when a seeking process is calling to port search statement (i.e., connection port search stimt) for
identify a portal computer through which it can connect to the a hole of a connected process through which the calling
broadcast channel. In decision block 1501, if this process is 5 process can connect to the broadcast channel. The routine
currently fully connected to the broadcast channel identified then returns.
in the message, then the routine continues at block 1502, else FIG. 17 is a flow diagram illustrating the processing of the
the routine continues at block 1503. In block 1502, the routine add neighbor routine in one embodiment. This routine adds
sets a message to indicate that this process is fully connected the process calling on the external port as a neighbor to this
to the broadcast channel and continues at block 1505. In block 10 process. In block 1701, the routine identifies the calling pro
1503, the routine sets a message to indicate that this process cess on the external port. In block 1702, the routine sets a flag
is not fully connected. In block 1504, the routine adds the to indicate that the neighbor has not yet received the broadcast
identification of the seeking process to a list of fellow seeking messages from this process. This flag is used to ensure that
processes. If this process is not fully connected, then it is there are no gaps in the messages initially sent to the new
attempting to connect to the broadcast channel. In block 15 neighbor. The external port becomes the internal port for this
1505, the routine sends the external message response (i.e., connection. In decision block 1703, if this process is in the
seeking connection resp) to the seeking process and then seeking connection state, then this process is connecting to its
returns. first neighbor and the routine continues at block 1704, else the
FIG. 16 is a flow diagram illustrating processing of the routine continues at block 1705. In block 1704, the routine
handle connection request call routine in one embodiment. sets the connection state of this process to partially connected.
This routine is invoked when the calling process wants this In block 1705, the routine adds the calling process to the list
process to initiate the connection of the process to the broad of neighbors of this process. In block 1706, the routine installs
cast channel. This routine either allows the calling process to an internal dispatcher for the new neighbor. The internal
establish an internal connection with this process (e.g., if in dispatcher is invoked when a message is received from that
the Small regime) or starts the process of identifying a process 25 new neighbor through the internal port of that new neighbor.
to which the calling process can connect. In decision block In decision block 1707, if this process buffered up messages
1601, if this process is currently fully connected to the broad while not fully connected, then the routine continues at block
cast channel, then the routine continues at block 1603, else the 1708, else the routine continues at block 1709. In one embodi
routine hangs up on the external port in block 1602 and ment, a process that is partially connected may buffer the
returns. In block 1603, the routine sets the number of holes 30 messages that it receives through an internal connection so
that the calling process should expect in the response mes that it can send these messages as it connects to new neigh
sage. In block 1604, the routine sets the estimated diameter in bors. In block 1708, the routine sends the buffered messages
the response message. In block 1605, the routine indicates to the new neighbor through the internal port. In decision
whether this process is ready to connect to the calling process. block 1709, if the number of holes of this process equals the
This process is ready to connect when the number of its holes 35 expected number of holes, then this process is fully connected
is greater than Zero and the calling process is not a neighbor of and the routine continues at block 1710, else the routine
this process. In block 1606, the routine sends to the calling continues at block 1711. In block 1710, the routine invokes
process an external message that is responsive to the connec the achieve connected routine to indicate that this process is
tion request call (i.e., connection request resp). In block fully connected. In decision block 1711, if the number of
1607, the routine notes the number of holes that the calling 40 holes for this process is Zero, then the routine continues at
process needs to fill as indicated in the request message. In block 1712, else the routine returns. In block 1712, the routine
decision block 1608, if this process is ready to connect to the deletes any pending edges and then returns. A pending edge is
calling process, then the routine continues at block 1609, else an edge that has been proposed to this process for edge pin
the routine continues at block 1611. In block 1609, the routine ning, which in this case is no longer needed.
invokes the add neighbor routine to add the calling process as 45 FIG. 18 is a flow diagram illustrating the processing of the
a neighbor. In block 1610, the routine decrements the number forward connection edge search routine in one embodiment.
of holes that the calling process needs to fill and continues at This routine is responsible for passing along a request to
block 1611. In block 1611, the routine hangs up on the exter connect a requesting process to a randomly selected neighbor
nal port. In decision block 1612, if is this process has no holes of this process through the internal port of the selected neigh
or the estimated diameter is greater than one (i.e., in the large 50 bor, that is part of the random walk. In decision block 1801, if
regime), then the routine continues at block 1613, else the the forwarding distance remaining is greater than Zero, then
routine continues at block 1616. In blocks 1613-1615, the the routine continues at block 1804, else the routine continues
routine loops forwarding a request for an edge through which at block 1802. In decision block 1802, if the number of
to connect to the calling process to the broadcast channel. One neighbors of this process is greater than one, then the routine
request is forwarded for each pair of holes of the calling 55 continues at block 1804, else this broadcast channel is in the
process that needs to be filled. In decision block 1613, if the small regime and the routine continues at block 1803. In
number of holes of the calling process to be filled is greater decision block 1803, if the requesting process is a neighbor of
than or equal to two, then the routine continues at block 1614, this process, then the routine returns, else the routine contin
else the routine continues at block 1616. In block 1614, the ues at block 1804. In blocks 1804-1807, the routine loops
routine invokes the forward connection edge search routine. 60 attempting to send a connection edge search call internal
The invoked routine is passed to an indication of the calling message (i.e., connection edge search call) to a randomly
process and the random walk distance. In one embodiment, selected neighbor. In block 1804, the routine randomly
the distance is twice in the estimated diameter of the broad selects a neighbor of this process. In decision block 1805, if
cast channel. In block 1614, the routine decrements the holes all the neighbors of this process have already been selected,
left to fill by two and loops to block 1613. In decision block 65 then the routine cannot forward the message and the routine
1616, if there is still a hole to fill, then the routine continues at returns, else the routine continues at block 1806. In block
block 1617, else the routine returns. In block 1617, the routine 1806, the routine sends a connection edge search call internal
US 7,412,537 B2
23 24
message to the selected neighbor. In decision block 1807, if process and then returns. In block 2007, the routine hangs up
the sending of the message is successful, then the routine the external connection. In block 2008, the routine invokes
continues at block 1808, else the routine loops to block 1804 the connect request routine to request that a process connect
to select the next neighbor. When the sending of an internal to one of the holes of this process. The routine then returns.
message is unsuccessful, then the neighbor may have discon FIG. 21 is a flow diagram illustrating the processing of the
nected from the broadcast channel in an unplanned manner. fill hole routine in one embodiment. This routine is passed an
Whenever such a situation is detected by the broadcaster indication of the requesting process. If this process is request
component, it attempts to find another neighbor by invoking ing to fill a hole, then this routine sends an internal message to
the fill holes routine to fill a single hole or the forward con other processes. If another process is requesting to fill a hole,
necting edge search routine to fill two holes. In block 1808, 10 then this routine invokes the routine to handle a connection
the routine notes that the recently sent connection edge search port search request. In block 2101, the routine initializes a
call has not yet been acknowledged and indicates that the connection port search Statement internal message (i.e., con
edge to this neighbor is reserved if the remaining forwarding nection port search stimt). In decision block 2102, if this
distance is less than or equal to one. It is reserved because the process is the requesting process, then the routine continues at
selected neighbor may offer this edge to the requesting pro 15 block 2103, else the routine continues at block 2104. In block
cess for edge pinning. The routine then returns. 2103, the routine distributes the message to the neighbors of
FIG. 19 is a flow diagram illustrating the processing of the this process through the internal ports and then returns. In
handle edge proposal call routine. This routine is invoked block 2104, the routine invokes the handle connection port
when a message is received from a proposing process that search routine and then returns.
proposes to connect an edge between the proposing process FIG. 22 is a flow diagram illustrating the processing of the
and one of its neighbors to this process for edge pinning. In internal dispatcher routine in one embodiment. This routine is
decision block 1901, if the number of holes of this process passed an indication of the neighbor who sent the internal
minus the number of pending edges is greater than or equal to message. In block 2201, the routine receives the internal
one, then this process still has holes to be filled and the routine message. This routine identifies the message type and invokes
continues at block 1902, else the routine continues at block 25 the appropriate routine to handle the message. In block 2202,
1911. In decision block 1902, if the proposing process or its the routine assesses whether to change the estimated diameter
neighbor is a neighbor of this process, then the routine con of the broadcast channel based on the information in the
tinues at block 1911, else the routine continues at block 1903. received message. In decision block 2203, if this process is
In block 1903, the routine indicates that the edge is pending the originating process of the message or the message has
between this process and the proposing process. In decision 30 already been received (i.e., a duplicate), then the routine
block 1904, if a proposed neighbor is already pending as a ignores the message and continues at block 2208, else the
proposed neighbor, then the routine continues at block 1911, routine continues at block 2203A. In decision block 2203A, if
else the routine continues at block 1907. In block 1907, the the process is partially connected, then the routine continues
routine sends an edge proposal response as an external mes at block 2203B, else the routine continues at block 2204. In
Sage to the proposing process (i.e., edge proposal resp) indi 35 block 2203B, the routine adds the message to the pending
cating that the proposed edge is accepted. In decision block connection buffer and continues at block 2204. In decision
1908, if the sending of the message was successful, then the blocks 2204-2207, the routine decodes the message type and
routine continues at block 1909, else the routine returns. In invokes the appropriate routine to handle the message. For
block 1909, the routine adds the edge as a pending edge. In example, in decision block 2204, if the type of the message is
block 1910, the routine invokes the add neighbor routine to 40 broadcast statement (i.e., broadcast stimt), then the routine
add the proposing process on the external port as a neighbor. invokes the handle broadcast message routine in block 2205.
The routine then returns. In block 1911, the routine sends an After invoking the appropriate handling routine, the routine
external message (i.e., edge proposal resp) indicating that continues at block 2208. In decision block 2208, if the par
this proposed edge is not accepted. In decision block 1912, if tially connected buffer is full, then the routine continues at
the number of holes is odd, then the routine continues at block 45 block 2209, else the routine continues at block 2210. The
1913, else the routine returns. In block 1913, the routine broadcaster component collects all its internal messages in a
invokes the fill hole routine and then returns. buffer while partially connected so that it can forward the
FIG. 20 is a flow diagram illustrating the processing of the messages as it connects to new neighbors. If, however, that
handle port connection call routine in one embodiment. This buffer becomes full, then the process assumes that it is now
routine is invoked when an external message is received then 50 fully connected and that the expected number of connections
indicates that the sending process wants to connect to one was too high, because the broadcast channel is now in the
hole of this process. In decision block 2001, if the number of small regime. In block 2209, the routine invokes the achieve
holes of this process is greater than Zero, then the routine connection routine and then continues in block 2210. In deci
continues at block 2002, else the routine continues at block sion block 2210, if the application program message queue is
2003. In decision block 2002, if the sending process is not a 55 empty, then the routine returns, else the routine continues at
neighbor, then the routine continues at block 2004, else the block 2212. In block 2212, the routine invokes the receive
routine continues to block 2003. In block 2003, the routine response routine passing the acquired message and then
sends a port connection response external message (i.e., port returns. The received response routine is a callback routine of
connection resp) to the sending process that indicates that it the application program.
is not okay to connect to this process. The routine then 60 FIG. 23 is a flow diagram illustrating the processing of the
returns. In block 2004, the routine sends a port connection handle broadcast message routine in one embodiment. This
response external message to the sending process that indi routine is passed an indication of the originating process, an
cates that is okay to connect this process. In decision block indication of the neighbor who sent the broadcast message,
2005, if the sending of the message was successful, then the and the broadcast message itself. In block 2301, the routine
routine continues at block 2006, else the routine continues at 65 performs the out of order processing for this message. The
block 2007. In block 2006, the routine invokes the add neigh broadcaster component queues messages from each originat
bor routine to add the sending process as a neighbor of this ing process until it can send them in sequence number order to
US 7,412,537 B2
25 26
the application program. In block 2302, the routine invokes the routine continues at block 2802, else the routine continues
the distribute broadcast message routine to forward the mes at block 2813. In decision block 2802, if the forwarding
sage to the neighbors of this process. In decision block 2303, distance is greater than Zero, then the random walk is not
if a newly connected neighboris waiting to receive messages, complete and the routine continues at block 2803, else the
then the routine continues at block 2304, else the routine routine continues at block 2804. In block 2803, the routine
returns. In block 2304, the routine sends the messages in the invokes the forward connection edge search routine passing
correct order if possible for each originating process and then the identification of the requesting process and the decre
returns.
FIG. 24 is a flow diagram illustrating the processing of the mented forwarding distance. The routine then continues at
block 2815. In decision block 2804, if the requesting process
distribute broadcast message routine in one embodiment. 10
is a neighbor or the edge between this process and the sending
This routine sends the broadcast message to each of the neigh neighbor is reserved because it has already been offered to a
bors of this process, except for the neighbor who sent the process, then the routine continues at block 2805, else the
message to this process. In block 2401, the routine selects the routine continues at block 2806. In block 2805, the routine
next neighbor other than the neighbor who sent the message.
In decision block 2402, if all such neighbors have already 15 invokes the forward connection edge search routine passing
been selected, then the routine returns. In block 2403, the an indication of the requesting party and a toggle indicator
routine sends the message to the selected neighbor and then that alternatively indicates to continue the random walk for
loops to block 2401 to select the next neighbor. one or two more computers. The routine then continues at
FIG. 26 is a flow diagram illustrating the processing of the block 2815. In block 2806, the routine dials the requesting
handle connection port search statement routine in one process via the call-in port. In block 2807, the routine sends
embodiment. This routine is passed an indication of the an edge proposal call external message (i.e., edge proposal
neighbor that sent the message and the message itself Inblock call) and receives the response (i.e., edge proposal resp).
2601, the routine invokes the distribute internal message Assuming that the response is successfully received, the rou
which sends the message to each of its neighbors other than tine continues at block 2808. In decision block 2808, if the
the sending neighbor. In decision block 2602, if the number of 25 response indicates that the edge is acceptable to the request
holes of this process is greater than Zero, then the routine ing process, then the routine continues at block 2809, else the
continues at block 2603, else the routine returns. In decision routine continues at block 2812. In block 2809, the routine
block 2603, if the requesting process is a neighbor, then the reserves the edge between this process and the sending neigh
routine continues at block 2605, else the routine continues at bor. In block 2810, the routine adds the requesting process as
block 2604. In block 2604, the routine invokes the court 30 a neighbor by invoking the add neighbor routine. In block
neighbor routine and then returns. The court neighbor routine 2811, the routine removes the sending neighboras a neighbor.
connects this process to the requesting process if possible. In In block 2812, the routine hangs up the external port and
block 2605, if this process has one hole, then the neighbors continues at block 2815. In decision block 2813, if this pro
with empty ports condition exists and the routine continues at cess is the requesting process and the number of holes of this
block 2606, else the routine returns. In block 2606, the routine 35 process equals one, then the routine continues at block 2814.
generates a condition check message (i.e., condition check) else the routine continues at block 2815. In block 2814, the
that includes a list of this process neighbors. In block 2607, routine invokes the fill hole routine. In block 2815, the routine
the routine sends the message to the requesting neighbor. sends an connection edge search response message (i.e., con
FIG. 27 is a flow diagram illustrating the processing of the nection edge search response) to the sending neighbor
court neighbor routine in one embodiment. This routine is 40 indicating acknowledgement and then returns. The graphs are
passed an indication of the prospective neighbor for this pro sensitive to parity. That is, all possible paths starting from a
cess. If this process can connect to the prospective neighbor, node and ending at that node will have an even length unless
then it sends a port connection call external message to the the graph has a cycle whose length is odd. The broadcaster
prospective neighbor and adds the prospective neighbor as a component uses a toggle indicator to vary the random walk
neighbor. In decision block 2701, if the prospective neighbor 45 distance between even and odd distances.
is already a neighbor, then the routine returns, else the routine FIG. 29 is a flow diagram illustrating the processing of the
continues at block 2702. In block 2702, the routine dials the handle connection edge search response routine in one
prospective neighbor. In decision block 2703, if the number embodiment. This routine is passed as indication of the
of holes of this process is greater than Zero, then the routine requesting process, the sending neighbor, and the message. In
continues at block 2704, else the routine continues at block 50 block 2901, the routine notes that the connection edge search
2706. In block 2704, the routine sends a port connection call response (i.e., connection edge search resp) has been
external message (i.e., port connection call) to the prospec received and if the forwarding distance is less than or equal to
tive neighbor and receives its response (i.e., port connection one unreserves the edge between this process and the sending
resp). Assuming the response is successfully received, in neighbor. In decision block 2902, if the requesting process
block 2705, the routine adds the prospective neighbor as a 55 indicates that the edge is acceptable as indicated in the mes
neighbor of this process by invoking the add neighbor routine. sage, then the routine continues at block 2903, else the routine
In block 2706, the routine hangs up with the prospect and then returns. In block 2903, the routine reserves the edge between
returns. this process and the sending neighbor. In block 2904, the
FIG. 28 is a flow diagram illustrating the processing of the routine removes the sending neighbor as a neighbor. In block
handle connection edge search call routine in one embodi 60 2905, the routine invokes the court neighbor routine to con
ment. This routine is passed a indication of the neighbor who nect to the requesting process. In decision block 2906, if the
sent the message and the message itself. This routine either invoked routine was unsuccessful, then the routine continues
forwards the message to a neighbor or proposes the edge at block 2907, else the routine returns. In decision block 2907,
between this process and the sending neighbor to the request if the number of holes of this process is greater than Zero, then
ing process for edge pinning. In decision block 2801, if this 65 the routine continues at block 2908, else the routine returns.
process is not the requesting process or the number of holes of In block 2908, the routine invokes the fill hole routine and
the requesting process is still greater than or equal to two, then then returns.
US 7,412,537 B2
27 28
FIG. 30 is a flow diagram illustrating the processing of the continues at block 3403. If this process does not have one
broadcast routine in one embodiment. This routine is invoked hole, then the set of neighbors of this process is not the same
by the application program to broadcast a message on the as the set of neighbors of the sending process. In decision
broadcast channel. This routine is passed the message to be block 3402, if this process and the sending process have the
broadcast. In decision block 3001, if this process has at least same set of neighbors, then the broadcast channel is not in the
one neighbor, then the routine continues at block 3002, else small regime and the routine continues at block 3403, else the
the routine returns since it is the only process connected to be routine continues at block 3406. In decision block 3403, if
broadcast channel. In block 3002, the routine generates an this process has no holes, then the routine returns, else the
internal message of the broadcast statement type (i.e., broad routine continues at block 3404. In block 3404, the routine
cast stimt). In block 3003, the routine sets the sequence num 10 sets the estimated diameter for this process to one. In block
ber of the message. In block 3004, the routine invokes the 3405, the routine broadcasts a diameter reset internal message
distribute internal message routine to broadcast the message (i.e., diameter reset) indicating that the estimated diameter is
on the broadcast channel. The routine returns. one and then returns. In block 3406, the routine creates a list
FIG.31 is a flow diagram illustrating the processing of the of neighbors of this process. In block 3407, the routine sends
acquire message routine in one embodiment. The acquire 15 the condition check message (i.e., condition check stnt)
message routine may be invoked by the application program with the list of neighbors to the neighbor who sent the con
or by a callback routine provided by the application program. dition double check message and then returns.
This routine returns a message. In block 3101, the routine From the above description, it will be appreciated that
pops the message from the message queue of the broadcast although specific embodiments of the technology have been
channel. In decision block 3102, if a message was retrieved, described, various modifications may be made without devi
then the routine returns an indication of Success, else the ating from the spirit and scope of the invention. For example,
routine returns indication of failure. the communications on the broadcast channel may be
FIGS. 32-34 are flow diagrams illustrating the processing encrypted. Also, the channel instance or session identifier
of messages associated with the neighbors with empty ports may be a very large number (e.g., 128 bits) to help prevent an
condition. FIG.32 is a flow diagram illustrating processing of 25 unauthorized user to maliciously tap into abroadcast channel.
the handle condition check message in one embodiment. This The portal computer may also enforce security and not allow
message is sent by a neighbor process that has one hole and an unauthorized user to connect to the broadcast channel.
has received a request to connect to a hole of this process. In Accordingly, the invention is not limited except by the claims.
decision block 3201, if the number of holes of this process is The invention claimed is:
equal to one, then the routine continues at block 3202, else the 30 1. A method in a computer system for determining a diam
neighbors with empty ports condition does not exist any more eter of a broadcast channel, the broadcast channel having
and the routine returns. In decision block 3202, if the sending computers, each computer connected to at least three neigh
neighbor and this process have the same set of neighbors, the bor computers, the method comprising:
routine continues at block 3203, else the routine continues at receiving a message from a neighbor computer;
block 3205. In block 3203, the routine initializes a condition 35 identifying a distance traveled from the received message;
double check message (i.e., condition double check) with setting an estimated diameter based on the identified dis
the list of neighbors of this process. In block 3204, the routine tance traveled amount, wherein setting the estimated
sends the message internally to a neighbor other than sending diameter sets the estimated diameter to the distance trav
neighbor. The routine then returns. In block 3205, the routine eled whenever the identified distance traveled is greater
selects a neighbor of the sending process that is not also a 40 than the current estimated diameter;
neighbor of this process. In block 3206, the routine sends a incrementing the distance traveled in the message; and
condition repair message (i.e., condition repair stimt) exter sending the message with the incremented distance trav
nally to the selected process. In block 3207, the routine eled to a neighbor computer.
invokes the add neighbor routine to add the selected neighbor 2. The method of claim 1 wherein the computers of the
as a neighbor of this process and then returns. 45 broadcast channel forman m-regular and m-connected graph.
FIG. 33 is a flow diagram illustrating processing of the 3. The method of claim 2 wherein m is 4.
handle condition repair statement routine in one embodiment. 4. The method of claim 1 wherein each computer is con
This routine removes an existing neighbor and connects to the nected to its neighbor computers via point-to-point connec
tions.
process that sent the message. In decision block 3301, if this
process has no holes, then the routine continues at block3302, 50 5. The method of claim 1 including when the estimated
else the routine continues at block 3304. In block 3302, the diameter is set, broadcasting a message indicating a new
estimated diameter.
routine selects a neighbor that is not involved in the neighbors
with empty ports condition. In block 3303, the routine 6. The method of claim 1 including:
removes the selected neighbor as a neighbor of this process. receiving a message indicating a new estimated diameter;
Thus, this process that is executing the routine now has at 55 and
least one hole. In block 3304, the routine invokes the add when the new estimated diameter is greater than the cur
neighbor routine to add the process that sent the message as a rently estimated diameter, setting the estimated diameter
neighbor of this process. The routine then returns. to the new estimated diameter.
FIG. 34 is a flow diagram illustrating the processing of the 7. The method of claim 1 including:
handle condition double check routine. This routine deter
60 receiving a message indicated to reset the estimated diam
mines whether the neighbors with empty ports condition eter to a new estimated diameter, and
really is a problem or whether the broadcast channel is in the setting the estimated diameter to the new estimated diam
eter.
small regime. In decision block 3401, if this process has one
hole, then the routine continues at block 3402, else the routine
PATENT NO. : 7412,537 B2 Page 1 of 2

APPLICATIONNO. : 10/733669
DATED : August 12, 2008
INVENTOR(S) : Holt et al.
On the title page, item (56), under “Other Publications, in column 2, line 2, delete
“Internation and insert -- International --, therefor.
On page 2, item (56), under “Other Publications, in column 2, line 27, delete
“Communicaitons: Commecting and insert -- Communications: Connecting --,
therefor.
On page 2, item (56), under “Other Publications, in column 2, line 33, delete
“Annouces and insert -- Announces --, therefor.
In the Drawings: On sheet 23 of 39, figure 17, in box 1706, line 1, delete “interal and
insert -- internal --, therefor.
In the Drawings: On sheet 23 of 39, figure 17, in box 1708, line 1, delete “interal and
insert -- internal --, therefor.
In column 1, line 15, after “09/629,577, insert -- entitled --.
In column 1, line 22, after “09/629,023 insert --, --.
In column 1, line 36, delete “a.
In column 15, line 23, delete “implement and insert -- implemented --, therefor.
In column 21, line 49, after “if delete “is.
PATENT NO. : 7412,537 B2 Page 2 of 2

APPLICATIONNO. : 10/733669
DATED : August 12, 2008
INVENTOR(S) : Holt et al.
In column 25, line 22, after “itself insert -- . --.

Sixth Day of January, 2009
WDJ
JON. W. DUDAS
Director of the United States Patent and Trademark Office
CIVIL COVER SHEET
JS-CAND 44 (Rev. 06/17)
The JS-CAND 44 civil cover sheet and the information contained herein neither replace nor supplement the filing and service of pleadings or other papers as required by law,
except as provided by local rules of court. This form, approved in its original form by the Judicial Conference of the United States in September 1974, is required for the Clerk of
Court to initiate the civil docket sheet. (SEE INSTRUCTIONS ON NEXT PAGE OF THIS FORM.)
I. (a) PLAINTIFFS DEFENDANTS
EPIC GAMES, INC., a Maryland Corporation ACCELERATION BAY LLC, a Delaware Limited Liability Company
(b) County of Residence of First Listed Plaintiff Wake County of Residence of First Listed Defendant San Mateo, CA
(EXCEPT IN U.S. PLAINTIFF CASES) (IN U.S. PLAINTIFF CASES ONLY)
NOTE: IN LAND CONDEMNATION CASES, USE THE LOCATION OF
THE TRACT OF LAND INVOLVED.
(c) Attorneys (Firm Name, Address, and Telephone Number) Attorneys (If Known)
David P. Enzminger, Winston & Strawn LLP, 333 South Grand Avenue, 38th
Floor, Los Angeles, CA 90071-1543, 213-615-1700 (See Attachment A)
II. BASIS OF JURISDICTION (Place an “X” in One Box Only) III. CITIZENSHIP OF PRINCIPAL PARTIES (Place an “X” in One Box for Plaintiff
(For Diversity Cases Only) and One Box for Defendant)
PTF DEF PTF DEF
1 U.S. Government Plaintiff 3 Federal Question Citizen of This State 1 1 Incorporated or Principal Place 4 4
(U.S. Government Not a Party)
of Business In This State
Citizen of Another State 2 2 Incorporated and Principal Place 5 5
2 U.S. Government Defendant 4 Diversity of Business In Another State
(Indicate Citizenship of Parties in Item III)
Citizen or Subject of a 3 3 Foreign Nation 6 6
Foreign Country
IV. NATURE OF SUIT (Place an “X” in One Box Only)

CONTRACT TORTS FORFEITURE/PENALTY BANKRUPTCY OTHER STATUTES
110 Insurance PERSONAL INJURY PERSONAL INJURY 625 Drug Related Seizure of 422 Appeal 28 USC § 158 375 False Claims Act
120 Marine Property 21 USC § 881 423 Withdrawal 28 USC 376 Qui Tam (31 USC
310 Airplane 365 Personal Injury – Product
130 Miller Act Liability 690 Other § 157 § 3729(a))
315 Airplane Product Liability
140 Negotiable Instrument 367 Health Care/ LABOR PROPERTY RIGHTS 400 State Reapportionment
320 Assault, Libel & Slander
150 Recovery of Pharmaceutical Personal 410 Antitrust
330 Federal Employers’ 710 Fair Labor Standards Act 820 Copyrights
Overpayment Of Injury Product Liability 430 Banks and Banking
Liability 720 Labor/Management 830 Patent
Veteran’s Benefits 368 Asbestos Personal Injury 450 Commerce
340 Marine Relations 835 PatentņAbbreviated New
151 Medicare Act Product Liability
345 Marine Product Liability 740 Railway Labor Act Drug Application 460 Deportation
152 Recovery of Defaulted PERSONAL PROPERTY 470 Racketeer Influenced &
350 Motor Vehicle 751 Family and Medical 840 Trademark
Student Loans (Excludes 370 Other Fraud Corrupt Organizations
355 Motor Vehicle Product Leave Act
Veterans) 371 Truth in Lending SOCIAL SECURITY 480 Consumer Credit
Liability 790 Other Labor Litigation
153 Recovery of 380 Other Personal Property 861 HIA (1395ff)
360 Other Personal Injury 791 Employee Retirement 490 Cable/Sat TV
Overpayment Damage Income Security Act 862 Black Lung (923) 850 Securities/Commodities/
of Veteran’s Benefits 362 Personal Injury -Medical
Malpractice 385 Property Damage Product 863 DIWC/DIWW (405(g)) Exchange
160 Stockholders’ Suits Liability IMMIGRATION
864 SSID Title XVI 890 Other Statutory Actions
190 Other Contract 462 Naturalization
CIVIL RIGHTS PRISONER PETITIONS 865 RSI (405(g)) 891 Agricultural Acts
195 Contract Product Liability Application
440 Other Civil Rights HABEAS CORPUS FEDERAL TAX SUITS 893 Environmental Matters
196 Franchise 465 Other Immigration
441 Voting 463 Alien Detainee Actions 895 Freedom of Information
870 Taxes (U.S. Plaintiff or
REAL PROPERTY 442 Employment Defendant)
Act
510 Motions to Vacate
210 Land Condemnation 443 Housing/ Sentence 896 Arbitration
871 IRS–Third Party 26 USC
220 Foreclosure Accommodations 530 General § 7609 899 Administrative Procedure
445 Amer. w/Disabilities– Act/Review or Appeal of
230 Rent Lease & Ejectment 535 Death Penalty
Employment Agency Decision
240 Torts to Land OTHER 950 Constitutionality of State
245 Tort Product Liability 446 Amer. w/Disabilities–Other
540 Mandamus & Other Statutes
290 All Other Real Property 448 Education
550 Civil Rights
555 Prison Condition
560 Civil Detainee–
Conditions of
Confinement
V. ORIGIN (Place an “X” in One Box Only)

1 Original 2 Removed from 3 Remanded from 4 Reinstated or 5 Transferred from 6 Multidistrict 8 Multidistrict
Proceeding State Court Appellate Court Reopened Another District (specify) Litigation–Transfer Litigation–Direct File
VI. CAUSE OF Cite the U.S. Civil Statute under which you are filing (Do not cite jurisdictional statutes unless diversity):
28 U.S.C. § 2201
ACTION
Brief description of cause:
Declaratory judgment that neither Epic nor the Accused Products infringe any of the Asserted Patents under the Patent Laws of the United States, 35 U.S.C. §§ 101, et seq.
VII. REQUESTED IN CHECK IF THIS IS A CLASS ACTION DEMAND $ CHECK YES only if demanded in complaint:
COMPLAINT: UNDER RULE 23, Fed. R. Civ. P. JURY DEMAND: Yes No
VIII. RELATED CASE(S), JUDGE DOCKET NUMBER

IF ANY (See instructions):
IX. DIVISIONAL ASSIGNMENT (Civil Local Rule 3-2)
(Place an “X” in One Box Only) SAN FRANCISCO/OAKLAND SAN JOSE EUREKA-MCKINLEYVILLE
DATE 07/18/2019 SIGNATURE OF ATTORNEY OF RECORD /s/ Michael A. Tomasulo

JS-CAND 44 (rev. 07/16) Case 5:19-cv-04133-SVK Document 1-8 Filed 07/18/19 Page 2 of 3
INSTRUCTIONS FOR ATTORNEYS COMPLETING CIVIL COVER SHEET FORM JS-CAND 44
Authority For Civil Cover Sheet. The JS-CAND 44 civil cover sheet and the information contained herein neither replaces nor supplements the filings and
service of pleading or other papers as required by law, except as provided by local rules of court. This form, approved in its original form by the Judicial
Conference of the United States in September 1974, is required for the Clerk of Court to initiate the civil docket sheet. Consequently, a civil cover sheet is
submitted to the Clerk of Court for each civil complaint filed. The attorney filing a case should complete the form as follows:
I. a) Plaintiffs-Defendants. Enter names (last, first, middle initial) of plaintiff and defendant. If the plaintiff or defendant is a government agency, use
only the full name or standard abbreviations. If the plaintiff or defendant is an official within a government agency, identify first the agency and
then the official, giving both name and title.
b) County of Residence. For each civil case filed, except U.S. plaintiff cases, enter the name of the county where the first listed plaintiff resides at the
time of filing. In U.S. plaintiff cases, enter the name of the county in which the first listed defendant resides at the time of filing. (NOTE: In land
condemnation cases, the county of residence of the “defendant” is the location of the tract of land involved.)
c) Attorneys. Enter the firm name, address, telephone number, and attorney of record. If there are several attorneys, list them on an attachment, noting
in this section “(see attachment).”
II. Jurisdiction. The basis of jurisdiction is set forth under Federal Rule of Civil Procedure 8(a), which requires that jurisdictions be shown in
pleadings. Place an “X” in one of the boxes. If there is more than one basis of jurisdiction, precedence is given in the order shown below.
(1) United States plaintiff. Jurisdiction based on 28 USC §§ 1345 and 1348. Suits by agencies and officers of the United States are included here.
(2) United States defendant. When the plaintiff is suing the United States, its officers or agencies, place an “X” in this box.
(3) Federal question. This refers to suits under 28 USC § 1331, where jurisdiction arises under the Constitution of the United States, an amendment
to the Constitution, an act of Congress or a treaty of the United States. In cases where the U.S. is a party, the U.S. plaintiff or defendant code
takes precedence, and box 1 or 2 should be marked.
(4) Diversity of citizenship. This refers to suits under 28 USC § 1332, where parties are citizens of different states. When Box 4 is checked, the
citizenship of the different parties must be checked. (See Section III below; NOTE: federal question actions take precedence over diversity
cases.)
III. Residence (citizenship) of Principal Parties. This section of the JS-CAND 44 is to be completed if diversity of citizenship was indicated above.
Mark this section for each principal party.
IV. Nature of Suit. Place an “X” in the appropriate box. If the nature of suit cannot be determined, be sure the cause of action, in Section VI below, is
sufficient to enable the deputy clerk or the statistical clerk(s) in the Administrative Office to determine the nature of suit. If the cause fits more than
one nature of suit, select the most definitive.
V. Origin. Place an “X” in one of the six boxes.
(1) Original Proceedings. Cases originating in the United States district courts.
(2) Removed from State Court. Proceedings initiated in state courts may be removed to the district courts under Title 28 USC § 1441. When the
petition for removal is granted, check this box.
(3) Remanded from Appellate Court. Check this box for cases remanded to the district court for further action. Use the date of remand as the filing
date.
(4) Reinstated or Reopened. Check this box for cases reinstated or reopened in the district court. Use the reopening date as the filing date.
(5) Transferred from Another District. For cases transferred under Title 28 USC § 1404(a). Do not use this for within district transfers or
multidistrict litigation transfers.
(6) Multidistrict Litigation Transfer. Check this box when a multidistrict case is transferred into the district under authority of Title 28 USC
§ 1407. When this box is checked, do not check (5) above.
(8) Multidistrict Litigation Direct File. Check this box when a multidistrict litigation case is filed in the same district as the Master MDL docket.
Please note that there is no Origin Code 7. Origin Code 7 was used for historical records and is no longer relevant due to changes in statute.
VI. Cause of Action. Report the civil statute directly related to the cause of action and give a brief description of the cause. Do not cite jurisdictional
statutes unless diversity. Example: U.S. Civil Statute: 47 USC § 553. Brief Description: Unauthorized reception of cable service.
VII. Requested in Complaint. Class Action. Place an “X” in this box if you are filing a class action under Federal Rule of Civil Procedure 23.
Demand. In this space enter the actual dollar amount being demanded or indicate other demand, such as a preliminary injunction.
Jury Demand. Check the appropriate box to indicate whether or not a jury is being demanded.
VIII. Related Cases. This section of the JS-CAND 44 is used to identify related pending cases, if any. If there are related pending cases, insert the docket
numbers and the corresponding judge names for such cases.
IX. Divisional Assignment. If the Nature of Suit is under Property Rights or Prisoner Petitions or the matter is a Securities Class Action, leave this
section blank. For all other cases, identify the divisional venue according to Civil Local Rule 3-2: “the county in which a substantial part of the
events or omissions which give rise to the claim occurred or in which a substantial part of the property that is the subject of the action is situated.”
Date and Attorney Signature. Date and sign the civil cover sheet.
ATTACHMENT A
David P. Enzminger
Michael A. Tomasulo
WINSTON & STRAWN LLP
333 South Grand Avenue, 38th Floor
Los Angeles, CA 90071-1543
Telephone: (213) 615-1700
Louis L. Campbell
WINSTON & STRAWN LLP
275 Middlefield Road, Suite 205
Menlo Park, California 94025-4004
Telephone: (650) 858-6500

Acceleration

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Acceleration

Uploaded by

Copyright:

Available Formats

WARNING: AT LEAST ONE DOCUMENT COULD NOT BE INCLUDED!

You were not billed for these documents.

Document Number Document Description Pages Document Error

1 David P. Enzminger (SBN: 137065)

6 Louis L. Campbell (SBN:221282)

11 Attorneys for Plaintiff

13 UNITED STATES DISTRICT COURT

14 NORTHERN DISTRICT OF CALIFORNIA

15 EPIC GAMES, INC., a Maryland Corporation, Case No.: 3:19-cv-04133

COMPLAINT FOR DECLARATORY JUDGMENT OF PATENT NON-INFRINGEMENT - CASE NO. 3:19-CV-04133

2 against Defendant Acceleration Bay, LLC (“Acceleration” or “Defendant”) as follows:

3 PARTIES AND BACKGROUND

9 2. On information and belief, sometime around February 2015, Defendant Acceleration

14 Patents are attached as Exhibits 1-7.

18 “Patents Infringed By Epic Games.”

2 (hereinafter, the “Accused Products”).

6 advised that the ’537 Patent was at issue as well.

9 9. None of Acceleration’s communications provided an explanation for its repeated

12 named in the letters.

16 inventors, corporations and entrepreneurs to accelerate growth in creating innovative companies.”

21 acquiring and enforcing the Asserted Patents.

25 product or service or has any non-litigation-based income or revenue.

26 15. On information and belief, Acceleration is exclusively in the business of monetizing

27 patents acquired from third parties.

28 16. Epic’s games do not infringe the Asserted Patents.

2 computer network and/or broadcast channel with an m-regular, incomplete topology.

5 through a central server.

6 19. Certain of the claims are directed to a computer network.

8 the Asserted Patents.

9 21. Certain of the claims are directed to a broadcast channel.

14 the Asserted Patents.

15 24. The “Background” section of the Asserted Patents identifies “client/server

20 information.” See, e.g., Ex. 1, Background ’344 Patent at 1:58–60.

21 26. Client-server middleware systems where a server coordinates the communications

23 the alleged inventions of the Asserted Patents.

27 network must have connections to at least three other neighboring participants.

2 each participant is connected to the exact same number, m, of neighbor participants.

4 must be at least two less than the total number of participants.

11 to the alleged inventions of the Asserted Patents.

13 to the alleged inventions of the Asserted Patents.

20 section of ’344 Patent at 1:58–2:14.

24 38. A comparison of the three network types is shown below:

9 architecture that was well known in the prior art.

11 information only to/from a server.

13 exchange information with other Fortnite players.

15 infringe this patent.

22 the communications ports selected by the port ordering algorithm.”

25 ports selected by any non-random port ordering algorithm.

1 NATURE OF THE ACTION

4 its Fortnite products.

7 PARTIES, JURISDICTION AND VENUE

10 Crossroads Blvd, Cary, North Carolina, 27518.

28 traditional notions of fair play and substantial justice.

4 directed activities to this district.

10 Acceleration Bay patented technologies.”

12 Asserted Patents purportedly owned by Defendant.

18 business model is limited to enforcing these patents.

22 Plaintiff on the Asserted Patents.

4 60. This is an intellectual property action subject to district-wide assignment pursuant to

5 Local Rules 3-2(c) and 3-5(b).

7 FIRST CAUSE OF ACTION

8 (Declaratory Judgment of Non-Infringement of the ’344 Patent)

10 though fully set forth herein.