SQL to HTML: Automatic Generation

of HTML Database Schemas

Jon Finke
finkej@rpi.edu

Distributed Systems Group

Information Technology Services
Rensselaer Polytechnic Institute

LISA-IX-SQL-2-HTML Rensselaer Polytechnic Institute - Information Technology Services - Distributed Systems Group 1

Head Text
Box Text
Times Italic Bold Bold-Italic
Code Text
Box Title
 Defining the Problem

Hand
Written
Technical
/etc/passwd Programs New Documents
/etc/group Programs

/etc/hosts Generated
Simon Faims Technical
Unix Admin HR Documents

Document
Generator

LISA-IX-SQL-2-HTML Rensselaer Polytechnic Institute - Information Technology Services - Distributed Systems Group 2

Program Oracle Table Document

• Many of our systems administration tasks are programs built on top of an Oracle
database. From this system, we generate many of the configuration files that go
into systems operation such as the password file, the host file and so on.
• The problem comes when we go to write new programs, we would like to have some
technical documents to reference. Unfortunatly, these are all to often non existant.
• However, the database table descriptions can act as at least a partial technical
document, and so I am looking for ways to automate as much of this document
generation as I can.
• This problem has been approached in the past from the Program source side of
things, but my starting point is with the relational database.
 Why SQL and why HTML
• A Relational Database can be a very powerful sys
admin support tool. Many use SQL.
• Allows delegation of responsibility
• Handles large scale installations & automation
• Leads to other follow on applications.
• Emerging documentation standard
• Browsers available for many platforms
• Hypertext links are very nice for linking between
different documents and sources.
• Interesting synergy between SQL relations and
Hypertext links.

LISA-IX-SQL-2-HTML Rensselaer Polytechnic Institute - Information Technology Services - Distributed Systems Group 3

• We manage 500 systems with 10,000 users. All userid creation, expiration, disk
accounting, host naming (DNS), group file, netgroups, mail aliases and groups,
printcap configuration, etc.
• Many of these functions are delegated to other staff members, and in some cases, to
individual users (we can grant control of the membership in a specific unix group to a
specific user.
• Accounts created automatically based on info from Registrar, HR and alumni
development. Unix group membership tracks AFS/PTS group membership for some
groups. Mailing lists automatically remove members that expire, etc.
• Provides infrastructure for online directory maintenance, billing, ID card
management, student record systems, etc.
• Although crude and limited, it is becoming popular. Lends itself to online access,
and still allows for hardcopy. The online lookup seems easier to use than "man"
pages or hardcopy documents.
• Money for Unix workstations has run out at our site - current direction is to PCs.
• Links are helpful in delegating parts of a document.
• Since we can define a relationship between columns in different tables, we can
include those relationships as hypertext links when we mechanically generate a
document.
 Relations

Logins UID (the Unix user id number) is the

Username Primary Key in the Logins table.
UID

Groups GID (the Unix Group id number) is

Name the Primary Key of the Groups table.
GID

GroupMem Both UID and GID are Foreign

UID Keys, that relate to keys in the
Logins and Groups tables.
GID

LISA-IX-SQL-2-HTML Rensselaer Polytechnic Institute - Information Technology Services - Distributed Systems Group 4

• In this example, we look at how we build the /etc/group file and how the tables
interact. Since this IS an example, the tables have been simplified quite a bit. That
should not interfere with the example.
• The first table is the "logins" table. This information is used to generate the
/etc/passwd file amoung. The items of interest are the username (account name),
and the unix uid.
• The second table is the "groups" table. This holds the group name and gid, that is
used to make up the /etc/group file.
• The third table, is the "group members" table. This provides the link between the
logins table and the groups table for each username specifically listed in the
/etc/group file. By adding more GID/UID pairs here, we can put multiple people in
multiple groups. When we extract the /etc/group file, we look up the username to
include for each group. This means that if a username changes, it will be reflected
in the /etc/group file, and if a userid is deleted, it will automatically be removed
from the file as well.
 Relations Example
GroupMem Logins
2517 conway
Groups 100 2517
staff
100 GroupMem
12320
100 Logins
finkej
Groups GroupMem 12320
security 12320
120 120

staff::100:conway,finkej
security::120:finkej

LISA-IX-SQL-2-HTML Rensselaer Polytechnic Institute - Information Technology Services - Distributed Systems Group 5

• The same three tables, with two groups defined, staff and security in the groups
table, two people in the logins table, me and one of my co-workers and finally,
three group member records that put me into both of the groups and conway into
just the staff group. Put these all together and you get a group file as shown at
the bottom of the slide.
 Relations - HTML Output

Table Simon.Logins
All past and present accounts, with numeric uids, text userna
Index: I_Logins$Unixuid Logins(Uid)

Username char(8)
An 8 character identifier, aka login name, userid, account, etc
also used to reserve names.
Uid number - Primary Key
Referenced by: GroupMem(Uid), DiskVols(Uid),PwAdminL
The Unix uid - should not be re-used if possible.
Unixgid number
A group id if this user does not get the default group for this
Gecos char(64)
The finger or gecos information for this entry. Defaults to the

LISA-IX-SQL-2-HTML Rensselaer Polytechnic Institute - Information Technology Services - Distributed Systems Group 6

• When we generate a page, we use these relationships to create the hot links. In
this example, we have the Logins table, and in it, the UID column, which is
referenced by the GroupMem table, as well as the DiskVols and other tables.
• Although we didn’t show it in the previous slide, we also define a GID for each
user, and this references the Groups table.
• We also have a link within this page, from the index listing at the top, to the Uid
column. On larger tables, extra links like this are sometimes quite handy.
• Disclaimer - This is an example - The actual tables have more colujmns defined,
and from a relational theory standpoint, these relations are not all that good.
 Shortcomings of SQL to HTML

• Constraints have real meaning in Oracle

• Access control problems with other oracle users
• Tricky when dealing with other databases
• Complicates modifying tables
• Limited to Primary/Foreign Relationships
• Limited to Table-Row / Table-Row Relationships
• Does not handle sequences
• Limited to Database objects

LISA-IX-SQL-2-HTML Rensselaer Polytechnic Institute - Information Technology Services - Distributed Systems Group 7

• The first problem, is that Constraints have a real database meaning, and with
that, carry some restrictions on how they can be used. You need additional
access to a table, (reference, not just select). Other people may not want to give
you that access, as your "constraint", may prevent their changing their own table.
• Often times, our applications span multiple machines and databases.
• A constraint on a table (such as a Primary key, referenced by other tables) will
prevent you from deleting the table. While in a database sense, this is a feature, if
you are prototyping, this can be a real pain.
• You are also limited to defining a primary key, and other keys that reference it.
We have a number of cases where one table feeds into another, and neither really
"owns" the key. This is especially true with "external" data such as a SSN. (The
SSN makes a rotten key anyway.)
• Often times, you are not so much interested in showing a relationship between two
columns, but rather, a relationship between two tables.
• Some columns take their values from a sequence. These should be linked like
columns.
• But most of all, you want to be able to have links to objects OUTSIDE of the
database, such as program documentation, program source, operations manuals,
etc.
 Relations: Lifting Constraints

Uid_Count
NextVal

Logins GroupMem Groups

Username UID Name
UID GID GID
GID

Admin_Logins

LISA-IX-SQL-2-HTML Rensselaer Polytechnic Institute - Information Technology Services - Distributed Systems Group 8

Program Oracle Table Document

• We really want more than just the rigid relational constraints between these tables.
Besides just the GroupMem.Uid to Logins.Uid, and the GroupMem.Gid to
Groups.Gid, we also are interested in the "weak" link between Logins.Gid and
Groups.Gid.
• I should point out, that we are now going beyond what has been coded, and into areas
I would like to explore.
• We also want to encode the relationship between the Logins.Uid column and the
Oracle Sequence, Uid_Count.NextVal. This sequence looks like a table, but every
time you reference the column called "Nextval", you get the next number in the
sequence. This makes it easy to generate new Uids.
• We also "export" the contents of our Logins table to the admin database machine.
Since this is running on a different machine, we can not define a constraint to hold
this link. Also, the relationship is not just between a pair of columns, but rather
between the entire table.
• We need a facility to register both table and column references to other database
objects. In addition, we need to be able to make these links "absolute", as there may
be licensing or other restrictions on table definitions. For example, while the Simon
table definitions are available to anyone, the admin table descriptions are
proprietary and we need to protect them.
• This could also allow for additional table and column comments to avoid some of the
restrictions imposed by Oracle.
 Relations - Documents

group admin
user document

group admin
technical doc

Logins GroupMem Groups

Username UID Name
UID GID GID
GID

LISA-IX-SQL-2-HTML Rensselaer Polytechnic Institute - Information Technology Services - Distributed Systems Group 9

Program Oracle Table Document

• In addition to the other oracle objects, we want to reference HTML documents.

• The technical document (hand written HTML) can certainly reference the machine
generated table, it would also be nice to have machine generated table descriptions
point BACK to the technical document. This would actually be conditional, as in
this example, from the standpoint of the Logins table, we don’t really care that the
etc_group_admin program REFERENCES the Logins table, whereas, the Groups and
GroupMembers table certainly should point back to the tech doc, since that is the
program used to maintain these tables.
• The location of the Database table documents is known in advance, so they can
simply be hard coded into the HTML documents when they are written. However,
since the database table documents may be regenerated at any time, it is not
practical to add the document references by hand. What is needed, is a document
registry program that lets you specify a URL and key text that should be referenced
in either the Table documentation, or in the Table/Column documentation. In this
way, when a table document is regenerated, these additional links will be included
automatically.
• For table document references, you might want to have multiple classes of references,
such as primary references that are included before the column descriptions, and
secondary references that are tacked on at the end.
 Relations - Program Source
group admin
user document

etc_group_admin
group admin
technical doc

Logins GroupMem Groups

Username UID Name
UID GID GID
GID

PTS_Group_Shadow

LISA-IX-SQL-2-HTML Rensselaer Polytechnic Institute - Information Technology Services - Distributed Systems Group 10

Program Oracle Table Document

 Source Code 2 HTML

/* SQL_REF(Logins:Username,Uid) */
/* $Refs: groups/etc_group_admin, $ */
/* shadow/pts_groups_shadow */
int Username_To_UnixUid(char *username){
/***********************************************
* Given a username, look up and return the Uid
* -1 if not found
***********************************************/
int result;
result = sql_getnum("Logins.Username", username, "uid",
-1);
return result;
}

LISA-IX-SQL-2-HTML Rensselaer Polytechnic Institute - Information Technology Services - Distributed Systems Group 11

• If you click on "Logins", you jump to the logins table, click on Username or Uid and
go directly to that column definition in the logins table.
• Select "etc_group_admin, and you go to the source document for that program, or
possibly a hand written document.
• Since full filename paths would tend to get pretty long, some type of module
naming might be nice.... And of course, that module might have documentation.
• If you click on the Username_To_UnixUid, you go to the hand written HTML (or
other format) document for this routine, assuming one was registered of course.
• And last, click on sql_getnum and jump to the definition (or documentation) for
THAT routine.
• What started as a pretty small project, has grown, and isn’t really a database
project anymore (although a relational database could be VERY handy in some of
the back end support). If someone was to do this (and parts may have allready
been done) some really neat tools could result.
 Reality Check Time

• Extract and Link view definitions

• Add Multiple schema / HTML home support
• Add "softlink" support to other DB objects
• Add External URL support
• General DB Object document support
• SQL Script reader/linker
• C code reader for SQL_REF tags
• HTML code reader for SQL_DOC tags.

LISA-IX-SQL-2-HTML Rensselaer Polytechnic Institute - Information Technology Services - Distributed Systems Group 12

• It doesn’t seem likely that I am going to be able to find the time to do all of those neat
features, but I am hoping to make some smaller changes and enhancements.
• The first thing I plan on doing is extracting view definitions, and making appropriate
links (A view is a virtual table based on a selection from one or more tables)
• Next, I will add a table to track where in the HTML world, different schema pages live.
This will be handy in maintaining links between schemas.
• Building on the previous step, I want to support making "soft" links between database
objects. I expect that a small application program will be written to help manage
these. This will also allow linking to sequences.
• This will quickly be followed by a way to include refernces to EXTERNAL urls within a
table description. I expect a few options here to control WHERE in the page this
reference appears, and to specify the text that will be hot linked.
• Oracle 6 does not support comments on some objects (sequences) , and the rest are
limited to 250 characters. A more general text facility would allow for longer
comments where needed. This would also be helpful for databases that do not have
the comment facility.
• The code that reads and links view definitions will also be able to read SQL scripts and
record cross references to the table.
• A program to read C source code, and look for reference tags would be handy. This
reduces the work required to include a reference to a page by a program - just add a
comment to the current file (and not run a second program)
• Likewise, this might be a quick way of adding links TO HTML pages as you write the
HTML page, add a comment, and a scan program finds it and includes it.
 Availability

SQL to HTML
Jon Finke
finkej@rpi.edu
http://www.rpi.edu/~finkej

FTP: ftp.rpi.edu /pub/its-release/sql-tools-0.1/distrib/src

AFS: /afs/rpi.edu/campus/rpi/sql-tools/0.1/distrib/src
References under: http://www.rpi.edu/.....
Samples: campus/rpi/simon/misc/Tables/simon.Index.html
Papers: ~finkej/Papers.html

LISA-IX-SQL-2-HTML Rensselaer Polytechnic Institute - Information Technology Services - Distributed Systems Group 13

• The production source is available for FTP or AFS access. This is the actual
source that we are using, and may be changing as time goes on. I don’t consider
this an excellent example of programming style, it has been hacked and chopped
quite a bit since it first was generating man pages in nroff.
• If you want to actually run it, you also needs the "sql" and "sandbox" libraries,
which are also available in its-release.
• If you want to look at what we are currently generating, under the RPI web server
we have the index for the Simon project with lots of tables to look at (Logins or
People might be a good start)
• If you are interested in learning more about the Simon project, there are a bunch
of papers online as well.