The BIRCH Home Page has links to three indices for all documentation:

• Programs by Category

• Programs by Package

• Program Index

Within the indices, there is a hierarchy of web pages, listing all programs in a package, all programs in a category, and pages for each individual program. All of these pages are generated automatically from two databases: one database for BIRCH core software ($BIRCH/public_html/birchdocdb/birchdocdb.sqlite), and another database for locally-installed software ($BIRCH/local/public_html/birchdocdb/birchdocdb.local.sqlite). Changes to the birchdocdb directory should never be made at your BIRCH site. Changes should only be made to birchdocdb.local, as described in this tutorial.

 This tutorial gives an example of adding new documentation with the following steps:

1. run doc2birchdb.py to create database input for documentation in $local/doc

2. run dat2birchdb.py to create database input for documentation in $local/dat
3. create database input for packages and programs
   
4. run birchdocdbkit.py to add database input to the birchdocdb.local database
5. run sql2htmdoc.py to generate HTML index files from the database
6. Cleanup: remove dummy database objects from the birchdocdb.local database
   

4.1. Documentation files

Documentation files are added to the $local/doc directory. As a general guideline, it is best to put files into subdirectories, rather than having large numbers of files in the main $doc directory.  The $doc directory is organized with a separate directory for each package. For example, documentation files for the PHYLIP programs are found in the $doc/Phylip directory. Even when there is only one program and one documentation file, it is still best to create a separate directory for that program.

Note: remember to make your files world-readable and directories world-readable and world-executable.

This tutorial will be done in the directory called 'transactions', found in the containing the birchdocdb.local. To go to that directory, type

cd $BIRCH/local/public_html/birchdocdb/transactions

The next step is to create a file called docfiles.tsv, listing the documentation files for the package. As an example, we will demonstrate the steps for local installation of a package called 'dummy', which contains a program called 'bogus'. Documentation and ancillary datafiles for the bogus program are found, respectively in $BIRCH/local/doc/dummy and $BIRCH/local/dat/dummy.

doc2birchdb.py $local/doc/dummy

The output shows the details:

path= /home/birch/BIRCH/local/doc/dummy

doc2birchdb.py writes a file transaction for each file in doc/dummy that has a file extension indicating documentation (eg. .txt, .pdf, .docx, .odt and so on. ) The only documentation file in local/doc/dummy is a file called 'bogusman.txt', so docfiles.tsv looks like this:

--------------------------------

cat /home/birch/BIRCH/local/doc/dummy/docfiles.tsv > adddummy.tsv

Note that the Description line arbitrarily is set to 'Manual'. Depending on the document, it might be more appropriate to edit the description to read 'User's guide' or whatever best describes this document.

In this particular case, not all documentation is included in the files distributed with the package. You can add links to the adddummy.tsv using a text editor. For example, a link to the web site for the dummy package might be written as

--------------------------------
INSERT_INTO    FILE    http://www.dummy.org
Description    Web site
Command    $ACE_FILE_LAUNCHER    http://www.dummy.org
;

Note: To ensure that each File object has a unique name, we use the simple convention of naming each File object using the same path that is found on the Command line.

4.2 Data files

Many programs have supplementary datafiles. These might include things such as scoring matrices for similarity search programs, alternative genetic codes for translation programs, or even sample datasets. Data files are added to the $BIRCH/local/dat directory. As mentioned above for documentation, it is best to have a different subdirectory for each program or package, rather than having large numbers of files in the main $BIRCH/local/dat directory. Even when there is only one program and one data file, it is still best to create a separate directory for that program.

Note: remember to make your files world-readable and directories world-readable and world-executable.

The process for making a list of datafiles is similar to that for documentation. For example, if the datafiles for the dummy package are in $local/dat/dummy, type

dat2birchdb.py $local/dat/dummy

The terminal output is

path= /home/birch/BIRCH/local/dat/dummy
full_path= /home/birch/BIRCH/local/dat/dummy
dir_string= $local/dat/dummy
outlocation= /home/birch/BIRCH/local/dat/dummy/datfiles.tsv
dat2birchdb.py: Writing output to /home/birch/BIRCH/local/dat/dummy/datfiles.tsv

In this case, every file in $local/dat/dummy will be listed in datfiles.tsv. Since not every file is necessarily data, you may wish to delete some File objects before reading datfiles.tsv into birchdocdb.local.

Since there is no way to know in advance what the purpose of each file is, this is a good time to edit the Description lines in datfiles.tsv to be more explicit. For example

--------------------------------
INSERT_INTO    FILE    $/local/dat/dummy/bogusdat1.txt
Description    datafile
Command    $ACE_FILE_LAUNCHER    $local/dat/dummy/bogusdat1.txt
;
--------------------------------
INSERT_INTO    FILE    $local/dat/dummy/bogusdat2.txt
Description    sample input file
Command    $ACE_FILE_LAUNCHER    $local/dat/dummy/bogusdat2.txt
;
--------------------------------
INSERT_INTO FILE    $local/dat/dummy/bogusdat3.txt
Description    sample output file
Command    $ACE_FILE_LAUNCHER    $local/dat/dummy/bogusdat3.txt
;

Where there are a large number of datafiles, an alternative is to delete all File objects from datfiles.tsv, and replace them with a single object that simply points to the directory.

--------------------------------
INSERT_INTO    FILE    $local/dat/dummy
Description    sample data
Command    $ACE_FILE_LAUNCHER    $local/dat/dummy
;

When this link is read by a browser, the browser will show a view of the entire directory.

Next, append datfiles.tsv to the already-existing adddummy.tsv

cat /home/birch/BIRCH/local/dat/dummy/datfiles.tsv >> adddummy.tsv

(Note the double arrow (>>) operator is used to append output to a file.)

4.3 Transactions for Package and Program

Now that the File objects have been defined in adddummy.tsv, we need to add definitions for the Package and Program objects. The have to be entered by hand, or by copying and modifying other objects.

-------------------------------
-- add a package object
INSERT_INTO    PACKAGE    dummy
Description    definition for dummy package
Category    Miscellaneous
Platform    L:l:M:m
Installation    local
;
-------------------------------
-- add a program object
INSERT_INTO    PROGRAM    bogus
Description    dummy program
Launch    interactive bogus
Launch    bldna DNA/RNA --> bogus
Documentation    $local/doc/dummy/bogusman.txt
Data    $local/dat/dummy/bogusdat2.txt
SampleOutput    $local/dat/dummy/bogusdat3.txt
Package    dummy
Category    Miscellaneous
Platform    L:l:M:m
Installation    local

Several things are worth noting:

• Interface - This line says that the program is launched from the command line, and has a graphic interface. Command line arguments taken from the  manual pages are included.

• Documentation - This line was entered manually, since the file it refers to was not in the main $local/doc/staden directory, but rather in the manual subdirectory.

• Sample_input - It makes more sense to put this file here, rather than on the Data line.

Note that to indicate that the dummy package is locally-installed (ie. not part of the BIRCH distribution), Installation is set to 'local'.

4.4 Reading transactions into the birchdocdb.local database

The birchdocdb.local database is managed using birchdocdbkit.py. It is strongly recommended to do a test of your datafile before commiting the changes to the database. For example, the File, Program and Package commands
could all be combined into a single file, such as adddummy.example.tsv. To test this file,

birchdocdbkit.py adddummy.example.tsv ../birchdocdb.local.sqlite

The output details that changes that would be made if the transactions were committed:

At this point, no changes have been made to the database. This is your opportunity to correct errors in the input, if any. Once the input file is correct, re-run the command with the --commit option to commit the changes into the database

birchdocdbkit.py adddummy.example.tsv ../birchdocdb.local.sqlite --commit

The birchdocd.local database can be directly viewed and edited using sqlitestudio:

sqlitestudio $BIRCH/local/public_html/birchdocdb/birchdocdb.local.sqlite

You can also open the local database using birchadmin --> Documentation --> lbirchdocdb

The database might look something like this:

4.5. sql2htmldoc.py - Creating documentation index web pages

At this point, the new data objects have not been incorporated into the BIRCH web documentation. The final step is to Type

sql2htmldoc.py

The sql2htmldoc.py script reads both the core birchdocdb database and your birchdocdb.local database, and creates web documentation for the merged data. For an example of  how the documentation might be represented at a local BIRCH site, click here.

If a file is not readable from the web documentation, the most likely reason is that the file path is incorrect. This shouldn't happen, but there are occasions where symbolic links might give the wrong file path. You could quit birchdocdb.local without saving at this point, and edit the docfiles.tsv file to give the correct paths. Usually all paths can be changed in one step using a global Replace command. Alternatively, you could change each command line directly in the database using sqlitestudio. Keep in mind that changes to the database require re-running sql2htmldoc.py to take effect.

4.6.  Cleanup

Please send suggestions of comments regarding this page to frist@cc.umanitoba.ca