Tristan's blog

I've found the table that stores file information: mdas_ad_repl

MCAT can be set up to run a function when an INSERT, UPDATE or DELETE call is made on the table.

the function can be writen in a number of languages, including C and Python.

I've written a basic python test function that prints out the 'Trigger Data' dict object to a file.

Now when I do an Sput, Smv, etc the file gets appended with the data from that event.

Interestingly, when Sput is used, no matter how large the file, three events are triggered. an INSERT, which creates the entry for the new file, with a listed datasize of 0. then when the file is finished two UPDATE events are created. The first UPDATE seems to just update the file size to the correct file size. where the second UPDATE changes an attribute called 'is_dirty' to 2. The only description of this 'is_dirty' attribute is:

IS_DIRTY                      /* data has been changed in the container
					compared to other copies  */

which isn't very descriptive. I'm going to get on the SRB mailing list and fire off a question about this. But for now it looks like this could be the answer to telling when a file has finished uploading to the SRB.

Of course the issues Ian brought up still need to be addressed:

Weather the MCAT schema is stable, since changes to the database schema could break the triggers.

And what the deal is with using multiple MCAT servers. A test system might have to be set up to play with this

Why do we need to implement a trigger system in SRB when most databases (i.e. oracle and postgresql) used for the MCAT support triggers?

I'm pretty sure that file information is stored in the MCAT so we should be able to build triggers to preform operation (i.e. execute kepler workflows, send SOAP packets to an axis server which can be queried by kepler)

This should remove the need for the SRB directory poller system.

I was getting sick of having to kill and reload kepler every time i made a small change to an actor, so i decided to write a small unit testing system that didn't have to start the entire kepler environment to run.

I've successfully done this and will write a doc on the details soon.

I've been thinking about how the dart specific code is going to be rolled out when the time comes.

all the code at the moment is being compiled directly into the kepler build, and deploying our code like this would require having all the dart code inside the main kepler release.

This may cause problems with the licencing, and also all our code has to go through all the kepler channels to allow it to be included in the release.

I think it should be reletively simple to package all the dart code up into a package of it's own and link the main kepler jar files into it. this approach would make it easy to deploy our code, as well as allow other people to test the actors without them having access to kepler's CVS, it would also allow me to easily manage the code in the HPC SVN server (mixing SVN and CVS in the kepler tree has proved to be quite a pain) and there are probably another million advantages than i can't think of right now.

Changes made to kepler for cima related workflows:

CimaBean: a simple bean containing a byta array of data and a Hashtable of metadata.

CimaBeanListener: a server type actor that receives beans sent over a network socket.

Get Hashtable value: an actor to extract a value from a Hashtable given the value's key. used to extract specific metadata entries from CimaBean hashtable.

SRB Writer: created to write an array of bytes to a file in SRB.

SRB Add Metadata: modified to take a hashtable of metadata rather than an array or MetadataTokens.

A simple example workflow has also been created using these components, the only problem being a bug in ptolemy's ScalarToken and Expression actors where when converting the scalar to a string the string "0.0" is always returned.

Have been working with Mat to put together an actor to retrieve a JavaBean over a network link.

This has been a relatively simple task, the next step being to set up the transfer to use SSL. However this requires some googling, which is difficult without any internet :)

I've been playing around with the Software section on Plone and adding my Kepler actors as software packages (eg. Kepler SRB Actors). I've been pondering over what should be put up there. I've come to the conclusion that i'll just put usage documentation into Plone and leave all the implementation details to the javadocs.

If there was a simple way to embed javadocs into plone this would be fine, but it isn't very efficient to have 2 different sets of documents for the one thing.

I've begun creating some documentation by moving and cleaning up some of the documentation style blog entries. If you can think of any other topics that you would like to see documented add a comment to this entry.

it currently seems impossible to add a double quote in a StringToken. Something in the contstructor modifies the string token so that is rather than just showing ", it shows \". for now i just have to work around it and use single quotes instead.

I've written an actor which does directory polling of a SRB directory.

There is a problem with this SRB directory poller however. There is a large (around 8 second) delay in the checking for new fiels method.

I've spent some time re-structuring and java-docing the code I've written for kepler to better follow kepler's coding standard. I think it's important to note that in my java-docs i've given the reasons why I've re-written certain actors.

After figuring out how to use the vector graphics editing programs properly I re-made the actor icons for my dart actors. Now they all sport the two arrows from the dart logo, they look a lot more professional and they blend in well with the other kepler actors. Find them under https://svn.hpc.jcu.edu.au/svn/dart/kepler/lib/images/svg/

I've written a little proof of concept scenario for retreiving data from the labjack and getting it into kepler.

Since on the CIMA box the Labjack just outputs data to a file, i've simply written a small server program which reads that file and sends the data to any clients connected to the server (e.g. a telnet client or a kepler actor).

I then wrote a kepler actor to connect to this server and get the data, process it (which comtains humidity values i think) and outputs results to a graph.

I haven't been able to test this with realtime data yet, I just have my server reading a static file on my local machine. Mat wasn't willing to install java and allow me to place my server app on the CIMA box, so I'll have to wait till he has set up another box with a LabJack.

We have also come across a Java API for reading data from the LabJack, if it works really well it'll remove the need to have the server program read from a file and simply read from the LabJack itself.