Forgot your password?
Kepler
Unit testing actors
I was getting sick of having to kill and reload kepler every time i made a small change to an actor, so i decided to write a small unit testing system that didn't have to start the entire kepler environment to run.
I've successfully done this and will write a doc on the details soon.
Removing Dart specific code from main kepler build
I've been thinking about how the dart specific code is going to be rolled out when the time comes.
all the code at the moment is being compiled directly into the kepler build, and deploying our code like this would require having all the dart code inside the main kepler release.
This may cause problems with the licencing, and also all our code has to go through all the kepler channels to allow it to be included in the release.
I think it should be reletively simple to package all the dart code up into a package of it's own and link the main kepler jar files into it. this approach would make it easy to deploy our code, as well as allow other people to test the actors without them having access to kepler's CVS, it would also allow me to easily manage the code in the HPC SVN server (mixing SVN and CVS in the kepler tree has proved to be quite a pain) and there are probably another million advantages than i can't think of right now.
Cima workflows and actor changes
Changes made to kepler for cima related workflows:
CimaBean: a simple bean containing a byta array of data and a Hashtable of metadata.
CimaBeanListener: a server type actor that receives beans sent over a network socket.
Get Hashtable value: an actor to extract a value from a Hashtable given the value's key. used to extract specific metadata entries from CimaBean hashtable.
SRB Writer: created to write an array of bytes to a file in SRB.
SRB Add Metadata: modified to take a hashtable of metadata rather than an array or MetadataTokens.
A simple example workflow has also been created using these components, the only problem being a bug in ptolemy's ScalarToken and Expression actors where when converting the scalar to a string the string "0.0" is always returned.
CIMA Bean receiving actor
Have been working with Mat to put together an actor to retrieve a JavaBean over a network link.
This has been a relatively simple task, the next step being to set up the transfer to use SSL. However this requires some googling, which is difficult without any internet :)
IRC log of chat with Nandita Mangal about Matlab Actor
In the kepler group's latest meeting they did i code review of the MatlabExpression code that I had added to their CVS.
I've uploaded the presentation given by Nandita and the results of the code review here: Code Reviews
After I reviewed the document created from their code review i had the following chat with Nandita on the kepler irc channel. I thought i should copy and paste it here to keep a record of it.
note: Nandita seemed to have gone AFK after i started this rant. I'll append her comments later if she has any.
tristan: the network execution has been something i've been thinking about to. if there was some sort of matlab server then it would remove the need to start a process, and we could just send input and get output from network sockets
tristan: i've seen some things (that i've not looked into much detail yet) where someone has created a java network app that runs in matlab and controls talking to matlab over the network. Might be worth looking into. would be a fair bit of work tho, since you'd have to look into security and stuff
tristan: one thing i think we can change is the expression port. we can just change the expression parameter to a PortParameter. i'll work on that today
tristan: also, the p.waitFor() method blocks the execution until the process has ended. so there's no worries there
tristan: i'm adding a inputstream.close() call to close the inputstream
tristan: the log file used has a different name everytime the fire() method is called. i append it with a randomly generated number each time.
tristan: maybe rather than having a parameter for the switch -nodesktop, we should make it part of an interactive mode.
tristan: we'll have to come up with a use case where someone might want an interactive mode so we can properly look at what needs to be done there
MatlabExpression Actor
Matlab actor working
There may still me problems with the input and output, but it needs to go through more rigerous testing before I can figure out if there are problems.
One problem with it is that the output doesn't show the commands that were entered into matlab.
for example, where it should show:
>> 2 + 2
ans =
4
it only shows
>>
ans =
4
but this shouldn't be a problem yet, it's only needed for debugging purposes
For the moment this actor needs a working copy of Matlab installed on the local machine.
Mat has mentioned to me the existance of a networked version of matlab which allows users to talk to a matlab server over internet protocols. This may be a much better option for doing matlab stuff in kepler, and i'll look into it.
building Matlab actor
This actor will give the user the abbility to run a matlab expression (i.e. a .m file) taking input from input ports and output certain results from user specified output ports.
if a user names an input port called "aaa" then a variable will be created for use called "aaa" with the value that is received from an input token.
similarly if a user makes a port called "aaa" and there is something in the matlab expression along the lines of "aaa = 2 + 2" then the value "4" will be output on the aaa port.
It works by using the java class java.lang.Runtime to start an instance of matlab using the command line "matlab -nodisplay -nosplay" (which starts matlab without a gui). it then sets the input port variables, runs the specified expression, extracts the values for the output ports, then quits matlab.
This method is taken directly from the RExpression actor which does the same thing but using R.
The only hard part is parsing the output string from matlab to extract the values we need. currently i can read in Double's Integers and Strings. Arrays of those 3 types will be the next challenge (and should be the final one).
ptolemy StringTokens and \"
it currently seems impossible to add a double quote in a StringToken. Something in the contstructor modifies the string token so that is rather than just showing ", it shows \". for now i just have to work around it and use single quotes instead.
revised SRB actors
To fix this i've come up with a method of passing the SRB connection around to other actors without needing to link actors to a SRBConnect actor.
This is done by using a static class called SRBServerManger which stores connections to SRB servers and allows any actor that needs the SRB connection to query it and get the SRB connection.
One thing i had to keep in mind for this was the possibility of having multiple servers on a single workflow or multiple workflows that use the static class as the same time. I solved this by passing the workspace object and a SRB server id to SRBServerManger and have it store and retreive srb connections based on these values. The workspace object is availiable from any actor class. The server id is to be set by the user, and is only unique to each workflow (i.e. every seperate workflow can have a server id 0).
To avoid the added complexity of making the actors call SRBServerManger, i build a base class called SRBActor which any actors that require SRB access can extend. The SRBActor class adds a SRB server id parameter to the actor extending it, as well as a method for getting the SRBConnection, which handles all the talk with SRBServerManger.
Another actor was created called SRBServer, which has no input or output ports, but has Parameters for all the nessesary information to initiate an SRB connection and, during the pre-initialise stage of the workflow execution, sets up the connection to the SRB server through the SRBServerManger.
This method makes the workflows look a lot cleaner, and also makes creating new SRB actors very simple.
My only problem is that it removes the visible link between the SRBServer actor and the other SRB actors. However, if the SRBServer actor is thought of as a Director for other SRB actors than the need for a visible link is removed.
SRB directory polling, QA and MD proof of concept
SRB is set up with three directories: raw, data, rubbish. Raw is used for placing new data, and is the directory that the directory poller watches. when new data is found in the raw directroy, quality assurance and metadata creation is preformed and if all is successful the file is moved to the data directory and metadata is added. If the quality assurance failes the file is moved to the rubbish directory.
Invisibly linked Relations
I think a good method is to have a type set of relations which have an invisible link between them.
I've put this to the kepler-dev people and i'll try put together a proof of concept thing for this soon
SRB directory poller
I've written an actor which does directory polling of a SRB directory.
There is a problem with this SRB directory poller however. There is a large (around 8 second) delay in the checking for new fiels method.
this is caused by the method edu.sdsc.grid.io.srb.SRBFile#lastModified().FIXED: upgraded jargon from 1.4.17 to 1.4.19
following kepler coding standard
I've spent some time re-structuring and java-docing the code I've written for kepler to better follow kepler's coding standard. I think it's important to note that in my java-docs i've given the reasons why I've re-written certain actors.
Changed Dart icons for kepler actors
After figuring out how to use the vector graphics editing programs properly I re-made the actor icons for my dart actors. Now they all sport the two arrows from the dart logo, they look a lot more professional and they blend in well with the other kepler actors. Find them under https://svn.hpc.jcu.edu.au/svn/dart/kepler/lib/images/svg/
Labjack data stream in kepler
I've written a little proof of concept scenario for retreiving data from the labjack and getting it into kepler.
Since on the CIMA box the Labjack just outputs data to a file, i've simply written a small server program which reads that file and sends the data to any clients connected to the server (e.g. a telnet client or a kepler actor).
I then wrote a kepler actor to connect to this server and get the data, process it (which comtains humidity values i think) and outputs results to a graph.
I haven't been able to test this with realtime data yet, I just have my server reading a static file on my local machine. Mat wasn't willing to install java and allow me to place my server app on the CIMA box, so I'll have to wait till he has set up another box with a LabJack.
We have also come across a Java API for reading data from the LabJack, if it works really well it'll remove the need to have the server program read from a file and simply read from the LabJack itself.
subversioning kepler and CVS write access
I've started writing my kepler stuff to the HPC Subversion server
https://svn.hpc.jcu.edu.au/svn/dart/kepler/
It only contains the files i've created and modified, not the entire kepler source
I'm also gaining write access to the Kepler cvs server.
oh and my name is now on the kepler website: http://www.kepler-project.org/Wiki.jsp?page=GettingInvolved
new kepler icons
Kepler has just rolled out their new actor icon set, so I decided to create a few of my own icons for my actors rather than just using the default kepler one.
The images I created are just modified versions of the kepler ones which I used inkscape (due to the files being SVG files) to make, then i used GIMP to make the gif thumbnails which are used in the actor selection menu of kepler.
the SVG files are put in:
kepler/lib/images/svg
and the gif thumbnails in:
kepler/lib/images/actorthumbs
then the file
kepler/configs/ptolemy/configs/kepler/uiSVGIconMappingsByClass.properties
needs to be modified to use the images. I've added the following to the end of the file
....
org.dart.srb.SPut=dart-srb-file
org.dart.srb.SRBConnect=dart-srb
org.dart.srb.SRBAddMetadata=dart-srb
org.dart.BinaryFileReader=dart-file
org.dart.DirectoryPoller=dart-file
org.dart.BoolMultiplex=dart-arrow
kepler - srb - quality assurance - metadata
I've successfully build a Proof of Concept workflow in kepler which does the following:
1. finds new files in a directory
2. reads in new files
3. does a quality assurance test on the files
- if QA test successful
4. build metadata about the file
5. write file to srb
6. add metadata to file in srb
- if QA test unsuccessful
4. disguard the file
Adding new actor categories in kepler
I've added a Dart category in the actor selection tree in kepler.
this was done by adding the following to the file:
$KEPLER/configs/ptolemy/configs/kepler/ontologies/keplerActorOntology.owl
to include the following:
<owl:Class rdf:ID="Dart">This creates a Dart Actors category under the Actors category, and a SRB actors category under the Dart Actors category
<rdfs:label rdf:datatype="http://www.w3.org/2001/XMLSchema#string"
>Dart Actors</rdfs:label>
<rdfs:subClassOf>
<owl:Class rdf:about="urn:lsid:localhost:onto:1:1#Actor"/>
</rdfs:subClassOf>
</owl:Class>
<owl:Class rdf:ID="SRB">
<rdfs:subClassOf rdf:resource="urn:lsid:localhost:onto:1:1#Dart"/>
<rdfs:label rdf:datatype="http://www.w3.org/2001/XMLSchema#string"
>SRB Actors</rdfs:label>
</owl:Class>
<owl:Class rdf:ID="Dart_POC">
<rdfs:subClassOf rdf:resource="urn:lsid:localhost:onto:1:1#Dart"/>
<rdfs:label rdf:datatype="http://www.w3.org/2001/XMLSchema#string"
>Proof of Concept Actors</rdfs:label>
</owl:Class>
