where grayMatters meets Data

Finally, at it again

I finally have freed up some time to work on the Case procedures that I wrote for the PPDM years ago.  Lots of cobwebs there but I can see why the PPDM Association wants it done.  Saves lots and lots of time.  It will be interesting working on it from the lake.  Wireless to the internet and then VNC'ng to their network.  It has worked for other contracts, so it should work for this one.  NOT that this is a contract...  Damn, I'm good to some people.

Also found this:  ASuite   It should allow me to 'bake my own' since it's written in Delphi.  I have some new ideas around how to do the work I do, especially since my hard drive failure.  This might be one of them.

And let's not forget about some playtime.  More on that later.

Where has the time gone?

Wow, has it been nearly two months since I last added a post.  My bad.  No real excuse except that there always seemed something better to do.  So, you ask what changed?  What motivated me to finally hit the keyboard.  A major event, as usual.  As you know, I am a big fan of the HP TC1100 tablet/slate computer.  Well, last Thursday, I was merrily using my tablet when BAM, the hard drive went.  So, I had to go get a new one from Seagate.  I got a 100GB drive that spins at 7200 rpm.  Now, it's time to reload the O/S, the programs, and the backed up data.  And that's where it gets interesting.  You might think that your backups are working and that they are complete, but you don't really know until you HAVE to use them.
So, based on a previous post, I was able to reload the various programs that I have found and like to use in the tablet.  But, I found some holes in my backup routine.  The first was my wiki backup.  I use TiddlyWiki to track what I do professionaly each day and a personal copy of how the home computer network is set up.  Turns out that these files are a week out of date.  Not a great loss, but annoying.  Okay, that can be fixed.  Now, what else did I lose?
And this was a biggie.  I have been using OneNotes from Microsoft Office to capture all of my notes from the various meetings I have gone to.  It was finally starting to get some bulk and was turning into a good on-line reference as I thought it would.  Turns out I never backed up this file.  Heck, I don't even know where the file is.  Ouch.
So, I am blogging about it.  Design your backups, run them, test them and then review them every couple of months.  I know I will from now on.  This is just too painful.

If it's the May long weekend, there must be snow on the ground

Yup,
There is a bunch of white stuff on the ground here in Calgary.  Can't have a May Long without it!  So, it's time to finish off the indoor projects and whatnot so when the real spring finally shows up, we can enjoy it. 

Working with tablet and trying to change the way that I work has proven to be an interesting journey.  Change is always hard as one can find so many ways of defeating it or not embracing it totally.  I have dropped the use of the paper notebook entirely.  I find that storing what I do, who I do it for and how long it takes can be captured in the TiddlyWiki that I have set up.  I have used the tablet to go to meetings and take notes in OneNote.  I found a nice add-on that shows the date that the meetings notes were taken.  That's pretty neat as I can now flip through a calendar to see when meetings where to put it into the right context.  This weekend I decided to convert all of my written notes to text.  I found that it can be done but an extra screen is needed so hopping back and forth on a single, crowded screen need not happen.  I also found that if I am going to do this, I would have to do it every night or at least once a week, so I have the meeting information fresh in my mind.  A lot of it seems to be re-work, if I capture the information correctly on my wiki.  But, to many notes can never be a bad thing.

Speaking of too many or in this case, not enough notes was the problem up at our place in Sylvan.  We are in a condo park there and for the last couple of years, we have had new grounds managers.  The one that we had last year did a very bad job of winterizing and the new people (outside company rather than a park resident) are having a very difficult time understanding how things are set up.  At the AGM yesterday, that got me thinking that helping them create an operating manual would be a good thing for them, me and the park.  So I volunteered to help out.  Now to find some s/w like an open source CAD program or xl based data capture system?  I really don't know what but something has to be better than nothing.  This should prove interesting as data capture like this is exactly what I am faced with each day at work.

It's been a long time...

Hey, don't really have an excuse of why I have not posted here for awhile other than time flies when your having fun.  In the past 3 weeks here are some of the things that have happened:

• Got the new tires on the Mustang.  I know, where are the pictures.  They are coming after I wax the car which means we need some good weather here.  And it's a good thing that I did not post about my experience with getting the new rims/wheels on.
• Bought and then waited and waited for my second TC1100 tablet.  A series of circumstances caused it to show up 3 weeks after I bought it.  It's a sweet unit but 3 weeks?
• Discovered Craigslist for Calgary.  Should have bought a PC last weekend off of it because it was such a sweet deal but I have it bookmarked and it's quickly becoming one of my morning routines
• Bought a PS2 off of eBay.  Neat thing about it was the seller lives in Calgary.  That's pretty cool but when I found out that she was only 10 blocks away, that really made this a small world.  Needless to say, I did not pay for shipping.
• Made a quick run up to Sylvan to drop some stuff off.  Heading up there today for the May Long weekend; as usual the weather is less than perfect but who cares.
• Am having a devil of a time getting a consistent wireless signal at my house.  I have tried 3 different wireless routers (2 G's and 1 N) and none broadcast it continuously.  I have not been able to detect a pattern around telephone usage, microwave, TV, etc as to what the cause is.  My next option is to move it to a different room and try it there.
• As far as I am concerned, moving to a digital notebook/pad of paper has been a success.  I will talk about how it has worked for me in another posting.  But it has, hence me buying another tablet.

Okay, I guess I will have to be better at this posting thing because there are a lot of things I want to talk about and I just have to make it a habit.  And I will.  Tomorrow.

Group Therapy

That's the best words I heard on how to describe our session yesterday.  It was about 'Achieving a Trusted Well' and had a pretty good representation of people in Calgary that are working on data Management problems around a well.  I don't really have a lot of time to review what was said, but I did come away with two things:

1. Projects succeed when they start small, accomplish what they said they were going to do and then continue on with further successes as laid out in their original scope/plan
2. If at first you don't succeed, try try again.  I was not happy about how I talked with my part of the presentation but I know how to improve it.  I will and I will talk again.

One other thing happened yesterday as well.  I think I finally found my light weight screen capture tool.  It's called Snippy and while I have not used it very much, I think that installing it on my desktop(s) will give me the ability to capture what I need, when I need it.

Coming up, Round 2

Well, it's over.  The first round of the playoffs in this year's quest for the Stanley Cup.  I went to the double overtime game last night here in Calgary and it just showed who the best player for the home team was; their goalie.  He is due for a big raise after next year.  No Edmonton, but hey Vancouver won, so that's alright but what's with them playing again on Wednesday?  They should have had at least one more day off.  That's the Head Office of the NHL for you, never giving Canadian teams a break.

I went and bought another TC1100.  This ones for my youngest daughter.  This time I got a docking station for around the same price as before.  I really like my little guy and I think she will to too.  Time will tell.

We have our big presentation tomorrow around what a 'trusted well' is.  I am looking forward to it as I don't really know what to expect.  I am also looking forward to waxing my 'stang tomorrow night, so there you go.  The wheels should show up next week AND that ought to be fun.

It's Back

What a great weekend!
Okay, the weather was so-so and
the wedding paled in comparison to ...

Driving the Mustang home.

I'd forgotten what it does when you decide to pass someone.  You pass them right NOW.  Not in a few seconds or anything else.  It just goes.  Even went through a speed trap (yes, I was doing the speed limit).  So, now I have to start to clean it up, inside and out.  Get rid of those cat tracks that all over it (it was stored in my parent's farm machinery shed) and get the windshield replaced.  All good stuff.  As soon as I get the new tires on it, I will post some pictures.

My brother finally got married.  It would have been nice to see my thirty or so cousins but did not happen.  I had fun being the MC and even managed not to talk about several obvious items...  All in all, it was good, even freezing in the small church on a chilly April morning where the heater did not work.  I found that other people were pre-empting me with stories, so I had to do some fast thinking and storytelling.  Had a good laugh with my cell phone (to the tune of 'another bites the dust') call from an old girlfriend.

And to top the perfect weekend, I was watching the Flames lose this afternoon with the tablet on my lap on the easy chair.  Does it get any better than that?  Of course not.  That's what wireless technology is for.

It's spring already...

• Will it ever stop dropping the white stuff here in Calgary?  I mean, how many sunny days have we had since the first day of spring?  Two, maybe three?  Enough already, besides I go up to Edmonton this weekend to pick up the Mustang!  Can't wait and can't have snow on the ground, can we.  Really not looking forward to driving around all of the potholes and having all that road gravel fly around.  Oh well, it will get cleaned up by say, June.
• Ordered some tires for my rims, finally.  Got them at the same place I got the rims.  I looked around last weekend and decided that they were good enough.  Besides paying all that money for tires that have basically 1/2 the normal life expectancy just goes against my grain.
• I took the bus downtown earlier this week and spent the whole time on the tablet.  The time just flew by.  I was reading some articles and then taking notes in Mind Manager.  What a great workflow.  I also did that last night at the arena and again had a blast doing it.  It's great to carry around a stack of books & articles and then read what ever strikes my fancy.  I have to start to download some SF from here but the only thing I would like to have is a bookmark in these different files.  Maybe you can do that in the pdf reader I use but I have not found out how.
• Went to a meeting yesterday with my tablet and did not use it.  Kinda hard when you lead the meeting, so I had to use the normal big screen for everyone to see.  I used Mind Manager to capture the notes and work the meeting.  Again, a very interactive session with all thoughts captured electronically and ready for distribution with minimal editing effort from me.

Inventory, Part II

I forgot some of the applications that I wrote specifically for the OpenWorks data model and thought I should include them as well:

• Add On
  • Used to create the .prm files for DSL or the Data Store Loader.  I implemented DSL at several companies for load multiple projects and creating these various script files was a editing pain.  This was a script generating application, much like the code generation add-ons that you see in many IDEs.
  • Of course, I added on several other capabilities, such as:
    • A tab to check how populated the tables were
    • A tab to check if a well was created in two or more different projects
    • A tab to check for some basic business rules like
      • Picks > TD
      • Curves > TD
    • A tab to present the different sources in a project and how many wells they were the author of
    • A tab to see how often a well was reloaded into a project
  • all with the ability to dump to Excel
• dLogger
  • Shows all of the well lists in a project
  • Allows subordinate lists to be tagged to master
• dLogViewer
  • Designed to review well logs reference values loaded into the database and create .calm files for the Log List Loader Utility of which I can't remember the exact name of right now.
  • In my work with OpenWorks databases, the well log reference table comes pre-populated with a swac of log curve names.  Ones to cover all regions of the world.  Well, if your OpenWorks project only deals with Western Canada, then all you want to see is what is available from the data vendors in Western Canada.  To delete these from the table via manual editing is very time consuming but you can create a .calm file that has all of the necessary instructions.  Think of this file as a XML type file and you get the idea of what it does and the structure of it.
  • Log curves are grouped together by function and when a geologist wants to see a certain set of logs, they group them together and then just call up the group.  Sort of like log metadata.  This app allowed these groups to be created quickly because it should the end-user the number of times a log curve was present.  No point including a log curve if it is loaded 10 times in a 50,000 well project.
  • Has an analysis page that allows Excel files to be created and reviewed away from it by the geologist or project owner about what well log curves are really loaded
• dListViewer
  • Why stop at just well lists?  This program didn't. It allowed the user to see what was in all lists (Well, Seismic, etc) in the project, create an Excel dump for further analyst and even email it
• It should be noted that all of these applications were written to be run in a project that had a role like OW_ADMIN assigned to it.  There were a couple of non-OpenWorks tables that had to be created to hold the various meta-data needed to look into the client projects.

Inventory

I thought that I would create an inventory of the Delphi written Windows based application that I have written.  Most of them, ok all of them connect to Oracle and are focused on data Management, PPDM and the OpenWorks database.  Here goes:

• Compliance
  • Tested a data model against the released PPDM standard model of either v3.6 or v3.7
  • Went into a lot of detail around constraints, indexes, column data types which were all necessary for this type of comparison.
• ConMan
  • Short for Constraint Manager
  • Designed to help work with constraints in your data model.  Had a Outlook type of GUI.  In fact, most of what I have written has this.  It just seems like a good implementation for me.
  • Can review constraints assigned to a table, what the parent table was, review the type of constraint and columns.
  • Constraints could be enabled or disabled.
  • Has a reference editor tab.  Because the PPDM data model has a ton of reference tables and that most of them are exactly the same structure, a single simple interface is all that is needed.
  • Had the ability to edit OpenWorks reference tables as well.  They all have the same data structure (like the PPDM) so the editor was tweaked to work against them.  NOT that you are supposed to edit these reference tables anywhere but in an approved manner.
• ContentViewer
  • Designed to compliant a data analyst.  When reviewing a data model for content, there are a lot of similar queries that are run like:
    • select distinct(count(column)),column from table group by column &
    • select table_name,column_name from user_tab_columns where column_name like %
  • And so to avoid typing and typing and re-typing, I created this application.  There are pop-up windows that allow the analyst to drill into data and re-drill and drill.
  • By clicking on a table name, a pop-up window was created that contained the table definition.  By selecting any column, you could see the distinct values for that column OR you could see the other tables in the schema that contained that column.  From here you can re-drill.
  • If you clicked on a row from the distinct values, you were presented with another window that contained all of the rows from the table where that value was equal to the column value selected.
  • You could dump to Excel spreadsheets for reporting, etc.
  • This has to be one of my favorite programs and one that I will update, when I get Delphi 2007.  It's on my tablet, but that is another entry because it's working just great.
• dataMagic
  • This combined ContentViewer with the ability to look at Reference values or Data values.  It also was my first attempt at capturing business processes.
  • Tied in some of the PPDM's meta tables for better grouping
  • Used colour to denote if a reference value was used any where in tables that had it's Primary key as a foreign key.
• dM Tracker
  • This was to be a PPDM based application, but I became frustrated at the lack of multiple model support in the PPDM, so I ended up modifying the tables to suit this application.  Had a real good Outlook front end with approx 30 screens.  You could see the following:
    • Data model extensions (borrowed from Compliance a wee bit)
    • Maintain PPDM Groups (borrowed from Load of the Rings)
    • Look for data column within the database via dynamic SQL
    • Look at data / schema security (borrowed from ARM)
    • Review DDL changes.  This required that the DDL table and triggers be set up.
    • Had a reference editor (Does it seem that every app of mine has this?)
  • Export most of these queries to Excel
  • Had a .ini file for setup
  • Could work with multiple sources
• RCV (Reference Code Viewer)
  • simple Reference view/editor
  • released as freeware at the PPDM Fall Conference in 2003
  • can work against both OpenWorks and PPDM databases
  • creates Excel files as output
• Spot
  • Another attempt at allowing a data analyst to review the contents of an Oracle schema.  Designed to work against PPDM or OpenWorks databases.
  • Got the name from 'See Spot Run.
  • Can walk the referential tree that each model is based on.
  • Created a comprehensive help using HelpNDoc
  • I think this deserves to be on my tablet.
• ARM
  • Application Role Manager
  • I really wanted this to succeed as a PPDM application.  Maybe it tried to do too much or maybe it was not pushed / sold properly by dMCI or probably both.
  • Had a Outlook GUI with the ability to use the PPDM data model (3.7) to store information about
    • Applications
    • Roles
    • Users
    • Privileges
  • All of these are connected and again laid the basis for my fascination around processes, business rules and how they are implemented in a corporation.
  • Mandatory Reference Editor
  • Export to Excel
  • Had the ability to grant roles (if underlaying Oracle schema had the right) and sync it up to the database tables
  • This was the first time that I tried to store information about how a database looked today because it's different than what it was yesterday and it's going to be different than what it is going to be tomorrow.
• Load of the Rings
  • Written to help explain how the PPDM data model is connected and works.  If you think of the data model as a series of concentric rings with dependencies built on each layer, that gives you the general concept.  It uses the PPDM meta tables and is really useful for beginner programmers/developers because it SHOWS the dependencies around tables.
• Data Looker
  • Version of ContentViewer, basically.
  • Lot of the same pop-ups with exporting to Excel and the ability to drill into tables/data
  • Has a comprehensive help written
  • Certainly parts of this is available in TOAD but I believe that it can be a real useful tool when a defined data analysis process is done.  Hmmm, that gets me thinking....
  • This has a Data Analysis screen where SQL could be written, run and then exported to Excel.  No need to go to TOAD to do this!
• Chains
  • My current project.
  • Has a lot of the learnings from the above apps but with a serious business process bent, at least I hope to have it in there.

Well, that's it and I did not realize how many programs I have written, re-written and generally had a thought that they would be useful to me.  And if I was able to use them, then why wouldn't other O&G data analysts and architects.

But that is another entry.