where grayMatters meets Data

Where has the time gone?

Wow, has it been nearly two months since I last added a post.  My bad.  No real excuse except that there always seemed something better to do.  So, you ask what changed?  What motivated me to finally hit the keyboard.  A major event, as usual.  As you know, I am a big fan of the HP TC1100 tablet/slate computer.  Well, last Thursday, I was merrily using my tablet when BAM, the hard drive went.  So, I had to go get a new one from Seagate.  I got a 100GB drive that spins at 7200 rpm.  Now, it's time to reload the O/S, the programs, and the backed up data.  And that's where it gets interesting.  You might think that your backups are working and that they are complete, but you don't really know until you HAVE to use them.
So, based on a previous post, I was able to reload the various programs that I have found and like to use in the tablet.  But, I found some holes in my backup routine.  The first was my wiki backup.  I use TiddlyWiki to track what I do professionaly each day and a personal copy of how the home computer network is set up.  Turns out that these files are a week out of date.  Not a great loss, but annoying.  Okay, that can be fixed.  Now, what else did I lose?
And this was a biggie.  I have been using OneNotes from Microsoft Office to capture all of my notes from the various meetings I have gone to.  It was finally starting to get some bulk and was turning into a good on-line reference as I thought it would.  Turns out I never backed up this file.  Heck, I don't even know where the file is.  Ouch.
So, I am blogging about it.  Design your backups, run them, test them and then review them every couple of months.  I know I will from now on.  This is just too painful.

It's been a long time...

Hey, don't really have an excuse of why I have not posted here for awhile other than time flies when your having fun.  In the past 3 weeks here are some of the things that have happened:

• Got the new tires on the Mustang.  I know, where are the pictures.  They are coming after I wax the car which means we need some good weather here.  And it's a good thing that I did not post about my experience with getting the new rims/wheels on.
• Bought and then waited and waited for my second TC1100 tablet.  A series of circumstances caused it to show up 3 weeks after I bought it.  It's a sweet unit but 3 weeks?
• Discovered Craigslist for Calgary.  Should have bought a PC last weekend off of it because it was such a sweet deal but I have it bookmarked and it's quickly becoming one of my morning routines
• Bought a PS2 off of eBay.  Neat thing about it was the seller lives in Calgary.  That's pretty cool but when I found out that she was only 10 blocks away, that really made this a small world.  Needless to say, I did not pay for shipping.
• Made a quick run up to Sylvan to drop some stuff off.  Heading up there today for the May Long weekend; as usual the weather is less than perfect but who cares.
• Am having a devil of a time getting a consistent wireless signal at my house.  I have tried 3 different wireless routers (2 G's and 1 N) and none broadcast it continuously.  I have not been able to detect a pattern around telephone usage, microwave, TV, etc as to what the cause is.  My next option is to move it to a different room and try it there.
• As far as I am concerned, moving to a digital notebook/pad of paper has been a success.  I will talk about how it has worked for me in another posting.  But it has, hence me buying another tablet.

Okay, I guess I will have to be better at this posting thing because there are a lot of things I want to talk about and I just have to make it a habit.  And I will.  Tomorrow.

Inventory, Part II

I forgot some of the applications that I wrote specifically for the OpenWorks data model and thought I should include them as well:

• Add On
  • Used to create the .prm files for DSL or the Data Store Loader.  I implemented DSL at several companies for load multiple projects and creating these various script files was a editing pain.  This was a script generating application, much like the code generation add-ons that you see in many IDEs.
  • Of course, I added on several other capabilities, such as:
    • A tab to check how populated the tables were
    • A tab to check if a well was created in two or more different projects
    • A tab to check for some basic business rules like
      • Picks > TD
      • Curves > TD
    • A tab to present the different sources in a project and how many wells they were the author of
    • A tab to see how often a well was reloaded into a project
  • all with the ability to dump to Excel
• dLogger
  • Shows all of the well lists in a project
  • Allows subordinate lists to be tagged to master
• dLogViewer
  • Designed to review well logs reference values loaded into the database and create .calm files for the Log List Loader Utility of which I can't remember the exact name of right now.
  • In my work with OpenWorks databases, the well log reference table comes pre-populated with a swac of log curve names.  Ones to cover all regions of the world.  Well, if your OpenWorks project only deals with Western Canada, then all you want to see is what is available from the data vendors in Western Canada.  To delete these from the table via manual editing is very time consuming but you can create a .calm file that has all of the necessary instructions.  Think of this file as a XML type file and you get the idea of what it does and the structure of it.
  • Log curves are grouped together by function and when a geologist wants to see a certain set of logs, they group them together and then just call up the group.  Sort of like log metadata.  This app allowed these groups to be created quickly because it should the end-user the number of times a log curve was present.  No point including a log curve if it is loaded 10 times in a 50,000 well project.
  • Has an analysis page that allows Excel files to be created and reviewed away from it by the geologist or project owner about what well log curves are really loaded
• dListViewer
  • Why stop at just well lists?  This program didn't. It allowed the user to see what was in all lists (Well, Seismic, etc) in the project, create an Excel dump for further analyst and even email it
• It should be noted that all of these applications were written to be run in a project that had a role like OW_ADMIN assigned to it.  There were a couple of non-OpenWorks tables that had to be created to hold the various meta-data needed to look into the client projects.

Inventory

I thought that I would create an inventory of the Delphi written Windows based application that I have written.  Most of them, ok all of them connect to Oracle and are focused on data Management, PPDM and the OpenWorks database.  Here goes:

• Compliance
  • Tested a data model against the released PPDM standard model of either v3.6 or v3.7
  • Went into a lot of detail around constraints, indexes, column data types which were all necessary for this type of comparison.
• ConMan
  • Short for Constraint Manager
  • Designed to help work with constraints in your data model.  Had a Outlook type of GUI.  In fact, most of what I have written has this.  It just seems like a good implementation for me.
  • Can review constraints assigned to a table, what the parent table was, review the type of constraint and columns.
  • Constraints could be enabled or disabled.
  • Has a reference editor tab.  Because the PPDM data model has a ton of reference tables and that most of them are exactly the same structure, a single simple interface is all that is needed.
  • Had the ability to edit OpenWorks reference tables as well.  They all have the same data structure (like the PPDM) so the editor was tweaked to work against them.  NOT that you are supposed to edit these reference tables anywhere but in an approved manner.
• ContentViewer
  • Designed to compliant a data analyst.  When reviewing a data model for content, there are a lot of similar queries that are run like:
    • select distinct(count(column)),column from table group by column &
    • select table_name,column_name from user_tab_columns where column_name like %
  • And so to avoid typing and typing and re-typing, I created this application.  There are pop-up windows that allow the analyst to drill into data and re-drill and drill.
  • By clicking on a table name, a pop-up window was created that contained the table definition.  By selecting any column, you could see the distinct values for that column OR you could see the other tables in the schema that contained that column.  From here you can re-drill.
  • If you clicked on a row from the distinct values, you were presented with another window that contained all of the rows from the table where that value was equal to the column value selected.
  • You could dump to Excel spreadsheets for reporting, etc.
  • This has to be one of my favorite programs and one that I will update, when I get Delphi 2007.  It's on my tablet, but that is another entry because it's working just great.
• dataMagic
  • This combined ContentViewer with the ability to look at Reference values or Data values.  It also was my first attempt at capturing business processes.
  • Tied in some of the PPDM's meta tables for better grouping
  • Used colour to denote if a reference value was used any where in tables that had it's Primary key as a foreign key.
• dM Tracker
  • This was to be a PPDM based application, but I became frustrated at the lack of multiple model support in the PPDM, so I ended up modifying the tables to suit this application.  Had a real good Outlook front end with approx 30 screens.  You could see the following:
    • Data model extensions (borrowed from Compliance a wee bit)
    • Maintain PPDM Groups (borrowed from Load of the Rings)
    • Look for data column within the database via dynamic SQL
    • Look at data / schema security (borrowed from ARM)
    • Review DDL changes.  This required that the DDL table and triggers be set up.
    • Had a reference editor (Does it seem that every app of mine has this?)
  • Export most of these queries to Excel
  • Had a .ini file for setup
  • Could work with multiple sources
• RCV (Reference Code Viewer)
  • simple Reference view/editor
  • released as freeware at the PPDM Fall Conference in 2003
  • can work against both OpenWorks and PPDM databases
  • creates Excel files as output
• Spot
  • Another attempt at allowing a data analyst to review the contents of an Oracle schema.  Designed to work against PPDM or OpenWorks databases.
  • Got the name from 'See Spot Run.
  • Can walk the referential tree that each model is based on.
  • Created a comprehensive help using HelpNDoc
  • I think this deserves to be on my tablet.
• ARM
  • Application Role Manager
  • I really wanted this to succeed as a PPDM application.  Maybe it tried to do too much or maybe it was not pushed / sold properly by dMCI or probably both.
  • Had a Outlook GUI with the ability to use the PPDM data model (3.7) to store information about
    • Applications
    • Roles
    • Users
    • Privileges
  • All of these are connected and again laid the basis for my fascination around processes, business rules and how they are implemented in a corporation.
  • Mandatory Reference Editor
  • Export to Excel
  • Had the ability to grant roles (if underlaying Oracle schema had the right) and sync it up to the database tables
  • This was the first time that I tried to store information about how a database looked today because it's different than what it was yesterday and it's going to be different than what it is going to be tomorrow.
• Load of the Rings
  • Written to help explain how the PPDM data model is connected and works.  If you think of the data model as a series of concentric rings with dependencies built on each layer, that gives you the general concept.  It uses the PPDM meta tables and is really useful for beginner programmers/developers because it SHOWS the dependencies around tables.
• Data Looker
  • Version of ContentViewer, basically.
  • Lot of the same pop-ups with exporting to Excel and the ability to drill into tables/data
  • Has a comprehensive help written
  • Certainly parts of this is available in TOAD but I believe that it can be a real useful tool when a defined data analysis process is done.  Hmmm, that gets me thinking....
  • This has a Data Analysis screen where SQL could be written, run and then exported to Excel.  No need to go to TOAD to do this!
• Chains
  • My current project.
  • Has a lot of the learnings from the above apps but with a serious business process bent, at least I hope to have it in there.

Well, that's it and I did not realize how many programs I have written, re-written and generally had a thought that they would be useful to me.  And if I was able to use them, then why wouldn't other O&G data analysts and architects.

But that is another entry.

Getting Organized

• Found some backup software that looks like it will work.  Called SyncBack and I hope that it can replace SyncToy, RoboGUI, Windows scripts files and what ever other stuff I have accumulated over the years.
• Survived the weekend trip to Saskatoon.  The team did okay on the ice and great off.  That's what it's all about.  Having fun and having something to do/focus on.
• Going to start to work on the data/work flow as I have been thinking about.  The tablet is working out just fine.
• Found a wireless 'radar' detector.  Used it today and it was okay.  I think I need to walk around the downtown to see what I can find.

Keeping on top of things

• I posted earlier about the ebooks portal and how to use an RSS Reader for search engine news/blogs feeds.  Well, I have signed up for the feed from Ebooks into my RSS reader and wow.  Automatic notification of new books in the portal.  Updated twice a day.  Now, I don't have to browse through the volumes of digitized books, thinking I missed something or make a note to check this site every couple of days to see what happened.  Great us of Pull technology.

• Up in Saskatoon for a girls hockey tournament.  Last of the year, I hope.  All in all something for parent and child to do... like running through a hotel at all hours making sure the girls are in their rooms, oh wait, they did that when they were younger...  But do they ever really change?
  The game last night was one they let get away.  Had a ton of pressure during the first period and what they goalie did not stop, the posts did.  Got a tie out of it, so it was not a complete loss.  Two games up today.  That will make them sleep tonight!

• Cleanup up my PC(s) with the extra hurry up and wait time.  Looking at several s/w packages for folder synchronization.  Might have something that works, at least if I can make it a habit.
  
• Been working with Intervera on a 'Trusted Well Master' seminar.  This will be held later in April in Calgary.  Promises to be interesting and helpful to my peers as I see/hear a lot of the companies going through the same problems that I am experiencing.

Only on a Mac, you say

Found this real interesting s/w called NoteMind.  Too bad it is only available on a Mac as I would love to try out something like this.  I find that in my chosen field; Oil and Gas data Management, that pictures show the situation a lot better than a word document or excel spreadsheet.  That's one of the reasons that I like MindManager.  It allows me to zoom around at 50,000 feet, go down into the weeds at 500 and then do some more recon at 5,000.  I guess I might be a bit of a random abstract thinker so this really works for me.
Along the same lines, I found a Delphi component that might allow some type of this control.  I will try it out over the next couple of days and let you all know how it works.

Google.
Yahoo.
You have heard of them both.
Have you heard of Gahooyoogle?

It's a website that allows the user to search using both search engines at the same time.  The result set is displayed with each set of answers taking up 1/2 of your browser.  Need idea.  I have been trying it out for a couple of days and so far, one search engine has been giving me better answers.  It's nice to see what the other one does and there might be a gem or two in it, but ....

Day 1 of the new workflow

So today was the first day that I was truly going to try using my TC1100 as a portable notebook otherwise known as a tablet.  It started with the bus ride to work.  I sat down, fired up the tablet and reviewed a book that I downloaded from ebooksportal as well as tried to understand how to work M/S OneNote.  By the time my 1/2 hour was done, I was pumped.  What a great use of time.  So, it was on to conquer the day.

I had a 1 1/2 hour meeting in the morning where I captured notes in the tablet.  This was great.  I can put meetings in different folders, days in different books, clients in different notes.  All good.  The only downer was the comments from the various jealousy cats about how their paper notebooks don't need charging; if they get wet, theres can go in the dryer, etc.  All in good fun but all very good points.  After the meeting, I was down to approx 60% battery life, so I plugged it in over lunch to charge.

The afternoon started off with another meeting and the battery was fully charged and ready to go.  Again, I had to explain what the tablet was to the other attendees, but these people were much kinder to me...  Must have been the sugar high from cheap lunch swill.  The tablet worked as anticipated.  The best part was after the meeting when some people asked me about it and could not believe that the TC1100 is over 3 years old.  An even better thing then happened.  I stopped in a co-workers office to discuss a problem that he had and the answer was on the tablet.  So rather than having to go back to my desk, look it up on my machine and then phone/email him, we were able to carry on our discussion of the problem and take it to the next level.

So all in all, I consider it a pretty good 1st day with my TC1100.  I just hope that I don't drop the darn thing and that I get a decent backup plan working.  But that's another post.

Got it back

Well it took most of the evening, but I finally have my Table PC Input Panel back with the keypad.  Turns out the problem was fixed by successfully re-installing .Net Framework 1.1 and it's various service packs.  And to get that running took a bunch of installs/reboots and finger-crossing because there were a couple of times where I felt I was going backwards in solving the problem.
And now that I know where the M/S certificate of authenticity is, I can re-build the tablet when I want to.  I think that I am finally ready to tackle my objective for buying the tablet; re-working my workflow.

Getting a new machine up and running

Well, it has been a couple of days since I got my HP TC1100 from eBay.  Took it downtown and was very impressed with the battery life.  I was able to sit and take notes for just over 3 hrs worth of meetings and still had ~20% of battery life left.  Don't know how long it will take to fully recharge, but if it could do it over lunch, that would be sweet.

Now the big job, installing software and setting it up the way I want it to run.  And installing the many many XP updates took the better part of an evening.  Installing the individual software packages was not much better.  Rather than list out the s/w, I am showing a table that lists out the s/w and where I found it.

As you can tell from this list, most of the s/w is open source or freeware.  Running the number of PC's that we have in our household can get expensive.  And I find that open source just works.

As I stated before I am attempting to my tablet as a new way of working.  That means I need access to mainstream documents like word, excel & powerpoint (hence the MS Office 2007), Oracle (hence the OracleXE 10g instance & SQLDeveloper), PDF (FoxIt Reader) and all of the system software like CCleaner, JTDeFrag, etc.

I am just about set up and ready to do some real work except I lost my keypad for the pen.  It's blank, it was there and I lost it/overwrote it, whatever in the last couple of days.  Not a big surprise, when you consider all of the installing and removing I have been doing.

And let's not forget about backups.  Stuff on a computer is only as good as the copy that is accessible.  Lose a drive and you don't have a backup, your banging your head against the wall alot.  So, on most if not all machines, I have some rudimentary backup and maintenance scripts.   Just running the daily one now and will test out the weekly one as well.  Weekly is a bit more complicated because Oracle must be shutdown and then re-started.  Will set the jobs to run in the wee hours of the morning.  Always nice to have the machine work when I don't have to.