Well, it was hectic week. I went back to work again (on a cool project) and spent a lot of personal time on SaviCellsPro. Specifically on a new utility that I think is pretty darn exciting.
In the past, when you used SCP, you needed to write the XML for the product. Now, the XML is well-documented and I have provided help files and samples to make it easy but, let's face it, a lot of people are uncomfortable with XML.
A suggestion was made by a friend of mine at SAS to have a utility that would take an existing Excel spreadsheet and automatically generate the beginning XML so it doesn't have to be done from scratch. Well, I scratched my head and few times and decided, over the Christmas holiday, that this might be doable. Well, it was. SCP now has an XML creator that can take an existing workbook and describe it in XML. A SAS programmer can then use that as a basis and incorporate their data into the SCP XML. It makes it much easier to get started.
The utility supports cell values, styles (fonts, font sizes, borders, italics, bolds, etc.), formulas, graphs, images, print settings, and more. I will continue to add to it as demand is shown.
Expect a rollout next week on SAS-L, sasCommunity, and probably this blog. I have some testers looking at it first but I think this really rounds out SCP nicely. I will probably add in OLAP support and more robust graphing but for now, it is a cool utility that makes it easy to generate native Excel, PDF, and HTML files of your SAS data (or any other data for that matter). Since it uses XML, it can be run on any computer that produces text.
Look for my presentation at SGF on SaviCellsPro and how to use it. It is on Tues at 1:30.
Friday, January 15, 2010
Monday, January 04, 2010
.NET Coders and the sas7bdat dll
I have been spending the last week working on how the interface to the sas dataset can be used by a .NET developer. It is of particular concern because once the initial dll is released, it will be hard to change it. Getting the interface 'mostly' correct is important.
After a lot of thought, here is what I have so far:
//Initiate logging
Savian.SaviDataSet.Logging.StartLog(@"c:\temp\SaviDataSetErrors.log");
//Create SAS dataset object
SasDataSet ds = new SasDataSet();
//Add 4 variables
ds.AddVariable("AA1", "Var1");
ds.AddVariable("AA2", "Var2");
ds.AddVariable("AA3", "Var3", 10, SasVariableType.Character,"$5.","$7.");
ds.AddVariable("AA4", "Var4", 10, SasVariableType.Character);
//Add observations
//Bulk insertion
for (int i = 0; i < 1000; i++)
{
object[] values = new object[] {1, 2, "Test_" + (i + 1).ToString("00#"), "Test2", "Test3", "Test4", "Test5"};
ds.AddObservation(i, values );
}
//Modify observation - Similar to a .NET dataset
ds.Observations[0]["AA1"].Value = "Test";
//Write dataset
ds.WriteDataSet("TEMP", @"c:\temp\test.sas7bdat");
//Stop logging
Savian.SaviDataSet.Logging.CloseLog();
The area that was a challenge was adding observations since they are really an array across existing variable metadata.
The other area I am focused on is keeping the wording the same as how SAS would refer to things. Hence, observation instead of row and variable instead of column.
I have also been working on validation so that invalid data does not make it into the dataset. I can't prevent everything, but I am making a good faith effort to minimize it. Also, formats/informats have to be checked, name lengths, length values, dataset name, etc. all have to be verified to make sure they comply. So far, so good but more checking is underway.
While working on this, you realize how much effort has been put into the dataset by SAS over the years and how much work they have to go through to make things compliant and workable.
After a lot of thought, here is what I have so far:
//Initiate logging
Savian.SaviDataSet.Logging.StartLog(@"c:\temp\SaviDataSetErrors.log");
//Create SAS dataset object
SasDataSet ds = new SasDataSet();
//Add 4 variables
ds.AddVariable("AA1", "Var1");
ds.AddVariable("AA2", "Var2");
ds.AddVariable("AA3", "Var3", 10, SasVariableType.Character,"$5.","$7.");
ds.AddVariable("AA4", "Var4", 10, SasVariableType.Character);
//Add observations
//Bulk insertion
for (int i = 0; i < 1000; i++)
{
object[] values = new object[] {1, 2, "Test_" + (i + 1).ToString("00#"), "Test2", "Test3", "Test4", "Test5"};
ds.AddObservation(i, values );
}
//Modify observation - Similar to a .NET dataset
ds.Observations[0]["AA1"].Value = "Test";
//Write dataset
ds.WriteDataSet("TEMP", @"c:\temp\test.sas7bdat");
//Stop logging
Savian.SaviDataSet.Logging.CloseLog();
The area that was a challenge was adding observations since they are really an array across existing variable metadata.
The other area I am focused on is keeping the wording the same as how SAS would refer to things. Hence, observation instead of row and variable instead of column.
I have also been working on validation so that invalid data does not make it into the dataset. I can't prevent everything, but I am making a good faith effort to minimize it. Also, formats/informats have to be checked, name lengths, length values, dataset name, etc. all have to be verified to make sure they comply. So far, so good but more checking is underway.
While working on this, you realize how much effort has been put into the dataset by SAS over the years and how much work they have to go through to make things compliant and workable.
Monday, December 28, 2009
Progress on the sas7bdat
It is tough at times to be a consultant and get those side projects out of the way. That said, I finally had a chance to really work on the sas7bdat some more over the holidays. Not too much since my personal focus is my kids.
A really tough project interfered since the last time I played with it in September so I was looking forward to the holidays to pass another milestone.
Well, I just managed to create a 100 record dataset with 7 variables, char and numeric. My little Christmas gift to myself.
When I first started work on this project many moons back, I never expected that writing a dataset would be much more difficult than reading one. That was a major mistake. However, the writing has made me really discover a lot more about the structure and how it works plus I have had to step up my coding skills a bit to make it through.
Long road but finally happy to be here after so much time.
Happy New Year everyone.
Alan
A really tough project interfered since the last time I played with it in September so I was looking forward to the holidays to pass another milestone.
Well, I just managed to create a 100 record dataset with 7 variables, char and numeric. My little Christmas gift to myself.
When I first started work on this project many moons back, I never expected that writing a dataset would be much more difficult than reading one. That was a major mistake. However, the writing has made me really discover a lot more about the structure and how it works plus I have had to step up my coding skills a bit to make it through.
Long road but finally happy to be here after so much time.
Happy New Year everyone.
Alan
Tuesday, November 10, 2009
Calculate a SAS Observation Length
Well, I couldn't find this documented anywhere (someone correct me if I missed it) but the question for the day was how to calculate the SAS Observation Length. After some investigation, here is how it is done (pseudocode):
obsLength = Sum of all of your character lengths + Sum of all of your variable lengths;
if any numeric variables have a length of 8 then
round the obsLength to the nearest factor of 8.
Since the default for numeric variables is a length of 8, the rounding will almost always occur but it is not a guarantee.
I hope this helps others who may need to calculate this value for storage or performance reasons.
obsLength = Sum of all of your character lengths + Sum of all of your variable lengths;
if any numeric variables have a length of 8 then
round the obsLength to the nearest factor of 8.
Since the default for numeric variables is a length of 8, the rounding will almost always occur but it is not a guarantee.
I hope this helps others who may need to calculate this value for storage or performance reasons.
Thursday, August 27, 2009
What next?
As I can see that shimmer of light at the end of the tunnel I have to start thinking "what do I do with a binary SAS reader/writer?"
I originally started down this path with the goal of avoiding having to read/write SAS using XML Now that I am nearing the finish, I am trying to determine what comes after this (other than a quiet, non-working evening).
Where I am leaning right now is an Excel add-in or something in that space. I think Excel/SAS is one of the biggest areas of opportunity and an area that frankly does not get enough attention, IMO. SQL Server add-ins are another possible option so SAS can be supported in SSIS. It is early in this mulling period though so I would appreciate any thoughts people have on what they would like to see.
I originally started down this path with the goal of avoiding having to read/write SAS using XML Now that I am nearing the finish, I am trying to determine what comes after this (other than a quiet, non-working evening).
Where I am leaning right now is an Excel add-in or something in that space. I think Excel/SAS is one of the biggest areas of opportunity and an area that frankly does not get enough attention, IMO. SQL Server add-ins are another possible option so SAS can be supported in SSIS. It is early in this mulling period though so I would appreciate any thoughts people have on what they would like to see.
Sunday, July 26, 2009
Created my first SAS dataset without using SAS
Well, around 14 months ago, I started on a journey to understand the SAS dataset so I could read and write one independently. Originally, I needed the functionality for a client but it became an obsession to figure it out. Client work and personal matters interfered non-stop but I have finally written my first SAS dataset using C# and .NET.
It isn't pretty (no data is in it) but it is a dataset and that is what matters. Putting data in is minor so I am not concerned about that.
A lot of effort still remains to get it all worked out but I am happy to be at this position. After cleaning the code up and finishing it, I plan on making interoperability tools to simplify interfacing with SAS, expecially in areas where SAS needs help (i.e. SSIS, MS Office).
It is a good day but it has taken a brutal amount of hours to accomplish this feat. I would not suggest to others to spend this amount of time on this issue: it is simply mind-numbing.
[Update 07/27/2009]
I just wrote my first data into the dataset. It is only numerics right now but it is a start.
[Update 07/29/2009, 3AM]
I just created a dataset that contains text fields. I had an extraneous bug that made me think it was failing when it wasn't. Such is the life of a coder. Anyway, on to cleaning up code that I left hard-coded and making the system a bit more robust.
Let me be clear: I am tired of looking at hex code.
476F6F646E69676874
[Update 08/27/2009]
Progress continues. After I created my first dataset, the goal was to get rid of any hard-coded values and make it more dynamic. While doing that work, I hit a major logic issue and had to focus on figuring that out. I have made it through that and am now mopping up the remnants and tightening up the logic. Since the goal is to ship a .NET dll for developers, I also need to consider how a developer codes to the dll. Basically, I am in cleanup mode and testing additional variables, labels, etc. to see where things break or do not work as intended.
Stay tuned.
It isn't pretty (no data is in it) but it is a dataset and that is what matters. Putting data in is minor so I am not concerned about that.
A lot of effort still remains to get it all worked out but I am happy to be at this position. After cleaning the code up and finishing it, I plan on making interoperability tools to simplify interfacing with SAS, expecially in areas where SAS needs help (i.e. SSIS, MS Office).
It is a good day but it has taken a brutal amount of hours to accomplish this feat. I would not suggest to others to spend this amount of time on this issue: it is simply mind-numbing.
[Update 07/27/2009]
I just wrote my first data into the dataset. It is only numerics right now but it is a start.
[Update 07/29/2009, 3AM]
I just created a dataset that contains text fields. I had an extraneous bug that made me think it was failing when it wasn't. Such is the life of a coder. Anyway, on to cleaning up code that I left hard-coded and making the system a bit more robust.
Let me be clear: I am tired of looking at hex code.
476F6F646E69676874
[Update 08/27/2009]
Progress continues. After I created my first dataset, the goal was to get rid of any hard-coded values and make it more dynamic. While doing that work, I hit a major logic issue and had to focus on figuring that out. I have made it through that and am now mopping up the remnants and tightening up the logic. Since the goal is to ship a .NET dll for developers, I also need to consider how a developer codes to the dll. Basically, I am in cleanup mode and testing additional variables, labels, etc. to see where things break or do not work as intended.
Stay tuned.
Wednesday, July 15, 2009
PM Stories
These are stories I have collected over the years. i hope people enjoy.
Project Managers In Practice (PIMP…sorry, PMIP)
Our first story involves a young PM, new to the project and not very technical
Ok, so I (senior consultant guy) have a project and there are 3 changes requested by the team:
1. Change a label (minor)
2. Allow a user to save their selections made on dropdowns (these select what displays in a grid)
3. Allow a user to change a grid and submit the values to another program
That’s all fine and good. I bid 44 hours to make the changes.
Team now wants a user to not only make the grid changes (#3) but save those changes as well. So PM says change 2 & 3 are now merged. We need to save selections and grid entries.
I said, OK, slight increase in scope, 50 hours total.
PM says, wait, we merged 2 & 3 together so now we only have 2 changes not 3 so it should be a decrease in hours.
PM survival tip #1: Reduce scope by merging letters on the plan and hope no one notices.
Story 2 involves 4 experienced PMs
Project is in a major crisis. High-demanding customer, a lot of screaming, no time to deliver on time. Save team is called in to try and pull it out. Atmosphere is very tense.
Some of the project members (developer/consultants) have to work on all 4 segments of the overall project simultaneously. Each segment is given a PM. Customer wants to know what is happening all of the time so PMs demand status reports at least twice a day. For consultants working on all segments that is 8 status reports per day. When complaints are lodged, developers are told that it has to be that way and why aren’t we getting the work done.
Revolt of developers / consultants ensues after months of this activity.
PM survival tip #2: if you flood your customer with progress reports they will think you are doing your job. See tip #4 below.
Story 3 involves a fixed bid contract and some super savvy consultants
Two senior consultants figure out a clever way around a major stumbling block on a late project. They can change the process agreed upon slightly and save time, money, effort, AND bring in a late project on time. The customer also wants it done since the new process has a lot of compelling advantages.
Guess who says no? That’s right, our friendly neighborhood PM. It seems that the PM has decided that the original project, as scoped and laid out, is immutable for all time. No changes are allowed from the project plan WHATSOEVER.
PM Survival Tip #3: Minimize risk by making no changes. No changes, no new plans to build. Voila!
Story 4 involves a PM with a dangerous weapon…Outlook
A project, as usual, is in crisis. The PM decides that their main role is to send out emails…lots and lots of emails. On the day of ‘the incident’, the PM has sent close to 50 emails since 8am.
Then it happens…just before lunch. The PM sends an email saying that people are sending way too many emails.
Consultants laugh and break for lunch.
PM sends 10 more before they get back….
PM survival tip #4: Forward everything, comment on everything, make up stuff. The success of a PM is ultimately measured by the suffering of others.
Story 5 involves a PM with no sense what a PM does
So, the PM contacts the lead on a project and tells the lead that there are too many emails about the project and could the lead please summarize all of the open issues and assign priorities…
…isn’t this what a PM does?
Story 6 involves a PM…again…
The developer has been waiting on getting access to the client’s machine for almost a week. Technical issues, blah, blah, blah.
After the week is up, the PM wants to know timelines (this is good, they are doing their job). Developer says the development has been pushed by 3 days. The PM, who has been copied on every email between the client and the developer, says that they don’t understand why the project is delayed because of a lack of access…
Rule #1 in computers: it is hard to use them if you can’t access them.
Story 7 Guess who…
This story involves, SURPRISE, a PM who we will affectionately call Rum. Now Rum is a great PM because Rum thinks that EVERY piece of email and/or communication must flow through him. Rum insists that no direct communication happen between the technical people or the customer and that he will just forward on what is relevant.
I…AM…IRON MAN…DO, DO, DO DUH DO…
Story 8 Who wants it early?
So, the consultant finishes the application early (1 week) but they can’t test it because it isn’t scheduled to be deployed until the following week. They are told to wait until the project scheduled date so they meet the schedule.
Story 9 This is a loving one that I just title ‘Make the Sh*& up’
Well, the customer wants a project plan. Enter our superhero ‘The PM’. Rather than getting requirements or design done first, the PM decides ‘hey, let me have fun with little chain icons and links and just create a project plan! Oh boy!’.
So a project plan is created, with no document saying what needs to be done and no involvement from the programmers. Developers finally start working on the project and are told you must meet this date for this functionality. Programmers say WTF!?!? Where did this come from.
Well, it is in the project plan that we designed w/o any involvement from the people who do the work…
PM Survival Tip #5: When in doubt, just SWAG the dates and blame the developers.
Project Managers In Practice (PIMP…sorry, PMIP)
Our first story involves a young PM, new to the project and not very technical
Ok, so I (senior consultant guy) have a project and there are 3 changes requested by the team:
1. Change a label (minor)
2. Allow a user to save their selections made on dropdowns (these select what displays in a grid)
3. Allow a user to change a grid and submit the values to another program
That’s all fine and good. I bid 44 hours to make the changes.
Team now wants a user to not only make the grid changes (#3) but save those changes as well. So PM says change 2 & 3 are now merged. We need to save selections and grid entries.
I said, OK, slight increase in scope, 50 hours total.
PM says, wait, we merged 2 & 3 together so now we only have 2 changes not 3 so it should be a decrease in hours.
PM survival tip #1: Reduce scope by merging letters on the plan and hope no one notices.
Story 2 involves 4 experienced PMs
Project is in a major crisis. High-demanding customer, a lot of screaming, no time to deliver on time. Save team is called in to try and pull it out. Atmosphere is very tense.
Some of the project members (developer/consultants) have to work on all 4 segments of the overall project simultaneously. Each segment is given a PM. Customer wants to know what is happening all of the time so PMs demand status reports at least twice a day. For consultants working on all segments that is 8 status reports per day. When complaints are lodged, developers are told that it has to be that way and why aren’t we getting the work done.
Revolt of developers / consultants ensues after months of this activity.
PM survival tip #2: if you flood your customer with progress reports they will think you are doing your job. See tip #4 below.
Story 3 involves a fixed bid contract and some super savvy consultants
Two senior consultants figure out a clever way around a major stumbling block on a late project. They can change the process agreed upon slightly and save time, money, effort, AND bring in a late project on time. The customer also wants it done since the new process has a lot of compelling advantages.
Guess who says no? That’s right, our friendly neighborhood PM. It seems that the PM has decided that the original project, as scoped and laid out, is immutable for all time. No changes are allowed from the project plan WHATSOEVER.
PM Survival Tip #3: Minimize risk by making no changes. No changes, no new plans to build. Voila!
Story 4 involves a PM with a dangerous weapon…Outlook
A project, as usual, is in crisis. The PM decides that their main role is to send out emails…lots and lots of emails. On the day of ‘the incident’, the PM has sent close to 50 emails since 8am.
Then it happens…just before lunch. The PM sends an email saying that people are sending way too many emails.
Consultants laugh and break for lunch.
PM sends 10 more before they get back….
PM survival tip #4: Forward everything, comment on everything, make up stuff. The success of a PM is ultimately measured by the suffering of others.
Story 5 involves a PM with no sense what a PM does
So, the PM contacts the lead on a project and tells the lead that there are too many emails about the project and could the lead please summarize all of the open issues and assign priorities…
…isn’t this what a PM does?
Story 6 involves a PM…again…
The developer has been waiting on getting access to the client’s machine for almost a week. Technical issues, blah, blah, blah.
After the week is up, the PM wants to know timelines (this is good, they are doing their job). Developer says the development has been pushed by 3 days. The PM, who has been copied on every email between the client and the developer, says that they don’t understand why the project is delayed because of a lack of access…
Rule #1 in computers: it is hard to use them if you can’t access them.
Story 7 Guess who…
This story involves, SURPRISE, a PM who we will affectionately call Rum. Now Rum is a great PM because Rum thinks that EVERY piece of email and/or communication must flow through him. Rum insists that no direct communication happen between the technical people or the customer and that he will just forward on what is relevant.
I…AM…IRON MAN…DO, DO, DO DUH DO…
Story 8 Who wants it early?
So, the consultant finishes the application early (1 week) but they can’t test it because it isn’t scheduled to be deployed until the following week. They are told to wait until the project scheduled date so they meet the schedule.
Story 9 This is a loving one that I just title ‘Make the Sh*& up’
Well, the customer wants a project plan. Enter our superhero ‘The PM’. Rather than getting requirements or design done first, the PM decides ‘hey, let me have fun with little chain icons and links and just create a project plan! Oh boy!’.
So a project plan is created, with no document saying what needs to be done and no involvement from the programmers. Developers finally start working on the project and are told you must meet this date for this functionality. Programmers say WTF!?!? Where did this come from.
Well, it is in the project plan that we designed w/o any involvement from the people who do the work…
PM Survival Tip #5: When in doubt, just SWAG the dates and blame the developers.
Subscribe to:
Posts (Atom)