It is tough at times to be a consultant and get those side projects out of the way. That said, I finally had a chance to really work on the sas7bdat some more over the holidays. Not too much since my personal focus is my kids.
A really tough project interfered since the last time I played with it in September so I was looking forward to the holidays to pass another milestone.
Well, I just managed to create a 100 record dataset with 7 variables, char and numeric. My little Christmas gift to myself.
When I first started work on this project many moons back, I never expected that writing a dataset would be much more difficult than reading one. That was a major mistake. However, the writing has made me really discover a lot more about the structure and how it works plus I have had to step up my coding skills a bit to make it through.
Long road but finally happy to be here after so much time.
Happy New Year everyone.
Alan
Monday, December 28, 2009
Tuesday, November 10, 2009
Calculate a SAS Observation Length
Well, I couldn't find this documented anywhere (someone correct me if I missed it) but the question for the day was how to calculate the SAS Observation Length. After some investigation, here is how it is done (pseudocode):
obsLength = Sum of all of your character lengths + Sum of all of your variable lengths;
if any numeric variables have a length of 8 then
round the obsLength to the nearest factor of 8.
Since the default for numeric variables is a length of 8, the rounding will almost always occur but it is not a guarantee.
I hope this helps others who may need to calculate this value for storage or performance reasons.
obsLength = Sum of all of your character lengths + Sum of all of your variable lengths;
if any numeric variables have a length of 8 then
round the obsLength to the nearest factor of 8.
Since the default for numeric variables is a length of 8, the rounding will almost always occur but it is not a guarantee.
I hope this helps others who may need to calculate this value for storage or performance reasons.
Thursday, August 27, 2009
What next?
As I can see that shimmer of light at the end of the tunnel I have to start thinking "what do I do with a binary SAS reader/writer?"
I originally started down this path with the goal of avoiding having to read/write SAS using XML Now that I am nearing the finish, I am trying to determine what comes after this (other than a quiet, non-working evening).
Where I am leaning right now is an Excel add-in or something in that space. I think Excel/SAS is one of the biggest areas of opportunity and an area that frankly does not get enough attention, IMO. SQL Server add-ins are another possible option so SAS can be supported in SSIS. It is early in this mulling period though so I would appreciate any thoughts people have on what they would like to see.
I originally started down this path with the goal of avoiding having to read/write SAS using XML Now that I am nearing the finish, I am trying to determine what comes after this (other than a quiet, non-working evening).
Where I am leaning right now is an Excel add-in or something in that space. I think Excel/SAS is one of the biggest areas of opportunity and an area that frankly does not get enough attention, IMO. SQL Server add-ins are another possible option so SAS can be supported in SSIS. It is early in this mulling period though so I would appreciate any thoughts people have on what they would like to see.
Sunday, July 26, 2009
Created my first SAS dataset without using SAS
Well, around 14 months ago, I started on a journey to understand the SAS dataset so I could read and write one independently. Originally, I needed the functionality for a client but it became an obsession to figure it out. Client work and personal matters interfered non-stop but I have finally written my first SAS dataset using C# and .NET.
It isn't pretty (no data is in it) but it is a dataset and that is what matters. Putting data in is minor so I am not concerned about that.
A lot of effort still remains to get it all worked out but I am happy to be at this position. After cleaning the code up and finishing it, I plan on making interoperability tools to simplify interfacing with SAS, expecially in areas where SAS needs help (i.e. SSIS, MS Office).
It is a good day but it has taken a brutal amount of hours to accomplish this feat. I would not suggest to others to spend this amount of time on this issue: it is simply mind-numbing.
[Update 07/27/2009]
I just wrote my first data into the dataset. It is only numerics right now but it is a start.
[Update 07/29/2009, 3AM]
I just created a dataset that contains text fields. I had an extraneous bug that made me think it was failing when it wasn't. Such is the life of a coder. Anyway, on to cleaning up code that I left hard-coded and making the system a bit more robust.
Let me be clear: I am tired of looking at hex code.
476F6F646E69676874
[Update 08/27/2009]
Progress continues. After I created my first dataset, the goal was to get rid of any hard-coded values and make it more dynamic. While doing that work, I hit a major logic issue and had to focus on figuring that out. I have made it through that and am now mopping up the remnants and tightening up the logic. Since the goal is to ship a .NET dll for developers, I also need to consider how a developer codes to the dll. Basically, I am in cleanup mode and testing additional variables, labels, etc. to see where things break or do not work as intended.
Stay tuned.
It isn't pretty (no data is in it) but it is a dataset and that is what matters. Putting data in is minor so I am not concerned about that.
A lot of effort still remains to get it all worked out but I am happy to be at this position. After cleaning the code up and finishing it, I plan on making interoperability tools to simplify interfacing with SAS, expecially in areas where SAS needs help (i.e. SSIS, MS Office).
It is a good day but it has taken a brutal amount of hours to accomplish this feat. I would not suggest to others to spend this amount of time on this issue: it is simply mind-numbing.
[Update 07/27/2009]
I just wrote my first data into the dataset. It is only numerics right now but it is a start.
[Update 07/29/2009, 3AM]
I just created a dataset that contains text fields. I had an extraneous bug that made me think it was failing when it wasn't. Such is the life of a coder. Anyway, on to cleaning up code that I left hard-coded and making the system a bit more robust.
Let me be clear: I am tired of looking at hex code.
476F6F646E69676874
[Update 08/27/2009]
Progress continues. After I created my first dataset, the goal was to get rid of any hard-coded values and make it more dynamic. While doing that work, I hit a major logic issue and had to focus on figuring that out. I have made it through that and am now mopping up the remnants and tightening up the logic. Since the goal is to ship a .NET dll for developers, I also need to consider how a developer codes to the dll. Basically, I am in cleanup mode and testing additional variables, labels, etc. to see where things break or do not work as intended.
Stay tuned.
Wednesday, July 15, 2009
PM Stories
These are stories I have collected over the years. i hope people enjoy.
Project Managers In Practice (PIMP…sorry, PMIP)
Our first story involves a young PM, new to the project and not very technical
Ok, so I (senior consultant guy) have a project and there are 3 changes requested by the team:
1. Change a label (minor)
2. Allow a user to save their selections made on dropdowns (these select what displays in a grid)
3. Allow a user to change a grid and submit the values to another program
That’s all fine and good. I bid 44 hours to make the changes.
Team now wants a user to not only make the grid changes (#3) but save those changes as well. So PM says change 2 & 3 are now merged. We need to save selections and grid entries.
I said, OK, slight increase in scope, 50 hours total.
PM says, wait, we merged 2 & 3 together so now we only have 2 changes not 3 so it should be a decrease in hours.
PM survival tip #1: Reduce scope by merging letters on the plan and hope no one notices.
Story 2 involves 4 experienced PMs
Project is in a major crisis. High-demanding customer, a lot of screaming, no time to deliver on time. Save team is called in to try and pull it out. Atmosphere is very tense.
Some of the project members (developer/consultants) have to work on all 4 segments of the overall project simultaneously. Each segment is given a PM. Customer wants to know what is happening all of the time so PMs demand status reports at least twice a day. For consultants working on all segments that is 8 status reports per day. When complaints are lodged, developers are told that it has to be that way and why aren’t we getting the work done.
Revolt of developers / consultants ensues after months of this activity.
PM survival tip #2: if you flood your customer with progress reports they will think you are doing your job. See tip #4 below.
Story 3 involves a fixed bid contract and some super savvy consultants
Two senior consultants figure out a clever way around a major stumbling block on a late project. They can change the process agreed upon slightly and save time, money, effort, AND bring in a late project on time. The customer also wants it done since the new process has a lot of compelling advantages.
Guess who says no? That’s right, our friendly neighborhood PM. It seems that the PM has decided that the original project, as scoped and laid out, is immutable for all time. No changes are allowed from the project plan WHATSOEVER.
PM Survival Tip #3: Minimize risk by making no changes. No changes, no new plans to build. Voila!
Story 4 involves a PM with a dangerous weapon…Outlook
A project, as usual, is in crisis. The PM decides that their main role is to send out emails…lots and lots of emails. On the day of ‘the incident’, the PM has sent close to 50 emails since 8am.
Then it happens…just before lunch. The PM sends an email saying that people are sending way too many emails.
Consultants laugh and break for lunch.
PM sends 10 more before they get back….
PM survival tip #4: Forward everything, comment on everything, make up stuff. The success of a PM is ultimately measured by the suffering of others.
Story 5 involves a PM with no sense what a PM does
So, the PM contacts the lead on a project and tells the lead that there are too many emails about the project and could the lead please summarize all of the open issues and assign priorities…
…isn’t this what a PM does?
Story 6 involves a PM…again…
The developer has been waiting on getting access to the client’s machine for almost a week. Technical issues, blah, blah, blah.
After the week is up, the PM wants to know timelines (this is good, they are doing their job). Developer says the development has been pushed by 3 days. The PM, who has been copied on every email between the client and the developer, says that they don’t understand why the project is delayed because of a lack of access…
Rule #1 in computers: it is hard to use them if you can’t access them.
Story 7 Guess who…
This story involves, SURPRISE, a PM who we will affectionately call Rum. Now Rum is a great PM because Rum thinks that EVERY piece of email and/or communication must flow through him. Rum insists that no direct communication happen between the technical people or the customer and that he will just forward on what is relevant.
I…AM…IRON MAN…DO, DO, DO DUH DO…
Story 8 Who wants it early?
So, the consultant finishes the application early (1 week) but they can’t test it because it isn’t scheduled to be deployed until the following week. They are told to wait until the project scheduled date so they meet the schedule.
Story 9 This is a loving one that I just title ‘Make the Sh*& up’
Well, the customer wants a project plan. Enter our superhero ‘The PM’. Rather than getting requirements or design done first, the PM decides ‘hey, let me have fun with little chain icons and links and just create a project plan! Oh boy!’.
So a project plan is created, with no document saying what needs to be done and no involvement from the programmers. Developers finally start working on the project and are told you must meet this date for this functionality. Programmers say WTF!?!? Where did this come from.
Well, it is in the project plan that we designed w/o any involvement from the people who do the work…
PM Survival Tip #5: When in doubt, just SWAG the dates and blame the developers.
Project Managers In Practice (PIMP…sorry, PMIP)
Our first story involves a young PM, new to the project and not very technical
Ok, so I (senior consultant guy) have a project and there are 3 changes requested by the team:
1. Change a label (minor)
2. Allow a user to save their selections made on dropdowns (these select what displays in a grid)
3. Allow a user to change a grid and submit the values to another program
That’s all fine and good. I bid 44 hours to make the changes.
Team now wants a user to not only make the grid changes (#3) but save those changes as well. So PM says change 2 & 3 are now merged. We need to save selections and grid entries.
I said, OK, slight increase in scope, 50 hours total.
PM says, wait, we merged 2 & 3 together so now we only have 2 changes not 3 so it should be a decrease in hours.
PM survival tip #1: Reduce scope by merging letters on the plan and hope no one notices.
Story 2 involves 4 experienced PMs
Project is in a major crisis. High-demanding customer, a lot of screaming, no time to deliver on time. Save team is called in to try and pull it out. Atmosphere is very tense.
Some of the project members (developer/consultants) have to work on all 4 segments of the overall project simultaneously. Each segment is given a PM. Customer wants to know what is happening all of the time so PMs demand status reports at least twice a day. For consultants working on all segments that is 8 status reports per day. When complaints are lodged, developers are told that it has to be that way and why aren’t we getting the work done.
Revolt of developers / consultants ensues after months of this activity.
PM survival tip #2: if you flood your customer with progress reports they will think you are doing your job. See tip #4 below.
Story 3 involves a fixed bid contract and some super savvy consultants
Two senior consultants figure out a clever way around a major stumbling block on a late project. They can change the process agreed upon slightly and save time, money, effort, AND bring in a late project on time. The customer also wants it done since the new process has a lot of compelling advantages.
Guess who says no? That’s right, our friendly neighborhood PM. It seems that the PM has decided that the original project, as scoped and laid out, is immutable for all time. No changes are allowed from the project plan WHATSOEVER.
PM Survival Tip #3: Minimize risk by making no changes. No changes, no new plans to build. Voila!
Story 4 involves a PM with a dangerous weapon…Outlook
A project, as usual, is in crisis. The PM decides that their main role is to send out emails…lots and lots of emails. On the day of ‘the incident’, the PM has sent close to 50 emails since 8am.
Then it happens…just before lunch. The PM sends an email saying that people are sending way too many emails.
Consultants laugh and break for lunch.
PM sends 10 more before they get back….
PM survival tip #4: Forward everything, comment on everything, make up stuff. The success of a PM is ultimately measured by the suffering of others.
Story 5 involves a PM with no sense what a PM does
So, the PM contacts the lead on a project and tells the lead that there are too many emails about the project and could the lead please summarize all of the open issues and assign priorities…
…isn’t this what a PM does?
Story 6 involves a PM…again…
The developer has been waiting on getting access to the client’s machine for almost a week. Technical issues, blah, blah, blah.
After the week is up, the PM wants to know timelines (this is good, they are doing their job). Developer says the development has been pushed by 3 days. The PM, who has been copied on every email between the client and the developer, says that they don’t understand why the project is delayed because of a lack of access…
Rule #1 in computers: it is hard to use them if you can’t access them.
Story 7 Guess who…
This story involves, SURPRISE, a PM who we will affectionately call Rum. Now Rum is a great PM because Rum thinks that EVERY piece of email and/or communication must flow through him. Rum insists that no direct communication happen between the technical people or the customer and that he will just forward on what is relevant.
I…AM…IRON MAN…DO, DO, DO DUH DO…
Story 8 Who wants it early?
So, the consultant finishes the application early (1 week) but they can’t test it because it isn’t scheduled to be deployed until the following week. They are told to wait until the project scheduled date so they meet the schedule.
Story 9 This is a loving one that I just title ‘Make the Sh*& up’
Well, the customer wants a project plan. Enter our superhero ‘The PM’. Rather than getting requirements or design done first, the PM decides ‘hey, let me have fun with little chain icons and links and just create a project plan! Oh boy!’.
So a project plan is created, with no document saying what needs to be done and no involvement from the programmers. Developers finally start working on the project and are told you must meet this date for this functionality. Programmers say WTF!?!? Where did this come from.
Well, it is in the project plan that we designed w/o any involvement from the people who do the work…
PM Survival Tip #5: When in doubt, just SWAG the dates and blame the developers.
Wednesday, March 11, 2009
Windows 7
Ok, I am testing out Windows 7 and it seems great...except it didn't recognize my printer and wouldn't let me install the driver. It errored out with:
Unable to Install Printer. The printer driver is not compatible with a policy enabled on your computer that blocks Windows NT 4.0 drivers.
Ok, the solution (as is often the case under Vista) is to disable UAC. Don't go looking for a policy setting (it isn't there that I could find and I scoured all of them), don't try to install a local port like Microsoft advices for Windows Server 2003. Just disable UAC and you are good to go.
I know this one isn't SAS related but people really rely on some tips like these when they are beating their heads against the wall.
Unable to Install Printer. The printer driver is not compatible with a policy enabled on your computer that blocks Windows NT 4.0 drivers.
Ok, the solution (as is often the case under Vista) is to disable UAC. Don't go looking for a policy setting (it isn't there that I could find and I scoured all of them), don't try to install a local port like Microsoft advices for Windows Server 2003. Just disable UAC and you are good to go.
I know this one isn't SAS related but people really rely on some tips like these when they are beating their heads against the wall.
Tuesday, February 24, 2009
Opportunity
It will be very rare when I criticize a SAS PR release but this one was a bad release, in my opinion:
http://www.sas.com/news/preleases/AberdeenReport.html?utm_medium=RSS&utm_source=Corp_PressRelease
It infers that only mid-size businesses use spreadsheets and that it shows a level of immaturity. My current client, a SAS user, is 15x the size of SAS and uses a lot of spreadsheets. Why? because it is easy and it works. Also, let's not use a generic term spreadsheets. EVERYONE uses Excel so put a name on the usage and don't generalize.
I found this sentence the most galling:
Hatch said that the most successful midsized enterprises are much more likely to be using powerful and relevant technologies such as those from SAS, the leader in business analytics software and services.
A business can use SAS and be relevant while also using Excel. Trying to say that someone needs to use SAS end-to-end is the major issue I have today with SAS. SAS is extremely powerful but to say that their interfaces have to be the vehicle for BI is nonsense. Excel is a fantastic delivery mechanism for SAS BI.
Bad press release, IMO. I am unsure who it targets. The 'unsuccessful' midsized enterprises who love Excel and just want SAS to work with it or the ferociously succesful large businesses who have melded the 2 sides and view them as compatible.
Sell SAS guys but do so with an understanding of what clients want: SAS to work with Microsoft and display data in Excel, Word, Project, Powerpoint, SharePoint, etc. No need for a rift when the 2 sides work well together.
http://www.sas.com/news/preleases/AberdeenReport.html?utm_medium=RSS&utm_source=Corp_PressRelease
It infers that only mid-size businesses use spreadsheets and that it shows a level of immaturity. My current client, a SAS user, is 15x the size of SAS and uses a lot of spreadsheets. Why? because it is easy and it works. Also, let's not use a generic term spreadsheets. EVERYONE uses Excel so put a name on the usage and don't generalize.
I found this sentence the most galling:
Hatch said that the most successful midsized enterprises are much more likely to be using powerful and relevant technologies such as those from SAS, the leader in business analytics software and services.
A business can use SAS and be relevant while also using Excel. Trying to say that someone needs to use SAS end-to-end is the major issue I have today with SAS. SAS is extremely powerful but to say that their interfaces have to be the vehicle for BI is nonsense. Excel is a fantastic delivery mechanism for SAS BI.
Bad press release, IMO. I am unsure who it targets. The 'unsuccessful' midsized enterprises who love Excel and just want SAS to work with it or the ferociously succesful large businesses who have melded the 2 sides and view them as compatible.
Sell SAS guys but do so with an understanding of what clients want: SAS to work with Microsoft and display data in Excel, Word, Project, Powerpoint, SharePoint, etc. No need for a rift when the 2 sides work well together.
Tuesday, February 17, 2009
Microsoft Project and Inserting Columns
I am doing a LOT of work right now with SAS and Microsoft Project using the Microsoft Project interop, Base SAS, and SAS/IntrNet. One of the client requirements was to add in additional columns to the default MS Project columns. Well, I searched high and low and couldn't figure out how to do it. After some slogging, here is what I came up with:
I hope this helps some other interop developer out
there struggling with the same issue.
class MsProjectTest
{
Project p ;
object m = Missing.Value;
string file = @"c:\temp\test.mpp";
public void Process()
{
Application app = new Application();
app.Alerts(false);
p = app.Projects.Add(m,m,m);
Task t = p.Tasks.Add("Test", m);
t.Name = "Test Node";
t.Start = "03/20/2009";
t.Start7 = "03/31/2009";
p.Application.TableEdit("Entry", true, m, m, m, m,
"Start7", "Start Test", 10, PjAlignment.pjLeft,
true, true, 255, 1, 4, 1, m, m);
p.Application.TableApply("Entry");
p.SaveAs(file, PjFileFormat.pjMPP, m, m, m, m,
m, m, m, m, m, m, m, m, m);
app.Quit(PjSaveType.pjDoNotSave);
System.Diagnostics.Process.Start(file);
}
}
I hope this helps some other interop developer out
there struggling with the same issue.
Tuesday, January 06, 2009
Silverlight, WCF, SAS, and Demos
I had a client ask, very nicely, if I could showcase some of my work to help make their internal sale. Well, this is all good to me since I could possible get some business in 2009 and I needed to do it anyway.
Well, the only time available was the holiday season so I cranked out a lot of code, learned a LOT about WCF and Silverlight, but was able to get my demo site built.
The site utilizes WCF to talk to SAS on the backend and expose SAS out as a service on the front-end. Things are very secure right now so I have held a lot of the 'what is possible' back so that I don't expose a security hole.
Here is the site:
http://demos.savian.net
Silverlight has been the most challenge simply because the tools are still a bit immature and the coding examples are thin, thin, thin but I love the possibilities. The more I work with Silverlight, the more I like it even if the syntax leaves me scratching my head at times.
The examples I put on the web use SAS as a local resource but SAS could easily be sitting on another server and be exposed via IOM or IntrNet (REST).
Well, the only time available was the holiday season so I cranked out a lot of code, learned a LOT about WCF and Silverlight, but was able to get my demo site built.
The site utilizes WCF to talk to SAS on the backend and expose SAS out as a service on the front-end. Things are very secure right now so I have held a lot of the 'what is possible' back so that I don't expose a security hole.
Here is the site:
http://demos.savian.net
Silverlight has been the most challenge simply because the tools are still a bit immature and the coding examples are thin, thin, thin but I love the possibilities. The more I work with Silverlight, the more I like it even if the syntax leaves me scratching my head at times.
The examples I put on the web use SAS as a local resource but SAS could easily be sitting on another server and be exposed via IOM or IntrNet (REST).
Consultants, Idealism, and the Real World
Today, I hit an absolute programming wall. I was deep into my code, client asking for a deliverable date, and I hit a deal-breaker coding moment. I think, we have all had those things happen. You're there in the bowels and suddenly find some code that simply is not going to work and may jeopardize the entire design.
What to do? Well, Google and hope for a miracle. In this case, i got fortunate and finally stumbled onto a solution late into the evening.
However, the issue made me start doing some more research into the particular area I had struggled with and was reading a lot about static vs dynamic languages (this post is not about that). C$ 4.0 recognizes this dilemma and is implementing it but it isn't available now. However, what a firestorm on the web over the addition of dynamic capabilities.
What I realized is how far off the idealists / zealots are from the consultant/business coder.
If I tell my client that I have to rearchitect the 100s of hours of coding, what are they supposed to say? Instead of that Feb 1 deadline, I need to redo everything and deliver it in June? They will drop my contract and possible pursue action. Reality will sink in...quickly...
In the world of consulting, it is critical to deliver projects on time and to estimate accurately. When you hit those walls, there isn't a lot of time to debate the nuances of static/dynamic languages: I just have to have something that works...and NOW. Customers have a real allergy to paying a lot more money on top of what they already paid.
I dogged this problem, like always, and found an answer so I dodged a bullet: sometimes you aren't so lucky. There are times when you, as a consultant, have to take it in the shorts and I have done my time there. However, to ask a a client to pay because 'it isn't perfect' ignores the fact that nothing in computing is perfect: just make it work.
So, at the end of the day I will take tools that focus more on getting the job done than being perfect in theory. And I will also say an extra thank-you tonight for finding that one blog posting I really, really needed...
Catching pop flys,
Alan
What to do? Well, Google and hope for a miracle. In this case, i got fortunate and finally stumbled onto a solution late into the evening.
However, the issue made me start doing some more research into the particular area I had struggled with and was reading a lot about static vs dynamic languages (this post is not about that). C$ 4.0 recognizes this dilemma and is implementing it but it isn't available now. However, what a firestorm on the web over the addition of dynamic capabilities.
What I realized is how far off the idealists / zealots are from the consultant/business coder.
If I tell my client that I have to rearchitect the 100s of hours of coding, what are they supposed to say? Instead of that Feb 1 deadline, I need to redo everything and deliver it in June? They will drop my contract and possible pursue action. Reality will sink in...quickly...
In the world of consulting, it is critical to deliver projects on time and to estimate accurately. When you hit those walls, there isn't a lot of time to debate the nuances of static/dynamic languages: I just have to have something that works...and NOW. Customers have a real allergy to paying a lot more money on top of what they already paid.
I dogged this problem, like always, and found an answer so I dodged a bullet: sometimes you aren't so lucky. There are times when you, as a consultant, have to take it in the shorts and I have done my time there. However, to ask a a client to pay because 'it isn't perfect' ignores the fact that nothing in computing is perfect: just make it work.
So, at the end of the day I will take tools that focus more on getting the job done than being perfect in theory. And I will also say an extra thank-you tonight for finding that one blog posting I really, really needed...
Catching pop flys,
Alan
Subscribe to:
Posts (Atom)