SaviDataSet Alpha 1.0 On the Web

I am finally ready with my SAS dataset reader/writer for .NET. It is written in 100% managed code using .NET 3.5. The dlls can be found
here.

A sample .NET console application can be found in the program file entries after the installation.

Update 2/13/2010:

I am now testing this against a real-world project so I am finding little bugs here and there. These are being addressed for version 1.0.0.1. If you need a build sooner than it is released, let me know.

I have also found that you need at around 100+ observations for this to work correctly. I am investigating but keep that in mind while testing.

[LATE BREAKING] I fixed the obs issue so I went ahead and uploaded the fixed version. Watch for breaking changes which are in the Readme.txt file.

Update 2/29/2010

Based upon Chris' work, I added in a console application that exposed my existing work. It allows for export of the sas7bdat to Excel, delimited, and XML. I will add in import at some point and have designed the interface as such. For example, I put the parms in an XML file to provide enough flexibility to accomplish everything. That has pros and cons but I figured the pros outweighed the cons.

I am also doing something I have meant to do for a long time which is to build a data viewer. Hence, I don't want an interim release but will bundle it all together. I have spent some time this weekend starting that process. All of the components are here but I have to pull them together which I am hoping to do this week.

SaviCellsPro and Exciting Changes

Well, it was hectic week. I went back to work again (on a cool project) and spent a lot of personal time on SaviCellsPro. Specifically on a new utility that I think is pretty darn exciting.

In the past, when you used SCP, you needed to write the XML for the product. Now, the XML is well-documented and I have provided help files and samples to make it easy but, let's face it, a lot of people are uncomfortable with XML.

A suggestion was made by a friend of mine at SAS to have a utility that would take an existing Excel spreadsheet and automatically generate the beginning XML so it doesn't have to be done from scratch. Well, I scratched my head and few times and decided, over the Christmas holiday, that this might be doable. Well, it was. SCP now has an XML creator that can take an existing workbook and describe it in XML. A SAS programmer can then use that as a basis and incorporate their data into the SCP XML. It makes it much easier to get started.

The utility supports cell values, styles (fonts, font sizes, borders, italics, bolds, etc.), formulas, graphs, images, print settings, and more. I will continue to add to it as demand is shown.

Expect a rollout next week on SAS-L, sasCommunity, and probably this blog. I have some testers looking at it first but I think this really rounds out SCP nicely. I will probably add in OLAP support and more robust graphing but for now, it is a cool utility that makes it easy to generate native Excel, PDF, and HTML files of your SAS data (or any other data for that matter). Since it uses XML, it can be run on any computer that produces text.

Look for my presentation at SGF on SaviCellsPro and how to use it. It is on Tues at 1:30.

.NET Coders and the sas7bdat dll

I have been spending the last week working on how the interface to the sas dataset can be used by a .NET developer. It is of particular concern because once the initial dll is released, it will be hard to change it. Getting the interface 'mostly' correct is important.

After a lot of thought, here is what I have so far:

//Initiate logging

Savian.SaviDataSet.Logging.StartLog(@"c:\temp\SaviDataSetErrors.log");

//Create SAS dataset object

SasDataSet ds = new SasDataSet();

//Add 4 variables

ds.AddVariable("AA1", "Var1");
ds.AddVariable("AA2", "Var2");
ds.AddVariable("AA3", "Var3", 10, SasVariableType.Character,"$5.","$7.");
ds.AddVariable("AA4", "Var4", 10, SasVariableType.Character);

//Add observations
//Bulk insertion

for (int i = 0; i < 1000; i++)
{
object[] values = new object[] {1, 2, "Test_" + (i + 1).ToString("00#"), "Test2", "Test3", "Test4", "Test5"};
ds.AddObservation(i, values );
}

//Modify observation - Similar to a .NET dataset

ds.Observations[0]["AA1"].Value = "Test";


//Write dataset

ds.WriteDataSet("TEMP", @"c:\temp\test.sas7bdat");

//Stop logging

Savian.SaviDataSet.Logging.CloseLog();


The area that was a challenge was adding observations since they are really an array across existing variable metadata.

The other area I am focused on is keeping the wording the same as how SAS would refer to things. Hence, observation instead of row and variable instead of column.

I have also been working on validation so that invalid data does not make it into the dataset. I can't prevent everything, but I am making a good faith effort to minimize it. Also, formats/informats have to be checked, name lengths, length values, dataset name, etc. all have to be verified to make sure they comply. So far, so good but more checking is underway.

While working on this, you realize how much effort has been put into the dataset by SAS over the years and how much work they have to go through to make things compliant and workable.

Progress on the sas7bdat

It is tough at times to be a consultant and get those side projects out of the way. That said, I finally had a chance to really work on the sas7bdat some more over the holidays. Not too much since my personal focus is my kids.

A really tough project interfered since the last time I played with it in September so I was looking forward to the holidays to pass another milestone.

Well, I just managed to create a 100 record dataset with 7 variables, char and numeric. My little Christmas gift to myself.

When I first started work on this project many moons back, I never expected that writing a dataset would be much more difficult than reading one. That was a major mistake. However, the writing has made me really discover a lot more about the structure and how it works plus I have had to step up my coding skills a bit to make it through.

Long road but finally happy to be here after so much time.

Happy New Year everyone.

Alan

Calculate a SAS Observation Length

Well, I couldn't find this documented anywhere (someone correct me if I missed it) but the question for the day was how to calculate the SAS Observation Length. After some investigation, here is how it is done (pseudocode):

obsLength = Sum of all of your character lengths + Sum of all of your variable lengths;

if any numeric variables have a length of 8 then
round the obsLength to the nearest factor of 8.

Since the default for numeric variables is a length of 8, the rounding will almost always occur but it is not a guarantee.

I hope this helps others who may need to calculate this value for storage or performance reasons.

What next?

As I can see that shimmer of light at the end of the tunnel I have to start thinking "what do I do with a binary SAS reader/writer?"

I originally started down this path with the goal of avoiding having to read/write SAS using XML Now that I am nearing the finish, I am trying to determine what comes after this (other than a quiet, non-working evening).

Where I am leaning right now is an Excel add-in or something in that space. I think Excel/SAS is one of the biggest areas of opportunity and an area that frankly does not get enough attention, IMO. SQL Server add-ins are another possible option so SAS can be supported in SSIS. It is early in this mulling period though so I would appreciate any thoughts people have on what they would like to see.

Created my first SAS dataset without using SAS

Well, around 14 months ago, I started on a journey to understand the SAS dataset so I could read and write one independently. Originally, I needed the functionality for a client but it became an obsession to figure it out. Client work and personal matters interfered non-stop but I have finally written my first SAS dataset using C# and .NET.

It isn't pretty (no data is in it) but it is a dataset and that is what matters. Putting data in is minor so I am not concerned about that.

A lot of effort still remains to get it all worked out but I am happy to be at this position. After cleaning the code up and finishing it, I plan on making interoperability tools to simplify interfacing with SAS, expecially in areas where SAS needs help (i.e. SSIS, MS Office).

It is a good day but it has taken a brutal amount of hours to accomplish this feat. I would not suggest to others to spend this amount of time on this issue: it is simply mind-numbing.

[Update 07/27/2009]

I just wrote my first data into the dataset. It is only numerics right now but it is a start.

[Update 07/29/2009, 3AM]

I just created a dataset that contains text fields. I had an extraneous bug that made me think it was failing when it wasn't. Such is the life of a coder. Anyway, on to cleaning up code that I left hard-coded and making the system a bit more robust.

Let me be clear: I am tired of looking at hex code.
476F6F646E69676874

[Update 08/27/2009]

Progress continues. After I created my first dataset, the goal was to get rid of any hard-coded values and make it more dynamic. While doing that work, I hit a major logic issue and had to focus on figuring that out. I have made it through that and am now mopping up the remnants and tightening up the logic. Since the goal is to ship a .NET dll for developers, I also need to consider how a developer codes to the dll. Basically, I am in cleanup mode and testing additional variables, labels, etc. to see where things break or do not work as intended.

Stay tuned.