Tuesday, May 06, 2008

Microsoft Project Files, SAS, and .NET

Ok, another fun, fun time with SAS and an obscure area. Here's the scenario, the client needs data from an MPP file (Microsoft Project) converted into a SAS dataset. I tried lots of routes, all to no avail. As usual, I get to go through the mess of COM interop, lack of documentation on the web, and SAS not supporting write access in the local data provider. Ahhhh, the joys of consulting ;-]


I finally got the following C# code operational and it writes the data into a SAS-friendly XML format. I hope this helps someone else:

using System;
using System.Collections.Generic;
using System.Data;
using System.Data.OleDb;
using System.Linq;
using System.Text;
using Microsoft.Office.Interop.MSProject;
using System.Reflection;
using Savian.Core;
using Savian.DataManagement.Delimited;

namespace Client.ReadProjectFile
{
public class Main
{
List tasks = new List();

public void Process(string file)
{
ReadMppFile(file);
ConvertFileToSas();
Console.ReadLine();
}

private void ConvertFileToSas()
{
DateTime start = DateTime.Now;
Console.WriteLine("Start converting to SAS..." + start.ToShortTimeString());
DataTable dt = tasks.ToDataTable("TABLE");
dt.WriteXml(@"c:\temp\Client.xml");
DateTime end = DateTime.Now;
Console.WriteLine("Finished conversion..." + end.ToShortTimeString());
Console.WriteLine("Elapsed time: " + (end - start).TotalSeconds + " seconds");
}

private void ReadMppFile(string file)
{
ApplicationClass app = new ApplicationClass();
app.DisplayAlerts = false;
app.ScreenUpdating = false;
app.Visible = false;
app.MacroVirusProtection = false;
app.FileOpen(file, Missing.Value, Missing.Value, Missing.Value, Missing.Value,
Missing.Value, Missing.Value, Missing.Value, Missing.Value,
Missing.Value, Missing.Value, PjPoolOpen.pjDoNotOpenPool,
Missing.Value, Missing.Value, Missing.Value, Missing.Value);
var mppTasks = app.ActiveProject.Tasks;
int i = 0;
DateTime start = DateTime.Now;
Console.WriteLine("Start creating records..." + start.ToShortTimeString());
foreach (Task task in mppTasks)
{
ProjectInfo pi = GetProjectInfo(task);
tasks.Add(pi);
i++;
if (i % 100 == 0)
{
Console.Write("Records created: " + i);
Console.SetCursorPosition(0, 1);
}
}
Console.WriteLine("Records created: " + i);
DateTime end = DateTime.Now;
Console.WriteLine("Record read completed..." + end.ToShortTimeString());
Console.WriteLine("Elapsed time: " + (end-start).TotalSeconds + " seconds");
app.FileClose(PjSaveType.pjDoNotSave, Missing.Value);
}

private static ProjectInfo GetProjectInfo(Task task)
{
ProjectInfo pi = new ProjectInfo();
AssignFields(ref pi, task);
return pi;
}

private static void AssignFields(ref ProjectInfo pi, Task t)
{
pi.Name = t.Name;
pi.Start = (DateTime)t.Start;
pi.Finish = (DateTime)t.Finish;
pi.Id = t.ID;
pi.PercentComplete = Extensions.GetIntValue(t.PercentComplete);
pi.Duration = Extensions.GetIntValue(t.Duration);

}
}
}

class ProjectInfo
{
public int Id { get; set; }
public string Name { get; set; }
public DateTime Start { get; set; }
public DateTime Finish { get; set; }
public int? PercentComplete { get; set; }
public int? Duration { get; set; }
}

Sunday, March 23, 2008

Silverlight Lessons Learned

After a shotgun week of Silverlight 2.0, I thought I would share a few lessons learned.

Blank page

If you get a blank page after uploading your xap file, make sure that NETWORK SERVICE is in the security accounts.

XAP File

XAP is the only file that needs to be moved typically. As you do your code updates, just move the XAP file into place.

Dynamic XAML

Dynamic XAML requires a namespace attribute now:

<Grid xmlns="http://schemas.microsoft.com/client/2007">

Timing Issues

There seems to be a timing issue in Silverlight that is causing an event to get triggered before the data streams down from a site. This still requires investigation.

Visual Studio 2008 and Silverlight Debugging

For some reason, if a serious Silverlight error occurs, debugging will be disabled in the web application. This may make it seem like events are not firing when they are, just that the debugger is broken. Right-click properties on the web application --> Start Options --> Debuggers --> Check Silverlight

Overall

Overall, I love XAML and Silverlight. It was absolutely incredible to work with but it took a lot of time due to minimla information on the web right now. Hopefully, this post will help some other folks out.

Friday, March 07, 2008

Microsoft Silverlight

I have been in Vegas all week at the Microsoft MIX08 conference for the rollout of Silverlight 2.0. It is so revolutionary that it really got me thinking. That and SAS Global Forum made me mull a few thoughts:

1. Silverlight is going to revolutionize the world. Bold statement, I know.However, what I saw, as a web developer, absolutely stunned me. If you don't believe me, see what NBC will do with the Olympics. 2200 hours of hi-def video, 4 channels per user, VOD, and just much, much more.

However, check this out for something live now:

http://memorabilia.hardrock.com/

Data visualization and BI will move toward Silverlight. SAS should move there as well. as quickly as possible. SAS should be a leader here and not follow the inevitable.

2. Microsoft is innovating at an amazing pace. Windows Server 2008, Silverlight, Surface, Vista, IE8, IIS7, and on and on. Meanwhile we await a .07 release from SAS that is languishing once again. Basic project management: reduce scope, increase budget, adjust timeframes...hmmm, it seems like the only option at SAS is the latter and I honestly don't understand it. Hopefully, SAS Global Forum will provide some answers.

3. Silverlight controls are being released open source. Office formats are now open, the .NET source code has been released open source. I think SAS can learn from this and realize that money can be made without keeping everything tight to the vest.

4. Integration Technologies should be bundled with Base. All of this new technology requires it so why, as a customer, do I have to have a separate line item? Put SAS/IntrNet in that category as well. Why does web enablement cost extra from SAS. Pay more and realize it is 2008 and web enablement is part of Base SAS.

5. Microsoft actively encourages blogging even if it is critical of the company. People should be trusted and I have found most use discretion, especially if they are employees. Blogging and active user participation helps sell the company message and actively engages the user community. A once a year conference is not enough in today's fast-paced world. SAS Forums are fine, SAS-L operates via 'birdies' but wouldn't it be great to have blogs from Paul, Vince, Chris, Eric, etc. out in the wild? Open and free, give and take, user comments...

As a SAS advocate, user, former employee, and partner, I am asking for change. More openness, more releases, more interaction. Simple stuff that is doable today.

Wednesday, February 06, 2008

SAS LanguageService and You

For those of us who work with SAS LanguageService on a regular basis, it is frustrating. The entire SAS COM interface is very poorly documented and is light on functionality. It gets better over time but it would be much nicer to have a .NET interface so we don't have 50 COM nulls in a single method call.

Ok, that all said, I just came off of a marathon (10+ hours) bug fix that damn near drove me mad.

I was submitting a simple query via the Language Service:

proc sql;
create outdata.test as
select *
from DICTIONARY.COLUMNS
where …some criteria…
;
quit;


Ok, no problem. I could submt the query within the SAS Editor and it would work fine. However, DEPENDING ON WHERE I WAS IN MY .NET CODE, it would return 0 rows. Huh!?!? It wasn't the dataset, it wasn't the syntax for the call, it was where the call occured at during processing. Holy smokes Batman! That's a bear of a bug to catch.

You do the normal stuff and isolate, isolate, isolate. Well, it took a long time to isolate that it was where it was being called. Damn, damn, damn...one of the toughest bugs I have hit in years.

My answer for how to solve it:

Common.SAS.LanguageService.Reset();

(I am using a static class called Common).

Will this answer stand the test of time? Not sure. I have asked my buddy at SAS whether I am on the right track.

Back to the first point: better documentation would be welcome. Even better, an ADO.NET data provider WITHIN Base SAS. Don't make me buy IOM to use SQL processing.

Ah well, back out to the far corners of the ballpark I go...

Friday, December 07, 2007

SAS and LINQ - Part 1

So, I am just getting my feet wet with LINQ, the new integrated query language within .NET. There are loads of articles on LINQ elsewhere. What I want to do is to illustrate LINQ with SAS and this is my starting point.

Look at the following code:


using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Xml.Linq;
using System.Windows.Forms;

namespace WindowsFormsApplication2
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}

private void btnGetData_Click(object sender, EventArgs e)
{
XElement table =XElement.Parse(
@"<TABLE>
<CLASS>
<Name> Alfred </Name>
<Sex> M </Sex>
<Age> 14 </Age>
<Height> 69 </Height>
<Weight> 112.5 </Weight>
</CLASS>
<CLASS>
<Name> Henry </Name>
<Sex> M </Sex>
<Age> 14 </Age>
<Height> 63.5 </Height>
<Weight> 102.5 </Weight>
</CLASS>
<CLASS>
<Name> James </Name>
<Sex> M </Sex>
<Age> 12 </Age>
<Height> 57.3 </Height>
<Weight> 83 </Weight>
</CLASS>
<CLASS>
<Name> Jeffrey </Name>
<Sex> M </Sex>
<Age> 13 </Age>
<Height> 62.5 </Height>
<Weight> 84 </Weight>
</CLASS>
<CLASS>
<Name> John </Name>
<Sex> M </Sex>
<Age> 12 </Age>
<Height> 59 </Height>
<Weight> 99.5 </Weight>
</CLASS>
<CLASS>
<Name> Philip </Name>
<Sex> M </Sex>
<Age> 16 </Age>
<Height> 72 </Height>
<Weight> 150 </Weight>
</CLASS>
<CLASS>
<Name> Robert </Name>
<Sex> M </Sex>
<Age> 12 </Age>
<Height> 64.8 </Height>
<Weight> 128 </Weight>
</CLASS>
<CLASS>
<Name> Ronald </Name>
<Sex> M </Sex>
<Age> 15 </Age>
<Height> 67 </Height>
<Weight> 133 </Weight>
</CLASS>
<CLASS>
<Name> Thomas </Name>
<Sex> M </Sex>
<Age> 11 </Age>
<Height> 57.5 </Height>
<Weight> 85 </Weight>
</CLASS>
<CLASS>
<Name> William </Name>
<Sex> M </Sex>
<Age> 15 </Age>
<Height> 66.5 </Height>
<Weight> 112 </Weight>
</CLASS>
</TABLE>");


var obs = (from o in table.Elements("CLASS")
where (string)o.Element("Sex") == " M "
select new
{
Name = (string) o.Element("Name"),
Age = (string) o.Element("Age")
}
);
List observations = new List();
foreach (var ob in obs)
{
observations.Add(ob.Name) ;
}
}
}
}



The above is C# code.

Notice that the input data is a SAS XML representation (I am doing it inline right now as a demo. It could easily come from a file)? Notice the SQL-like code toward the bottom? That is LINQ.

One of the oft-used criticisms of low-level languages is that they were fine for a lot of things except processing data. For that, we needed languages such as SAS or SQL. LINQ doesn't substitute for the database but what it does do is enable us to combine data processing with the full-blown power of a language like C#. The syntax isn't as elegant as SAS (by any means) but what it does provide is a way to juice up the power of data processing against SAS data when needed.

SAS is OleDb and ODBC compliant. As of yet, Microsoft has not provided support for either one within LINQ. When it does, I will post examples of using LINQ against a SAS dataset directly rather than via XML.

The above represents a fraction of the power of LINQ and it is in its first iteration. Using the same query language against any data source (SAS, XML, SQL Server, etc.) and then combine that with the power of a language such as C# holds a lot of potential. Power SAS users, especially on the ETL side, should keep an eye on LINQ because it offers up a great way to do data processing.

Monday, November 26, 2007

Demarcation

Good systems practice is to have demarcations between systems layers. This involves separation of the following layers (at a minimum):

Application
Business logic
Data access
Data

SAS programmers though tend to muddle all together:

libname indata ...; <-- Data Access
data mydata...; <--- Business Logic
ods html; proc report...; <--- Application Layer / UI

A better way to handle is to treat all layers as standalone and get them out of each other's space. The easiest way for a SAS coder to accomplish this is to use macro libraries to get it all started:

%GetFinanceData() ;
%DetermineProfit() ;
%OutputResults() ;

However, even this is constrained since everything is called together and a single program controls what happens top-to-bottom. Rather than doing the final piece as a part of the batch job, consider doing it on demand and perhaps through other means:

%GetFinanceData() ;
%DetermineProfit() ;

...wait until use requests information....

Get information on demand and make the presentation logic extraneous to SAS. Someone could even keep it in SAS ODS but the idea would be to pull it out to an application layer that is completely separated from how the information was formulated.

SAS programmers would be better off putting in extremely strong lines of demarcations between layers. This makes code easier to maintain, easier to change, and easier to understand.

Longer term, web service calls will simplify this even more but the concept of processing silos can be done now. Break your code apart so everything isn't glued together so tightly that code reuse is hard and code libraries are non-existent or little used.

C# and DBF files

I have a client that asked me to read DBF files generated by their SAS application. Well, this posting is to explain to someone what i have learned in reading DBF files using C#. The main thing I learned was no spaces in the file path.

Here is the code that worked for me:

string connectionString = "Driver={Microsoft dBase Driver (*.dbf)};SourceType=DBF;SourceDB=C:\projects\Client\Data\WeeklyAvailableComparison;Exclusive=No; Collate=Machine;NULL=NO;DELETED=NO;BACKGROUNDFETCH=NO;";
string selectCommand = @"SELECT * FROM C:\projects\Client\Data\WeeklyAvailableComparison\ac_141.dbf";
DataTable dt = new DataTable();

OdbcConnection oConn = new OdbcConnection();
oConn.ConnectionString = connectionString;
oConn.Open();
OdbcCommand oCmd = oConn.CreateCommand();
oCmd.CommandText = selectCommand;

dt.Load(oCmd.ExecuteReader());

I tried doing a 8.3 conversion and it wouldn't work for the select clause.

I hope this helps someone.

Alan

SAS throwing RPC error

If you are doing code in C#  and get this error when creating a LanguageService: The RPC server is unavailable. (Exception from HRESULT:...