Category Archives: SSIS

Building Flat File Connectors Dynamically For SSIS

Building Connectors Is Crap in SSIS – Updated

What else can I say. I finally broke when I had to build a flat file connector with 258 columns that needed to be imported into a staging database. 258 columns… I almost had a stroke. Not only is it mind numbing it’s also error prone. I do dabble in c# so I build a little tool to automate this for me.

Introducing SSISConnectionBuilder

SSISConnectionBuilder is a simple command line tool to ease the burden of building flat file connectors for SSIS. It is WAY WAY alpha but I am working on cleaning up the code. You can pick it up here on codeplex.

I have only tested this on Windows 8/7 64 bit. If you get an error about cannot find DLL you need to install the client development stuff from the SQL Server installer.

You need to generate an excel spreadsheet with four columns. Column,Type,Precision,Scale.

The program loops through the sheet and kicks out an SSIS dtsx file with a single flat file connector defined.

You can choose a delimiter, package name and set the csv file name for the connector. The csv file name doesn’t have to be valid. If you know the csv file will be unicode you need to pass the -u or you will have errors with your connector with the error column being ntext instead of text.
Command line options:
-s, –schemafile =VALUE Your excel schema definition file.
-d, –delimiter=VALUE The column separator you wish to use, usually a comma or pipe.
-p, –packagename=VALUE Name of the dtsx file that will have your connection in.
-c, –csvfilename=VALUE Name of the csv file that your connection will use.
-u, –unicode csv file is Unicode.
-?, -h, –help show this message and exit

Update 7/23/2013

Removed the dependencies on the SSIS SDK the current release doesn’t require any external dll’s to run!

Quick Tip: SSIS and SAP BW Round 2

Again, frustration.

After almost breaking my arm patting myself on the back getting past my last SAP BW issue I found that SAP BW Connector and SQL Server 2012 were punishing me again. I was building a second package against the same SAP instance on the same SQL Server 2012 instance when I hit something more than flip this field from 3 to 1 kind of thing.

NO SOUP FOR YOU!

“Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: Index”

Oh mister SSIS this means war! Looking at the complete output log I saw that SAP BW had actually delivered the data but the SAP BW source component had thrown a hard error. It was a general error and searching Bing yielded next to nothing. I did find posts like this one that pointed to some bugs inside and outside the connector. This, of course, builds a ton of confidence that its going to work in the future.

Desperation Leads To A Fix.

After searching dozens of KB articles around the SAP BW 1.1 and 1.0 connector I finally just pulled the trigger and installed service pack 1 for SQL Server 2012 and the 2012 SP1 feature pack SAP BW 1.1 connector. I hate when the standard tech support advice works…. The scary thing is there are still outstanding bugs that are fixed in CU3 and CU4 that aren’t included in SP1. The worst thing is I never could find an exact KB that matched my error codes. I don’t know why it was broken and I don’t know if there was a specific patch or CU that addressed my bug. This isn’t the first time something like this has happened ether. Sometimes bug fixes don’t get listed when a CU or service pack is made available.

Quick Tip: SSIS and SAP BW

Frustration.

If you have ever had to work with Integration Services and data sources other than the Microsoft variety you know how frustrating it can be. I recently started a project using SQL Server 2012 and SAP BW 7. The customer had already decided that the other commercial options weren’t viable and settled on the Microsoft Connector 1.1 for SAP BW. A test package was built and worked.

I had already read through Using SQL Server 2008 Integration Services with SAP BI 7.0 and felt comfortable with the instructions. Even though we were using SSIS 2012 I knew there was very little differences between the 1.0 connector and the 1.1 connector. I built out our new package and all seemed well. The SAP BW connection in the connection manager hooked up and tested out. The SAP BW source also tested out pulling the metadata and even delivered preview data just fine. Next, I attempted to run the package. Just as I expected it ran and I got the data I expected. The next three attempts all resulted in SAP BW connector timeouts. I kept poking and prodding it. The SAP admin didn’t see any issues ether.

So, I did what I always do. I stopped tussling with the package and walked through all the steps again. I went back and read the doc again. I made a few notes and shot an email off to the SAP administrator. I settled on one thing pointed out in the doc:

“We want to keep the number of parallel processes to a reasonable value for the overall DTP process type DTP_LOAD, but this parallelism can lead to a timeout error during the Open Hub DTP extraction through Microsoft Connector for SAP BI”

Sure enough, it was set to something other than 1.

VICTORY!

Now, we get all our records and don’t have the timeout issues. The downside seems to be a bit slower performance. If the choice is a faster data pump that only runs once every five or so attempts or one that runs slower but runs every time I’ll slow up a bit. The fact that everything verified and I even got preview data on every attempt was hugely frustrating and time consuming. Knowing when to stop and reevaluate the problem and having a good relationship with the SAP administrator probably saved me hours if not days of cursing and shaking my fist at both Microsoft and SAP.

SSIS and Oracle All Your Non-options

Why do some things have to be so hard?

I have been asking myself that question for the better part of two weeks as I wrestle with SSIS 2008 R2 and getting data out of Oracle and into SQL Server.

It’s like the shell game, only with drivers.

Like, go native man!

Like anyone else working in SSIS and dealing with Oracle I started with the default drivers that ship with SSIS. Technically, they work. There are a couple of glaring caveats. They only work in 32 bit mode. Huge non starter on our 64 bit system. They are slow. I know that’s like saying the sky is blue, especially if you don’t have any context. Well I do have some context. Migrating packages from DTS to SSIS I’ve got historical run times and also did some test runs before actually converting the packages. The native 32 bit drivers were slower or just equal to the equivalent on the SQL Server 2000 box running the Oracle 8 drivers. They don’t return the proper metadata column data types. Everything comes back as a wide string a.k.a. st_wstr or varchar for you table creating types. Decimal(18,2)? Thats a varchar(50) for you. varchar(10)? You guessed it, varchar(50) should do it! This beyond anything else was probably the biggest problem.

No, only use what Oracle provides!

I decided to install the Oracle drivers. Let the pain begin!
First, you have to create an account on the Oracle Developer Network site. Really? I just need some drivers. I guess it could have been worse, like a sharp stick to the eye. You need to download the 32 bit and the 64 bit driver packs. Each one weighs in at 700MB compressed. They do include a ton of tools that I have no clue how to use, thats a bonus. Here is your sharp stick to the eye as you get to use of one of the worst installers in the history of installers. After about a dozen tries I finally found out the magic combination to get only the drivers cutting out about 1.3GB from being piled onto my server. Oh, and you get to do it twice. Next, as you look at where it put the drivers at you realize that each install is named client, client_1 and so on. No clue at all which is the 32 bit or 64 bit install bits. Get that figured out, you can go fix your borked path. The installer will gladly stick its path right at the beginning giving your hours of fun trying to figure out exactly what is broken. That is if your path isn’t already too long and it just skips this bit for you. And finally, you get to manually add a system variable pointing to your tnsnames.ora file usually stored in /app/<nt login>/11.2/client/network/tnsnames.ora

After all your hard work you are rewarded with Ole Db, ADO and ADO.Net drivers HUZZAH!
First thing I found out is the Ole Db drivers work as well as the native drivers as far as metadata is concerned. Performance was better. The ADO and ADO.Net do take it up a notch. You do get some additional metadata goodness from these drivers. I did get decimal and float types back but pretty much every string came back as st_wstr again. It would size correctly them that was nice. It did mean I had to add conversion from wide string to string. On a table with 86 columns and 64 million records this also was a bad combination. You do get to run in both 32 bit and 64 bit.

Use what random people on twitter recommend!

Well, not quite that bad. I posted a 14o character version of this post to the mighty #sqlhelp hashtag and lo’ my friend Merrill Aldrich (twitter|blog) and he simply said “Can you use the Attunity connector? #sqlhelp” Huh? When I did a search for just Attunity it brought me to their website http://www.attunity.com/ I didn’t see exaclty what Merrill was talking about. Doing a search for Attunity Connector brought me to the gold I’d been looking for http://www.attunity.com/products/attunity-connect/ssis-connectors-for-oracle-and-teradata. Apparently, Attunity makes connectors for Oracle and Teradata and releases them for F R E E. There are two versions currently 1.2 for SSIS 2008 R2 and 2.0 for SSIS 2012. The installer is rough but not Oracle rough. One of the nice things is installing the 64 bit drivers also installs the 32 bit drivers. Unless you have a problem with the installer and the 64 bit installer only installs the 32 bit drivers. After a few searches and a few more failed install attempts I found that you need to install the Visual C++ 2008 SP1 redistributable package. I only needed to do this on my server since I had Visual Studio 2010 already installed on my laptop. Once that was done the installer worked just fine. Except it really didn’t “install” everything. You still have to manually add them to your toolbox sidebar for data flows before you will see the source and destination connectors for Oracle. It’s totally worth it.

First off with a bang, all the metadata returned was 100% spot on. No more fussing with conversion steps or guessing what the data type should be. They are faster. Not by a small margin ether. On 50~ packages they were around 25% or more faster than the Oracle provided drivers. You still need to have a TNS names file you can’t use the machine name, port and service name directly. (that I know of)

And there was much joy to be had.

I do think it is sad that neither Microsoft or Oracle has a good solution to this issue. I’m glad Microsoft is supporting Attunity I wished they would ship them by default. As for Oracle, I know why Oracle developers and DBA’s get paid so much. If getting drivers installed was this hard I can’t imagine getting the whole database setup and going, ugh.