Combining XML Files using SSIS or T-SQL etc.

Loop?
Peter DeBetta, MVP - SQL Server
http://sqlblog.com
"Terry" <Terry@.discussions.microsoft.com> wrote in message
news:187DB17A-FA31-4236-844A-7068BE9C072E@.microsoft.com...
> How can I combine all my xml files so it can be processed by SSIS in order
> that I may create a table containing the imported xml files?
>
What is Loop?
Could you please explain more in detail.
Thank you in advance for your assistance.
"Peter W. DeBetta" wrote:

> Loop?
> --
> Peter DeBetta, MVP - SQL Server
> http://sqlblog.com
> --
> "Terry" <Terry@.discussions.microsoft.com> wrote in message
> news:187DB17A-FA31-4236-844A-7068BE9C072E@.microsoft.com...
>
>
|||SSIS now has a Control Flow Item called Foreach Loop Container. You use this
to loop through all the xml files in a specified directory. You will also
need to create a Data Flow Task in the Foreach Loop Container. The samples
that come with SQL Server 2005 have an example of using this container
control, and you can find more info in BOL.
Peter DeBetta, MVP - SQL Server
http://sqlblog.com
"Terry" <Terry@.discussions.microsoft.com> wrote in message
news:5EAFCB63-AC58-4592-A7DF-786AF5CABBBF@.microsoft.com...[vbcol=seagreen]
> What is Loop?
> Could you please explain more in detail.
> Thank you in advance for your assistance.
> "Peter W. DeBetta" wrote:

combining multiple tables into a single flat file destination

Hi,

I want to combine a series of outputs from tsql queries into a single flat file destination using SSIS.

Does anyone have any inkling into how I would do this.

I know that I can configure a flat file connection manager to accept the output from the first oledb source, but am having difficulty with subsequent queries.

e.g. output

personID, personForename, personSurname,
1, pf, langan

***Roles
roleID, roleName
1, developer
2, architect
3, business analyst
4, project manager
5, general manager
6, ceo

***joinPersonRoles
personID,roleID
1,1
1,2
1,3
1,4
1,5
1,6

Use Merge Joins to join your data sources. You'll need to use more than one because the Merge Join uses two inputs only.

http://msdn2.microsoft.com/en-us/library/ms141775.aspx

|||

You can do the merge join transformations as phil stated, or use a series of lookup transformations, or you could simply do a join on your initial data source query...

select out.personID, out.personForename, out.personSurname, role.roleID, role.roleName

from output as out

INNER JOIN

joinPersonRoles as pr

ON

pr.personID = out.personID

INNER JOIN

roles as role

ON

role.roleID = pr.roleID

Do you want your output to have a single row per person or are multiple rows ok? If you need it all in a single row you will also need to use a pivot transformation (or use pivot in your t-sql statement).

|||Performing the join in the source query via T-SQL as EWisdahl stated is the best route because it lets the database engine do the work.|||Ah, but you're missing my question.

I know how to do joins in order to produce a single recordset to output to a flat file.

I don't want to output a single set of records. I want to produce 3 sets, and output them all to the same file.

the example output I provided is exactly as I want the flat file.

Basically I want to run 3 data flow tasks in sequential order that appends their own output to the flat file destination.|||

So make three data flow tasks and three flat file connection managers, each referencing the same file.

Hook the data flow tasks up together in the control flow to enforce precedence.

What's the problem, I guess? It sounds like you've got it figured out: "Basically I want to run 3 data flow tasks in sequential order that appends their own output to the flat file destination."

|||

This is an odd request, however, you could potentially build up your own csv record.

Have each record be a single column text field and use a derived column transformation to string all of your current columns together for each of the sources.

After you pull these together do a union all.

If needed, you can do a select to grab the metadata information (i.e. table and column names) to push into the output as well. (i.e. select 'tablename'; select 'column1, column2, columnN'; etc...)

:edit - phil once again provided a working answer above while I was typing ... :

|||

great.. thank you so much!

ColumnNamesInFirstDataRow Expression

I have a SSIS package I am trying to create that will accept two types of files. They are the same exact file except for one contains a header and one does not. So I setup a conneciton manager to the csv file. I then set a variable of bool type, and then assigned that to the ColumNamesInFirstDataRow expressions property of the conneciton manager.

So the pacakge runs. Loops a directory, runs a script that sets the Header variable to true/false based on the file name. But when it gets to the data flow it always ignores the property and never bypasses the header row. The variable is being set. I tried datarowstoskip and set it to 1 instead of the above mentioned property, and that does not work either.

How can I accomplish this?

There is a known bug in evaluation of flat file connection properties that might be causing this. IT is scheduled to be fixed for the next release. Unfortunately, I do not see an easy workaround...

-Bob

|||Is there any know work around to this with .net code or anything else?|||

You could set up two connection managers (one with headers, one without) and two dataflows to match, and use a script task to check the file for a header row. Then use precedence constraints to pick the data flow to execute.

Or you could parse the whole file in a script source, but that could be a lot of work, depending on the complexity of your file.

ColumnNamesInFirstDataRow Expression

I have a SSIS package I am trying to create that will accept two types of files. They are the same exact file except for one contains a header and one does not. So I setup a conneciton manager to the csv file. I then set a variable of bool type, and then assigned that to the ColumNamesInFirstDataRow expressions property of the conneciton manager.

So the pacakge runs. Loops a directory, runs a script that sets the Header variable to true/false based on the file name. But when it gets to the data flow it always ignores the property and never bypasses the header row. The variable is being set. I tried datarowstoskip and set it to 1 instead of the above mentioned property, and that does not work either.

How can I accomplish this?

There is a known bug in evaluation of flat file connection properties that might be causing this. IT is scheduled to be fixed for the next release. Unfortunately, I do not see an easy workaround...

-Bob

|||Is there any know work around to this with .net code or anything else?|||

You could set up two connection managers (one with headers, one without) and two dataflows to match, and use a script task to check the file for a header row. Then use precedence constraints to pick the data flow to execute.

Or you could parse the whole file in a script source, but that could be a lot of work, depending on the complexity of your file.

Column mapping in SSIS

Hi,

My Destination columns are more than source columns....

So, how to do column mapping if my source and destination columns are different?

Thanx,

Ruja

Even if the number of columns is different, the mapping should be done automatically if the names of the destination columns are the same of the source.

For all the others destination columns ... you have to do it manually.

Cosimo

|||

Ruja wrote:

Hi,

My Destination columns are more than source columns....

So, how to do column mapping if my source and destination columns are different?

Thanx,

Ruja

I don't understand very well. Are you taking about diffrences in data types? for that you have to cast them to the proper data type using data Conversion or derived column transforms.

If the diffrence is just on the name; you have to create the mapping manually (drag and drop to create the arrows)...

If you have more columns on source than in the target....you shoud know what to do, since you are the only one that knows your data

|||If you simply have more columns in the destination than the source, you can "ignore" the destination columns that don't map to the source, and they'll simply get NULLs (or the default value if defined in the database schema) inserted into those fields that you ignore.

You don't have to match on the number of columns.|||

Ruja wrote:

Hi,

My Destination columns are more than source columns....

So, how to do column mapping if my source and destination columns are different?

Thanx,

Ruja

How can anyone except yourself possibly answer that question?

If you have more columns in your destination than in your source and this is a problem to you - change either your source or your destination.

-Jamie