Detect Missing Records in Flat File

I am importing records from a flat file to a database table. If a record is in the table but NOT in the flat file, I need to update a date column in the table.

Any ideas?

First page of this forum, "Checking to see if a record exists, if so update, else insert"

http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1211340&SiteID=1|||

Thanks Phil, I read that post before asking my question.

Since the row exists in the table but NOT in the flat file, a lookup doesn't work. Unless you know of a way to connect a Lookup to a flat file. As I mentioned, I need to know when a row in the database doesn't exist in a flat file.

|||A lookup does too work, you just have to load a staging table first with the contents of the flat file.

You can have more than one data flow task in a given package, so in the first data flow, load up the staging table with the file, and then in the second data flow, use an OLE DB source against your table and then use the lookup transformation against your staging table.|||

Thanks again Phil.

I seemed to have accomplished the same thing by using a Merge Join setup as a left outer join.

Do you see any performance advantages to using the staging table with a lookup instead?

|||

skaszyk wrote:

Thanks again Phil.

I seemed to have accomplished the same thing by using a Merge Join setup as a left outer join.

Do you see any performance advantages to using the staging table with a lookup instead?

If the file is hugh, something on the order of +1,000,000 rows, I'd say yes, but flat files are very fast to process, and I would not think you'd gain anything by using your method versus mine. For most things I do, though, I always push data into staging tables first, before working with the data. Having the data in a table offers the ability to perform SQL upfront while leveraging the database engine, among other things.|||

dear friend,

In my opinion, you can do that using a MERGE JOIN.

Add 2 sources, one OLEDB Datasource in order to get all the IDs, and a second to get the records from flatfile.

Add a merge join and configure it as right or left join depend on the sources order.

Helped?

Regards!

|||

PedroCGD wrote:

dear friend,

In my opinion, you can do that using a MERGE JOIN.

Add 2 sources, one OLEDB Datasource in order to get all the IDs, and a second to get the records from flatfile.

Add a merge join and configure it as right or left join depend on the sources order.

Helped?

Regards!

Pedro, he's already done that. Please read the entire thread first.|||

Sorry phil, but my page wasn't refreshed and didn't see some posts...

Regards!

Desperatly need help to import .csv file using Bulk Insert

Hi all-

I am in need of some help importing a .CSV file into a SQL Server 2005 Enterprise Edition.

The problem is I already implemented Bulk Insert task in SSIS but it is not importing any data. My detailed layout is as follows :
In SSIS package1 -
In Control Flow Bulk Insert Task has been inserted
Properties of Bulk Insert Task:
Connection adtc009d.ganny
Destination Table ganny.dbo.t4
Format
Format Specify
Row Delimiter {CR}
Column Delimiter Comma{,}
Source Connection
File r.csv
Options
Options Check Constraints
Maxerrors 20

This bulk insert task is connected to Data flow task, if we click edit to data flow task, data flow section will come, here Flat file source & OLE DB Destination is there. Flat file source is connected to OLE DB Destination.

Properties of Flat File
Connection Manager
Flat file connection Manager
here by clicking new link flat file properties to this.
Preview
by clicking preview all data are visible
Properties of OLE DB Destination editor
Oledb connection manager adtc009d.ganny
Data access mode: Table or View - fast load
Name of Table or view dbo.t4

After designing all this then if I start debugging I could able to get records are imported to a table.
Please suggest me where I am going wrong.

Thanks in advance
Karna

Karna,

There are several ways to find out what's wrong, the easiest is

looking at the errors reported by SSIS when you run the package.

The other way is to look at the file source and preview the rows to see if your Column delimiters and row delimiters are working correctly.

Try these and let me know

|||On the flat connection manager, you might need to put a text qualifier (like "). Also you might want to try {CR}{LF} on the row delimeter.|||Dear all,

Thanks for reply.

I solved this problem by using only Bulk Insert option alone, without using data flow task.

My biggest problem is whether we have any tool to check validations before inserting records to a table. It has to check duplicates, if duplicates are existing just insert only only real records not the duplicate ones.

I tried removing redundancy by inserting all records from csv data file to a table called t1, create another table which is copy of table t1 but has primary key called table as t2. Use insert into t2 by selecting only distinct records from t1.

My question apart from above option whether we have any tool which does all related job in SSIS.

Thanks in advance
Karna|||

The short answer is No.

The best option is to import into a 'staging' table. then you can have as many 'clean-up' and data modification steps as is necessary. For example, you may wish to create a output (in some form) of the rows that fail the concurrency test.