Showing posts with label logic. Show all posts
Showing posts with label logic. Show all posts

Wednesday, March 21, 2012

Detecting an attach or detach

Does anyone know of a way to have logic that responds to an attach or detach event?

Thanks,

Russ

The best that I can determine is setting up an Alert to monitor the application log for EventID 17137, 'Starting up database'.

Other than than, let us know if you find a method.

Friday, February 24, 2012

Design question from data mining newbie

Hi,

We currently have about 900 stored procedures which have logic to group healthcare claims into different 'Edit Groups' depending on the logic within each 'Edit' stored procedure.

Examples of the logic for the Edit stored procedures would be something like:

Edit1: Find all claims from same patient and same provider (matching on SubsriberID and ProviderID) which has a procedure code in (P1, P2, P3....P345) and a diagnosis code in (D1, D2,...D123) and does NOT have a modifer code in (M1, M2, M3)

Edit2: Find all claims from same patient and same provider (matching on SubsriberID and ProviderID) which has a procedure code in (P7, P8, P9....Pxxx) and a diagnosis code in (D1, D2,...Dyyy) and has a modifer code in (M3, M4, M7), which are dated within 120 days of each other.

Do you think one of the SQL Server 2005 Data mining algorithms (Clustering or Classification or Association Rules) could play some part in this? Most of the 900 stored procs can be grouped based on logic, I mean the logic is similar for each group and only the parameters (in brackets above) vary for each stored proc within the same group.

We're totally new to data mining, although we do have some moderately complex cubes running. Which algorithm (if any) would be the most appropriate for our needs?

Thanks for any help,

JGP

Hi,

From you examples, you data is of multimensional nature. It is definitely helpful to use Microsoft Analysis Services to create OLAP cube(s) to efficiently support browse and explore your edit groups. If you are only interested in querying your edit groups, you don't need data mining.

Data mining is good at generalizing knowledge(such as patterns) from data. It can then use learned knowldege to analyze your (new) data. For example, if you have a given set of claims described by the following table:

ClaimID Income HaveInurance City Fraud

1 x0,000 Yes FairyLand No

2 y0,000 No FraudLand Yes

.......

Suppose you can collect the above data from your claim database. Now let's say that you are interested in predict whether or not a new claim is fraud. You can train a model with one of Microsoft data mining algorithms (such as Microsoft Decision Trees). After training, you can use your model to predict new claims like this:

ClaimID Income HaveInurance City

10001 a0,000 Yes FairyLand

1002 b0,000 No FraudLand

.......

Depends on the query you use, you can get some result like this:

ClaimID Income HaveInurance City Predicted Result of Fraud

10001 a0,000 Yes FairyLand No

1002 b0,000 No FraudLand Yes

.......

In general, data mining can play a role when you need to learn patterns from your data, and then apply the patterns to analyze (new) data.

|||

Thanks for the example.

Are you saying that Data mining is relevant only in predicting stuff and not in finding relationships based on pre-defined rules, like what I initially explained?

I did see come cases where classification was done, where based on a bunch of parameters, historic claims can be classified into Edit1(cluster1) , Edit2(cluster2) etc.

Isn't this possible?

Thanks,

JGP

|||

My post is just using predicting as an example. Data mining is relavant to predicting as well as finding relation among data. For example, you can use clustering algorithm to cluster you data, and check whether there is a natural mapping between your edit groups and the cluster1 you found (as you mentioned above). This process can help you understand you data better. For example, if an Edit group can be naturally mapped to some cluster, this Edit group can be considered as well defined, since it really maps to some existing grouping (or cluster) of your data. On the other hand, you might consider merge a few groups if they belong to the same cluster, etc.

On the other hand, data mining works on the basis of probability. In other words, it can not be 100% correct most of the times. Say, for a problem with a set of rules, you can already classify each case (such as each claim) into each target group 100% correct. You don't want to use data mining, because you can not do any better than 100% correct, and data mining does not come free. But, if you need to find something unknown about your data, such as a claim is/isn't fraud or if it belongs to some unknown group, you should resort to data mining.

Good luck,

|||

Thanks, I think I'm getting the picture now.

Just to get some hands on, would you happen to know of any good tutorials\books for clustering\classification that is available for newbies?

Preferably one that works with 'well-defined' groups....

|||

A good book is Data Mining Techniques by Berry and Linhoff.

I think in your situation you can use data mining for data discovery to learn quite a bit about your data sets. It sounds like each of your "edits" are a fairly complicated set of rules and it may be difficult to determine which rules end up being more important, or which "edits" are related (if I am correct, a single record could have multiple "edits", no?)

With this is mind, you could "reverse engineer" the edits with a simple classification model -e.g. trees - to predict what factors are the "most important" in determining an edit. If all of your edits are "and" conditions, this won't do much, but if you have any with "or" conditions, you may find that a majority of records recieve an edit for only a few of the possible conditions. Using this to classify old records, as Yimin notes, is not going to be 100% accurate though, since you already have an encoding of the 100% accurate rules.

If you created a table that had for each record all of the record data and all of the possible "edits" for that record (you would likely need a nested table) you could predict "edits" based on the record data plus other "edits". This would end up in a relationship diagram describing how record data and edits are related. If you made such a model including only the edits, you would see how they are interrelated independent of record data. You could perform a similar operation using Clustering, to see if groups of edits cluster together.

In the end you could end up with a greater understanding of your data - potentially removing redundant code or streamlining in other ways. The good thing about data mining, it that it's painless - it's kind of fun to play with and you can get some good insights, but it doesn't cause any harm in the meantime....

Enjoy, and feel free to post any follow up questions.

-Jamie

Sunday, February 19, 2012

Design question - stored prosedures vs. components

It is a design question which affects SQL Server.
Scenario A. Business logic in SQL Server stored procedures. No middle tier
components is used. As a result the stored procedures are very long, up to
5,000 lines.
Scenario B. Business logic is in components. Stored procedures are used for
elementary operations, insert, update, delete and therefore are very short.
Components have code that glues everything together. Components can run on a
separate server.
In my opinion, scenario A doesn't scale well and causes very high cpu
utilization on SQL Server but I would like to listen somebody else's opinion
on that ...
Thanks,
-Stan"Stan" <nospam@.yahoo.com> wrote in message
news:eUc1IQpjDHA.708@.TK2MSFTNGP10.phx.gbl...
> It is a design question which affects SQL Server.
> Scenario A. Business logic in SQL Server stored procedures. No middle tier
> components is used. As a result the stored procedures are very long, up to
> 5,000 lines.
> Scenario B. Business logic is in components. Stored procedures are used
for
> elementary operations, insert, update, delete and therefore are very
short.
> Components have code that glues everything together. Components can run on
a
> separate server.
> In my opinion, scenario A doesn't scale well and causes very high cpu
> utilization on SQL Server but I would like to listen somebody else's
opinion
> on that ...
> Thanks,
> -Stan
Stan,
Let me give you my opinion. We have a similar set up at our location.
We developed a system where all business logic laid in SQLServer SPs,
triggers, etc, and no middle tier. Only light validations were performed at
the UI and the heavy stuff took place at the database server. Pros: Any
change in business rules affected 80%-100% the database layer (SPs) and
0%-20% on the client side. This made it a lot easier for us to modify or
correct code, save and go, without having too much impact on the client
layer. Also, having all database objects at hand (tables, views, other SPs,
etc.) is always a great advantage if you place your business logic in SPs.
Cons: As you said, it causes very high CPU utilization, but that will always
depend on the hardware you are using, and how intense will be used your
application. This drawback may be compensated with a good state-of-the-art
server, if feasible.
Having business logic in a middle tier also has its great advantages, and
may free your database server machine of hardwork. But then, in my opinion,
you may have to deal with C++/VB (or whatever language your are using) code,
that definitely is more difficult to maintain than SQL statements.
In conclusion, there is actually no better approach between these two. That
will depend on the requirements specifications for your system (how
complicate it will be, what would fit better based on your requirements) and
the expected usage.
C.|||I agree entirely with Carlitos, but would like to add... for MAXIMUM
scalability you must go to a middle tier... Like Carlitos, I find loading
the back end, easy and quick to create and maintain... However, if you need
HIGHEST scalability, when you have already purchased the largest SQL Server
box that money can buy, what do you do?... you have to move work OFF of the
SQL Server onto another box - middle tier...
There is no best answer - except the one that meets your needs
--
Wayne Snyder, MCDBA, SQL Server MVP
Computer Education Services Corporation (CESC), Charlotte, NC
www.computeredservices.com
(Please respond only to the newsgroups.)
I support the Professional Association of SQL Server (PASS) and it's
community of SQL Server professionals.
www.sqlpass.org
"Stan" <nospam@.yahoo.com> wrote in message
news:eUc1IQpjDHA.708@.TK2MSFTNGP10.phx.gbl...
> It is a design question which affects SQL Server.
> Scenario A. Business logic in SQL Server stored procedures. No middle tier
> components is used. As a result the stored procedures are very long, up to
> 5,000 lines.
> Scenario B. Business logic is in components. Stored procedures are used
for
> elementary operations, insert, update, delete and therefore are very
short.
> Components have code that glues everything together. Components can run on
a
> separate server.
> In my opinion, scenario A doesn't scale well and causes very high cpu
> utilization on SQL Server but I would like to listen somebody else's
opinion
> on that ...
> Thanks,
> -Stan
>|||I concur with Wayne in what he added.
C.
"Wayne Snyder" <wsnyder@.computeredservices.com> wrote in message
news:uiic90pjDHA.3504@.TK2MSFTNGP11.phx.gbl...
> I agree entirely with Carlitos, but would like to add... for MAXIMUM
> scalability you must go to a middle tier... Like Carlitos, I find loading
> the back end, easy and quick to create and maintain... However, if you
need
> HIGHEST scalability, when you have already purchased the largest SQL
Server
> box that money can buy, what do you do?... you have to move work OFF of
the
> SQL Server onto another box - middle tier...
> There is no best answer - except the one that meets your needs
> --
> Wayne Snyder, MCDBA, SQL Server MVP
> Computer Education Services Corporation (CESC), Charlotte, NC
> www.computeredservices.com
> (Please respond only to the newsgroups.)
> I support the Professional Association of SQL Server (PASS) and it's
> community of SQL Server professionals.
> www.sqlpass.org
>
> "Stan" <nospam@.yahoo.com> wrote in message
> news:eUc1IQpjDHA.708@.TK2MSFTNGP10.phx.gbl...
> > It is a design question which affects SQL Server.
> >
> > Scenario A. Business logic in SQL Server stored procedures. No middle
tier
> > components is used. As a result the stored procedures are very long, up
to
> > 5,000 lines.
> >
> > Scenario B. Business logic is in components. Stored procedures are used
> for
> > elementary operations, insert, update, delete and therefore are very
> short.
> > Components have code that glues everything together. Components can run
on
> a
> > separate server.
> >
> > In my opinion, scenario A doesn't scale well and causes very high cpu
> > utilization on SQL Server but I would like to listen somebody else's
> opinion
> > on that ...
> >
> > Thanks,
> >
> > -Stan
> >
> >
>|||I agree - it all depends on situation.
I am trying to feel the magnitude of a difference between these two
scenarios.
It is amazingly easy, though, to load 8-way the-best-money-can-buy SQL
server box
with stored procedures that are long, run often and use a lot of temp tables
...
Despite that, there are a lot of vendors and developers who opt for keeping
business logic on SQL Server...
"Wayne Snyder" <wsnyder@.computeredservices.com> wrote in message
news:uiic90pjDHA.3504@.TK2MSFTNGP11.phx.gbl...
> I agree entirely with Carlitos, but would like to add... for MAXIMUM
> scalability you must go to a middle tier... Like Carlitos, I find loading
> the back end, easy and quick to create and maintain... However, if you
need
> HIGHEST scalability, when you have already purchased the largest SQL
Server
> box that money can buy, what do you do?... you have to move work OFF of
the
> SQL Server onto another box - middle tier...
> There is no best answer - except the one that meets your needs
> --
> Wayne Snyder, MCDBA, SQL Server MVP
> Computer Education Services Corporation (CESC), Charlotte, NC
> www.computeredservices.com
> (Please respond only to the newsgroups.)
> I support the Professional Association of SQL Server (PASS) and it's
> community of SQL Server professionals.
> www.sqlpass.org
>
> "Stan" <nospam@.yahoo.com> wrote in message
> news:eUc1IQpjDHA.708@.TK2MSFTNGP10.phx.gbl...
> > It is a design question which affects SQL Server.
> >
> > Scenario A. Business logic in SQL Server stored procedures. No middle
tier
> > components is used. As a result the stored procedures are very long, up
to
> > 5,000 lines.
> >
> > Scenario B. Business logic is in components. Stored procedures are used
> for
> > elementary operations, insert, update, delete and therefore are very
> short.
> > Components have code that glues everything together. Components can run
on
> a
> > separate server.
> >
> > In my opinion, scenario A doesn't scale well and causes very high cpu
> > utilization on SQL Server but I would like to listen somebody else's
> opinion
> > on that ...
> >
> > Thanks,
> >
> > -Stan
> >
> >
>|||You have to get rid of the temp tables then, they are expensive resource
wise. A lot of cases I have seen where temporary tables were used in stored
procedures they could be rewritten into one SQL statement, usually with
derived tables, and performance went up by what I think is the rule-of-thumb
for these kind of scenarios: 50 times. Yes, you read that correct, writing
proper SQL code instead of using a procedural approach and temp tables (and
even worse cursors) makes your code run 50 times as fast.
Jacco Schalkwijk
SQL Server MVP
"Stan" <nospam@.yahoo.com> wrote in message
news:O8Nbr9pjDHA.3688@.TK2MSFTNGP11.phx.gbl...
> I agree - it all depends on situation.
> I am trying to feel the magnitude of a difference between these two
> scenarios.
> It is amazingly easy, though, to load 8-way the-best-money-can-buy SQL
> server box
> with stored procedures that are long, run often and use a lot of temp
tables
> ...
> Despite that, there are a lot of vendors and developers who opt for
keeping
> business logic on SQL Server...
>
> "Wayne Snyder" <wsnyder@.computeredservices.com> wrote in message
> news:uiic90pjDHA.3504@.TK2MSFTNGP11.phx.gbl...
> > I agree entirely with Carlitos, but would like to add... for MAXIMUM
> > scalability you must go to a middle tier... Like Carlitos, I find
loading
> > the back end, easy and quick to create and maintain... However, if you
> need
> > HIGHEST scalability, when you have already purchased the largest SQL
> Server
> > box that money can buy, what do you do?... you have to move work OFF of
> the
> > SQL Server onto another box - middle tier...
> >
> > There is no best answer - except the one that meets your needs
> >
> > --
> > Wayne Snyder, MCDBA, SQL Server MVP
> > Computer Education Services Corporation (CESC), Charlotte, NC
> > www.computeredservices.com
> > (Please respond only to the newsgroups.)
> >
> > I support the Professional Association of SQL Server (PASS) and it's
> > community of SQL Server professionals.
> > www.sqlpass.org
> >
> >
> > "Stan" <nospam@.yahoo.com> wrote in message
> > news:eUc1IQpjDHA.708@.TK2MSFTNGP10.phx.gbl...
> > > It is a design question which affects SQL Server.
> > >
> > > Scenario A. Business logic in SQL Server stored procedures. No middle
> tier
> > > components is used. As a result the stored procedures are very long,
up
> to
> > > 5,000 lines.
> > >
> > > Scenario B. Business logic is in components. Stored procedures are
used
> > for
> > > elementary operations, insert, update, delete and therefore are very
> > short.
> > > Components have code that glues everything together. Components can
run
> on
> > a
> > > separate server.
> > >
> > > In my opinion, scenario A doesn't scale well and causes very high cpu
> > > utilization on SQL Server but I would like to listen somebody else's
> > opinion
> > > on that ...
> > >
> > > Thanks,
> > >
> > > -Stan
> > >
> > >
> >
> >
>|||On Thu, 9 Oct 2003 21:57:14 +0100, "Jacco Schalkwijk"
<NOSPAMjaccos@.eurostop.co.uk> wrote:
>You have to get rid of the temp tables then, they are expensive resource
>wise. A lot of cases I have seen where temporary tables were used in stored
>procedures they could be rewritten into one SQL statement, usually with
>derived tables, and performance went up by what I think is the rule-of-thumb
>for these kind of scenarios: 50 times. Yes, you read that correct, writing
>proper SQL code instead of using a procedural approach and temp tables (and
>even worse cursors) makes your code run 50 times as fast.
I just want to concur with what so far is an amazingly consistent
thread!
Putting everything in the back end is probably extreme, as is putting
nothing in the back end. I will share this wisdom with you, that what
the back end is all about is not tables, but a data model. If all
your middle tier asks of the database is simple stuff, it is extreme
in another dimension, and doing things the hard way.
5,000 lines is way too long for a single SP, I'd guestimate from just
that, that something is majorly wrong. Lots of cursors, too, I'll
bet, unnecessary and horribly slow cursors, is what I mean. Clean up
the database tier, and the entire problem might just vanish. I've
heard that it's easy to load up an 8-way system with SQLServer because
neither Wintel nor SQLServer is very good at it, but the only case
I've actually seen had such crude code running on it, that it was
really inappropriate to bitch about the hardware.
No easy answers.
Joshua Stern|||Hi Stan,
Yes, there is no easy answer. I'm including some supplemental information
below:
Where Should Business Logic Go?
http://www.4guysfromrolla.com/webtech/121501-1.shtml
This document contains references to a third party World Wide Web site.
Microsoft is providing this information as a convenience to you. Microsoft
does not control these sites and has not tested any software or information
found on these sites; therefore, Microsoft cannot make any representations
regarding the quality, safety, or suitability of any software or
information found there. There are inherent dangers in the use of any
software found on the Internet, and Microsoft cautions you to make sure
that you completely understand the risk before retrieving any software from
the Internet.
Thanks for using MSDN newsgroup.
Regards,
Michael Shao
Microsoft Online Partner Support
Get Secure! - www.microsoft.com/security
This posting is provided "as is" with no warranties and confers no rights.

Friday, February 17, 2012

design of the classes of data access layer

hi all

for designing the data access logic layer,if i have 3 table that relate to each other ,for example contact,address,sometime with creating the customer address should be created too.

should i design one class for each of them seperately and design 1 class(in bussiness layer or data access layer(i am not sure)) for calling the create stored procedures of 2 tables and put the 2 stored procedures in one transaction or not ,should I design 1 class for both tables and add the transaction in sp part?

because i need also the sp of creating each table seperately.

thanks

I wouldn't use a stored procedure for creating a table, if that is what you are suggesting. If you do not want the data to persist, use a temporay table, then drop it when it is no longer in use. As for the classes, the best and easiest option is to create a class (business layer) for each table, and a dataset class (data access layer)for each table. Then the class interacts with the dataset to validate data, insert update and delete data, and should contain constants for the procedure names, and separate methods which call stored procedures to do the inserting, deleting and updating of data, as well as loading the data. An instance of the data access class can be passed into these methods, or form a member variable of the business layer class. If you are updating several tables at a time, then use one transaction, ensuring that all the changes to all the related table will be rolled back on error.

However, there are many ways to approach this, whatever way works best for your data model and business model is best.

HTH

|||

thank you for your answere ,but maybe i have displayed something wrong ,my purpose of create table means inserting the data in table ,according to that I want to know that how do i design my classes that if update of one table relate to update another table ?

should i put the transaction in the store procedure in that case the seperate class for each table is not sutable?,otherwise i need the seperate clase for each table ,should i have third class for joining 2 class (in bussiness layer or data layer?)

thanks

|||You can interact with multiple classes from the front end inside one single transaction. ALternatively, you can enforce referential integrity across tables using relationships, or create update triggers to update other tables based on updates to one table.