8

Given the following data:

id      |   user_id |   started             |   closed              |   dead
-------------------------------------------------------------------------------------------
7714    |   238846  |   2015-01-27 15:14:50 |   2015-02-02 14:14:13 |   NULL
7882    |   238846  |   2015-01-28 13:25:58 |   NULL                |   2015-05-15 12:16:07
13190   |   259140  |   2015-03-17 10:11:44 |   NULL                |   2015-03-18 07:31:57
13192   |   259140  |   2015-03-17 10:12:17 |   NULL                |   2015-03-18 11:46:46
13194   |   259140  |   2015-03-17 10:12:53 |   NULL                |   2015-03-18 11:46:36
14020   |   259140  |   2015-03-23 14:32:16 |   2015-03-24 15:57:32 |   NULL
17124   |   242650  |   2015-04-16 16:19:08 |   2015-04-16 16:21:06 |   NULL
19690   |   238846  |   2015-05-15 13:17:31 |   NULL                |   2015-05-27 13:56:43
20038   |   242650  |   2015-05-19 15:38:17 |   NULL                |   NULL
20040   |   242650  |   2015-05-19 15:39:58 |   NULL                |   2015-05-21 12:01:02
20302   |   242650  |   2015-05-21 13:09:06 |   NULL                |   NULL
20304   |   242650  |   2015-05-21 13:09:54 |   NULL                |   NULL
20306   |   242650  |   2015-05-21 13:10:19 |   NULL                |   NULL
20308   |   242650  |   2015-05-21 13:12:20 |   NULL                |   NULL
21202   |   238846  |   2015-05-29 16:47:29 |   NULL                |   NULL
21204   |   238846  |   2015-05-29 16:47:56 |   NULL                |   NULL
21208   |   238846  |   2015-05-29 17:05:15 |   NULL                |   NULL
21210   |   238846  |   2015-05-29 17:05:55 |   NULL                |   NULL
21918   |   242650  |   2015-06-04 17:04:29 |   NULL                |   2015-06-12 15:47:23

I need to build a data set that meets the following rules:

  1. Groups are defined first by user_id so we should be comparing only records from the same user_id
  2. All records that started at least within 15 days of when any other record was started, closed or dead should be counted as group.
  3. Of each group, the end should be calculated as either the first record closed or all records have a value for dead and we take the greatest date of the dead column.
  4. If a record did not start within 15 days of the start or end of another group, then it begins a new grouping.

Tentatively, I believe my data should look like this:

user_id | started               | end
------------------------------------------------------
238846  | 2015-01-27 15:14:50   | 2015-02-02 14:14:13
259140  | 2015-03-23 14:32:16   | 2015-03-24 15:57:32
242650  | 2015-04-16 16:19:08   | 2015-04-16 16:21:06
242650  | 2015-05-21 13:09:06   | NULL
238846  | 2015-05-15 13:17:31   | NULL

Can anyone provide some guidance on how to build a query to meet these conditions?

Here is a link to the DDL and DML statements for the data presented in this question.

Alternatively, we could skip rules #2 and #4 and more simply state that only records that overlap each other should be included. The more important rule is that in a given set, if there is a closed date then that becomes the end of the set and not the greatest dead date.

Improve this question
2
  • This would be easier with a schema change. There's no need for the two columns, closed and dead. Just have an "ended" column and then a reason for the ending.
    – Andrew Brennan
    Commented Oct 13, 2016 at 9:33
  • Your first 3 examples can be encoded as "If an id is 'closed', then it is a group unto itself. Since that does not seem to highlight all your rules, please add more examples.
    – Rick James
    Commented Oct 15, 2016 at 18:10

2 Answers 2

Reset to default
3
+150

Due to the lack of clarity in the question, I came up with four different solutions. The solutions differ on:

  1. Whether you are to "cascade" per Chris's answer
  2. When you have a closed date, whether you use the earliest date for that group or the started date for the record that is closed.

Please note this is done in SQL Server, not MySQL. Other than some very minor syntax changes, it should work the same.

Common setup and sample data for all four methods

CREATE TABLE #example 
(
    id int NOT NULL DEFAULT '0',
    borrower_id int NOT NULL,
    started datetime NULL DEFAULT NULL,
    closed datetime NULL DEFAULT NULL,
    dead datetime NULL DEFAULT '0000-00-00 00:00:00'
);

CREATE TABLE #result 
(   
    borrower_id int NOT NULL DEFAULT '0',    
    started datetime NULL DEFAULT NULL,    
    ended datetime NULL DEFAULT NULL 
);    

INSERT INTO #example 
    (id, borrower_id, started, closed, dead) 
VALUES 
    (7714,238846,'2015-01-27 15:14:50','2015-02-02 14:14:13',NULL), 
    (7882,238846,'2015-01-28 13:25:58',NULL,'2015-05-15 12:16:07'), 
    (13190,259140,'2015-03-17 10:11:44',NULL,'2015-03-18 07:31:57'), 
    (13192,259140,'2015-03-17 10:12:17',NULL,'2015-03-18 11:46:46'), 
    (13194,259140,'2015-03-17 10:12:53',NULL,'2015-03-18 11:46:36'), 
    (14020,259140,'2015-03-23 14:32:16','2015-03-24 15:57:32',NULL), 
    (17124,242650,'2015-04-16 16:19:08','2015-04-16 16:21:06',NULL), 
    (19690,238846,'2015-05-15 13:17:31',NULL,'2015-05-27 13:56:43'), 
    (20038,242650,'2015-05-19 15:38:17',NULL,NULL), 
    (20040,242650,'2015-05-19 15:39:58',NULL,'2015-05-21 12:01:02'), 
    (20302,242650,'2015-05-21 13:09:06',NULL,NULL), 
    (20304,242650,'2015-05-21 13:09:54',NULL,NULL), 
    (20306,242650,'2015-05-21 13:10:19',NULL,NULL), 
    (20308,242650,'2015-05-21 13:12:20',NULL,NULL), 
    (21202,238846,'2015-05-29 16:47:29',NULL,NULL), 
    (21204,238846,'2015-05-29 16:47:56',NULL,NULL), 
    (21208,238846,'2015-05-29 17:05:15',NULL,NULL), 
    (21210,238846,'2015-05-29 17:05:55',NULL,NULL), 
    (21918,242650,'2015-06-04 17:04:29',NULL,'2015-06-12 15:47:23');

1. CASCADING - USING CLOSED RECORD solution

This is the solution I believe the asker is looking for & matches his results.

select *
into #temp1
from #example

while (select count(1) from #temp1)>0
begin
    --Grab only one user's records and place into a temp table to work with
    declare @curUser int
    set @curUser=(select min(borrower_id) from #temp1)

    select * 
    into #temp2
    from #temp1 t1
    where t1.borrower_id=@curUser

    while(select count(1) from #temp2)>0
    begin
        --Grab earliest start date and use as basis for 15 day window (#2 rule)
        --Use the record as basis for rules 3 and 4
        declare @minTime datetime
        set @minTime=(select min(started) from #temp2)

        declare @maxTime datetime
        set @maxTime=@minTime

        declare @curId int
        set @curId=(select min(id) from #temp2 where started=@minTime)

        select * 
        into #temp3
        from #temp2 t2
        where t2.id=@curId

        --Remove earliest record from pool of potential records to check rules against
        delete 
        from #temp2 
        where id=@curId

        --Insert all records within 15 days of start date, then remove record from pool
        while (select count(1) 
                from #temp2 t2 
                where t2.started<=DATEADD(day,15,@maxTime) 
                    or t2.closed<=DATEADD(day,15,@maxTime) 
                    or t2.dead<=DATEADD(day,15,@maxTime)  )>0
        begin
            insert into #temp3
            select *
            from #temp2 t2
            where t2.started<=DATEADD(day,15,@maxTime)  or t2.closed<=DATEADD(day,15,@maxTime)  or t2.dead<=DATEADD(day,15,@maxTime) 

            delete
            from #temp2
            where started<=DATEADD(day,15,@maxTime)  or closed<=DATEADD(day,15,@maxTime)  or dead<=DATEADD(day,15,@maxTime) 

            --set new max time from any column
            if (select max(started) from #temp3)>@maxTime
                set @maxTime=(select max(started) from #temp3)
            if (select max(closed) from #temp3)>@maxTime
                set @maxTime=(select max(started) from #temp3)
            if (select max(dead) from #temp3)>@maxTime
                set @maxTime=(select max(started) from #temp3)

        end

        --Calculate end time according to rule #3
        declare @end datetime 
        set @end = null
        set @end=(select min(closed) from #temp3)

        if @end is not null
        begin
            set @minTime=(select started from #temp3 where closed=@end)
        end

        if @end is null
        begin
            if(select count(1) from #temp3 where dead is null)=0
            set @end= (select max(dead) from #temp3)
        end

        insert into #result (borrower_id,started,ended)
        values (@curUser,@minTime,@end)

        drop table #temp3
    end

    --Done with the one user, remove him from temp table and iterate thru to the next user
    delete  
    from #temp1 
    where borrower_id=@curUser    

    drop table #temp2

end

drop table #temp1

drop table #example

select * from #result order by started

drop table #result

2. NON-CASCADING - USING CLOSED RECORD solution

Start calculated by first closed date when available, then by earliest start date.

select *
into #temp1
from #example

while (select count(1) from #temp1)>0
begin
    --Grab only one user's records and place into a temp table to work with
    declare @curUser int
    set @curUser=(select min(borrower_id) from #temp1)

    select * 
    into #temp2
    from #temp1 t1
    where t1.borrower_id=@curUser

    while(select count(1) from #temp2)>0
    begin
        --Grab earliest start date and use as basis for 15 day window (#2 rule)
        --Use the record as basis for rules 3 and 4
        declare @minTime datetime
        set @minTime=(select min(started) from #temp2)

        declare @curId int
        set @curId=(select min(id) from #temp2 where started=@minTime)

        select * 
        into #temp3
        from #temp2 t2
        where t2.id=@curId

        --Remove earliest record from pool of potential records to check rules against
        delete 
        from #temp2 
        where id=@curId

        --Insert all records within 15 days of start date, then remove record from pool
        insert into #temp3
        select *
        from #temp2 t2
        where t2.started<=DATEADD(day,15,@minTime)

        delete
        from #temp2
        where started<=DATEADD(day,15,@minTime)

        --Insert all records within 15 days of closed, then remove record from pool
        insert into #temp3
        select *
        from #temp2 t2
        where t2.closed<=DATEADD(day,15,@minTime)

        delete
        from #temp2
        where closed<=DATEADD(day,15,@minTime)

        --Insert all records within 15 days of dead, then remove record from pool
        insert into #temp3
        select *
        from #temp2 t2
        where t2.dead<=DATEADD(day,15,@minTime)

        delete
        from #temp2
        where dead<=DATEADD(day,15,@minTime)

        --Calculate end time according to rule #3
        declare @end datetime 
        set @end = null
        set @end=(select min(closed) from #temp3)

        if @end is not null
        begin
            set @minTime=(select started from #temp3 where closed=@end)
        end

        if @end is null
        begin
            if(select count(1) from #temp3 where dead is null)=0
            set @end= (select max(dead) from #temp3)
        end

        insert into #result (borrower_id,started,ended)
        values (@curUser,@minTime,@end)

        drop table #temp3
    end

    --Done with the one user, remove him from temp table and iterate thru to the next user
    delete  
    from #temp1 
    where borrower_id=@curUser


    drop table #temp2

end

drop table #temp1

drop table #example

select * from #result

drop table #result

3. NON-CASCADING - USING EARLIEST DATE solution

Start calculated by earliest date only.

select *
into #temp1
from #example

while (select count(1) from #temp1)>0
begin
    --Grab only one user's records and place into a temp table to work with
    declare @curUser int
    set @curUser=(select min(borrower_id) from #temp1)

    select * 
    into #temp2
    from #temp1 t1
    where t1.borrower_id=@curUser

    while(select count(1) from #temp2)>0
    begin
        --Grab earliest start date and use as basis for 15 day window (#2 rule)
        --Use the record as basis for rules 3 and 4
        declare @minTime datetime
        set @minTime=(select min(started) from #temp2)

        declare @curId int
        set @curId=(select min(id) from #temp2 where started=@minTime)

        select * 
        into #temp3
        from #temp2 t2
        where t2.id=@curId

        --Remove earliest record from pool of potential records to check rules against
        delete 
        from #temp2 
        where id=@curId

        --Insert all records within 15 days of start date, then remove record from pool
        insert into #temp3
        select *
        from #temp2 t2
        where t2.started<=DATEADD(day,15,@minTime) or t2.closed<=DATEADD(day,15,@minTime) or t2.dead<=DATEADD(day,15,@minTime)

        delete
        from #temp2
        where started<=DATEADD(day,15,@minTime) or closed<=DATEADD(day,15,@minTime) or dead<=DATEADD(day,15,@minTime)

        --Calculate end time according to rule #3
        declare @end datetime 
        set @end = null

        set @end=(select min(closed) from #temp3)

        if @end is null
        begin
            if(select count(1) from #temp3 where dead is null)=0
            set @end= (select max(dead) from #temp3)
        end

        insert into #result (borrower_id,started,ended)
        values (@curUser,@minTime,@end)

        drop table #temp3
    end

    --Done with the one user, remove him from temp table and itterate thru to the next user
    delete  
    from #temp1 
    where borrower_id=@curUser    

    drop table #temp2

end

drop table #temp1

drop table #example

select * from #result

drop table #result

4. CASCADING - USING EARLIEST DATE solution

Start calculated by earliest date only.

select *
into #temp1
from #example

while (select count(1) from #temp1)>0
begin
--Grab only one user's records and place into a temp table to work with
declare @curUser int
set @curUser=(select min(borrower_id) from #temp1)

select * 
into #temp2
from #temp1 t1
where t1.borrower_id=@curUser

while(select count(1) from #temp2)>0
begin
    --Grab earliest start date and use as basis for 15 day window (#2 rule)
    --Use the record as basis for rules 3 and 4
        declare @minTime datetime
    set @minTime=(select min(started) from #temp2)


    declare @maxTime datetime
    set @maxTime=@minTime

    declare @curId int
    set @curId=(select min(id) from #temp2 where started=@minTime)

    select * 
    into #temp3
    from #temp2 t2
    where t2.id=@curId

    --Remove earliest record from pool of potential records to check rules against
    delete 
    from #temp2 
    where id=@curId

    --Insert all records within 15 days of start date, then remove record from pool
    while (select count(1) 
            from #temp2 t2 
            where t2.started<=DATEADD(day,15,@maxTime) 
                or t2.closed<=DATEADD(day,15,@maxTime) 
                or t2.dead<=DATEADD(day,15,@maxTime)  )>0
    begin
        insert into #temp3
        select *
        from #temp2 t2
        where t2.started<=DATEADD(day,15,@maxTime)  or t2.closed<=DATEADD(day,15,@maxTime)  or t2.dead<=DATEADD(day,15,@maxTime) 

        delete
        from #temp2
        where started<=DATEADD(day,15,@maxTime)  or closed<=DATEADD(day,15,@maxTime)  or dead<=DATEADD(day,15,@maxTime) 

        --set new max time from any column
        if (select max(started) from #temp3)>@maxTime
            set @maxTime=(select max(started) from #temp3)
        if (select max(closed) from #temp3)>@maxTime
            set @maxTime=(select max(started) from #temp3)
        if (select max(dead) from #temp3)>@maxTime
            set @maxTime=(select max(started) from #temp3)

    end

    --Calculate end time according to rule #3
    declare @end datetime 
    set @end = null

    set @end=(select min(closed) from #temp3)

    if @end is null
    begin
        if(select count(1) from #temp3 where dead is null)=0
        set @end= (select max(dead) from #temp3)
    end

    insert into #result (borrower_id,started,ended)
    values (@curUser,@minTime,@end)

    drop table #temp3
end

--Done with the one user, remove him from temp table and iterate thru to the next user
delete  
from #temp1 
where borrower_id=@curUser

drop table #temp2

end

drop table #temp1

drop table #example

select * from #result order by started

drop table #result
Improve this answer
0
-2

I'm concerned that we may not have a clear picture of how a group is defined. I only say this because, depending on some unstated conditions, the dates above will either form one giant single group, or 3 groups where one group dominates the set.

Missing grouping conditions?

1) Does this 15 day rule cascade? If a record Y starts 10 days after another record X, and then there is another record Z started 10 days after that, then does this form one group of three records X,Y,Z, or two groups each containing two records X,Y and Y,Z? I made the assumption that the 15 day rules cascades to form larger groups.

2) Are the dates inclusive? For example, if one record has a start date and then a dead date many months later, do all days within that range get merged into the group? I treat both possibilities in my quick analysis below.

Potential Groupings

So, if we begin with id 7714, we see that the start date is 1/27. Clearly, the next entry 7882 starting on 1/28 falls in this group. Notice however that 7882 ends on 5/15, so anything which starts within 15 days of 5/15 must be added to the group.

Thus, 19690through 21210 get added to the group, which via cascading leads to 21918 being subsequently added to the group. The cascading has consumed nearly all entries in the set. Call this GROUP A.

If, however, the grouping is date inclusive as well, all entries from 13190 up to 17124 must also belong to GROUP A, and now all ids are in a single group.

If the dates from GROUP A aren't inclusive, but actually strictly adhere to the '15 day after' rule with cascading, then instead you would have a second group composed of 13190 through 14020, and a third group with a single entry, 17124.

Essentially, my question is, do any of these match your intended grouping, or is there some other information we're missing in the group definition? I'm sorry for such a long-winded answer, but it doesn't appear that your tentative requested output meets your grouping definition.

With clarifications, I'm sure that we can sort this problem out.

Improve this answer
3
  • What if I got rid of the 15 day rule altogether? Would that simplify the problem?
    – Noah Goodrich
    Commented Jun 20, 2015 at 8:39
  • 2
    Also, I think you missed the bit about giving precedence to the first closed date over the last dead date. As a result, for the first grouping starting on 1/27, the close date of 2/2 becomes the end of the group and not 5/15.
    – Noah Goodrich
    Commented Jun 20, 2015 at 8:49
  • Yikes, you're right, I did mis-interpret what you said about the first closed/last dead... Sorry, I was working on this last night around 12:30 at night Pacific time, so I may have been a little sleepy. :) Also, the additional grouping by user data may help, I think. I'll give it just a bit more thought, and try to get back to you.
    – Chris
    Commented Jun 20, 2015 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.