0

I've got a pile of json lines files that I'm trying to process into a set of Sql Server database tables. Can't use OPENJSON() because jsonlines is not valid in total, only line by line. Plus the files are mammoth so I was trying to work up a way to process it line by line without reading the whole file in as a CLOB and then parsing it.

My first thought was to use OPENROWSET with a format file that declared only one column - one giant string. Then I was going to put a cursor on that and loop through.

The individual bits are working great, but when I put a cursor on an OPENROWSET like this, the cursor fetches one line and stops (breaks out of the WHILE loop).

I googled for a while but couldn't find anything precisely like this.

Is this an inherent limitation of trying to use OPENROWSET and a cursor together? Or are there some different cursor options I could use that might handle it better?

I was hoping for an all-in-sql approach, but I suppose I could knock together some script that will read the lines and feed them in a query individually.

Thanks

DECLARE @line nvarchar(max)
DECLARE JsonLCursor CURSOR LOCAL FAST_FORWARD for
SELECT [Object] FROM OPENROWSET (BULK 'G:\input\944.jsonl', FORMATFILE='G:\jsonl.fmt') as j

OPEN JsonLCursor
FETCH NEXT FROM JsonLCursor INTO @Line

WHILE @@FETCH_STATUS = 0
BEGIN
    EXEC [dbo].[JsonLProcess] @line = @line
    FETCH NEXT FROM JsonLCursor INTO @Line
END

CLOSE JsonLCursor
DEALLOCATE JSonLCursor

JsonL.fmt

14.0
1
1   SQLCHAR 0   1000000 "\r\n"  1   Object  Latin1_General_100_CI_AS_SC_UTF8
4
  • Good point. My smaller samples were done through notepad++ so the \n -> \r\n might have gotten normalized there. I just changed the format file to \n and got a lot farther. If you want to post this as a reply, I'll mark it as the answer Commented Aug 16, 2021 at 21:04
  • There should be a neat way to make it into a giant JSON array. What about adding brackets [] beginning and end, and perhaps a comma at the end of each line? Either way, you should be able to use OPENJSON or whatever code you have in that stored procedure, directly on the OPENROWSET result, which would be far easier and perform better. Please add the code for the procedure in the question Commented Aug 16, 2021 at 21:47
  • Aaron - I got passed the initial problem and PRINT outputs a number for all of the lines but then it says the query the "Query completed with Errors" and there are some missing rows from the table count. Charlieface, I could try and dress it up, but the files are almost 700 meg. I was trying to avoid sucking it all into memory at once. Commented Aug 16, 2021 at 22:09
  • Okay, just fixed a couple of errors and got through the full 700 meg file in about 4 3/4 minutes Commented Aug 16, 2021 at 22:22

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.