I was requested at work to learn ETL. I have already created a database without using ETL with some dimensions and consolidations, etc, containing above 1 billion numerical values. Now I have to create and fill a similar database (at least 100 million values) using ETL through Jedox Palo Suite. I read some manuals and looked on a sample project and I understand the principles but it looks like ETL is made only for well formatted data files that contain names of the elements on 1st row and their values on the following rows (or something similar).
My source data files consist of only a single column. The structure is as follows (values are numerical, only 1 on line except 1st line):
1152 576 //sizes of following structure (longitude and latitude, x and y) 1st_value //1st value, I need to store it with coordinate like [0,0,0] 2nd_value .. 1152th_value //last value of the first y from 576, coordinate [1151,0,0] 1153th_value //first value of the second y, coordinate [0,1,0] .. 663552th_value //last value of the last y, coordinate [1151,575,0] 1152 576 //again sizes (always 1152 576) 1st_value //first value of next element in the 3rd dimension [1151,575,1] .. .. last_value //[1151,575,63]
I need to read the data and place it in the appropriate cell of my cube. How can I do this in ETL? I think the TableTransform can be used with its mode Normalization/Denormalization which can change rows to columns or vice-versa but it looks like this Transform is for different uses. Another problem is how to count the rows. RowNumber and getRowCount count the values and I can use some variables to store the number of cycles but I don't know how to increase them and also how to then increase the element of dimension.
How can I best read the data from 1 column to different dimensions?