Hot answers tagged

7

Traditionally the use-case for Hadoop is when you need to split your data storage over dozens or more of machines, and you're not using a traditional RDBMS solution. When you only have one machine, you're likely to negate any potential gains the use of Hadoop would have provided. Additionally, 20 columns * 5 million rows is considered by most DBA to be a ...


3

Notes There are several things to go over here: Just to get this out of the way: gz[ip] and 7zip.exe have nothing to do with any of this. Compression does not alter encoding or any of the original bytes (else compression / decompression wouldn't be reliable). Well, theoretically there can be a bug in one of them such that the decompressed output is somehow ...


3

The first primer which will save you research is by Martin Fowler. NoSQL Distilled. Available on Kindle from Amazon. Reviews the types of NoSQL databases. Key Value Pair {Redis, Raik}, Uses {Web Sessions, Shopping Cart, User Profiles} Document {MongoDB, CouchDB...}, Uses {EventLogging, CMS, Blog} Column {Cassandra, Hbase...}, Uses {EventLogging, CMS, Blog}...


3

You cannot run SQL Server under Hadoop but you can query Hadoop data via T-SQL in SQL Server 2016. This next edition of SQL Server includes a technology called PolyBase which allows access of Hadoop data and Azure blob storage from T-SQL. Tutorials are available from Microsoft at the bottom of the linked page. SSIS for SQL Server 2016 has Hadoop connection ...


2

3 different things. Hadoop is a framework. Something you use to develop an entire application. MongoDb is a db. A nosql one. It is where you store data. BigData is a concept. It is related to huge quantity of data. Where huge is not a fixed parameter. 4 mb was huge in 1960. Now huge means several terabyte.


2

If you foresee that moving to Cassandra is definitely in your future, it will be easier to do while your dataset is still small and manageable. Also, as you learn and get a feel for Cassandra, a small dataset is a better one to make mistakes on (and thus, easier to correct them). That way your data model is solid by the time your dataset gets big, and that'...


2

Creation and maintenance of PolyBase statistics is in the hands of the DBA. The "getting started" guide provides several steps to establish PolyBase, the last of which is CREATE STATISTICS referencing the external table. The documentation on CREATE STATISTICS states: Statistics for external tables When creating external table statistics, SQL Server ...


2

This might be too broad of a question but as with any system, the less steps and dependencies you have, the easier it is to support it as long as it meets current and future (or future enough) business goals. In your use case scenario it doesn't look like you have any use for HDFS and SQOOP. In a lot of environments they might. For example they might ...


2

The fastest way to import data into SQL Data Warehouse is to use PolyBase... https://azure.microsoft.com/en-gb/documentation/articles/sql-data-warehouse-overview-load/ I do not think the same recommendation exists for SQL Server 2016, but it still makes sense. The two options are slightly different in that Polybase allows you to access Hadoop or Azure ...


2

Hadoop is a mistake. You only talk about "quantity" of bits, you don't say at all how you're going to make use of them "25GB of data (images, log files, xlsx, ppts)." An RDBMS, Hadoop, and Spark all make shitty file systems. If what you want is a file system, use a filesystem -- check out ZFS. There are real concrete reasons for this. For those reasons, I ...


1

HBase is simply a non-relational database that runs on HDFS which is essentially a filesystem abstraction layer on top of Hadoop which is map-reduce framework. Apache HBase provides Bigtable-like capabilities on top of Hadoop and HDFS. So the breakdown is like this, HDFS is a distributed file system that is well suited for the storage of large files. ...


1

UTF-8 encoded files can optionally include a byte order mark (BOM). This is a "magic number" to indicate to consuming software that the file is UTF encoded. To further complicate things the BOM can be big-endian or little-endian. Notepad++ makes all of these visible through the Encoding menu. I'd suggest you fiddle around with these for your file (or a cut-...


1

I got a similar error in dev environment, then I added export HBASE_MANAGES_ZK=true to conf/hbase-env.sh and it started working for me. Source - https://askubuntu.com/a/599094 and https://stackoverflow.com/a/22664366


1

I am not sure which SQL is used by Apache Phoenix and I cannot exclude that your code above contains syntax error but what I know is that the MINUS operator can only be used in Oracle SQL. I suggest trying it with the EXCEPT operator which is used in Transact SQL: https://docs.microsoft.com/en-us/sql/t-sql/language-elements/set-operators-except-and-...


1

Cassandra definitely is more suitable for storage of metadata only, when you have big payloads, it's performance isn't very good. Similar stuff was for HBase when I did use it several years ago. For storage of binary data itself, I maybe would go with something S3-compatible.


1

MSDN has updated the documentation Option 6: Cloudera 5.1, 5.2, 5.3, 5.4, 5.5, and 5.9 on Linux https://msdn.microsoft.com/en-us/library/mt143174.aspx


1

As you're accessing all the columns, then consider row based rather than columnar. Partitioning will help. Pick a column that is easy to divide into around 100 ranges, or that has 100 or so distinct values. As you are not joining to any other tables, you might as well distribute randomly. This may be faster than hashing a text column. Nothing beats trying ...


1

There are tools to extract data from XtraBackup data files. But don't forget about a redo log. Without it the Xtrabackup copy is incomplete. To make the backup usable you have to apply the redo log which means you need to restore the backup. I would look at mydumper or mysqldump with -T option and then upload the dumps to hive.


1

I think it could be helpful for you to think of MapReduce as (essentially) a distributed query engine. I know it isn't one-to-one, but with SELECT and aggregate functions such as sum(), max(), etc being very much like a Map() operation, and GROUP BY being very much like a Reduce(), there is quite a bit of similarity. In your case, on the same hardware, I ...


1

I think here Sqoop is not able to find the connection manager. If you are using CDH, then try to add following parameter in your list-tables command. --connection-manager "com.cloudera.connector.teradata.TeradataManager"


1

If you are interested in returning all the values as columns, you need to be try something very adventurous. First look at your query SELECT CONCAT(id, '-', date), MAX(IF(`num` = 0, avg, NULL)) num0 FROM table GROUP BY id, date; If would be a big mess to get MySQL to execute it as a query by giving MySQL the query with each column ...


1

The Hadoop Support section of the Cassandra Wiki says: Cassandra's Hadoop support implements the same interface as HDFS to achieve input data locality (see cluster configuration for details on data locality and how to split your analytic and realtime read loads). So the answer appears to be, "yes".


1

It is my understanding that impala does not automatically refresh the metadata. If you create new tables etc through impala (1.2 and higher) it will update the other impala nodes, however if you create new tables etc in hive you will have to manually refresh the impala metadata. You can manually refresh the metadata on either a table basis with 'REFRESH ...


1

I recommend Vertica. You can get the free community edition that allows up to 1TB of data. If you normalize your web logs when you load them in, chances are they'd compress down and may fit under 1TB, as Vertica has a pretty powerful data compression engine itself. If not, I'd still recommend trying out the platform, but the license fee isn't the cheapest ...


Only top voted, non community-wiki answers of a minimum length are eligible