Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [hadoop]

Questions related to Apache Hadoop.

Filter by
Sorted by
Tagged with
0 votes
0 answers
44 views

Is it necessary to optimize join in hdfs?

What is the most optimal way to query in Hive (Datalake, based on hdfs)? Establishing filters in the tables prior to join them, select* from (select code from table_1 where type="a") a ...
Daniel_DS's user avatar
0 votes
0 answers
58 views

Hive Mysql Metastore Row lock increases

Below is the query causing row locks in Mariadb Galera Cluster 10.2.32 (3Node Cluster). select MT_COMMENT from AUX_TABLE where MT_KEY1='Cleaner' and MT_KEY2=0 for update Hive is a framework, we ...
Mannoj's user avatar
  • 1,561
2 votes
1 answer
65 views

Can we use Cassandra in place of Hadoop with Spark?

Considering we have a backend written in NodeJS and uses MySQL and Cassandra as it's databases, if we want to add Spark to the system to do some data analyzing stuff like recommendation, can we do it ...
user20551429's user avatar
1 vote
1 answer
661 views

How to check total allotted space inside a HDFS 'group'

Our DBA has created a schema for our team in HDFS/HIVE. Not sure if 'schema' is the right word, they call it a 'group'. Anyway, we can only write to the data lake inside this schema, whether it is ...
Victor's user avatar
  • 127
0 votes
1 answer
29 views

How to get percentage after grouping?

I have a table like this: person_id food data 1 bread xxx 1 bread xxx 1 fruite xxx 2 bread xxx 2 fruite xxx food can have only three values (bread, ...
Marco Dinatsoli's user avatar
0 votes
0 answers
23 views

Field with Top Ranking Field Name

Let's imagine a table structured like this: Bucket Red Blue Green First 1 3 4 Second 6 5 2 What I'm trying to achieve is based on the values within each bucket, I'd like to generate another set of ...
Franco Buhay's user avatar
0 votes
3 answers
92 views

Data storage for analytics [closed]

I have to store some amount of data for analytical purposes. The data source produces 2TB data per month. Data is collected on a monthly basis (not real-time). Data is fully structured. There are 100+...
Leeloo's user avatar
  • 111
0 votes
1 answer
309 views

Creating external tables from SQL Server using JDBC Drivers

I am trying to query Hive tables AND Kylin tables from SQL Server 2019. I was doing some research and found that MSSQL provide PolyBase as a form of interface to Hadoop storage. However, the examples ...
user859385's user avatar
1 vote
0 answers
199 views

how to access hadoop slaves which have the same IP address but different port numbers

My question is, I'm wondering if it's possible Hadoop can do ssh access with a different port number(not 22) for slaves when it makes a Hadoop cluster. Detail) I tried to make Hadoop cluster with two ...
eyeballs's user avatar
1 vote
0 answers
117 views

mongo-hadoop connector makes duplicate data by numbers of Mongo Sharding

I'm using mongo-hadoop connector which makes that Hadoop can get data from MongoDB and save into MongoDB. I found MongoDB data is duplicated after Hadoop Map-Reduce job with MongoDB data. environment:...
eyeballs's user avatar
0 votes
1 answer
783 views

Does HBase support spatial functionality?

I see mention for spatial functional in HBase. For example "HBaseSpatial: A Scalable Spatial Data Storage Based on HBase". What spatial functionality does HBase support and where is this documented?
e7lT2P's user avatar
  • 175
2 votes
1 answer
582 views

Directly go to Hadoop or use SQLite/Postgres as a stepping stone towards Hadoop/Spark? [closed]

In our organization, some people are working on setting up Hadoop with a lot of security restrictions etc. By the rate of progress, this seems complex, especially considering security restrictions, ...
alpha_989's user avatar
  • 139
1 vote
0 answers
740 views

Apache Phoenix Upsert not working while trying to update null/empty columns

Step 1: Update a column with null value. Upsert into abc.tablename (Rowkeycombo1, Rowkeycombo2, columnname) values (123,to_date('1988-08-06'),null); This works fine and the column then appears to ...
Ananya Antony's user avatar
5 votes
2 answers
5k views

Why is my UTF-8 document raising UTF-8 encoding errors in Azure Data Lake Analytics?

I have a document that was compressed in gunzip from a unknown source system. It was downloaded and decompressed using a 7zip console application. The document is a CSV file that appears to be encoded ...
Fastidious's user avatar
-1 votes
1 answer
214 views

Suggestions for type of DB to use (Elasticsearch vs MongoDB vs Hadoop)

I am hoping some of you highly skilled DBAs can provide some insight as to the best DB to use for this dataset. This is a relatively simple dataset with no relationship needed between the entries. ...
Scott Summers's user avatar
0 votes
1 answer
4k views

"ZooKeeper exists failed after 4 attempts" when launching Hbase

When launching Hbase I have the following error mike@mike-thinks:~/hbase-1.2.6/bin$ ./hbase shell 2017-11-30 17:26:42,137 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for ...
Revolucion for Monica's user avatar
5 votes
1 answer
208 views

How to installation spark standalone mode in ubuntu

I am trying to install spark standalone but show error. How i can solve this problem. Java version :- 1.8.0_131 Spark:- 2.2.0 Hadoop: 2.7.4 bashrc file setting Hadoop file location in local System: /...
Shiva Manhar's user avatar
0 votes
1 answer
670 views

User specific default database in Hive

I am running a Oracle Big Data Appliance platform that has Cloudera EDH 5.9.x running under the hood. My users are mainly using beeswax, which is the Hive query editor app within Hue. When a user ...
rustycodemonkey's user avatar
1 vote
1 answer
137 views

Hive metrastore Database in Postgresql External [closed]

We recently migrated hive metastore database from embedded postgresql(coming with cloudera hadoop) to external postgresql(standalone in vm server). After that we notified that there is increase of 5 ...
Mughil's user avatar
  • 493
3 votes
1 answer
348 views

Apache Phoenix: Using MINUS throws an error

I'm using Apache Phoenix to query Hbase. I tried to use a simple MINUS operator like we do in good old SQL but it produces an error that I couldn't wrap my head around. Here's the query: select * ...
Youcef Chihi's user avatar
2 votes
1 answer
300 views

Binary storage in Cassandra, HBase

I am looking at some implementations of Cassandra and HBase for medium-sized data sets (~1M resources) to be exposed to clients as graphs (via e.g. Tinkerpop). I would also like to store binaries in ...
gattu marrudu's user avatar
0 votes
1 answer
125 views

Polybase with Cloudera 5.9

We are working on a Proof Of Concept using Polybase with Cloudera. In the polybase documentation it says we can connect to Cloudera 5.5: https://msdn.microsoft.com/en-us/library/mt143174.aspx We ...
Jeremiah's user avatar
3 votes
1 answer
2k views

How does PolyBase collect statistics from external tables?

Books online has this to say about query optimisation in PolyBase: The query optimizer makes a cost-based decision to push computation to Hadoop when doing so will improve query performance. It ...
Michael Green's user avatar
2 votes
1 answer
2k views

PostgreSQL to HDFS to Spark, or PostgreSQL directly to Spark SQL?

If the end goal is to perform computations with Spark, what are the reasons one would first transfer Postgres data to an HDFS (with Sqoop) vs. just using Spark SQL with Postgres directly (with the ...
selegnasol's user avatar
1 vote
1 answer
473 views

Can SQL Server 2016 R Services be used to connect to Hadoop data?

I've only been experimenting with SQL Server 2016 recently. So, please correct me if my assumptions are incorrect: From some research on SQL Server R Services, I see that RxHDFSConnect and ...
Minu's user avatar
  • 391
-1 votes
1 answer
688 views

Slow select from one billion rows GreenPlum DB

I've created the following table on GreenPlum: CREATE TABLE data."CDR" ( mcc text, mnc text, lac text, cell text, from_number text, to_number text, cdr_time timestamp ...
Dor Cohen's user avatar
0 votes
1 answer
71 views

Tools for working with xtrabackup files?

I have database backups in xtrabackup format. I would like to load them into hive without restoring to a mysql instance and use sqoop to then do the transfer. If there are libraries to work with the ...
Marcin's user avatar
  • 121
3 votes
1 answer
2k views

select two tables and print the columns side by side with matching key in hive

I have the below tables: table_1 name_1 id_1 value_1 avi 1 21 adi 2 X table_2 name_2 id_2 value_2 avi 1 21 adi 2 22 metadata_info: can be read from ...
avinash's user avatar
  • 31
2 votes
1 answer
138 views

Is it possible to have a SQL Server cluster with Hadoop?

I know how Hadoop works. Is there any possible way to have SQL Server run under Hadoop to build a cluster based on SQL Server and not Drill or Hive. Just a wild question. Any good tutorial/...
Data-Base's user avatar
  • 357
0 votes
1 answer
277 views

Fastest way to sync (or keep import) 3.5TB data from hadoop to sharded mongodb cluster

There are 3.5TB of data in our Hadoop cluster(yes on hdfs). And we have newly built a sharded MongoDB cluster(the latest 3.x) with 3 mongos, 3 configdb and 3 shards(each shard has 1 primary and 2 ...
nntp's user avatar
  • 13
2 votes
0 answers
166 views

Options for copying data out of continuously updating tables

I work with a pretty large (O(10^1) TB) DB2 (LUW, v. 9.7) database. It's got data coming in on a continuous basis via Golden Gate replication from another DB2 database. It's used for business ...
user1071847's user avatar
2 votes
1 answer
741 views

Postgres or MapReduce to Match Reciprocal Needle Pairs from a Stack of Needles?

I have a Postgres (latest version) table with about a million lines (a_lat, a_lon, b_lat, b_lon) that each have about a dozen criteria. These lines go together in reciprocal pairs (A - B, B - A), ...
user1517922's user avatar
2 votes
0 answers
254 views

Exporting an HBase table which is larger than the available HDFS space?

I have a 1TB hbase table which I need to export. Unfortunately I only have 600GB free in HDFS, so I was hoping to export it to a mounted file share. I've learned that the hbase export can only export ...
alienth's user avatar
  • 121
1 vote
1 answer
465 views

What is the difference between hadoop, mongoDB, Big data? [closed]

What is the difference between Hadoop, MongoDB, Big data ?
Rani's user avatar
  • 37
2 votes
0 answers
360 views

Can we query Accumulo data from Apache Drill

Accumulo uses HDFS where as HBase also works on top of HDFS. We can query HBase data through Drill. So can we query Accumulo data also from Apache Drill ?
user2710961's user avatar
3 votes
0 answers
245 views

Alternatives for storing 365TB of info

I need to upload daily CSV dumps to a blackbox of my choice, and it should be possible to run queries or create filters over the uploaded data. Daily dumps weight around 1TB and are basically id, ...
ffflabs's user avatar
  • 345
2 votes
1 answer
4k views

Small startup: Cassandra + Hadoop or ElasticSearch

To give some context, I am from a small start up (4 team members) and we work in the education space. We've developed an e-learning platform that several schools are already using. We currently have ...
Sneaksta's user avatar
  • 123
1 vote
1 answer
4k views

Problem while connecting to teradata with sqoop

I am trying to connect to Teradat using Sqoop with following command: sqoop import -libjars /usr/lib/sqoop/lib/tdgssconfig.jar,/usr/lib/sqoop/lib/terajdbc4.jar -driver com.teradata.jdbc.TeraDriver -...
Mangesh Jogade's user avatar
0 votes
1 answer
913 views

MySQL Rows to Columns to HBase (using Sqoop)

I have a table that looks like this in MySQL: id | date | num | avg I have found that I can use this for rows to columns and it works so far: SELECT CONCAT(id, '-', date), MAX(IF(`num` = ...
dunce1's user avatar
  • 201
0 votes
1 answer
284 views

Cassandra with Hadoop

Cassandra uses Cassandra File System (CFS) and Hadoop uses Hadoop Distributed File System (HDFS) I'm already using Pig over HDFS and i would like to install Cassandra. Having additional servers for ...
Jossef Harush Kadouri's user avatar
2 votes
1 answer
2k views

How often does the Impala StateStore refresh?

I am using Hive and Impala on the same cluster. I find that when I create new tables, the Impala StateStore does not refresh automatically even after a few hours. I know that I can accomplish this by ...
John Chilton's user avatar
1 vote
0 answers
640 views

Enabling JMX For Hadoop HDFS & Also MapReduce

I'm having a hard time figuring out how to enable JMX to submit metrics for HDFS and MapReduce jobs in Hadoop (CDH4). I've seen several links and read through 'The Definitive Guide' and 'Hadoop ...
Ali Razeghi - AWS's user avatar
0 votes
1 answer
385 views

Storing 5TB web server access log vs OLAP DB

We have over 5TB compressed web server log in a HDFS and we often analyse using Hadoop. It is painful to run map reduce on 5TB of data and most importantly, not many developer are familiar with it. ...
Yoga's user avatar
  • 549
1 vote
1 answer
342 views

MapReduce performance on a single PC [closed]

I have heard that Hadoop has better performance than MySQL. Until now, I have used relational databases so this is really new technology for me. I have a 6 core PC. Suppose I have a table with 20 ...
Mokus's user avatar
  • 1,017
1 vote
1 answer
672 views

Books to start with big data [closed]

I would like to start learning about the big data technologies. I work in this area in the future. Does anyone know good books to start learning about it? Hadoop, HBase. Beginner - intermediate - ...
Gunter Amorim's user avatar