F1, Networking, Coding, Photography & Minitatures

Big data and example use. July 11, 2017

Ok, something came to me while playing with Hadoop, and Cloudera.

Mobile phone networks.

How do they track your details on what tower you are on?

This is an example of how they could do it using big data and Hadoop.

Firstly Hadoop is a distributed file system, that breaks each file up into chunks over multiple “nodes” (basically cheap servers with lots of storage).

Its basically a EXT file system used by linux.

In this case, how it could track your mobile from tower to tower would be quite simple.


We’d start off with a directory /Australia

Under that we’d have the states, in this case we’ll use NSW. (/Australia/NSW)

Each state would be divided up into territories to simplify the amount of towers in each territory (we’ll call them nodes). (/Australia/NSW/Node1 – in this case Node one is say Macquarie Park.).

Then under that we’d have each tower in that node. So we’ll say we are on Tower 1 in Node 1. (/Australia/NSW/Node1/Tower1).

Now, generally speaking the user data would be quite anonymous, it wouldnt be easy to tie the data to a particular owner – it would be possible but not necessarily easy (mainly due to the privacy act). Now at this point the user could be a file, or they could be their own directory, containing their imsi, mobile number, sim card number, and signal strength. Now generally speaking an Imsi and Sim card are assigned to a single user and can be used to easily find the owner, but unless you have access to the database of users, its almost impossible. (/Australia/NSW/Node1/Tower1/IMSI#)

Now why would this be useful (aside from billing purposes)? Say there was a state of emergency called in a particular area, the mobile company could quickly run a script and send an alert to all the customers in that particular node/tower. It would be quick and easy. Pulling live data from a tower is a fraction harder for various reasons I wont go into here.

Anyways, thats enough from me now. Cya’s

No Comments on Big data and example use.
Categories: Uncategorized

Hadoop – Permission denied when putting files in the cluster July 10, 2017

If you get the following error:

[root@hadoop2 ~]# hadoop fs -mkdir /user/root
mkdir: Permission denied: user=root, access=WRITE, inode=”/user”:hdfs:supergroup:drwxr-xr-x

There is a trick to fixing it:

From the terminal enter the following:

sudo -u hdfs hadoop fs -mkdir /user/root
sudo -u hdfs hadoop fs -chown root /user/root

Basically the file system is under hdfs ownership and as root you cannot put any files to it (you get the above error). To correct it you need to use sudo -u to change the user you are running the command as, in this case we are running it as hdfs, thus we can create the directory, and then we change the owner of the newly created directory to that of root.

No Comments on Hadoop – Permission denied when putting files in the cluster
Categories: Uncategorized

Cloudera Install.

Ok, so I’ve been playing with Cloudera for Hadoop. Tried the quick start but it has some issues which I havent been able to clear through. However did a manual install of a mini cluster using: &

One thing that came up as a problem was being able to access the cloudera manager web interface. I could ‘curl’ the http://localhost:7180 and I could see the HTML, however from the other nodes, I couldnt see the main node either (nor could I see the other nodes).

Im mainly a Ubuntu dude, and havent had much exposure to other distros.

Looks like iptables which is something that a lot of cloudera tutorials talk about stopping dont really exist on cloudera 7.

So what firewall is stopping my shit.


So after a quick “systemctl disable firewalld” & “systemctl stop firewalld” on all nodes, access was possible.

Also: edit /etc/sysconfig/network-scripts/ifcfg-ethX (or -ens for virtual adapters under VMware) with NM_CONTROLLED=no then update /etc/resolv.conf with nameserver x.x.x.x 

Hope that helps some body else.

No Comments on Cloudera Install.
Categories: Uncategorized

Welcome Back! July 9, 2017

Here we go folks!

We are back. Thanks for sticking with me. Content will be added regularly. And I will be focusing on F1, 40k, Photography and IT (Networking and shit).


No Comments on Welcome Back!
Categories: Uncategorized