How to install Apache Spark on Mac OS X Yosemite

Hello data scientists,

This is a quick installation guide to install the Apache Spark on your local machine. I found the documentation on the website little confusing.

1. Download the Apache Spark tar file from the [Choose any version you would like from the dropdown menu. I recommend anything 1.3.1 or above]

2. Unzip the file into your home directory.

3. Open your terminal and go to the spark directory by doing cd spark-1.3.1 [Assuming you are in your home directory]

4. Now, simply run

build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package

5. It takes at least 10 minutes to complete the whole build.

6. After the build’s completed. It should look something like the following:

Build Success

7. Now run

./bin/run-example SparkPi 10

8. You should see something like this:

Screen Shot 2015-07-24 at 1.29.07 PM

As you can see here, it says the Job has been finished which means you have successfully made it running 🙂

Note: I am assuming you have Java installed properly on your machine. This is very important.

How to update Ruby 2.0 to the latest version of Ruby on Mac OS X Yosemite?


Mac OS X Yosemite comes included with Ruby 2.0. But when you are working with the latest version of Rails[4.2], you might be in need of the latest version of Ruby [2.2] to have better performance. So, how do you update Ruby? There is a simple solution for this is to use Ruby Version Manager (RVM), which is a standard for upgrading ruby on OSX.

To do this. Open your terminal and run

\curl -L | bash -s stable –ruby

When this is complete, you need to restart your terminal for the rvm to work.

Now, run rvm list known 

This shows the list of versions of the ruby.

Now, run rvm install ruby-2.2.0

If you type ruby -v in the terminal, it still shows you ruby 2.0.

To use the latest installed version. Run rvm use ruby-2.2.0

Now, when you look at the version, it shows you the following:

ruby 2.2.0p0 (2014-12-25 revision 49005) [x86_64-darwin14]

There you go, you have the latest version of ruby now in use.



Installing Windows 8.1 on VMware Fusion on Mac


This post demonstrates how to install Windows on your VMware Fusion on Mac. Frankly speaking, it’s a piece of cake if you do everything right. I did good amount of research on the internet reading various posts and reviews on the performance issues one might have by installing windows on virtual machine.

I have even personally experienced the Windows 8.1 installed using the Boot Camp Assistant on a separate partition. One thing is clear, personally I felt dual booting was a pain. I didn’t like it at all. I have to install all the software applications again in the other partition. I can’t access files from Mac. I can’t utilize my trackpad to the fullest. I ran out of memory in a couple days after installing Visual Studio and I had only 7 GB of space left. I allocated 60 GB for Windows partition because I only have 256 GB Flash drive. I couldn’t update Visual Studio further and I couldn’t install SQL server on the partition because of low disk space. It was terrible. So I got rid of it. One more thing, I spent over an hour to install windows using the boot camp.  One might be thinking, why did I install windows using boot camp instead of installing it on virtual machine. Because, I thought there would be performance issues and I was trolled for doing that.

Okay coming to the configuration of my machine.

I have a MacBook Pro Retina 15 inch Mid 2014 (latest) currently running OS X Yosemite Version 10.10.2

Processor: 2.2 GHz Intel Core i7

Memory: 16 GB  1600 MHz DDR3

Graphics: Intel Iris Pro 1536 MB

Storage: 256 GB Flash

So even though you have an i5 processor with 8 GB of RAM, its good to go.

This post assumes that you have VMware Fusion already installed on your machine. If you don’t have it, download and install it. As I am a student enrolled at a university. I got it for freeeeeeeeeee!! Good stuff!! Anyways, following are the steps to be followed to install Windows 8.1 on your VMware Fusion on a Mac Machine.

Step 1: Fire up VMware Fusion and click on “+” and select “New“.

Screen 1

 Step 2: In the installation method. Select the “Install from disc or image” option and click on continue.

Screen 2

Step 3: Click on “Use another disc or disc imageScreen 3 Step 4: Select the iso image of Windows 8.1 from the file chooser. Once you select it, it appears as the following. Click on continue.

Screen 4

 Step 5: Check Easy Install and give your details and enter the Windows registration key.

Screen 5

Step 6: Select “More Seamless

Screen 6

Step 7:  Here the Virtual Machine automatically assigns the memory, to configure it, you should click on “Customize Settings” and save the virtual machine.

Screen 7

Step 8: The VMware fires up as soon as you save the virtual machine as below: Click on the “Processors and Memory” option in the System Settings.

Screen 8

Step 9: Select “processor cores” as following and allocate the memory depending on your machine’s memory. Also, 4 GB is recommended as Visual Studio and other developer tools need minimum of 4 GB RAM as a requirement. So, beware of that.

Screen 9

Step 10: After you close the settings page. Click on the play button on the screen which looks like the following. As soon as you do it, the windows setup starts.

Screen 10

Step 11: Setup screen

Screen 11

Step 12: It hardly took 4 to 5 minutes for the installation to be completed. I was really amazed at the speed of installation.

Screen 12

I skipped the McAfee antivirus installation. Now, you can simply login with the password which you have given on the easy installation screen.

Screen 13

And there you go. You have your Windows 8.1 on your VM on Mac. If you want to install Visual Studio. Just download the PowerISO on your windows VM and extract the files from its image and install. As simple as that.

I found this link very interesting on the performance of visual studio on the VMware Fusion. Do check it out.

Have fun.



Removing Windows from Mac using Boot Camp


This is a short post demonstrating the complete removal of windows from Mac which has been installed using Boot Camp native application. Once you remove Windows, the bootcamp automatically restores the space and reverts it to a single OS X partition like before.

Step 1: Open the bootcamp application by navigating to Applications -> Utilities -> Boot Camp Assistant.

Boot Camp Assistant

Step 2:  Click on Continue

Screen 1

Step 3: Check Remove Windows 7 or later version and click Continue

Screen 2

Step 4: If you haven’t backed up your files on Windows partition. Please do it. Because continuing on this step would completely remove all the files on the windows partition and cannot be recovered.

Click Restore to continue the process.

Screen 3

Step 5: Restoration of Partition starts

Screen 4

Step 6: Done. The Windows partition is removed successfully.

Screen 5

You can check the disk size after quitting this application.



Excitement didn’t last long!

Looking at the title you might be thinking what is this all about. Let me describe you a sweet story, exactly 5 days back, the world’s most costliest and the colossal company who won several hearts of the customers all over the world with their jaw dropping innovative products has launched an Operating System. Yes, I am speaking  about the mighty and it’s recent launch of OSX Mavericks. The demo at the conference was brilliant, they announced the OS upgrade as free, they made it available on AppStore for the upgrade on the same day itself. Isn’t that cool? Yes, it was an awesome announcement for everyone who is an Apple Fan, and who owns a Mac.They can’t wait to rush to their home to upgrade this sweet OS to explore it.

Mavericks Installation Gruesome

But guess what! The excitement didn’t last long. People were unable to upgrade it properly, they faced many problems downloading it.Some people got stuck with the installation in the mid way and are going mad thinking of the situation because it’s all about the important data they are mostly worried about which is on their drives on their computers. As an apple fan boy, even I was very excited to install the new OS. I installed it successfully. It was a very huge update when compared to the Mountain Lion. It was about 5.29 GB and after downloading the operating system, it took around one more hour to install the OS and boot with the exciting OS. I was very happy having it installed.The OS was looking seraphic. It was smooth and quick,the Safari got even more better than ever.The Maps was awesome, Calendar was looking great.

I opened the AppStore,downloaded the iBooks, it was one sweet application. I downloaded many books and the iCloud sync was excellent and all the books were on my iPad mini on the very next second. I was enjoying every moment after the update and was eagerly waiting for the iWork and iLife updates. During this stint, I clicked on the Dictionary app… to my surprise, it didn’t open.I tried to open it again, BAM.. it didn’t open. It just bounced once, flashed and stopped. In trepidation, I tried to open the other applications like Calendar,Reminders,Messages,Maps,iBooks,Terminal,FaceTime which were not responding like the Dictionary app. To my biggest surprise, even the AppStore was not getting opened up. I thought I was doomed. Safari, iTunes, Notes, Preview and Activity Monitor were working properly.

Immediately, I tried to open up other applications. All the other third party applications were working fine except some of the native applications.I googled the problem as “Applications are not working on Mac“. I was guided to a the Apple Support Community where many people share their problems and the fellow users post the solutions to the problems. It’s a very social and amiable community. I saw a post “Most of the applications fail to launch (specifically after the Mavericks update)“by a person named SoapiestHeart35 ”  ” There was a hell lot of discussion on this thread.But no one couldn’t solve this problem yet.I was one of the active user on this thread(I go by name ShellZero)

Today, even Preview was not working. One of the Mac user approached the Apple Store with this problem and they gave a solution which solved the problem but the problem persisted again after few hours. The solution was “They recommended a re-install of the OS. You can do this by holding down option as soon as you boot up your Mac. This should open up the bootloader. There’s a recovery partition next to the main startup disk. If you select this partition you’ll see the option to re-install Mavericks.Oh, and most importantly: it is not an actual clean re-install. Your files and programs will still be on your Mac.

Few hours later, the same guy posted “Never mind. The apps worked fine for a little while, but now they’re not launching anymore. It’s just weird that initially they do work and after a while they don’t.“I didn’t try this yet, but I am going to try out my luck.

Finally, I request Apple to please fix the issue and send us an update which can be directly downloaded and installed as we are not even able to open the AppStore through which the downloads usually happen. Because, here we are talking about a immense loss of personal data(photos and videos) and secure data(work stuff).

Hope the Apple Developers fix it as soon as possible.

How to copy and paste the files from mac to an External HD

Hi mac users,

If you own a mac from a long time, you must have already experienced some of the most crucial problems of writing the files into the windows formatted NTFS hard drives. In clear language, one cannot copy and paste or drag and drop the files on to the external hard drives which you have connected to your mac. The driver that is implemented on OS X is simply not capable of writing to NTFS formatted drives.It is however quite annoying for a user that wants to exchange files from a Windows NTFS drive.

So here is a pretty simple fix for this problem. There are several commercial applications which were developed to resolve this problem such as Tuxera NTFS and Paragon NTFS which cost you some dollars. But there is a remarkable application which is known by the name MACFUSE which comes for no cost.

So go ahead and **download** the latest release of macfuse from the website and install it.

MacFuse allows you to enable file system drivers, without the need of adding them to your Unix-kernel. It is a so-called user side framework that adds support for several file formats.

After the installation of the MacFuse you need to download and install the NTFS 3G which is a system driver application which needs to be added to the MacFuse Framework.You have the choice between two versions, the stable one ‘No caching’ which  is the most secure and the ‘ublio’ version which is the quickest. We suggest that you choose the ‘stable’ version, since the ‘ublio’ version can corrupt your NTFS volume when you remove a USB hard-drive from your Mac without unmounting it properly.

After you completed the installation of MacFuse and NTFS-3G, restart your Mac and you should be able to read and write to NTFS volumes. MacFuse automatically mounts your drive, so no additional steps are required. You can now also format drives with a NTFS file system, by using the OS X Disk Utility.

You will have a new option in System Preferences that enables you to write to NTFS drives.

On clicking the NTFS-3G icon in the preferences pane. You will be redirected to the following screen.

On clicking the Mac Fuse icon, It asks you to re open the System Preferences and shows this screen.

If you want to check your NTFS volumes or restore, you can make use of the terminal application and the ntfsfix command. For all the available options just type: ‘man ntfsfix’ in your terminal window.

Happy Copying 🙂

Importance of /var folder in MacOSX

Hi Mac Users,

varis one of the most important directory on the Mac OS.

Importance of var folder:

  • /var – This is actually a hidden folder in mac! Until and unless you make your mac to show the hidden files, this file will not be visible. This is actually under the /private/var.The directory which you see is the shortcut.It is just a symbolic link.Now lets see what is its significance! The folder contains the processes controlled by the operating system.Processes like printing and the programs that store the log files will use the sub directories in the /var directory to store those files.It also holds a fair bit of configuration information [especially /var/db]
  • /var/backups – Used to store backups of critical system information.
  • /var/db – Holds various databases of the system information.The most notable are the netinfo databases(stored in /var/db/netinfo), shadow password files(in /var/db/shadow/hash),and the system’s network configuration database (/var/db/SystemConfiguration/preferences.xml although it moved to /Library/Preferences/SystemConfiguration/ in 10.6), which together store much of the system and network configuration information that a traditional unix admin would expect to find in /etc, and a Mac OS 10 admin would expect to find in System Folder:Preferences.
  • /var/log – This is where many of the system event logs are kept (others are kept in /Library/Logs).
  • /var/root -The root (superuser) account’s home directory. Note that this directory will exist even if you haven’t enabled the root account.
  • /var/run – Stores various status information about processes (especially daemons) running on the system.
  • /var/tmp – A place for programs to store temporary data, just like /tmp. Some programs use one, some use the other, so Mac OS X provides both.
  • /var/vm – Used to store the swap files for Mac OS X’s virtual memory.
  • /var/vm/app_profile – Holds information about various applications’ virtual memory usage.

Never delete this directory because it leads to many severe problems such as data loss and non functionality of your applications! Check out the post in case if you have deleted the /var folder accidentally.